feat: add shortcut module `optree.pytree` and `optree.treespec` #189

lqhuang · 2025-02-13T11:59:28Z

A much simpler way to add new modules pytree and treespec.

Check PR #186 for more history context and motivation.

The major difference between them:

PR refactor(module)!: A proposal to reorg modules and functions with new tree module and treespec module #186 has concrete and refined docs and migrate to new API design like what jax does.
This PR leverages name re-export with lightweight tests.

Merge of current PR will close #186

…from `ops` module

XuehaiPan

Generally looks good with some nit suggestions. Thanks!

optree/treespec.py

README.md

optree/pytree.py

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

optree/pytree.py

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

lqhuang · 2025-02-13T12:49:51Z

The failing lint says there is a spelling error.

optree/treespec.py:11:0: C0402: Wrong spelling of a word 'versionadded' in a docstring:
.. versionadded:: 0.14.1
   ^^^^^^^^^^^^
Did you mean: ''version added' or 'version-added' or 'versioned' or 'diversionary''? (wrong-spelling-in-docstring)

According to https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-versionadded, it's the right directive.

XuehaiPan · 2025-02-13T12:51:21Z

The failing lint says there is a spelling error.

You can add versionadded to docs/source/spelling_wordlist.txt.

lqhuang · 2025-02-13T13:42:02Z

Failed again

pylint...................................................................Failed
- hook id: pylint
- exit code: 4

************* Module optree.pytree
optree/pytree.py:36:0: W0622: Redefining built-in 'all' (redefined-builtin)
optree/pytree.py:37:0: W0622: Redefining built-in 'any' (redefined-builtin)
optree/pytree.py:48:0: W0622: Redefining built-in 'iter' (redefined-builtin)
optree/pytree.py:50:0: W0622: Redefining built-in 'map' (redefined-builtin)
optree/pytree.py:56:0: W0622: Redefining built-in 'max' (redefined-builtin)
optree/pytree.py:57:0: W0622: Redefining built-in 'min' (redefined-builtin)
optree/pytree.py:62:0: W0622: Redefining built-in 'sum' (redefined-builtin)
************* Module optree.treespec
optree/treespec.py:32:0: W0622: Redefining built-in 'dict' (redefined-builtin)
optree/treespec.py:35:0: W0622: Redefining built-in 'list' (redefined-builtin)
optree/treespec.py:40:0: W0622: Redefining built-in 'tuple' (redefined-builtin)

-----------------------------------
Your code has been rated at 9.94/10

make: *** [Makefile:191: pre-commit] Error 1

Actually, I tried to run make lint local but with error

$ make pylint
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pip show pylint &>/dev/null || (cd && /Users/lqhuang/Git/optree/.venv/bin/python3 -m pip install --upgrade pylint[spelling])
Collecting pylint[spelling]
  Using cached pylint-3.3.4-py3-none-any.whl.metadata (12 kB)
Collecting dill>=0.3.6 (from pylint[spelling])
  Using cached dill-0.3.9-py3-none-any.whl.metadata (10 kB)
Collecting platformdirs>=2.2.0 (from pylint[spelling])
  Using cached platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: astroid<=3.4.0-dev0,>=3.3.8 in ./Git/optree/.venv/lib/python3.12/site-packages (from pylint[spelling]) (3.3.8)
Collecting isort!=5.13.0,<7,>=4.2.5 (from pylint[spelling])
  Using cached isort-6.0.0-py3-none-any.whl.metadata (11 kB)
Collecting mccabe<0.8,>=0.6 (from pylint[spelling])
  Using cached mccabe-0.7.0-py2.py3-none-any.whl.metadata (5.0 kB)
Collecting tomlkit>=0.10.1 (from pylint[spelling])
  Using cached tomlkit-0.13.2-py3-none-any.whl.metadata (2.7 kB)
Requirement already satisfied: pyenchant~=3.2 in ./Git/optree/.venv/lib/python3.12/site-packages (from pylint[spelling]) (3.2.2)
Using cached dill-0.3.9-py3-none-any.whl (119 kB)
Using cached isort-6.0.0-py3-none-any.whl (94 kB)
Using cached mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB)
Using cached platformdirs-4.3.6-py3-none-any.whl (18 kB)
Using cached tomlkit-0.13.2-py3-none-any.whl (37 kB)
Using cached pylint-3.3.4-py3-none-any.whl (522 kB)
Installing collected packages: tomlkit, platformdirs, mccabe, isort, dill, pylint
Successfully installed dill-0.3.9 isort-6.0.0 mccabe-0.7.0 platformdirs-4.3.6 pylint-3.3.4 tomlkit-0.13.2
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pip show pyenchant &>/dev/null || (cd && /Users/lqhuang/Git/optree/.venv/bin/python3 -m pip install --upgrade pyenchant)
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pylint --version
pylint 3.3.4
astroid 3.3.8
Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pylint optree
usage: pylint [options]
pylint: error: argument --spelling-dict: invalid choice: 'en_US' (choose from '')
make: *** [pylint] Error 32

I searched the error, but google redirect me to an unsolved issue pylint-dev/pylint#5092

XuehaiPan · 2025-02-13T13:45:18Z

I tried to run make lint local but with error

Run

brew install enchant
pip3 install -U --pre pyenchant

lqhuang · 2025-02-13T13:47:03Z

I tried to run make lint local but with error

Run
brew install enchant
pip3 install -U --pre pyenchant

Doesn't work at all indeed without --pre.

So the --pre is the key?

…pec`

lqhuang · 2025-02-13T13:52:50Z

Oh man, it works. @XuehaiPan

Please approve again, sorry for that.

$ make pylint
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pip show pylint &>/dev/null || (cd && /Users/lqhuang/Git/optree/.venv/bin/python3 -m pip install --upgrade pylint[spelling])
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pip show pyenchant &>/dev/null || (cd && /Users/lqhuang/Git/optree/.venv/bin/python3 -m pip install --upgrade pyenchant)
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pylint --version
pylint 3.3.4
astroid 3.3.8
Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
/Users/lqhuang/Git/optree/.venv/bin/python3 -m pylint optree

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 9.95/10, +0.05)

XuehaiPan · 2025-02-13T13:56:56Z

So the --pre is the key?

The patch is only included in the rc release. You will need to pass --pre to install the pre-release package.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (1b47694) to head (cb3679b).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##              main      #189    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           36        42     +6     
  Lines         3789      3945   +156     
==========================================
+ Hits          3789      3945   +156

Flag	Coverage Δ
unittests	`100.00% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

XuehaiPan · 2025-02-13T21:32:03Z

@lqhuang Thanks for the contribution!

lqhuang · 2025-02-14T02:25:12Z

Seriously? You wake up at 6 a.m.?

Never mind, optree is obviously underrated, your work is like a gem 👋

feat: Add two new modules pytree and treespec re-exporting utils …

73afbb3

…from `ops` module

lqhuang requested a review from XuehaiPan as a code owner February 13, 2025 11:59

bugfix: Add missing function name to __all__

22906dc

lqhuang force-pushed the new-modules-simpler-impl branch from 33b44be to 22906dc Compare February 13, 2025 12:01

lqhuang added 2 commits February 13, 2025 20:09

test: Add testcase for re-export shortcut

34c4a44

lint: Fix test function name

02654aa

XuehaiPan approved these changes Feb 13, 2025

View reviewed changes

optree/treespec.py Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Show resolved Hide resolved

optree/pytree.py Outdated Show resolved Hide resolved

lqhuang mentioned this pull request Feb 13, 2025

refactor(module)!: A proposal to reorg modules and functions with new tree module and treespec module #186

Closed

29 tasks

lqhuang and others added 3 commits February 13, 2025 20:31

docs: Update optree/treespec.py

ed171b9

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

Update README.md

a6b3cdd

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

Update optree/pytree.py

1897493

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

XuehaiPan reviewed Feb 13, 2025

View reviewed changes

optree/pytree.py Outdated Show resolved Hide resolved

lqhuang and others added 2 commits February 13, 2025 20:37

Update README.md for pt.leaves demo cases

3a72c2e

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

Update optree/pytree.py: simplify doctest

624ca40

Co-authored-by: Xuehai Pan <XuehaiPan@outlook.com> Signed-off-by: Lanqing Huang <lqhuang@outlook.com>

docs: Add versionadded into spelling wordlist

66c636b

XuehaiPan added enhancement New feature or request py Something related to the Python source code labels Feb 13, 2025

lint: Disable pylint rule redefined-builtin

872c143

lint: disable redefined-builtin from pylint for pytree and `trees…

4f8e7ba

…pec`

XuehaiPan added 2 commits February 13, 2025 23:22

style: nit updates

37aaeba

lint: disable pylint false positive

7e793e2

XuehaiPan changed the title ~~refactor!(module): A proposal to reorg modules and functions with new tree module and treespec module~~ feat: add shortcut module optree.pytree and optree.treespec Feb 13, 2025

XuehaiPan added 2 commits February 14, 2025 01:00

feat: also export registry APIs

dc01df6

docs: improve sphinx cross-reference

a65aecd

XuehaiPan force-pushed the new-modules-simpler-impl branch from 9267dff to a65aecd Compare February 13, 2025 18:47

XuehaiPan force-pushed the main branch from a9f02e4 to 1150954 Compare February 13, 2025 21:20

XuehaiPan added 2 commits February 14, 2025 05:28

docs: fix docs

ba534eb

Merge branch 'main' into pr/lqhuang/189

cb3679b

XuehaiPan merged commit f442bec into metaopt:main Feb 13, 2025
28 of 29 checks passed

lqhuang deleted the new-modules-simpler-impl branch February 14, 2025 02:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add shortcut module `optree.pytree` and `optree.treespec` #189

feat: add shortcut module `optree.pytree` and `optree.treespec` #189

lqhuang commented Feb 13, 2025 •

edited

Loading

XuehaiPan left a comment

lqhuang commented Feb 13, 2025

XuehaiPan commented Feb 13, 2025

lqhuang commented Feb 13, 2025 •

edited

Loading

XuehaiPan commented Feb 13, 2025

lqhuang commented Feb 13, 2025

lqhuang commented Feb 13, 2025

XuehaiPan commented Feb 13, 2025

codecov bot commented Feb 13, 2025 •

edited

Loading

XuehaiPan commented Feb 13, 2025

lqhuang commented Feb 14, 2025

feat: add shortcut module optree.pytree and optree.treespec #189

feat: add shortcut module optree.pytree and optree.treespec #189

Conversation

lqhuang commented Feb 13, 2025 • edited Loading

XuehaiPan left a comment

Choose a reason for hiding this comment

lqhuang commented Feb 13, 2025

XuehaiPan commented Feb 13, 2025

lqhuang commented Feb 13, 2025 • edited Loading

XuehaiPan commented Feb 13, 2025

lqhuang commented Feb 13, 2025

lqhuang commented Feb 13, 2025

XuehaiPan commented Feb 13, 2025

codecov bot commented Feb 13, 2025 • edited Loading

Codecov Report

XuehaiPan commented Feb 13, 2025

lqhuang commented Feb 14, 2025

feat: add shortcut module `optree.pytree` and `optree.treespec` #189

feat: add shortcut module `optree.pytree` and `optree.treespec` #189

lqhuang commented Feb 13, 2025 •

edited

Loading

lqhuang commented Feb 13, 2025 •

edited

Loading

codecov bot commented Feb 13, 2025 •

edited

Loading