Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing dependency for cmake build in documentation. #7711

Closed
bluejack opened this issue Jan 16, 2025 · 10 comments · Fixed by #7838
Closed

Missing dependency for cmake build in documentation. #7711

bluejack opened this issue Jan 16, 2025 · 10 comments · Fixed by #7838
Assignees
Labels
module: doc Issues related to documentation, both in docs/ and inlined in code module: user experience Issues related to reducing friction for users

Comments

@bluejack
Copy link

bluejack commented Jan 16, 2025

📚 The doc issue

A dependency for the cmake build:

pip install zstd

Build will fail in a confusing way without this dependency prepared.

Suggest a potential alternative/fix

Add to documentation, probably here:

https://pytorch.org/executorch/stable/runtime-build-and-cross-compilation.html#one-time-setup-to-prepare-for-cmake-build

cc @mergennachin @byjlw

@mcr229 mcr229 added the module: doc Issues related to documentation, both in docs/ and inlined in code label Jan 17, 2025
@byjlw byjlw added the module: user experience Issues related to reducing friction for users label Jan 18, 2025
@github-project-automation github-project-automation bot moved this to To triage in ExecuTorch DevX Jan 18, 2025
@byjlw
Copy link
Contributor

byjlw commented Jan 18, 2025

Thanks for reporting it.

@swolchok
Copy link
Contributor

swolchok commented Jan 18, 2025

Weird, zstd is listed as a build-system dep in pyproject.toml. I wonder what's going wrong...

@swolchok
Copy link
Contributor

@bluejack can you provide the error you get if you don't have zstd installed?

@bluejack
Copy link
Author

Well, gosh! Now I cannot repro this problem with the release build or with the repo head. I hit it with build attempts on both branches, so whatever I was doing wrong I was repeating my mistake quite effectively.

I apologize for the confusion!

@mergennachin
Copy link
Contributor

Thanks @bluejack for raising the issue. Due to limited information, I will close this issue for now, but if you encounter similar issue please let us know. Thank you so much!

@github-project-automation github-project-automation bot moved this from To triage to Done in ExecuTorch DevX Jan 21, 2025
@swolchok
Copy link
Contributor

@shoumikhin just had this happen, reopening

@swolchok swolchok reopened this Jan 22, 2025
@github-project-automation github-project-automation bot moved this from Done to Backlog in ExecuTorch DevX Jan 22, 2025
@swolchok
Copy link
Contributor

I bet the problem is --no-build-isolation in install_executorch.py preventing dependencies from getting used...

@shoumikhin
Copy link
Contributor

Here's the log:

./install_executorch.sh --pybind xnnpack

Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/nightly/cpu
Collecting torch==2.6.0.dev20250104
  Downloading https://download.pytorch.org/whl/nightly/cpu/torch-2.6.0.dev20250104-cp312-none-macosx_11_0_arm64.whl.metadata (28 kB)
Collecting torchvision==0.22.0.dev20250104
  Downloading https://download.pytorch.org/whl/nightly/cpu/torchvision-0.22.0.dev20250104-cp312-cp312-macosx_11_0_arm64.whl (1.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 8.5 MB/s eta 0:00:00
Collecting torchaudio==2.6.0.dev20250104
  Downloading https://download.pytorch.org/whl/nightly/cpu/torchaudio-2.6.0.dev20250104-cp312-cp312-macosx_11_0_arm64.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 28.7 MB/s eta 0:00:00
Collecting timm==1.0.7 (from -r requirements-examples.txt (line 3))
  Using cached timm-1.0.7-py3-none-any.whl.metadata (47 kB)
Collecting torchsr==1.0.4 (from -r requirements-examples.txt (line 4))
  Using cached torchsr-1.0.4-py3-none-any.whl.metadata (12 kB)
Collecting transformers==4.47.1 (from -r requirements-examples.txt (line 5))
  Downloading transformers-4.47.1-py3-none-any.whl.metadata (44 kB)
Collecting filelock (from torch==2.6.0.dev20250104)
  Downloading filelock-3.17.0-py3-none-any.whl.metadata (2.9 kB)
Collecting typing-extensions>=4.10.0 (from torch==2.6.0.dev20250104)
  Using cached https://download.pytorch.org/whl/nightly/typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Collecting networkx (from torch==2.6.0.dev20250104)
  Using cached https://download.pytorch.org/whl/nightly/networkx-3.4.2-py3-none-any.whl (1.7 MB)
Collecting jinja2 (from torch==2.6.0.dev20250104)
  Using cached jinja2-3.1.5-py3-none-any.whl.metadata (2.6 kB)
Collecting fsspec (from torch==2.6.0.dev20250104)
  Using cached fsspec-2024.12.0-py3-none-any.whl.metadata (11 kB)
Collecting setuptools (from torch==2.6.0.dev20250104)
  Using cached setuptools-75.8.0-py3-none-any.whl.metadata (6.7 kB)
Collecting sympy==1.13.1 (from torch==2.6.0.dev20250104)
  Using cached https://download.pytorch.org/whl/nightly/sympy-1.13.1-py3-none-any.whl (6.2 MB)
Collecting numpy (from torchvision==0.22.0.dev20250104)
  Downloading numpy-2.2.2-cp312-cp312-macosx_14_0_arm64.whl.metadata (62 kB)
Collecting pillow!=8.3.*,>=5.3.0 (from torchvision==0.22.0.dev20250104)
  Using cached pillow-11.1.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (9.1 kB)
Collecting pyyaml (from timm==1.0.7->-r requirements-examples.txt (line 3))
  Using cached https://download.pytorch.org/whl/nightly/PyYAML-6.0.2-cp312-cp312-macosx_11_0_arm64.whl (173 kB)
Collecting huggingface_hub (from timm==1.0.7->-r requirements-examples.txt (line 3))
  Using cached huggingface_hub-0.27.1-py3-none-any.whl.metadata (13 kB)
Collecting safetensors (from timm==1.0.7->-r requirements-examples.txt (line 3))
  Using cached safetensors-0.5.2-cp38-abi3-macosx_11_0_arm64.whl.metadata (3.8 kB)
Requirement already satisfied: packaging>=20.0 in ./.venv/lib/python3.12/site-packages (from transformers==4.47.1->-r requirements-examples.txt (line 5)) (24.2)
Collecting regex!=2019.12.17 (from transformers==4.47.1->-r requirements-examples.txt (line 5))
  Using cached regex-2024.11.6-cp312-cp312-macosx_11_0_arm64.whl.metadata (40 kB)
Collecting requests (from transformers==4.47.1->-r requirements-examples.txt (line 5))
  Using cached https://download.pytorch.org/whl/nightly/requests-2.32.3-py3-none-any.whl (64 kB)
Collecting tokenizers<0.22,>=0.21 (from transformers==4.47.1->-r requirements-examples.txt (line 5))
  Using cached tokenizers-0.21.0-cp39-abi3-macosx_11_0_arm64.whl.metadata (6.7 kB)
Collecting tqdm>=4.27 (from transformers==4.47.1->-r requirements-examples.txt (line 5))
  Using cached tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
Collecting mpmath<1.4,>=1.1.0 (from sympy==1.13.1->torch==2.6.0.dev20250104)
  Using cached https://download.pytorch.org/whl/nightly/mpmath-1.3.0-py3-none-any.whl (536 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch==2.6.0.dev20250104)
  Using cached MarkupSafe-3.0.2-cp312-cp312-macosx_11_0_arm64.whl.metadata (4.0 kB)
Collecting charset-normalizer<4,>=2 (from requests->transformers==4.47.1->-r requirements-examples.txt (line 5))
  Using cached charset_normalizer-3.4.1-cp312-cp312-macosx_10_13_universal2.whl.metadata (35 kB)
Collecting idna<4,>=2.5 (from requests->transformers==4.47.1->-r requirements-examples.txt (line 5))
  Using cached https://download.pytorch.org/whl/nightly/idna-3.10-py3-none-any.whl (70 kB)
Collecting urllib3<3,>=1.21.1 (from requests->transformers==4.47.1->-r requirements-examples.txt (line 5))
  Using cached urllib3-2.3.0-py3-none-any.whl.metadata (6.5 kB)
Collecting certifi>=2017.4.17 (from requests->transformers==4.47.1->-r requirements-examples.txt (line 5))
  Using cached certifi-2024.12.14-py3-none-any.whl.metadata (2.3 kB)
Downloading https://download.pytorch.org/whl/nightly/cpu/torch-2.6.0.dev20250104-cp312-none-macosx_11_0_arm64.whl (66.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.7/66.7 MB 35.9 MB/s eta 0:00:00
Using cached timm-1.0.7-py3-none-any.whl (2.3 MB)
Using cached torchsr-1.0.4-py3-none-any.whl (31 kB)
Downloading transformers-4.47.1-py3-none-any.whl (10.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.1/10.1 MB 32.0 MB/s eta 0:00:00
Using cached huggingface_hub-0.27.1-py3-none-any.whl (450 kB)
Using cached fsspec-2024.12.0-py3-none-any.whl (183 kB)
Downloading numpy-2.2.2-cp312-cp312-macosx_14_0_arm64.whl (5.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.1/5.1 MB 40.3 MB/s eta 0:00:00
Using cached pillow-11.1.0-cp312-cp312-macosx_11_0_arm64.whl (3.1 MB)
Using cached regex-2024.11.6-cp312-cp312-macosx_11_0_arm64.whl (284 kB)
Using cached safetensors-0.5.2-cp38-abi3-macosx_11_0_arm64.whl (408 kB)
Using cached tokenizers-0.21.0-cp39-abi3-macosx_11_0_arm64.whl (2.6 MB)
Using cached tqdm-4.67.1-py3-none-any.whl (78 kB)
Downloading filelock-3.17.0-py3-none-any.whl (16 kB)
Using cached jinja2-3.1.5-py3-none-any.whl (134 kB)
Using cached setuptools-75.8.0-py3-none-any.whl (1.2 MB)
Using cached certifi-2024.12.14-py3-none-any.whl (164 kB)
Using cached charset_normalizer-3.4.1-cp312-cp312-macosx_10_13_universal2.whl (196 kB)
Using cached MarkupSafe-3.0.2-cp312-cp312-macosx_11_0_arm64.whl (12 kB)
Using cached urllib3-2.3.0-py3-none-any.whl (128 kB)
Installing collected packages: mpmath, urllib3, typing-extensions, tqdm, sympy, setuptools, safetensors, regex, pyyaml, pillow, numpy, networkx, MarkupSafe, idna, fsspec, filelock, charset-normalizer, certifi, requests, jinja2, torch, huggingface_hub, torchvision, torchaudio, tokenizers, transformers, torchsr, timm
Successfully installed MarkupSafe-3.0.2 certifi-2024.12.14 charset-normalizer-3.4.1 filelock-3.17.0 fsspec-2024.12.0 huggingface_hub-0.27.1 idna-3.10 jinja2-3.1.5 mpmath-1.3.0 networkx-3.4.2 numpy-2.2.2 pillow-11.1.0 pyyaml-6.0.2 regex-2024.11.6 requests-2.32.3 safetensors-0.5.2 setuptools-75.8.0 sympy-1.13.1 timm-1.0.7 tokenizers-0.21.0 torch-2.6.0.dev20250104 torchaudio-2.6.0.dev20250104 torchsr-1.0.4 torchvision-0.22.0.dev20250104 tqdm-4.67.1 transformers-4.47.1 typing-extensions-4.12.2 urllib3-2.3.0
Processing ./third-party/ao
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: torchao
  Building wheel for torchao (pyproject.toml) ... done
  Created wheel for torchao: filename=torchao-0.8.0+git11333ba-py3-none-any.whl size=677359 sha256=790ad33717ced6f077330be8481f38100890f5ce3433ee28c206797196585924
  Stored in directory: /private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-ephem-wheel-cache-1pd_elr0/wheels/02/9b/94/d870dbecb7d5b85280802e4b26e82c3425ff1d7adf2c543326
Successfully built torchao
Installing collected packages: torchao
Successfully installed torchao-0.8.0+git11333ba
Using pip 24.3.1 from /Users/shoumikhin/Temp/executorch/.venv/lib/python3.12/site-packages/pip (python 3.12)
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/nightly/cpu
Processing /Users/shoumikhin/Temp/executorch
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch.egg-info
  writing /private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch.egg-info/dependency_links.txt
  writing entry points to /private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch.egg-info/entry_points.txt
  writing requirements to /private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch.egg-info/requires.txt
  writing top-level names to /private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch.egg-info/top_level.txt
  writing manifest file '/private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file '/private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch.egg-info/SOURCES.txt'
  creating '/private/var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/pip-modern-metadata-e4xtwalz/executorch-0.6.0a0+cd9ab87.dist-info'
  Preparing metadata (pyproject.toml) ... done
Collecting expecttest (from executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for expecttest from https://files.pythonhosted.org/packages/27/fb/deeefea1ea549273817ca7bed3db2f39cc238a75a745a20e3651619f7335/expecttest-0.3.0-py3-none-any.whl.metadata
  Using cached expecttest-0.3.0-py3-none-any.whl.metadata (3.8 kB)
Collecting flatbuffers (from executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for flatbuffers from https://files.pythonhosted.org/packages/55/75/ed93520bceceabd78305717b3500224625f92e03617ba1f8e08d60e8eefb/flatbuffers-25.1.21-py2.py3-none-any.whl.metadata
  Downloading flatbuffers-25.1.21-py2.py3-none-any.whl.metadata (875 bytes)
Collecting hypothesis (from executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for hypothesis from https://files.pythonhosted.org/packages/a3/e1/80f1819e65a9a0b1b744b453cd26803764fc192cff8eaea202e76f6f5a65/hypothesis-6.124.2-py3-none-any.whl.metadata
  Downloading hypothesis-6.124.2-py3-none-any.whl.metadata (6.3 kB)
Requirement already satisfied: mpmath==1.3.0 in ./.venv/lib/python3.12/site-packages (from executorch==0.6.0a0+cd9ab87) (1.3.0)
  Link requires a different Python (3.12.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/3a/be/650f9c091ef71cb01d735775d554e068752d3ff63d7943b26316dc401749/numpy-1.21.2.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
  Link requires a different Python (3.12.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/5f/d6/ad58ded26556eaeaa8c971e08b6466f17c4ac4d786cd3d800e26ce59cc01/numpy-1.21.3.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
  Link requires a different Python (3.12.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
  Link requires a different Python (3.12.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/c2/a8/a924a09492bdfee8c2ec3094d0a13f2799800b4fdc9c890738aeeb12c72e/numpy-1.21.5.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
  Link requires a different Python (3.12.4 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/45/b7/de7b8e67f2232c26af57c205aaad29fe17754f793404f59c8a730c7a191a/numpy-1.21.6.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
Collecting numpy==2.0.0 (from executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for numpy==2.0.0 from https://files.pythonhosted.org/packages/3f/ab/1dc9f176d3084a2546cf76eb213dc61586d015ef59b3b17947b0e40038af/numpy-2.0.0-cp312-cp312-macosx_14_0_arm64.whl.metadata
  Using cached numpy-2.0.0-cp312-cp312-macosx_14_0_arm64.whl.metadata (60 kB)
Requirement already satisfied: packaging in ./.venv/lib/python3.12/site-packages (from executorch==0.6.0a0+cd9ab87) (24.2)
Collecting pandas==2.2.2 (from executorch==0.6.0a0+cd9ab87)
  Using cached https://download.pytorch.org/whl/nightly/pandas-2.2.2-cp312-cp312-macosx_11_0_arm64.whl (11.3 MB)
Collecting parameterized (from executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for parameterized from https://files.pythonhosted.org/packages/00/2f/804f58f0b856ab3bf21617cccf5b39206e6c4c94c2cd227bde125ea6105f/parameterized-0.9.0-py2.py3-none-any.whl.metadata
  Using cached parameterized-0.9.0-py2.py3-none-any.whl.metadata (18 kB)
Collecting pytest (from executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for pytest from https://files.pythonhosted.org/packages/11/92/76a1c94d3afee238333bc0a42b82935dd8f9cf8ce9e336ff87ee14d9e1cf/pytest-8.3.4-py3-none-any.whl.metadata
  Using cached pytest-8.3.4-py3-none-any.whl.metadata (7.5 kB)
Collecting pytest-xdist (from executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for pytest-xdist from https://files.pythonhosted.org/packages/6d/82/1d96bf03ee4c0fdc3c0cbe61470070e659ca78dc0086fb88b66c185e2449/pytest_xdist-3.6.1-py3-none-any.whl.metadata
  Using cached pytest_xdist-3.6.1-py3-none-any.whl.metadata (4.3 kB)
Requirement already satisfied: pyyaml in ./.venv/lib/python3.12/site-packages (from executorch==0.6.0a0+cd9ab87) (6.0.2)
Collecting ruamel.yaml (from executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for ruamel.yaml from https://files.pythonhosted.org/packages/c2/36/dfc1ebc0081e6d39924a2cc53654497f967a084a436bb64402dfce4254d9/ruamel.yaml-0.18.10-py3-none-any.whl.metadata
  Using cached ruamel.yaml-0.18.10-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: sympy in ./.venv/lib/python3.12/site-packages (from executorch==0.6.0a0+cd9ab87) (1.13.1)
Collecting tabulate (from executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for tabulate from https://files.pythonhosted.org/packages/40/44/4a5f08c96eb108af5cb50b41f76142f0afa346dfa99d5296fe7202a11854/tabulate-0.9.0-py3-none-any.whl.metadata
  Using cached tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
Requirement already satisfied: typing-extensions in ./.venv/lib/python3.12/site-packages (from executorch==0.6.0a0+cd9ab87) (4.12.2)
Collecting python-dateutil>=2.8.2 (from pandas==2.2.2->executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for python-dateutil>=2.8.2 from https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas==2.2.2->executorch==0.6.0a0+cd9ab87)
  Using cached https://download.pytorch.org/whl/nightly/pytz-2024.2-py2.py3-none-any.whl (508 kB)
Collecting tzdata>=2022.7 (from pandas==2.2.2->executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for tzdata>=2022.7 from https://files.pythonhosted.org/packages/0f/dd/84f10e23edd882c6f968c21c2434fe67bd4a528967067515feca9e611e5e/tzdata-2025.1-py2.py3-none-any.whl.metadata
  Downloading tzdata-2025.1-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting attrs>=22.2.0 (from hypothesis->executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for attrs>=22.2.0 from https://files.pythonhosted.org/packages/89/aa/ab0f7891a01eeb2d2e338ae8fecbe57fcebea1a24dbb64d45801bfab481d/attrs-24.3.0-py3-none-any.whl.metadata
  Using cached attrs-24.3.0-py3-none-any.whl.metadata (11 kB)
Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis->executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for sortedcontainers<3.0.0,>=2.1.0 from https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl.metadata
  Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting iniconfig (from pytest->executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for iniconfig from https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl.metadata
  Using cached iniconfig-2.0.0-py3-none-any.whl.metadata (2.6 kB)
Collecting pluggy<2,>=1.5 (from pytest->executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for pluggy<2,>=1.5 from https://files.pythonhosted.org/packages/88/5f/e351af9a41f866ac3f1fac4ca0613908d9a41741cfcf2228f4ad853b697d/pluggy-1.5.0-py3-none-any.whl.metadata
  Using cached pluggy-1.5.0-py3-none-any.whl.metadata (4.8 kB)
Collecting execnet>=2.1 (from pytest-xdist->executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for execnet>=2.1 from https://files.pythonhosted.org/packages/43/09/2aea36ff60d16dd8879bdb2f5b3ee0ba8d08cbbdcdfe870e695ce3784385/execnet-2.1.1-py3-none-any.whl.metadata
  Using cached execnet-2.1.1-py3-none-any.whl.metadata (2.9 kB)
Collecting ruamel.yaml.clib>=0.2.7 (from ruamel.yaml->executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for ruamel.yaml.clib>=0.2.7 from https://files.pythonhosted.org/packages/48/41/e7a405afbdc26af961678474a55373e1b323605a4f5e2ddd4a80ea80f628/ruamel.yaml.clib-0.2.12-cp312-cp312-macosx_14_0_arm64.whl.metadata
  Using cached ruamel.yaml.clib-0.2.12-cp312-cp312-macosx_14_0_arm64.whl.metadata (1.2 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas==2.2.2->executorch==0.6.0a0+cd9ab87)
  Obtaining dependency information for six>=1.5 from https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl.metadata
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Using cached numpy-2.0.0-cp312-cp312-macosx_14_0_arm64.whl (5.0 MB)
Using cached expecttest-0.3.0-py3-none-any.whl (8.2 kB)
Downloading flatbuffers-25.1.21-py2.py3-none-any.whl (30 kB)
Downloading hypothesis-6.124.2-py3-none-any.whl (482 kB)
Using cached parameterized-0.9.0-py2.py3-none-any.whl (20 kB)
Using cached pytest-8.3.4-py3-none-any.whl (343 kB)
Using cached pytest_xdist-3.6.1-py3-none-any.whl (46 kB)
Using cached ruamel.yaml-0.18.10-py3-none-any.whl (117 kB)
Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Using cached attrs-24.3.0-py3-none-any.whl (63 kB)
Using cached execnet-2.1.1-py3-none-any.whl (40 kB)
Using cached pluggy-1.5.0-py3-none-any.whl (20 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached ruamel.yaml.clib-0.2.12-cp312-cp312-macosx_14_0_arm64.whl (133 kB)
Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Downloading tzdata-2025.1-py2.py3-none-any.whl (346 kB)
Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: executorch
  Running command Building wheel for executorch (pyproject.toml)
  running bdist_wheel
  running build
  command options for 'CustomBuild':
    build_base = pip-out
    build_purelib = pip-out/lib
    build_platlib = pip-out/lib.macosx-10.9-universal2-cpython-312
    build_lib = pip-out/lib.macosx-10.9-universal2-cpython-312
    build_scripts = pip-out/scripts-3.12
    build_temp = pip-out/temp.macosx-10.9-universal2-cpython-312
    plat_name = macosx-10.9-universal2
    compiler = None
    parallel = 15
    debug = None
    force = None
    executable = /Users/shoumikhin/Temp/executorch/.venv/bin/python3
  creating /Users/shoumikhin/Temp/executorch/pip-out/temp.macosx-10.9-universal2-cpython-312/cmake-out
  deleting /Users/shoumikhin/Temp/executorch/pip-out/temp.macosx-10.9-universal2-cpython-312/cmake-out/CMakeCache.txt
  cmake -S /Users/shoumikhin/Temp/executorch -B /Users/shoumikhin/Temp/executorch/pip-out/temp.macosx-10.9-universal2-cpython-312/cmake-out -DBUCK2= -DPYTHON_EXECUTABLE=/Users/shoumikhin/Temp/executorch/.venv/bin/python3 -DCMAKE_PREFIX_PATH=/Users/shoumikhin/Temp/executorch/.venv/lib/python3.12/site-packages -DCMAKE_BUILD_TYPE=Release -DEXECUTORCH_ENABLE_LOGGING=ON -DEXECUTORCH_LOG_LEVEL=Info -DCMAKE_OSX_DEPLOYMENT_TARGET=10.15 -DEXECUTORCH_SEPARATE_FLATCC_HOST_PROJECT=OFF -DEXECUTORCH_BUILD_PYBIND=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON -DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON -DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON -DEXECUTORCH_BUILD_XNNPACK=ON
  -- The C compiler identification is AppleClang 16.0.0.16000026
  -- The CXX compiler identification is AppleClang 16.0.0.16000026
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /Applications/Xcode_16.2.0_16C5023f_fb.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /Applications/Xcode_16.2.0_16C5023f_fb.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  CMake Deprecation Warning at backends/xnnpack/third-party/FXdiv/CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
    Compatibility with CMake < 3.10 will be removed from a future version of
    CMake.

    Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
    to tell CMake that the project requires at least <min> but has been updated
    to work with policies introduced by <max> or earlier.


  CMake Deprecation Warning at backends/xnnpack/third-party/cpuinfo/CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
    Compatibility with CMake < 3.10 will be removed from a future version of
    CMake.

    Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
    to tell CMake that the project requires at least <min> but has been updated
    to work with policies introduced by <max> or earlier.


  CMake Deprecation Warning at backends/xnnpack/third-party/pthreadpool/CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
    Compatibility with CMake < 3.10 will be removed from a future version of
    CMake.

    Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
    to tell CMake that the project requires at least <min> but has been updated
    to work with policies introduced by <max> or earlier.


  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
  -- Found Threads: TRUE
  -- Using python executable '/Users/shoumikhin/Temp/executorch/.venv/bin/python3'
  CMake Warning at build/Utils.cmake:278 (message):
    Failed to resolve buck2.
  Call Stack (most recent call first):
    CMakeLists.txt:372 (resolve_buck2)


  CMake Warning at build/Utils.cmake:279 (message):
    Traceback (most recent call last):

      File "/Users/shoumikhin/Temp/executorch/build/resolve_buck.py", line 21, in <module>
        import zstd

    ModuleNotFoundError: No module named 'zstd'

  Call Stack (most recent call first):
    CMakeLists.txt:372 (resolve_buck2)


  -- Killing buck2 daemon
  'buck2 killall'
  -- executorch: Generating source lists
  -- executorch: Generating source file list /Users/shoumikhin/Temp/executorch/pip-out/temp.macosx-10.9-universal2-cpython-312/cmake-out/executorch_srcs.cmake
  Error while generating /Users/shoumikhin/Temp/executorch/pip-out/temp.macosx-10.9-universal2-cpython-312/cmake-out/executorch_srcs.cmake. Exit code: 1
  Output:

  Error:
  Traceback (most recent call last):
    File "/Users/shoumikhin/Temp/executorch/build/buck_util.py", line 26, in run
      cp: subprocess.CompletedProcess = subprocess.run(
                                        ^^^^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['buck2', 'cquery', "inputs(deps('//runtime/executor:program'))"]' returned non-zero exit status 1.

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/Users/shoumikhin/Temp/executorch/build/extract_sources.py", line 232, in <module>
      main()
    File "/Users/shoumikhin/Temp/executorch/build/extract_sources.py", line 217, in main
      target_to_srcs[name] = sorted(target.get_sources(graph, runner, buck_args))
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/shoumikhin/Temp/executorch/build/extract_sources.py", line 121, in get_sources
      sources: set[str] = set(runner.run(["cquery", query] + buck_args))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/shoumikhin/Temp/executorch/build/buck_util.py", line 34, in run
      raise RuntimeError(ex.stderr.decode("utf-8")) from ex
  RuntimeError: Error: Could not read dotslash file /Users/shoumikhin/Temp/executorch/.buck2: No such file or directory (os error 2)


  CMake Error at build/Utils.cmake:230 (message):
    executorch: source list generation failed
  Call Stack (most recent call first):
    CMakeLists.txt:381 (extract_sources)


  -- Configuring incomplete, errors occurred!
  error: command '/Library/Frameworks/Python.framework/Versions/3.12/bin/cmake' failed with exit code 1
  error: subprocess-exited-with-error

  × Building wheel for executorch (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/shoumikhin/Temp/executorch/.venv/bin/python3 /Users/shoumikhin/Temp/executorch/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /var/folders/6h/sjyn7zg5105dw8nc5h3xk8d00000gn/T/tmp3j8b3hja
  cwd: /Users/shoumikhin/Temp/executorch
  Building wheel for executorch (pyproject.toml) ... error
  ERROR: Failed building wheel for executorch
Failed to build executorch
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (executorch)
Traceback (most recent call last):
  File "/Users/shoumikhin/Temp/executorch/./install_executorch.py", line 142, in <module>
    main(sys.argv[1:])
  File "/Users/shoumikhin/Temp/executorch/./install_executorch.py", line 120, in main
    subprocess.run(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/shoumikhin/Temp/executorch/.venv/bin/python3', '-m', 'pip', 'install', '.', '--no-build-isolation', '-v', '--extra-index-url', 'https://download.pytorch.org/whl/nightly/cpu']' returned non-zero exit status 1.

@swolchok
Copy link
Contributor

I can repro. Steps:

./install_executorch.sh --clean # zstd is only needed for first-time cmake runn
rm -rf buck2-bin buck-out # zstd is needed to download buck
./install_executorch.sh --pybind xnnpack

@swolchok
Copy link
Contributor

I think the problem here is the --no-build-isolation flag. Removing it prevents gen_oplist.py from importing torchgen. Not sure why "don't install dependencies at all" and "prefer installed dependencies if available" are coupled.

The much-debated --only-deps flag to pip install (which doesn't exist) would solve our problem here nicely. In its absence, I guess we need to go back to duplicating dev dependencies.

swolchok added a commit that referenced this issue Jan 22, 2025
Installation apparently couldn't find buck from a clean checkout. (I'm not sure how it managed to work for anybody, actually.)

Fixes #7711.

Test Plan: ./install_executorch.sh --clean
rm -rf buck2-bin buck-out
./install_executorch.sh --pybind xnnpack
@swolchok swolchok self-assigned this Jan 22, 2025
@mergennachin mergennachin moved this from Backlog to In progress in ExecuTorch DevX Jan 22, 2025
@github-project-automation github-project-automation bot moved this from In progress to Done in ExecuTorch DevX Jan 22, 2025
YIWENX14 pushed a commit that referenced this issue Jan 28, 2025
Fix install

Installation apparently couldn't find buck from a clean checkout. (I'm not sure how it managed to work for anybody, actually.)

Fixes #7711.

Test Plan: ./install_executorch.sh --clean
rm -rf buck2-bin buck-out
./install_executorch.sh --pybind xnnpack
zonglinpeng pushed a commit to zonglinpeng/executorch that referenced this issue Jan 30, 2025
Fix install

Installation apparently couldn't find buck from a clean checkout. (I'm not sure how it managed to work for anybody, actually.)

Fixes pytorch#7711.

Test Plan: ./install_executorch.sh --clean
rm -rf buck2-bin buck-out
./install_executorch.sh --pybind xnnpack
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module: doc Issues related to documentation, both in docs/ and inlined in code module: user experience Issues related to reducing friction for users
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

6 participants