Poetry not pulling git submodules when installing scipy from source

[JPFrancoia]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Docker

• Poetry version: 1.1.11

Issue

This isn't about installing scipy, I have no problem installing this dependency when it's the latest version from Pypi (i.e: when the wheel is already built).

When installing scipy from source, I'm running into a problem that seems to be because poetry didn't do a git submodule update --init when cloning the source.

I will provide minimal Docker files to reproduce the issue.

This is the baseline, where I install a branch of scipy from source. This works:

FROM python:3.8

RUN apt-get update \
    && apt-get install -y make gcc g++ curl libpq-dev python3-dev libpng-dev libxft-dev gfortran python3-dev libopenblas-dev liblapack-dev

COPY . /app

WORKDIR app/

RUN pip install poetry

RUN pip install git+/~https://github.com/mckib2/scipy@boost-ibeta-derivative-overflow

However, when I try this instead:

FROM python:3.8

RUN apt-get update \
    && apt-get install -y make gcc g++ curl libpq-dev python3-dev libpng-dev libxft-dev gfortran python3-dev libopenblas-dev liblapack-dev

COPY . /app

WORKDIR app/

RUN pip install poetry

RUN poetry config virtualenvs.create false \
  && pip install wheel \
  && poetry install -vv -n --no-dev

I end up with the following error:

Traceback (most recent call last):
#9 785.8         File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
#9 785.8           main()
#9 785.8         File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
#9 785.8           json_out['return_val'] = hook(**hook_input['kwargs'])
#9 785.8         File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 151, in prepare_metadata_for_build_wheel
#9 785.8           return hook(metadata_directory, config_settings)
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 166, in prepare_metadata_for_build_wheel
#9 785.8           self.run_setup()
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 258, in run_setup
#9 785.8           super(_BuildMetaLegacyBackend,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
#9 785.8           exec(compile(code, __file__, 'exec'), locals())
#9 785.8         File "setup.py", line 622, in <module>
#9 785.8           setup_package()
#9 785.8         File "setup.py", line 618, in setup_package
#9 785.8           setup(**metadata)
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/core.py", line 137, in setup
#9 785.8           config = configuration()
#9 785.8         File "setup.py", line 528, in configuration
#9 785.8           config.add_subpackage('scipy')
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
#9 785.8           config_list = self.get_subpackage(subpackage_name, subpackage_path,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
#9 785.8           config = self._get_configuration_from_setup_py(
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
#9 785.8           config = setup_module.configuration(*args)
#9 785.8         File "scipy/setup.py", line 20, in configuration
#9 785.8           config.add_subpackage('sparse')
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
#9 785.8           config_list = self.get_subpackage(subpackage_name, subpackage_path,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
#9 785.8           config = self._get_configuration_from_setup_py(
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
#9 785.8           config = setup_module.configuration(*args)
#9 785.8         File "scipy/sparse/setup.py", line 14, in configuration
#9 785.8           config.add_subpackage('linalg')
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
#9 785.8           config_list = self.get_subpackage(subpackage_name, subpackage_path,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
#9 785.8           config = self._get_configuration_from_setup_py(
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
#9 785.8           config = setup_module.configuration(*args)
#9 785.8         File "scipy/sparse/linalg/setup.py", line 14, in configuration
#9 785.8           config.add_subpackage('_propack')
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
#9 785.8           config_list = self.get_subpackage(subpackage_name, subpackage_path,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
#9 785.8           config = self._get_configuration_from_setup_py(
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
#9 785.8           config = setup_module.configuration(*args)
#9 785.8         File "scipy/sparse/linalg/_propack/setup.py", line 36, in configuration
#9 785.8           check_propack_submodule()
#9 785.8         File "scipy/sparse/linalg/_propack/setup.py", line 13, in check_propack_submodule
#9 785.8           raise RuntimeError("Missing the `PROPACK` submodule! Run "
#9 785.8       RuntimeError: Missing the `PROPACK` submodule! Run `git submodule update --init` to fix this.
#9 785.8       ----------------------------------------
#9 785.8   WARNING: Discarding file:///usr/local/src/scipy. Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpk70ej4xc Check the logs for full command output.
#9 785.8   ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpk70ej4xc Check the logs for full command output.
#9 785.8   WARNING: You are using pip version 21.2.4; however, version 21.3 is available.
#9 785.8   You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#9 785.8
#9 785.8
#9 785.8   at /usr/local/lib/python3.8/site-packages/poetry/utils/env.py:1183 in _run
#9 785.9       1179│                 output = subprocess.check_output(
#9 785.9       1180│                     cmd, stderr=subprocess.STDOUT, **kwargs
#9 785.9       1181│                 )
#9 785.9       1182│         except CalledProcessError as e:
#9 785.9     → 1183│             raise EnvCommandError(e, input=input_)
#9 785.9       1184│
#9 785.9       1185│         return decode(output)
#9 785.9       1186│
#9 785.9       1187│     def execute(self, bin, *args, **kwargs):
#9 785.9
------
executor failed running [/bin/sh -c poetry config virtualenvs.create false   && pip install wheel   && poetry install -vv -n --no-dev]: exit code: 1

And here is the pyproject.toml that I used:

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = ">=3.8,<3.10"
scipy = { git = "/~https://github.com/mckib2/scipy.git", branch = "boost-ibeta-derivative-overflow" }
# scipy = { git = "ssh://git@github.com/mckib2/scipy.git", branch = "boost-ibeta-derivative-overflow" }

[tool.poetry.dev-dependencies]

[build-system]
requires = ["setuptools", "poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Do you have any idea what could be causing this issue?

[abn]

Can you please try the fix at #5428.

Using pipx

pipx install --suffix=@5428 'poetry @ git+/~https://github.com/python-poetry/poetry.git@refs/pull/5428/head'

Using a container (podman | docker)

podman run --rm -i --entrypoint bash docker.io/python:3.8 <<EOF
set -xe
apt-get update \
    && apt-get install -y make gcc g++ curl libpq-dev python3-dev libpng-dev libxft-dev gfortran python3-dev libopenblas-dev liblapack-dev
python -m pip install -q git+/~https://github.com/python-poetry/poetry.git@refs/pull/5428/head
poetry new foobar
pushd foobar
poetry add git+/~https://github.com/mckib2/scipy#boost-ibeta-derivative-overflow
EOF

Created package foobar in foobar
/foobar /
Creating virtualenv foobar-lWDpn5M1-py3.8 in /root/.cache/pypoetry/virtualenvs

Updating dependencies
Resolving dependencies... (14.6s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing numpy (1.22.3)
  • Installing scipy (1.8.0.dev0+0.1d1ccf6 1d1ccf6)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.