support for multiple sources #716

ghost · 2017-09-26T21:41:31Z

I have tried to define two sources in Pipfile, something like this:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://myprivatepypi.com/simple"
verify_ssl = true
name = "internal"

[requires]
python_version = "2.7"

[dev-packages]
"coverage" = {version="*", index="pypi"}

[packages]
"my-private-package" = {version="*", index="internal"}

However, when I try pipenv install I run into an error, which includes a 404, saying that pipenv is looking for 'my-private-package' in https://pypi.python.org/simple.

/~https://github.com/kennethreitz/pipenv/blob/1faec7f251780e318264dee0699e6936e0c2000b/pipenv/project.py#L412-L413

Seems like the code might just be grabbing the first source in the Pipfile?

The text was updated successfully, but these errors were encountered:

kennethreitz · 2017-09-26T21:54:09Z

let me take a look

kennethreitz · 2017-09-26T21:55:50Z

are you using the latest version?

kennethreitz · 2017-09-26T22:01:41Z

this is confirmed as working on the latest version.

kennethreitz · 2017-09-26T22:02:53Z

please ensure you're using the latest version and enclose the full traceback if so. Closing until then.

ghost · 2017-09-26T22:31:08Z

Thanks for taking a look. Here's the full set of commands:

egustavson /home/egustavson/projects/dea-core 
(master)$ pipenv --version
pipenv, version 8.0.5


egustavson /home/egustavson/projects/dea-core 
(master)$ cat Pipfile 
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://pypicloud.test.netflix.net:7002/pypi"
verify_ssl = true
name = "netflix"

[requires]
python_version = "2.7"

[dev-packages]
"nflx-kragle-scripts" = {version="*", index="netflix"}
"nflx-cloudsol-python-libs" = {version="*", index="netflix"}
"coverage" = {version="*", index="pypi"}

[packages]
"nflx-kragle" = {version="*", index="netflix"}
"nflx-kragle-quinto" = {version="*", index="netflix"}


egustavson /home/egustavson/projects/dea-core 
(master)$ cat Pipfile.lock
cat: Pipfile.lock: No such file or directory


egustavson /home/egustavson/projects/dea-core 
(master)$ pipenv install
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
404 Client Error: Not Found (nflx-cloudsol-python-libs does not exist) for url: https://pypi.python.org/simple/nflx-cloudsol-python-libs/

kennethreitz · 2017-09-26T22:37:23Z

Ah, interesting.

kennethreitz · 2017-09-26T22:38:54Z

Do as the warning suggests, and use install --skip-lock until we get to the bottom of this.

ghost · 2017-09-26T22:44:04Z

I tried it and got a stacktrace:

egustavson /home/egustavson/projects/dea-core 
(master)$ pipenv install --skip-lock
Installing dependencies from Pipfile…
Traceback (most recent call last):▉▉▉▉ 0/2 — 00:00:00
  File "/home/egustavson/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pipenv/cli.py", line 1721, in install
    do_init(dev=dev, allow_global=system, ignore_pipfile=ignore_pipfile, system=system, skip_lock=skip_lock, verbose=verbose, concurrent=concurrent, deploy=deploy)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pipenv/cli.py", line 1238, in do_init
    skip_lock=skip_lock, verbose=verbose, concurrent=concurrent)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pipenv/cli.py", line 794, in do_install_dependencies
    index=index
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pipenv/cli.py", line 1260, in pip_install
    if [x for x in requirements.parse(package_name.split('--hash')[0])][0].vcs:
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/requirements/parser.py", line 50, in parse
    yield Requirement.parse(line)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/requirements/requirement.py", line 220, in parse
    return cls.parse_line(line)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/requirements/requirement.py", line 198, in parse_line
    pkg_req = Req.parse(line)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3038, in parse
    req, = parse_requirements(s)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2986, in parse_requirements
    yield Requirement(line)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2995, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'--truste'"
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/2 — 00:00:00

kennethreitz · 2017-09-26T22:52:21Z

try latest, v8.0.6. I think I fixed it.

ghost · 2017-09-26T23:05:17Z

Seems to be working now, thanks!

kennethreitz · 2017-09-26T23:05:31Z

excellent news!

kennethreitz · 2017-09-26T23:05:44Z

i think you're our first customer using that feature haha

helderco · 2017-09-29T10:21:59Z

Hey @kennethreitz, second customer here :)

Been giving pipenv another shot since yesterday, but install hangs on some packages. I've tried removing my second source (we have internal, private packages), and no longer hangs.

$ pipenv --version
pipenv, version 8.2.0

Tried in Mac and Ubuntu.

helderco · 2017-09-29T10:24:52Z

One thing I did differently than @egustavson is I only specified my private index and left pypi untouched:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://pypi.example.org/simple"
verify_ssl = true
name = "internal"

[packages]
django = "~=1.11.5"
mysqlclient = "*"
django-environ = "*"
django-mfcookie = {version="*", index="internal"}

helderco · 2017-09-29T10:33:22Z

Tried this and still hangs:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://pypi.example.org/simple"
verify_ssl = true
name = "internal"

[packages]
django =  {version="~=1.11.5", index="pypi"}
mysqlclient =  {version="*", index="pypi"}
django-environ = {version="*", index="pypi"}
django-mfcookie = {version="*", index="internal"}

Here's the output:

$ pipenv --three install --verbose
Virtualenv already exists!
Removing existing virtualenv…
Creating a virtualenv for this project…
Using /usr/local/bin/python3 to create virtualenv…
⠋Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/python3.6
Also creating executable in /Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A
Installing dependencies from Pipfile.lock…
Installing 'pytz==2017.2 --hash=sha256:c883c2d6670042c7bc1688645cac73dd2b03193d1f7a6847b6154e96890be06d  --hash=sha256:03c9962afe00e503e2d96abab4e8998a0f84d4230fa57afe1e0528473698cdd9  --hash=sha256:487e7d50710661116325747a9cd1744d3323f8e49748e287bc9e659060ec6bf9  --hash=sha256:43f52d4c6a0be301d53ebd867de05e2926c35728b3260157d274635a0a947f1c  --hash=sha256:d1d6729c85acea5423671382868627129432fba9a89ecbb248d8d1c7a9f01c67  --hash=sha256:54a935085f7bf101f86b2aff75bd9672b435f51c3339db2ff616e66845f2b8f9  --hash=sha256:39504670abb5dae77f56f8eb63823937ce727d7cdd0088e6909e6dcac0f89043  --hash=sha256:ddc93b6d41cfb81266a27d23a79e13805d4a5521032b512643af8729041a81b4  --hash=sha256:f5c056e8f62d45ba8215e5cb8f50dfccb198b4b9fbea8500674f3443e4689589'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-nD8Lmf-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-nD8Lmf-requirement.txt --require-hashes -i https://pypi.example.org/simple --exists-action w
Installing 'django-environ==0.4.4 --hash=sha256:e9c171b9d5f540e6f3bc42866941d9cd0bd77fb110a7c13a7c4857a2c08cfa40  --hash=sha256:ee2f8405d83137e3328b26b3de01bd715b5395fca22feb919dcc905fb6099cfa'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-aWT2Lj-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
Installing 'six==1.11.0 --hash=sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb  --hash=sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-E_7UsF-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-E_7UsF-requirement.txt --require-hashes -i https://pypi.example.org/simple --exists-action w
Installing 'mysqlclient==1.3.12 --hash=sha256:b3b1a7e4468180afb79289b54069d9499242946a4cedf3928cbf6b2a13800016  --hash=sha256:d56e379c03efad746e84705cbb97401f60d1f98b05e11a27f2d9c2d043936974  --hash=sha256:371df79d000af56b4e540b7ce2120d1c9afb04b751bfce25a1eb609c50fd10ff  --hash=sha256:1e85e48b167e2af3bb08f273fdbd1ad6401cbe75057fa6513f97387dc7b282dc  --hash=sha256:2d9ec33de39f4d9c64ad7322ede0521d85829ce36a76f9dd3d6ab76a9c8648e5'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-GU9uWO-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
Installing 'django-mfcookie==0.3.3'▉▉▉ 4/6 — 00:00:00
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-zn2odJ-requirement.txt -i https://pypi.example.org/simple --exists-action w
Installing 'django==1.11.5 --hash=sha256:89162f70a74aac62a53f975128faba6099a7ef2c9d8140a41ae9d6210bda05cd  --hash=sha256:1836878162dfdf865492bacfdff0321e4ee8f1e7d51d93192546000b54982b29'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-szL0JQ-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w

kennethreitz · 2018-03-13T12:16:18Z

Confirmed with this Pipfile:

[[source]]

url = "http://pypi.home.kennethreitz.org/simple"
verify_ssl = false
name = "home"

[[source]]

url = "https://testpypi.python.org/pypi"
verify_ssl = true
name = "test"


[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"




[packages]

flask = {version="*", index="home"}
"cs18-sidecar" = {version="*", index="test"}
django = {version="*", index="pypi"}


[dev-packages]

kennethreitz · 2018-03-13T12:20:48Z

Hmm, everything should be working as expected:

['flask -i http://pypi.home.kennethreitz.org/simple --trusted-host pypi.home.kennethreitz.org', 'cs18-sidecar -i https://testpypi.python.org/pypi', 'django -i https://pypi.python.org/simple']

digging in...

techalchemy · 2018-03-13T12:48:15Z

@kennethreitz I have some code on this as of yesterday which started to handle the cross-index resolution problem. It worked actually but it wrote the wrong info to the lockfile. I did note that we currently write no info to the lockfile

techalchemy · 2018-03-13T12:48:31Z

About indexes that is

kennethreitz · 2018-03-13T12:49:13Z

@techalchemy please focus on the issues i have assigned to you :)

kennethreitz · 2018-03-13T12:49:25Z

Those are the areas i need help with currently :)

techalchemy · 2018-03-20T22:48:25Z

I think we are supporting this properly now, closing!

jonlil · 2018-04-11T20:19:04Z

I think we are supporting this properly now, closing!

@techalchemy - can you elaborate why you think this is supported now?

techalchemy · 2018-04-11T23:36:10Z

@jonlil this is supported because the example Pipfile can be installed, locked, etc without issue:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://testpypi.python.org/pypi"
verify_ssl = true
name = "testpypi"

[packages]
pipenv-test-private-package = {version = "*", index = "testpypi"}
requests = "*"

kennethreitz closed this as completed Sep 26, 2017

kennethreitz reopened this Sep 26, 2017

ghost closed this as completed Sep 26, 2017

kimvais mentioned this issue Mar 13, 2018

Support for multiple sources is broken #1695

Closed

kennethreitz reopened this Mar 13, 2018

techalchemy closed this as completed Mar 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support for multiple sources #716

support for multiple sources #716

ghost commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

ghost commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

ghost commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

ghost commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

helderco commented Sep 29, 2017 •

edited

Loading

helderco commented Sep 29, 2017 •

edited

Loading

helderco commented Sep 29, 2017

kennethreitz commented Mar 13, 2018

kennethreitz commented Mar 13, 2018

techalchemy commented Mar 13, 2018

techalchemy commented Mar 13, 2018

kennethreitz commented Mar 13, 2018

kennethreitz commented Mar 13, 2018

techalchemy commented Mar 20, 2018

jonlil commented Apr 11, 2018

techalchemy commented Apr 11, 2018 •

edited

Loading

support for multiple sources #716

support for multiple sources #716

Comments

ghost commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

ghost commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

ghost commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

ghost commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

kennethreitz commented Sep 26, 2017

helderco commented Sep 29, 2017 • edited Loading

helderco commented Sep 29, 2017 • edited Loading

helderco commented Sep 29, 2017

kennethreitz commented Mar 13, 2018

kennethreitz commented Mar 13, 2018

techalchemy commented Mar 13, 2018

techalchemy commented Mar 13, 2018

kennethreitz commented Mar 13, 2018

kennethreitz commented Mar 13, 2018

techalchemy commented Mar 20, 2018

jonlil commented Apr 11, 2018

techalchemy commented Apr 11, 2018 • edited Loading

helderco commented Sep 29, 2017 •

edited

Loading

helderco commented Sep 29, 2017 •

edited

Loading

techalchemy commented Apr 11, 2018 •

edited

Loading