Cross-platform Pipenv.lock? #3902

AlexP11223 · 2019-08-23T08:29:25Z

I am creating a cross-platform Python app.

While trying to build the app on Windows and Linux I noticed that Pipenv.lock is platform-specific.

For example, I create a project on Windows and add pytest, it has colorama with "markers": "sys_platform == 'win32'" in the lock file. But when modifying the lock file on Linux, such as adding a new lib (pipenv install hello-world), this dependency will disappear.

diff ../py-win/Pipfile.lock Pipfile.lock 
4c4
<             "sha256": "828b8ad012f4c8773e6e61e3ac2be0ffcd7540fd7ed175a8355676c8e31c4d3d"
---
>             "sha256": "c4a1bdfad8e2d208344003d9c704432100fbcb9bd87f713178895637222767a4"
33c33
<         "colorama": {
---
>         "hello-world": {
35,36c35
<                 "sha256:05eed71e2e327246ad6b38c540c4a3117230b19679b875190486ddd2d721422d",
<                 "sha256:f8ac84de7840f5b9c4e3347b3c1eaa50f7e49c2b07596221daec5edaabbd7c48"
---
>                 "sha256:f236ab36224613dff8db190dc4a02b3787c3d19e95f7e0876e3098966118bc1e"
38,39c37,38
<             "markers": "sys_platform == 'win32'",
<             "version": "==0.4.1"
---
>             "index": "pypi",
>             "version": "==0.2"

Here is the full output on both systems: https://gist.github.com/AlexP11223/6457fd4c98272c526c855c63088c00d4

Is it supposed to work like that? If yes, what is the best way to handle that for cross-platform apps? Is there any better solution than just committing lock files for all platforms to the Git repo (such as Pipenv-win.lock, Pipenv-linux.lock) and renaming one of them to Pipenv.lock when building?

Pipenv version: '2018.11.26'

Pipenv location: 'C:\\Users\\user\\AppData\\Roaming\\Python\\Python37\\site-packages\\pipenv'

Python location: 'c:\\python37\\python.exe'

Python installations found:

3.7.3: C:\Python37\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.3',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.17763',
 'python_full_version': '3.7.3',
 'python_version': '3.7',
 'sys_platform': 'win32'}

Pipenv version: '2018.11.26'

Pipenv location: '/home/alex/.local/lib/python3.7/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

3.7.3: /usr/bin/python3.7
3.7.3: /usr/bin/python3.7m
2.7.16: /usr/bin/python

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.0.0-21-generic',
 'platform_system': 'Linux',
 'platform_version': '#22+system76-Ubuntu SMP Tue Jul 16 19:57:52 UTC 2019',
 'python_full_version': '3.7.3',
 'python_version': '3.7',
 'sys_platform': 'linux'}

The text was updated successfully, but these errors were encountered:

frostming · 2019-08-23T09:03:59Z

use --keep-outdated to keep the unneeded dependencies on a platform.

AlexP11223 · 2019-08-23T09:16:16Z

it doesn't seem to work

alex@pop-os:~/py1$ pipenv install --keep-outdated
Creating a virtualenv for this project…
Pipfile: /home/alex/py1/Pipfile
Using /usr/bin/python3.7 (3.7.3) to create virtualenv…
⠋ Creating virtual environment...

⠙ Creating virtual environment...

⠴ Creating virtual environment...Using base prefix '/usr'
New python executable in /home/alex/.local/share/virtualenvs/py1-MRCLG4IM/bin/python3.7
Also creating executable in /home/alex/.local/share/virtualenvs/py1-MRCLG4IM/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python3.7

✔ Successfully created virtual environment! 
Virtualenv location: /home/alex/.local/share/virtualenvs/py1-MRCLG4IM
Installing dependencies from Pipfile.lock (1c4d3d)…
Ignoring colorama: markers 'sys_platform == "win32"' don't match your environment
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
alex@pop-os:~/py1$ pipenv install --keep-outdated hello-world
Installing hello-world…
Adding hello-world to Pipfile's [packages]…
✔ Installation Succeeded 
Pipfile.lock (2767a4) out of date, updating to (1c4d3d)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success! 
Updated Pipfile.lock (2767a4)!
Installing dependencies from Pipfile.lock (2767a4)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

alex@pop-os:~/py1$ diff ../py-win/Pipfile.lock Pipfile.lock
4c4
<             "sha256": "828b8ad012f4c8773e6e61e3ac2be0ffcd7540fd7ed175a8355676c8e31c4d3d"
---
>             "sha256": "c4a1bdfad8e2d208344003d9c704432100fbcb9bd87f713178895637222767a4"
33c33
<         "colorama": {
---
>         "hello-world": {
35,36c35
<                 "sha256:05eed71e2e327246ad6b38c540c4a3117230b19679b875190486ddd2d721422d",
<                 "sha256:f8ac84de7840f5b9c4e3347b3c1eaa50f7e49c2b07596221daec5edaabbd7c48"
---
>                 "sha256:f236ab36224613dff8db190dc4a02b3787c3d19e95f7e0876e3098966118bc1e"
38,39c37,38
<             "markers": "sys_platform == 'win32'",
<             "version": "==0.4.1"
---
>             "index": "pypi",
>             "version": "==0.2"

frostming · 2019-08-23T09:22:03Z

Would you like to try the master branch?

AlexP11223 · 2019-08-23T09:53:48Z

What's the simplest way to install it?

frostming · 2019-08-23T09:58:21Z

@AlexP11223 If you are on mac: brew install pipenv --HEAD
Otherwise: pip install --user -e git+https://github.com/pypa/pipenv.git@master

AlexP11223 · 2019-08-23T10:56:38Z

Looks like it works in master:

alex@pop-os:~/py2$ python3 -m pipenv install --keep-outdated
Creating a virtualenv for this project…
Pipfile: /home/alex/py2/Pipfile
Using /usr/bin/python3.7m (3.7.3) to create virtualenv…
⠙ Creating virtual environment...Using base prefix '/usr'
New python executable in /home/alex/.local/share/virtualenvs/py2-5HLUPQ8M/bin/python3.7m
Also creating executable in /home/alex/.local/share/virtualenvs/py2-5HLUPQ8M/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python3.7m

✔ Successfully created virtual environment! 
Virtualenv location: /home/alex/.local/share/virtualenvs/py2-5HLUPQ8M
Installing dependencies from Pipfile.lock (1c4d3d)…
Ignoring colorama: markers 'sys_platform == "win32"' don't match your environment
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:11
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
alex@pop-os:~/py2$ python3 -m pipenv install --keep-outdated hello-world
Installing hello-world…
Adding hello-world to Pipfile's [packages]…
✔ Installation Succeeded 
Pipfile.lock (1c4d3d) out of date, updating to (2767a4)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (2767a4)!
Installing dependencies from Pipfile.lock (2767a4)…
Ignoring colorama: markers 'sys_platform == "win32"' don't match your environment
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 14/14 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
alex@pop-os:~/py2$ python3 -m pipenv --version
pipenv, version 2018.11.27.dev0

alex@pop-os:~/py2$ diff ../py-win/Pipfile.lock Pipfile.lock
4c4
<             "sha256": "828b8ad012f4c8773e6e61e3ac2be0ffcd7540fd7ed175a8355676c8e31c4d3d"
---
>             "sha256": "c4a1bdfad8e2d208344003d9c704432100fbcb9bd87f713178895637222767a4"
39a40,46
>         },
>         "hello-world": {
>             "hashes": [
>                 "sha256:f236ab36224613dff8db190dc4a02b3787c3d19e95f7e0876e3098966118bc1e"
>             ],
>             "index": "pypi",
>             "version": "==0.2"

AlexP11223 · 2019-09-10T08:10:33Z

btw looks like Poetry handles it better by default, it doesn't remove dependencies for other platforms https://gist.github.com/AlexP11223/afbe247175efdcc897930d8c6ab04184

Maybe pipenv should consider this too?

frostming · 2019-09-10T08:46:12Z

@AlexP11223 --keep-outdated does the trick, as said above

AlexP11223 · 2019-09-10T08:59:47Z

I mean that it should be the default behavior (otherwise someone will forget to add it sooner or later), not hidden under some switch with an unrelated name (which probably does more than just keeping dependencies marked as for other platforms).

gruns · 2020-05-14T11:43:21Z

Platform dependent Pipenv.lock problem popped up again. Also with colorama. See

Convert colorama into an optional dependency gruns/icecream#40

cf pypa/pipenv#3902

jamesstidard · 2021-03-24T12:41:00Z

For anyone that may have initially thought to use --keep-outdated as I did and doing a pipenv install --dev --keep-outdated pytest this will not reinstall and capture the pytest dependancies for that platform. Instead, you can used it with the lock command and get all the platform specific dependancies for your whole project:

pipenv lock --keep-outdated

Probably most people read it that way to begin with, but not I, for whatever reason.

Maybe worth scanning your the diff of your Pipfile.lock though, to make sure no unexpected versions get changed or removed.

- Updated Pipfile and Pipfile.lock - Added `pipenv lock --keep-outdated` to automated workflows to support cross-platform deployment. See pypa/pipenv#3902.

RomanSteinberg · 2022-06-29T08:25:52Z

@jamesstidard Sorry, but it is not clear for me how to handle cross-platform dependencies. Let me introduce my case.

I installed project dependencies on Ubuntu (pipenv install).
Copied Pipfile and Pipfile.lock to MacOS.
Ran pipenv install on MacOS. Result: "ERROR: No matching distribution found for tflite-runtime==2.8.0"
Deleted Pipfile.lock.
Ran pipenv install on MacOS. Result: success.
Copied Pipfile and Pipfile.lock back to Ubuntu.
Ran pipenv install on Ubuntu. Result: "ERROR: No matching distribution found for tflite-runtime==2.5.0"

So, I concluded that I need different dependency trees for different platform. By the way, I checked sys_platform option according to PEP 509 and haven't found workaround. Anyway I tried this:
8. Run pipenv lock --keep-outdated on Ubuntu as you recommended.
9. Run pipenv install. Result: "ERROR: No matching distribution found for tflite-runtime==2.5.0"

Am I doing something wrong or it is another case?

jamesstidard · 2022-06-29T08:37:35Z

@RomanSteinberg Looks like you're doing the right steps. The only thing I can thing is: are you running pipenv lock --keep-outdated on the platform you are wanting to also support? seems like, from your description, you maybe did:

ubuntu $ pipenv install x
ubuntu $ pipenv lock --keep-outdated
transfer lock file
macOS $ pipenv sync

Instead of:

ubuntu $ pipenv install x
transfer lock file
macOS $ pipenv lock --keep-outdated
macOS $ pipenv sync

Otherwise, I'm not sure, looks like you're doing the same steps as me. Sorry.

EDIT: on rereading, both your ubuntu and macOS are failing with the same missing dependancy, so this is maybe unrelated to it being a platform specific dependancy and maybe a problem with the package itself.

RomanSteinberg · 2022-06-29T11:11:40Z

Using your recipe I failed again. :(

EDIT: on rereading, both your ubuntu and macOS are failing with the same missing dependency, so this is maybe unrelated to it being a platform specific dependency and maybe a problem with the package itself.

As far as I understand, this package has different versions for different platforms. May be it is maintainers problem but it becomes mine. So, I should store Pipfile.lock for each platform separately.

matteius · 2022-06-29T12:22:09Z

I suspect you may be reporting about this issue: #5130

triage-new-issues bot added the triage label Aug 23, 2019

frostming closed this as completed Aug 23, 2019

AlexP11223 mentioned this issue Aug 23, 2019

Cannot install Pipenv from Git #3903

Closed

christophe-david mentioned this issue Jan 30, 2020

Jupyter dependency not properly installing with pipenv on Linux fast-aircraft-design/FAST-OAD#91

Closed

ritou11 mentioned this issue Mar 23, 2020

Locking package dependencies raises pexpect.exceptions.TIMEOUT #3752

Closed

gruns mentioned this issue May 14, 2020

Convert colorama into an optional dependency gruns/icecream#40

Open

isomorpheme mentioned this issue Jul 23, 2020

Pipenv should prioritize more cross-platform sys_platform condition between packages when lock #4101

Open

nealkruis mentioned this issue Jul 30, 2020

How to manage platform specific sub-dependencies #4408

Closed

Systemcluster mentioned this issue Sep 11, 2020

Sub-dependencies of platform-specific dependencies are not added to the lockfile #4461

Closed

spk added a commit to spk/template-builder that referenced this issue Oct 21, 2020

README: add info for Linux install

536d81d

cf pypa/pipenv#3902

spk mentioned this issue Oct 21, 2020

README: add info for Linux install platformsh/template-builder#357

Closed

spk added a commit to spk/template-builder that referenced this issue Oct 21, 2020

README: add info for Linux install

9b1dde0

cf pypa/pipenv#3902

spk added a commit to spk/template-builder that referenced this issue Oct 21, 2020

README: add info for Linux install

6794969

cf pypa/pipenv#3902

ssantichaivekin mentioned this issue Jun 5, 2021

Update pipfile & pipfile.lock ssantichaivekin/empress#230

Merged

jaredmcgrath mentioned this issue Feb 10, 2022

Improve dependency management and testing. bryan-hoang/mthe-493-eng-math-group-a2-project#7

Merged

This was referenced Jun 7, 2023

Lambda Docker image failing when trying to install boto3 and pysqlite3-binary coleifer/pysqlite3#55

Closed

Using pysqlite3-binary fails on install --deploy when locked on M1 and installed on Linux #5723

Closed

jaecktec mentioned this issue Nov 9, 2023

Pipenv --keep-outdated flag no longer supported serverless/serverless-python-requirements#779

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross-platform Pipenv.lock? #3902

Cross-platform Pipenv.lock? #3902

AlexP11223 commented Aug 23, 2019

frostming commented Aug 23, 2019

AlexP11223 commented Aug 23, 2019

frostming commented Aug 23, 2019

AlexP11223 commented Aug 23, 2019

frostming commented Aug 23, 2019

AlexP11223 commented Aug 23, 2019

AlexP11223 commented Sep 10, 2019

frostming commented Sep 10, 2019

AlexP11223 commented Sep 10, 2019

gruns commented May 14, 2020 •

edited

Loading

jamesstidard commented Mar 24, 2021

RomanSteinberg commented Jun 29, 2022 •

edited

Loading

jamesstidard commented Jun 29, 2022 •

edited

Loading

RomanSteinberg commented Jun 29, 2022

matteius commented Jun 29, 2022

Cross-platform Pipenv.lock? #3902

Cross-platform Pipenv.lock? #3902

Comments

AlexP11223 commented Aug 23, 2019

frostming commented Aug 23, 2019

AlexP11223 commented Aug 23, 2019

frostming commented Aug 23, 2019

AlexP11223 commented Aug 23, 2019

frostming commented Aug 23, 2019

AlexP11223 commented Aug 23, 2019

AlexP11223 commented Sep 10, 2019

frostming commented Sep 10, 2019

AlexP11223 commented Sep 10, 2019

gruns commented May 14, 2020 • edited Loading

jamesstidard commented Mar 24, 2021

RomanSteinberg commented Jun 29, 2022 • edited Loading

jamesstidard commented Jun 29, 2022 • edited Loading

RomanSteinberg commented Jun 29, 2022

matteius commented Jun 29, 2022

gruns commented May 14, 2020 •

edited

Loading

RomanSteinberg commented Jun 29, 2022 •

edited

Loading

jamesstidard commented Jun 29, 2022 •

edited

Loading