Jupyter dependency not properly installing with pipenv on Linux

[ScottDelbecq]

After setting up the developer environment using pipenv.

When running fastoad notebooks, the following error is returned:
"
Traceback (most recent call last):
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/bin/fastoad", line 6, in
from pkg_resources import load_entry_point
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/pkg_resources/init.py", line 3252, in
@_call_aside
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/pkg_resources/init.py", line 3236, in _call_aside
f(*args, **kwargs)
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/pkg_resources/init.py", line 3265, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/pkg_resources/init.py", line 584, in _build_master
ws.require(requires)
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/pkg_resources/init.py", line 901, in require
needed = self.resolve(parse_requirements(requirements))
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/pkg_resources/init.py", line 787, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pexpect; sys_platform != "win32"' distribution was not found and is required by ipython
"
Running jupyter lab works. However, once opening a notebook the error returned is:
"
[W 10:21:44.519 LabApp] Notebook src/fastoad/notebooks/tutorial/01_tutorial.ipynb is not trusted
[I 10:21:45.397 LabApp] Kernel started: a52e9341-e923-4f34-9866-fe51a663a1ea
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/ipykernel_launcher.py", line 15, in
from ipykernel import kernelapp as app
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/ipykernel/init.py", line 2, in
from .connect import *
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/ipykernel/connect.py", line 13, in
from IPython.core.profiledir import ProfileDir
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/init.py", line 55, in
from .core.application import Application
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/core/application.py", line 25, in
from IPython.core import release, crashhandler
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/core/crashhandler.py", line 27, in
from IPython.core import ultratb
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/core/ultratb.py", line 119, in
from IPython.utils import path as util_path
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/utils/path.py", line 17, in
from IPython.utils.process import system
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/utils/process.py", line 19, in
from ._process_posix import system, getoutput, arg_split, check_pid
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/utils/_process_posix.py", line 23, in
import pexpect
ModuleNotFoundError: No module named 'pexpect'
"""

Running jupyter notebook the terminal renders:
"""
(FAST-OAD) sdelbecq@scott-xps:~/PycharmProjects/FAST-OAD$ jupyter notebook
[W 10:26:38.743 NotebookApp] Terminals not available (error was No module named 'winpty')
[I 10:26:38.763 NotebookApp] JupyterLab extension loaded from /home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/jupyterlab
[I 10:26:38.763 NotebookApp] JupyterLab application directory is /home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/share/jupyter/lab
[I 10:26:38.765 NotebookApp] Serving notebooks from local directory: /home/sdelbecq/PycharmProjects/FAST-OAD
[I 10:26:38.765 NotebookApp] The Jupyter Notebook is running at:
[I 10:26:38.765 NotebookApp] http://localhost:8888/?token=c55ff51999397eadb6bca865fba834a4f6342ac8dd8512ea
[I 10:26:38.765 NotebookApp] or http://127.0.0.1:8888/?token=c55ff51999397eadb6bca865fba834a4f6342ac8dd8512ea
[I 10:26:38.765 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 10:26:38.782 NotebookApp]

To access the notebook, open this file in a browser:
    file:///home/sdelbecq/.local/share/jupyter/runtime/nbserver-28015-open.html
Or copy and paste one of these URLs:
    http://localhost:8888/?token=c55ff51999397eadb6bca865fba834a4f6342ac8dd8512ea
 or http://127.0.0.1:8888/?token=c55ff51999397eadb6bca865fba834a4f6342ac8dd8512ea

[W 10:26:43.040 NotebookApp] 404 GET /api/kernels/865f91ef-1df4-4fc6-8afd-43f90d01ae2f/channels?session_id=b2b15ef60c0f4c27a3b1c1f3cb27dbf6 (127.0.0.1): Kernel does not exist: 865f91ef-1df4-4fc6-8afd-43f90d01ae2f
[W 10:26:43.047 NotebookApp] 404 GET /api/kernels/865f91ef-1df4-4fc6-8afd-43f90d01ae2f/channels?session_id=b2b15ef60c0f4c27a3b1c1f3cb27dbf6 (127.0.0.1) 12.31ms referer=None
[W 10:26:52.039 NotebookApp] Replacing stale connection: 865f91ef-1df4-4fc6-8afd-43f90d01ae2f:b2b15ef60c0f4c27a3b1c1f3cb27dbf6

"""

When opening a notebook the following error is return:

"""
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/ipykernel_launcher.py", line 15, in
from ipykernel import kernelapp as app
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/ipykernel/init.py", line 2, in
from .connect import *
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/ipykernel/connect.py", line 13, in
from IPython.core.profiledir import ProfileDir
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/init.py", line 55, in
from .core.application import Application
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/core/application.py", line 25, in
from IPython.core import release, crashhandler
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/core/crashhandler.py", line 27, in
from IPython.core import ultratb
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/core/ultratb.py", line 119, in
from IPython.utils import path as util_path
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/utils/path.py", line 17, in
from IPython.utils.process import system
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/utils/process.py", line 19, in
from ._process_posix import system, getoutput, arg_split, check_pid
File "/home/sdelbecq/.local/share/virtualenvs/FAST-OAD-3a9hL4Al/lib/python3.7/site-packages/IPython/utils/_process_posix.py", line 23, in
import pexpect
ModuleNotFoundError: No module named 'pexpect'

"""

The module pexpect appears in the FAST-OAD pipenv packages (version 4.6.0 , the latest is 4.7.0).

Manually installing pexpect using pipenv install pexpect==4.6.0 solves the 3 above issues.

[ScottDelbecq]

The problem is solved by running pipenv update --dev on the Linux machine. The Wiki for developper install has thus been updated for this case. I am closing the issue.

[ScottDelbecq]

I have been testing things to find a generic solution:

• No pexpect dependency in setup.cfg, generated PipFile.lock with Windows -> Installation works only on Windows (missing pexpect for Linux)
• No pexpect dependency in setup.cfg, generated PipFile.lock with Linux -> Installation works only on Linux (missing colorama for Windows)
• Added pexpect dependency in setup.cfg, generated PipFile.lock with Windows -> Installation works only on Windows AND on Linux
• Added pexpect dependency in setup.cfg, generated PipFile.lock with Linux -> Installation works only on Linux (missing colorama for Windows)

Therefore, the easiest, but not optimal, way to maintain the PipFile.lock would be to add the pexpect dependency in the setup.cfg and generate the file on Windows. The less easy, and less optimal but totally generic, would be to add also the colorama dependency and generate the PipFile.lock on either Windows or Linux. What is your feeling on this @christophe-david?

[christophe-david]

What do you do exactly when you talk about "generating Pipfile.lock"?

On my side, keeping the Pipfile.lock you created on Linux, I do pipenv lock --dev (like pipenv install --dev excepts it does not modify my installed packages) and I still see pexpect in the file.

[ScottDelbecq]

I did pipenv lock --dev but after deleting the previous version. If I do not delete the previous version as you suggests it works.

However, if we follow the Wiki and run pipenv sync --dev or use PyCharm to create the virtual environment. The PipFile.lock is not updated and the installation is not correct and leads to the following error:

Traceback (most recent call last):
File "C:\Users\s.delbecq.virtualenvs\FAST-OAD-test-rnZn6qQz\Scripts\fastoad-script.py", line 6, in
from pkg_resources import load_entry_point
File "c:\users\s.delbecq.virtualenvs\fast-oad-test-rnzn6qqz\lib\site-packages\pkg_resources_init_.py", line 3252, in
@call_aside
File "c:\users\s.delbecq.virtualenvs\fast-oad-test-rnzn6qqz\lib\site-packages\pkg_resources_init.py", line 3236, in call_aside
f(*args, **kwargs)
File "c:\users\s.delbecq.virtualenvs\fast-oad-test-rnzn6qqz\lib\site-packages\pkg_resources_init.py", line 3265, in initialize_master_working_set
working_set = WorkingSet.build_master()
File "c:\users\s.delbecq.virtualenvs\fast-oad-test-rnzn6qqz\lib\site-packages\pkg_resources_init.py", line 584, in build_master
ws.require(requires)
File "c:\users\s.delbecq.virtualenvs\fast-oad-test-rnzn6qqz\lib\site-packages\pkg_resources_init.py", line 901, in require
needed = self.resolve(parse_requirements(requirements))
File "c:\users\s.delbecq.virtualenvs\fast-oad-test-rnzn6qqz\lib\site-packages\pkg_resources_init.py", line 787, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'colorama; sys_platform == "win32"' distribution was not found and is required by ipython

I see three solutions:

1. Add to the Wiki the step were you run pipenv lock --dev before the pipenv sync --dev

2. Do from time to time a pipenv lock --dev to generate a new PipFile.lock that is commited to the repository (hoping nobody deletes the file before generating it)

3. Add the colorama and pexpect dependencies to the setup.cfg

I would suggest combining solution 1 and 2. Thus, updating the Wiki and also updating the PipFile.lock on the repository so it is compatible by default with Windows and Linux.

[christophe-david]

The Wiki part about pipenv only deals with the way to add high-level packages (which could indeed be said more clearly).

It also does not aim to be a full pipenv course and provides a link to the full documentation. But feel free to amend it to have this particular case covered :)

About your solutions:

1. This is the good one and can be done in one step with pipenv update --dev
2. I would not like someone to do the lock and commit without doing the sync on its machine and running the tests. But yes, from time to time, a pipenv update --dev followed by a full run of tests can be useful.
3. ... on my dead body!

To conclude, there is a need to modify automated tests in GitHub actions to have Linux tested (and maybe macOS). This will help seeing OS-related problems.

[christophe-david]

To keep track of what we discussed live: once correctly tested, pipenv update --dev actually does not work, unless maybe using the --keep-outdated option with the master of their GitHub repo instead of the released version, as stated here. But the option is not intended for that in the first place, so it looks a bit awkward.

[christophe-david]

I just realized there is a 4th solution, or a "3bis" that consists in adding colorama and pexpect in the [packages]' section of Pipfile`.

I would accept it more easily than the setup.cfg, because it would just affect our development environment without impacting the packaging process, that rely only on setup.cfg.

But the real solution is probably to give up on pipenv, that looks half-dead (no release since end of 2018) and go for Poetry, that looks well alive and natively able to handle multiple platforms. But it looks like it would need the adoption of pyproject.toml instead of setuptools, which will require some work.

[christophe-david]

Just tried using Poetry. The poetry.lock file generated on Windows already contains the dependency to pexpect for non-Windows platforms (and same for colorama for Windows-only platforms). Nice!

Packaging looks easy from Poetry. I'll test it.

The only drawback is that Sphinx and Pytest still cannot be piloted from a pyproject.toml file (but it is considered by Pytest devs). So moving to Poetry will need to have both setup.cfg and pyproject.toml files. Not really a regression from current status, as we currently live with setup.cfg and Pipfile, and dependencies are separated in both files, whereas all of them should be only in pyproject.toml.

Stay tuned...