opensim for python 3.8 #2869
Comments
|
@kidzik Can you clarify what is the workflow that's supposed to work here:
|
|
Based on a recent post on the Moco forum, there may be issues with installing OpenSim in Python 3.8: https://simtk.org/plugins/phpBB/viewtopicPhpbb.php?f=1815&t=12387&p=0&start=0&view=&sid=0e54eee37b6d796a511ab14df68a3750. Moco issue: opensim-org/opensim-moco#655. @mcmips are you able to install on Python 3.7? |
|
There are two relatively easy ways to run OpenSim in colab.
Both installation methods are described here in the context of reinforcement learning with osim-rl @aymanhab answering your questions:
It's probably good to completely scratch my builds and start off from @chrisdembia code conda-forge/staged-recipes#4693 @mcmips to answer your question, my conda builds won't work with python 3.8 so for that version you would have to recompile OpenSim. Depending what you want to do, the easiest way to make things work could be to just use conda and my OpenSim build with python 3.6 |
|
@nickbianco do you know if this was specific to Anaconda or to the standard python 3.8? Trying to use Anaconda and 3.8 locally I get the errors below: @kidzik Does the opensim-rl environment depend on numpy? and do we need to be on 3.8? Thanks |
|
The user on the forum was using standard Python 3.8. He got an error when the Python install script tries to import the simbody Python module but was unable to find it. Module imports happen using Python's |
|
This appears to be SWIG related (release notes for SWIG 4.0.1 explicitly specifies adding support for python 3.8), so it makes no sense to try to fix this anywhere else at the moment, instead we should plan upgrading SWIG first. How time critical is this? @kidzik @nickbianco |
|
I think most users can revert to 3.7 (as did the user on the Moco forum). Unless we have users who specifically need 3.8 features it's probably not super time critical. |
|
It's not time-critical for my use cases (NeurIPS challenge and RL workshop). Also, in the tutorial I'm listing exact versions with which everything is functioning well. Numpy is indeed needed for opensim-rl. We could make it leaner and depend only on OpenSim. Our users will almost surely use numpy anyway. |
|
The next release of opensim will depend on numpy. |
|
As an additional datapoint, I just spent a while trying to get python to work on my Windows machine for a researcher because of this issue. The reason why I found the install pretty confusing is because using Python with OpenSim right now involves non-standard library setup steps (e.g., not using
|
|
To further compound the issue, after I just tried to get Anaconda working with the researcher:
|
|
Thanks for the feedback @adamkewley You have to remember that:
|
|
I might try a 3.8 custom build out to satisfy the researcher on our side.
It might be that swig generates mostly low-3 source-level bindings but the
problem is likely to be the native bindings they hook into (if python 3.8
changed things). I'll report back after I try this.
For pip et al, it would potentially be enough for us to ship a very minimal
package that effectively checks the python version, abi, and runtime
searches for our preinstalled SDK package, followed by forwarding or
installing the SDK. That would effectively be a python-friendly installer
that politely asks the user to go and install opensim from simtk, set such
and such a variable, "sorry, your python won't work", etc. I might be able
to whip this up in an unofficial package as a prototype and then I can get
back to you once I know the pitfalls.
The main issue is that typical python developers are used to their package
manager railroading loads of stuff. Even if we provide a minimal package
that is clear about what the problem is (by showing error messages etc) it
might eradicate a bunch of stuff we otherwise need to manually document and
hope people read carefully.
…On Thu, Sep 24, 2020, 7:07 PM Ayman Habib ***@***.***> wrote:
Thanks for the feedback @adamkewley <https://github.com/adamkewley> You
have to remember that:
1. Python changed how packages are initialized in version 3.8, the
documentation was written before that change so it assumed python will
maintain backward compatibility which is false in hindsight (We tested
against 3.4-3.7). If you can update the documentation until we upgrade that
would be great.
2. Based on your investigation, would bindings created for python 3.8
work with 3.7 and earlier? or we have to choose between the two? Ideally we
don't force users on python 3.7 and earlier to upgrade just to please 3.8
users 😸
3. Not sure what needs to be changed on our side to use pip instead of
the current mechanism, any clues?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2869 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABEC5ST26NMEACZOXYRVGBLSHN4EZANCNFSM4QPT6TPA>
.
|
|
As an update for this, I manually stripped all traces of other python versions from the PC and built OpenSim with anaconda's python3.8. It seems to build + install fine, but it's now having issues loading the native binaries in the Which might've been similar to message I was getting when using 4.1 with python3.7, so it could be that there's some module-loading-path missing (the module is right there, in the zip package, though). I Had similar issues with loading However, if I use the package in its unpacked form (e.g. by manually setting So the trick for Anaconda /w 3.8 seems to be:
You'd think that, because the Anaconda version of OpenSim is built against 3.8, opensim would "just work" with barebones Python 3.8. But going through the exact same steps yields: They are slightly different versions of It could also be because (as far as I can tell) the OpenSim build requires locating NumPy at configure time, which it probably uses it for some of the python bindings (perf?). The (working) anaconda version has the advantage of being the exact version built against with the exact numpy built against. Either way, for my purposes, I might be able to internally ship the custom build /w instructions for using it with Anaconda as a temporary bandaid. |
|
@adamkewley would you be interested in picking up the work of creating an opensim conda package? This is the correct long term solution. A lot of the work is already done: conda-forge/staged-recipes#11894 . I could answer any questions you have. |
|
Additional update: we now have a working |
|
Thanks for the update @adamkewley The main question I have is what version of SWIG you used to create the bindings and on which platforms. If you needed to make changes to get the latest source to work with python 3.8 it would be great to have these on our master branch. Did it work also with generic python 3.8 from python.org? Thank you |
|
@aymanhab , for our workstation's opensim (the one on which the researcher is using Anaconda /w Python 3.8): The researcher is using python3.7, installed as an additional environment. Hacky fix I added into For the actual build, I built opensim with the scapulo joint merged. Can't remember why, could be because one of the researchers needs access to the joint for their models and I didn't want the hassle of setting up the library paths etc.: Build script (not my finest work: something I had lying around from months ago): #!/usr/bin/env bash
# (before) checkout this project
# git clone https://github.com/opensim-org/opensim-core.git
# cd opensim-core/
# (before) install package-managed dependencies
#apt-get install git cmake cmake-curses-gui \
# freeglut3-dev libxi-dev libxmu-dev \
# liblapack-dev swig python-dev default-jdk
# CMAKE_BUILD_TYPE: what type of build to run
# Debug Unoptimized (10x slower) build with debug symbols
# Release Optimized without debug symbols
# RelWithDebInfo Optimized build with debug symbols (default)
# MinSizeRel Optimized **for size** build without debug symbols
opensim_project="${PWD}"
# where build intermediates are placed
opensim_build_dir="${opensim_project}/opensim-core-build"
# where built binaries are placed
# same, but for vendored dependencies
dependencies_project="${opensim_project}/dependencies"
dependencies_build_dir="${opensim_project}/opensim-core-dependencies-build"
# number of build targets to build in parallel
num_build_jobs=$(nproc)
# build vendored dependencies that OpenSim uses
mkdir -p "${dependencies_build_dir}"
pushd "${dependencies_build_dir}"
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \
"${dependencies_project}"
make -j"${num_build_jobs}"
popd
# build OpenSim
mkdir -p "${opensim_build_dir}"
pushd "${opensim_build_dir}"
cmake -DOPENSIM_DEPENDENCIES_DIR="${opensim_project}/opensim_dependencies_install" -DBUILD_JAVA_WRAPPING=OFF \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
"${opensim_project}"
make -j "${num_build_jobs}"
ctest -j "${num_build_jobs}"
make install
popd |
|
Development builds as of PR #2987 are made against python 3.7 and numpy 1.19 |
|
@aymanhab What is the status of this issue? |
|
We used Anaconda for the Moco workshop at TGCS, using python 3.8 so I assume this is fixed already, but I haven't used the colab environment directly, maybe others did |
|
If this issue is strictly about OpenSim working with 3.8, then we should at least have this tested well with Windows and Mac environments. @aymanhab have you tried on Linux? I haven't tried using OpenSim with colab yet. |
|
I was using it in a workshop, but I used a workaround hack -- participants had local binaries of OpenSim (not in google's cloud) and colab connected to these local binaries |
|
Successful run in browser with 3.8 and OpenSim 4.3 Beta as shown here |
|
Slow response... but this issue is solved my perspective. I just went through the opensim python documentation page again to update to source code changes for conda and the documentation page was clear for me. A trick for building from source where you want OpenSim to find a custom Anaconda environment: where opensim_jam is the custom environment name. |
|
I've also needed to set Python3_ROOT_DIR when building OpenSim from source to set my desired conda environment. We should document this in the updated build instructions (related: #3178). |
|
I'd be grateful if someone could explain how Python documentation seems to indicate that setting the
The OpenSim documentation also says the same thing:
But this is not what I observe in practice: it is still necessary to set the PATH environment variable! I might be missing something. But at the moment, I suspect this might be causing confusion to OpenSim end-users. |
|
My conjecture is that while python decided to invent its own mechanism for
locating dlls, the underlying windows system is still using it since this
how all other programs are distributed, so it's possible this is still
necessary. You may want to add dll directory for simbody as well to see
what happens.
Due to the wide variety of python distributions, we've created a self
contained conda package in release 4.4, I'd recommend you give that a try.
…On Mon, Jun 13, 2022, 7:54 AM ParticularMiner ***@***.***> wrote:
I'd be grateful if someone could explain how os.add_dll_directory(...)
works.
Python documentation
<https://docs.python.org/3/whatsnew/3.8.html#bpo-36085-whatsnew:~:text=Specifically%2C%20PATH,loading%20your%20library.>
seems to indicate that setting the PATH environment variable in Windows
to "C:\OpenSim 4.3\bin" in order to enable the file-access of _simbody.lib
is not required for Python versions 3.8+:
Specifically, PATH and the current working directory are no longer used,
and modifications to these will no longer have any effect on normal DLL
resolution. If your application relies on these mechanisms, you should
check for add_dll_directory()
<https://docs.python.org/3/library/os.html#os.add_dll_directory> and if
it exists, use it to add your DLLs directory while loading your library.
The OpenSim documentation
<https://simtk-confluence.stanford.edu:8443/display/OpenSim/Scripting+in+Python#:~:text=Starting%20at%20version%203.8%2C%20Python%20changed%20how%20dll%27s%20are%20located%20on%20windows%2C%20in%20particular%20PATH%20is%20not%20used%20any%20more%2C%20instead%20every%20environment%20should%20set%20the%20path%20to%20locate%20the%20dlls%20using%20this%20snippet>
also says the same thing:
Starting at version 3.8, Python changed how dll's are located on windows,
in particular PATH is not used any more, instead every environment should
set the path to locate the dlls using ...
But this is not what I observe in practice: it is still necessary to set
the PATH environment variable!
I might be missing something. But at the moment, I suspect this might be
causing confusion to end-users.
—
Reply to this email directly, view it on GitHub
<#2869 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA6JY4HCNM25Z7EP7N3TW3TVO5DTJANCNFSM4QPT6TPA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Thank you very much @aymanhab for the pointers! I'm also excited to learn about the new conda release. That would certainly make things much easier. So I executed the following commands without any problems: But got the following error on attempting to import Is this expected behavior? I had expected the conda installation script to set the dll paths automatically so that I wouldn't have to do it myself (by running |
|
FYI, following my previous message, I modified the following line in my conda environment's to if (os.path.exists(os.path.join(sys.prefix, 'Lib'))):This cleared the runtime error above, but raised another error (the old one): I also tried setting environment variable None of the above attempts worked. Question: Do I need to install the full OpenSim 4.4 (non-python) application somewhere on my system, in addition to the conda package opensim installation? |
|
You don't need to install OpenSim elsewhere, this is supposed to be self contained. |
Thanks for the info, @aymanhab. However, removing and recreating the environment with
It seems that the conda-detection line failed on my system because my conda installation does not have a subdirectory named "conda". I guess this is the default configuration for conda environments. But correct me if I'm wrong. |
|
Some suggestions:
|
|
Installation in the base environment yielded the same results as above: This is my first time using dependency walker. Could you direct me on how to use it in this case, please. My efforts to pass |
|
Modify DLL_PATH to the folder that contains all the opensim dlls first |
|
Or you could change the line to |
|
Thanks for your help!
Yes! That worked for the base environment. 🎉 But not for other environments. I see you've already pushed the 'conda-meta' correction to the conda repository. However, I'm now getting the following errors when installing |
|
Can you clarify:
|
|
For dependencies, I get Dependencies from here https://github.com/lucasg/Dependencies and drag dlls or pyds into it to see if anything is missing from the packaging but that doesn't appear to be the case |
|
In my base environment,
In the other environments, Unfortunately I have other packages that depend on that Strange that you haven't pushed anything to the conda repository. I'm seeing that |
|
For the Moco side, the default package doesn't include Moco, we'll provide this soon, so no surprise (the anaconda website shows two packages opensim, opensim-moco though the latter is rather old). |
I guess so. I'm not sure about the exact version though. |
|
Can you do conda list and send output? |
|
Sure. Click to see conda list output |
|
I see you have numpy 1.22.4 while our build uses numpy 1.20 exclusively, this could be the problem. You can use "conda search package_name --info" to see the dependencies of suspected culprits. |
|
I'm not sure what happened, but |
|
Wow, that's wonderful. If using 'conda-meta' works consistently I can update the code as well on our side to make sure other users get it as well. |
|
Thanks again for your help @aymanhab ! |
|
@aymanhab The beta has 'conda' not 'conda-meta'. Didn't work for me with 'conda', seems to work with 'conda-meta'. Perhaps this change can be included in the release or at least documented somewhere more visible to save users time in the future ⏰ |
|
To update the scenario: I just had the "importError: DLL load failed while importing _simbody:" problem trying to use a python 3.9 conda env, but it is working fine in 3.8 after mannualy inserting DLL path to system path. |
How can I get opensim to install in python 3.8 for google colab? Please help.
The text was updated successfully, but these errors were encountered: