Download IDAKLU from pybammsolvers (#4487)

* Removing IDAKLU, local install of pybammsolvers * Fix test with local install * remove test option, temp change to nox file * Uncomment pybammsolvers dependency * Start removing release and docs references * Minor changes * Fix for casadi import * Cleanup * Second attempt * Fix regex error in fix * Change path method * Fix changelog * Style * Skip some tests * style: pre-commit fixes * Refactor tests * Fixing some warnings * Skip a few more tests * Another skip * More skips * Remove excess skips * Docs update * Add a note to docs * Update required dependencies * Apply suggestions from code review Co-authored-by: Agriya Khetarpal <[email protected]> * Update pyproject.toml Co-authored-by: Agriya Khetarpal <[email protected]> * Update scripts/Dockerfile Co-authored-by: Agriya Khetarpal <[email protected]> * Cleanup from code review * Fix missed message --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Agriya Khetarpal <[email protected]>
pybamm-team · Jan 7, 2025 · 2de780d · 2de780d
1 parent 9cff00d
commit 2de780d
Show file tree

Hide file tree

Showing 82 changed files with 250 additions and 7,564 deletions.
diff --git a/.github/release_workflow.md b/.github/release_workflow.md
@@ -57,11 +57,6 @@ If a new release is required after the release of `vYY.MM.{x-1}` -
 
 Some other essential things to check throughout the release process -
 
-- If updating our custom vcpkg registry entries
-  [sundials-vcpkg-registry][SUNDIALS_VCPKG]
-  or [casadi-vcpkg-registry][CASADI_VCPKG] (used to build Windows
-  wheels), make sure to update the baseline of the registries in
-  vcpkg-configuration.json to the latest commit id.
 - Update jax and jaxlib to the latest version in `pybamm.util` and
   `pyproject.toml`, fixing any bugs that arise.
 - If changes are made to the API, console scripts, entry points, new optional
@@ -79,7 +74,5 @@ Some other essential things to check throughout the release process -
   PR **must** bump the build number in `meta.yaml` and **must** be from a
   personal fork of the repository.
 
-[SUNDIALS_VCPKG]: https://github.com/pybamm-team/sundials-vcpkg-registry
-[CASADI_VCPKG]: https://github.com/pybamm-team/casadi-vcpkg-registry
 [PYBAMM_FEED]: https://github.com/conda-forge/pybamm-feedstock
 [FEED_GUIDE]: https://conda-forge.org/docs/maintainer/updating_pkgs.html#updating-the-feedstock-repository
diff --git a/.github/workflows/benchmark_on_push.yml b/.github/workflows/benchmark_on_push.yml
@@ -31,12 +31,8 @@ jobs:
 
       - name: Install python dependencies
         run: |
-          python -m uv pip install --upgrade pip wheel setuptools wget cmake casadi numpy
           python -m uv pip install asv[virtualenv]
 
-      - name: Install SuiteSparse and SUNDIALS
-        run: python scripts/install_KLU_Sundials.py
-
       - name: Fetch base branch
         run: |
           # This workflow also runs for merge commits

diff --git a/.github/workflows/periodic_benchmarks.yml b/.github/workflows/periodic_benchmarks.yml
@@ -39,19 +39,12 @@ jobs:
 
       - name: Install python dependencies
         run: |
-          python -m uv pip install --upgrade pip wheel setuptools wget cmake casadi numpy
           python -m uv pip install asv[virtualenv]
 
-      - name: Install SuiteSparse and SUNDIALS
-        run: python scripts/install_KLU_Sundials.py
-
       - name: Run benchmarks
         run: |
           asv machine --machine "GitHubRunner"
           asv run --machine "GitHubRunner" NEW --show-stderr -v
-        env:
-          SUNDIALS_INST: $HOME/.local
-          LD_LIBRARY_PATH: $HOME/.local/lib
 
       - name: Upload results as artifact
         uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0

diff --git a/.github/workflows/publish_pypi.yml b/.github/workflows/publish_pypi.yml
@@ -48,9 +48,6 @@ jobs:
           with open(output_file, "a", encoding="utf-8") as output_stream:
               output_stream.write(f"count={num_cpus}\n")
 
-      - name: Clone pybind11 repo (no history)
-        run: git clone --depth 1 --branch v2.12.0 https://github.com/pybind/pybind11.git -c advice.detachedHead=false
-
       - name: Cache packages installed through vcpkg on Windows
         uses: actions/cache@1bd1e32a3bdc45362d1e726936510720a7c30a57 # v4.2.0
         env:
@@ -83,7 +80,6 @@ jobs:
           CIBW_REPAIR_WHEEL_COMMAND: delvewheel repair --add-path C:/Windows/System32 -w {dest_dir} {wheel}
           CIBW_TEST_EXTRAS: "all,dev,jax"
           CIBW_TEST_COMMAND: |
-            python -c "import pybamm; print(pybamm.IDAKLUSolver())"
             python -m pytest -m cibw {project}/tests/unit
       - name: Upload Windows wheels
         uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
@@ -104,24 +100,17 @@ jobs:
         with:
           python-version: 3.11
 
-      - name: Clone pybind11 repo (no history)
-        run: git clone --depth 1 --branch v2.12.0 https://github.com/pybind/pybind11.git -c advice.detachedHead=false
-
       - name: Build wheels on Linux
         run: pipx run cibuildwheel --output-dir wheelhouse
         env:
           CIBW_ENVIRONMENT: >
             PYBAMM_DISABLE_TELEMETRY="true"
           CIBW_ARCHS_LINUX: x86_64
-          CIBW_BEFORE_ALL_LINUX: >
-            yum -y install openblas-devel lapack-devel &&
-            bash scripts/install_sundials.sh 6.0.3 6.5.0
-          CIBW_BEFORE_BUILD_LINUX: python -m pip install cmake casadi setuptools wheel
+          CIBW_BEFORE_BUILD_LINUX: python -m pip install setuptools
           CIBW_REPAIR_WHEEL_COMMAND_LINUX: auditwheel repair -w {dest_dir} {wheel}
           CIBW_TEST_EXTRAS: "all,dev,jax"
           CIBW_TEST_COMMAND: |
             set -e -x
-            python -c "import pybamm; print(pybamm.IDAKLUSolver())"
             python -m pytest -m cibw {project}/tests/unit
 
       - name: Upload wheels for Linux
@@ -144,9 +133,6 @@ jobs:
         with:
           python-version: '3.11'
 
-      - name: Clone pybind11 repo (no history)
-        run: git clone --depth 1 --branch v2.12.0 https://github.com/pybind/pybind11.git -c advice.detachedHead=false
-
       - name: Install cibuildwheel
         run: python -m pip install cibuildwheel
 
@@ -155,95 +141,17 @@ jobs:
         run: |
             set -e -x
 
-            # Set LLVM-OpenMP URL
-            if [[ $(uname -m) == "x86_64" ]]; then
-              OPENMP_URL="https://anaconda.org/conda-forge/llvm-openmp/11.1.0/download/osx-64/llvm-openmp-11.1.0-hda6cdc1_1.tar.bz2"
-            elif [[ $(uname -m) == "arm64" ]]; then
-              OPENMP_URL="https://anaconda.org/conda-forge/llvm-openmp/11.1.0/download/osx-arm64/llvm-openmp-11.1.0-hf3c4609_1.tar.bz2"
-            fi
-
-            # Download gfortran with proper macOS minimum version (11.0)
-            if [[ $(uname -m) == "x86_64" ]]; then
-              GFORTRAN_URL="https://github.com/isuruf/gcc/releases/download/gcc-11.3.0-2/gfortran-darwin-x86_64-native.tar.gz"
-              KNOWN_SHA256="981367dd0ad4335613e91bbee453d60b6669f5d7e976d18c7bdb7f1966f26ae4 gfortran.tar.gz"
-            elif [[ $(uname -m) == "arm64" ]]; then
-              GFORTRAN_URL="https://github.com/isuruf/gcc/releases/download/gcc-11.3.0-2/gfortran-darwin-arm64-native.tar.gz"
-              KNOWN_SHA256="84364eee32ba843d883fb8124867e2bf61a0cd73b6416d9897ceff7b85a24604 gfortran.tar.gz"
-            fi
-
-            # Validate gfortran tarball
-            curl -L $GFORTRAN_URL -o gfortran.tar.gz
-            if ! echo "$KNOWN_SHA256" != "$(shasum --algorithm 256 gfortran.tar.gz)"; then
-              echo "Checksum failed"
-              exit 1
-            fi
-
-            mkdir -p gfortran_installed
-            tar -xv -C gfortran_installed/ -f gfortran.tar.gz
-
-            if [[ $(uname -m) == "x86_64" ]]; then
-              export FC=$(pwd)/gfortran_installed/gfortran-darwin-x86_64-native/bin/gfortran
-              export PATH=$(pwd)/gfortran_installed/gfortran-darwin-x86_64-native/bin:$PATH
-            elif [[ $(uname -m) == "arm64" ]]; then
-              export FC=$(pwd)/gfortran_installed/gfortran-darwin-arm64-native/bin/gfortran
-              export PATH=$(pwd)/gfortran_installed/gfortran-darwin-arm64-native/bin:$PATH
-            fi
-
-            # link libgfortran dylibs and place them in $HOME/.local/lib
-            # and then change rpath to $HOME/.local/lib for each of them
-            # Note: libgcc_s.1.dylib not available on macOS arm64; skip for now
-            mkdir -p $HOME/.local/lib
-            if [[ $(uname -m) == "x86_64" ]]; then
-              lib_dir=$(pwd)/gfortran_installed/gfortran-darwin-x86_64-native/lib
-              for lib in libgfortran.5.dylib libgfortran.dylib libquadmath.0.dylib libquadmath.dylib libgcc_s.1.dylib libgcc_s.1.1.dylib; do
-                cp $lib_dir/$lib $HOME/.local/lib/
-                install_name_tool -id $HOME/.local/lib/$lib $HOME/.local/lib/$lib
-                codesign --force --sign - $HOME/.local/lib/$lib
-              done
-            elif [[ $(uname -m) == "arm64" ]]; then
-              lib_dir=$(pwd)/gfortran_installed/gfortran-darwin-arm64-native/lib
-              for lib in libgfortran.5.dylib libgfortran.dylib libquadmath.0.dylib libquadmath.dylib libgcc_s.1.1.dylib; do
-                cp $lib_dir/$lib $HOME/.local/lib/
-                install_name_tool -id $HOME/.local/lib/$lib $HOME/.local/lib/$lib
-                codesign --force --sign - $HOME/.local/lib/$lib
-              done
-            fi
-
-            export SDKROOT=${SDKROOT:-$(xcrun --show-sdk-path)}
-
-            # Can't download LLVM-OpenMP directly, use conda/mamba and set environment variables
-            brew install miniforge
-            mamba create -n pybamm-dev $OPENMP_URL
-            if [[ $(uname -m) == "x86_64" ]]; then
-              PREFIX="/usr/local/Caskroom/miniforge/base/envs/pybamm-dev"
-            elif [[ $(uname -m) == "arm64" ]]; then
-              PREFIX="/opt/homebrew/Caskroom/miniforge/base/envs/pybamm-dev"
-            fi
-
-            # Copy libomp.dylib from PREFIX to $HOME/.local/lib, needed for wheel repair
-            cp $PREFIX/lib/libomp.dylib $HOME/.local/lib/
-            install_name_tool -id $HOME/.local/lib/libomp.dylib $HOME/.local/lib/libomp.dylib
-            codesign --force --sign - $HOME/.local/lib/libomp.dylib
-
-            export CC=/usr/bin/clang
-            export CXX=/usr/bin/clang++
-            export CPPFLAGS="$CPPFLAGS -Xpreprocessor -fopenmp"
-            export CFLAGS="$CFLAGS -I$PREFIX/include"
-            export CXXFLAGS="$CXXFLAGS -I$PREFIX/include"
-            export LDFLAGS="$LDFLAGS -L$PREFIX/lib -lomp"
-
             # cibuildwheel not recognising its environment variable, so set manually
             export CIBUILDWHEEL="1"
 
-            python scripts/install_KLU_Sundials.py
             python -m cibuildwheel --output-dir wheelhouse
         env:
           CIBW_ENVIRONMENT: >
             PYBAMM_DISABLE_TELEMETRY="true"
           # 10.13 for Intel (macos-13), 11.0 for Apple Silicon (macos-14 and macos-latest)
           MACOSX_DEPLOYMENT_TARGET: ${{ matrix.os == 'macos-14' && '11.0' || '10.13' }}
           CIBW_ARCHS_MACOS: auto
-          CIBW_BEFORE_BUILD: python -m pip install cmake casadi setuptools delocate
+          CIBW_BEFORE_BUILD: python -m pip install setuptools delocate
           CIBW_REPAIR_WHEEL_COMMAND: |
             if [[ $(uname -m) == "x86_64" ]]; then
               delocate-listdeps {wheel} && delocate-wheel -v -w {dest_dir} {wheel}
@@ -255,7 +163,6 @@ jobs:
           CIBW_TEST_EXTRAS: "all,dev,jax"
           CIBW_TEST_COMMAND: |
             set -e -x
-            python -c "import pybamm; print(pybamm.IDAKLUSolver())"
             python -m pytest -m cibw {project}/tests/unit
 
       - name: Upload wheels for macOS (amd64, arm64)

diff --git a/.github/workflows/run_periodic_tests.yml b/.github/workflows/run_periodic_tests.yml
@@ -9,8 +9,6 @@ on:
 env:
   PYBAMM_DISABLE_TELEMETRY: "true"
   FORCE_COLOR: 3
-  PYBAMM_IDAKLU_EXPR_CASADI: ON
-  PYBAMM_IDAKLU_EXPR_IREE: ON
 
 concurrency:
   # github.workflow: name of the workflow, so that we don't cancel other workflows
@@ -70,11 +68,6 @@ jobs:
       - name: Install nox
         run: python -m uv pip install nox[uv]
 
-      - name: Install SuiteSparse and SUNDIALS on GNU/Linux and macOS
-        timeout-minutes: 10
-        if: matrix.os != 'windows-latest'
-        run: python -m nox -s pybamm-requires
-
       - name: Run unit tests for ${{ matrix.os }} with Python ${{ matrix.python-version }}
         if: matrix.os != 'ubuntu-latest' || matrix.python-version != '3.12'
         run: python -m nox -s unit
@@ -92,7 +85,6 @@ jobs:
       - name: Run integration tests for ${{ matrix.os }} with Python ${{ matrix.python-version }}
         run: python -m nox -s integration
 
-  # Skips IDAKLU module compilation for speedups, which is already tested in other jobs.
   run_doctests:
     runs-on: ubuntu-latest
     strategy:
@@ -153,10 +145,6 @@ jobs:
       - name: Install nox
         run: python -m uv pip install nox[uv]
 
-      - name: Install SuiteSparse and SUNDIALS on GNU/Linux
-        timeout-minutes: 10
-        run: python -m nox -s pybamm-requires
-
       - name: Run example notebooks tests for GNU/Linux with Python 3.12
         run: python -m nox -s examples
 
@@ -186,9 +174,5 @@ jobs:
       - name: Install nox
         run: python -m uv pip install nox[uv]
 
-      - name: Install SuiteSparse and SUNDIALS on GNU/Linux
-        timeout-minutes: 10
-        run: python -m nox -s pybamm-requires
-
       - name: Run example scripts tests for GNU/Linux with Python 3.12
         run: python -m nox -s scripts
diff --git a/.github/workflows/test_on_push.yml b/.github/workflows/test_on_push.yml
@@ -7,8 +7,6 @@ on:
 env:
   PYBAMM_DISABLE_TELEMETRY: "true"
   FORCE_COLOR: 3
-  PYBAMM_IDAKLU_EXPR_CASADI: ON
-  PYBAMM_IDAKLU_EXPR_IREE: ON
 
 concurrency:
   # github.workflow: name of the workflow, so that we don't cancel other workflows
@@ -95,24 +93,6 @@ jobs:
       - name: Install nox
         run: python -m uv pip install nox[uv]
 
-      - name: Cache pybamm-requires nox environment for GNU/Linux and macOS
-        uses: actions/cache@1bd1e32a3bdc45362d1e726936510720a7c30a57 # v4.2.0
-        if: matrix.os != 'windows-latest'
-        with:
-          path: |
-            # Repository files
-            ${{ github.workspace }}/pybind11/
-            ${{ github.workspace }}/install_KLU_Sundials/
-            # Headers and dynamic library files for SuiteSparse and SUNDIALS
-            ${{ env.HOME }}/.local/lib/
-            ${{ env.HOME }}/.local/include/
-          key: nox-${{ matrix.os }}-pybamm-requires-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/install_KLU_Sundials.py', '**/noxfile.py', '**/test_on_push.yml') }}
-
-      - name: Install SuiteSparse and SUNDIALS on GNU/Linux and macOS
-        timeout-minutes: 10
-        if: matrix.os != 'windows-latest'
-        run: python -m nox -s pybamm-requires
-
       - name: Run unit tests for ${{ matrix.os }} with Python ${{ matrix.python-version }}
         if: matrix.os != 'ubuntu-latest' || matrix.python-version != '3.12'
         run: python -m nox -s unit
@@ -130,7 +110,6 @@ jobs:
       - name: Run integration tests for ${{ matrix.os }} with Python ${{ matrix.python-version }}
         run: python -m nox -s integration
 
-  # Skips IDAKLU module compilation for speedups, which is already tested in other jobs.
   run_doctests:
     runs-on: ubuntu-latest
     strategy:
@@ -211,22 +190,6 @@ jobs:
       - name: Install nox
         run: python -m uv pip install nox[uv]
 
-      - name: Cache pybamm-requires nox environment for GNU/Linux
-        uses: actions/cache@1bd1e32a3bdc45362d1e726936510720a7c30a57 # v4.2.0
-        with:
-          path: |
-            # Repository files
-            ${{ github.workspace }}/pybind11/
-            ${{ github.workspace }}/install_KLU_Sundials/
-            # Headers and dynamic library files for SuiteSparse and SUNDIALS
-            ${{ env.HOME }}/.local/lib/
-            ${{ env.HOME }}/.local/include/
-          key: nox-${{ matrix.os }}-pybamm-requires-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/install_KLU_Sundials.py', '**/noxfile.py', '**/test_on_push.yml') }}
-
-      - name: Install SuiteSparse and SUNDIALS on GNU/Linux
-        timeout-minutes: 10
-        run: python -m nox -s pybamm-requires
-
       - name: Run example notebooks tests for GNU/Linux with Python 3.12
         run: python -m nox -s examples
 
@@ -266,21 +229,5 @@ jobs:
       - name: Install nox
         run: python -m uv pip install nox[uv]
 
-      - name: Cache pybamm-requires nox environment for GNU/Linux
-        uses: actions/cache@1bd1e32a3bdc45362d1e726936510720a7c30a57 # v4.2.0
-        with:
-          path: |
-            # Repository files
-            ${{ github.workspace }}/pybind11/
-            ${{ github.workspace }}/install_KLU_Sundials/
-            # Headers and dynamic library files for SuiteSparse and SUNDIALS
-            ${{ env.HOME }}/.local/lib/
-            ${{ env.HOME }}/.local/include/
-          key: nox-${{ matrix.os }}-pybamm-requires-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/install_KLU_Sundials.py', '**/noxfile.py', '**/test_on_push.yml') }}
-
-      - name: Install SuiteSparse and SUNDIALS on GNU/Linux
-        timeout-minutes: 10
-        run: python -m nox -s pybamm-requires
-
       - name: Run example scripts tests for GNU/Linux with Python 3.12
         run: python -m nox -s scripts
diff --git a/.gitignore b/.gitignore
@@ -78,14 +78,6 @@ pyvenv.cfg
 .vscode
 .ruff_cache/
 
-# sundials
-sundials
-sundials4
-sundials-*
-SuiteSparse-*
-build_sundials
-KLU_module_deps
-
 # downloads
 *.gz
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -19,6 +19,9 @@
 package to install PyBaMM with only the required dependencies. ([conda-forge/pybamm-feedstock#70](https://github.com/conda-forge/pybamm-feedstock/pull/70))
 - Separated extrapolation options for `pybamm.BoundaryValue` and `pybamm.BoundaryGradient`, and updated the default to be "linear" for the value and "quadratic" for the gradient. ([#4614](https://github.com/pybamm-team/PyBaMM/pull/4614))
 - Double-layer SEI models have been removed (with the corresponding parameters). All models assume now a single SEI layer. ([#4470](https://github.com/pybamm-team/PyBaMM/pull/4470))
+- Moved the IDAKLU solver to a standalone `pybammsolvers` package. This will
+  make PyBaMM a pure Python package and make installing and using the solver
+  easier. ([#4487](https://github.com/pybamm-team/PyBaMM/pull/4487))
 - Wycisk OCP model now requires an parameter to set the initial condition. ([#4374](https://github.com/pybamm-team/PyBaMM/pull/4374))
 
 ## Bug fixes