From 2baf1ebde13ea32c5756ca3ed30f993a309e0662 Mon Sep 17 00:00:00 2001 From: paugier Date: Fri, 5 Jan 2024 23:54:40 +0100 Subject: [PATCH] Update doc after Meson --- doc/build-from-source.md | 234 ++++++++++++++++++++++++++++++++++++ doc/faq.md | 15 +-- doc/index.md | 1 + doc/install.md | 253 +++++++++++---------------------------- site.cfg.default | 21 ---- 5 files changed, 313 insertions(+), 211 deletions(-) create mode 100644 doc/build-from-source.md delete mode 100644 site.cfg.default diff --git a/doc/build-from-source.md b/doc/build-from-source.md new file mode 100644 index 000000000..d6bcb6f0c --- /dev/null +++ b/doc/build-from-source.md @@ -0,0 +1,234 @@ +# Build from source + +## Requirements + +To build Fluidsim from source, we need: + +- A modern Python with Python headers and with `pip` + +- A decent amount of RAM (at least few GB available). + +- A C++ compiler fully compliant with the C++-11 standard (clang or GCC, currently + not Intel compilers) + +Moreover, you might also want to compile Fluidfft, mpi4py and a parallel version +of h5py. + +## Get the source + +Note that we use for Fluidsim development the revision control software +[Mercurial] and modern Mercurial extensions like [Evolve] and \[Topic\]. The main +repository is hosted [here](https://foss.heptapod.net/fluiddyn/fluidsim) in +[Heptapod](https://heptapod.net/). + +There are other ways to get the source but we are going here to assume that you +can install [Mercurial]. It can be useful when working with Fluidsim source to +[fully setup Mercurial with these extensions and learn a bit of Mercurial](http://fluiddyn.readthedocs.org/en/latest/mercurial_heptapod.html). +Then, the Fluidsim repository can be cloned with + +```sh + +hg clone https://foss.heptapod.net/fluiddyn/fluidsim + +``` + +```{admonition} Why Mercurial/Heptapod and not simply Git/Github? + +We consider that modern Mercurial is really great. Much better in some aspects +that Git. Moreover, we do not think that it is a good thing that the whole +open-source ecosystem depends on Github, a close-source project owned by +Microsoft. + +Thanks to [Octobus](https://octobus.net/) and [Clever +Cloud](https://www.clever-cloud.com) for providing https://foss.heptapod.net! + +``` + +## Installing from the repository + +### Simple installation from source + +We recommend to create a clean virtual environment, for example with: + +```sh +cd fluidsim +python3 -m venv .venv +. .venv/bin/activate +pip install pip -U +``` + +Then, let us just install the local project (fluidsim) with `pip`: + +```sh +pip install . -v +``` + +```{note} + +`-v` toggles the verbose mode of `pip` so that we see the compilation log and +can check that everything goes well. + +``` + +However, the build (which uses [Meson]) can be controlled through environment +variables (for the C++ compilation) and options. The particular build options for +Fluidsim are defined in the file `meson.options` which contains: + +```{literalinclude} ../meson.options +``` + +To choose a value different from the default value, one can use this ugly syntax: + +```sh +pip install . -v --config-settings=setup-args=-Dtransonic-backend=python +# or +pip install . -v -C setup-args=-Dtransonic-backend=python +``` + +```{admonition} Let's decompose this syntax! + +There are 3 levels: + +- `--config-settings` / `-C` is a `pip` option to pass configuration to the PEP + 517 build backend (for Fluidsim [meson-python]). + +- `setup-args` is [understood by + meson-python](https://meson-python.readthedocs.io/en/latest/reference/config-settings.html) + +- `transonic-backend` is a Fluidsim build option. But one needs to add the `-D` + for [Meson]! + +``` + +Of course, one can also change values for +[other buildin Meson options](https://meson-python.readthedocs.io/en/latest/reference/config-settings.html). + +````{warning} +(compile-args-j2)= + +Fluidsim binaries are builds in parallel. This speedups the build process a lot on +most computers. However, it can be a very bad idea on computers with not enough +memory. One can control the number of processes launched in parallel with: + +```sh +pip install . -C compile-args=-j2 +``` + +```` + +```{todo} + +- How to use `-march=native`? How to differentiate a native build from a + regular build to produce binaries usable on other computers? + +- How to know which compiler and compilation flags are used? + +- How to produce a wheel for other architectures (cross-compilation)? + +- How to check if XSIMD was indeed used? + +``` + +### Setup a full developer environment with editable installation + +Let us first present the tools that are used for Fluidsim development. + +- [PDM] is a modern Python package and dependency manager, + +- [Meson] is an open source build system (in particular used by Scipy), + +- [Nox] is a command-line tool that automates testing in multiple Python + environments, + +- [Pytest] is the most popular testing framework for Python, + +- [pip] is the official package installer for Python, + +- [Pythran] is an ahead of time compiler for a subset of the Python language, with + a focus on scientific computing, + +- [Transonic] is a pure Python package to accelerate modern Python-Numpy code with + different accelerators (in particular Pythran). + +Fluidsim is built with [Meson]. We use [PDM] for Fluidsim development. [Pytest] +and [Nox] are used for testing. We use [Pythran] through [Transonic] to accelerate +some numerical kernels written in Python. + +#### Standard Python from Python.org + +We present here how one can build Fluidsim from source as the main developers and +users. + +##### Install PDM + +A first step is to install [PDM] as an external independant application. I (Pierre +Augier) usually use \[pipx\] for that but +[there are other methods](https://pdm-project.org/latest/#installation). + +```sh +python3 -m pip install pipx +pipx install pdm -U +``` + +Installing in editable mode is a bit particular with Meson, since all build +dependencies have to be installed in the main virtual environment, i.e. editable +installations are incompatible with isolated builds! Fortunatelly, it's not too +difficult with [PDM]: + +```sh +pdm install --no-self +pdm run pip install -e . --no-build-isolation +``` + +### Python installed with conda/mamba/conda-forge + +??? + +## Run the tests + +You can run some unit tests by running `make tests` (shortcut for +`fluidsim-test -v`) or `make tests_mpi` (shortcut for +`mpirun -np 2 fluidsim-test -v`). Alternatively, you can also run `pytest` from +the root directory or from any of the source directories. + +## Advice for developers + +(pythranrc)= + +### About using Pythran to compile functions + +When developing with Transonic and Pythran, it is good to have a `~/.pythranrc` +file, with for example something like (see +[the dedicated section in Pythran documentation](https://pythran.readthedocs.io/en/latest/MANUAL.html#customizing-your-pythranrc)): + +```sh + +[pythran] +complex_hook = True + +[compiler] +CXX=clang++ +CC=clang + +``` + +### Set the MESONPY_EDITABLE_VERBOSE mode + +It can be useful to set this environment variable when using the editable mode. + +```sh +export MESONPY_EDITABLE_VERBOSE=1 +``` + +See +https://meson-python.readthedocs.io/en/latest/how-to-guides/editable-installs.html#verbose-mode + +[evolve]: https://www.mercurial-scm.org/doc/evolution/ +[mercurial]: https://www.mercurial-scm.org/ +[meson]: https://mesonbuild.com +[nox]: https://nox.thea.codes +[pdm]: https://pdm-project.org +[pip]: https://pip.pypa.io +[pytest]: https://docs.pytest.org +[pythran]: https://pythran.readthedocs.io +[transonic]: https://transonic.readthedocs.io diff --git a/doc/faq.md b/doc/faq.md index 7dedd4e37..9607eebe3 100644 --- a/doc/faq.md +++ b/doc/faq.md @@ -16,11 +16,11 @@ instance {mod}`fluidsim.base.basilisk`, {mod}`fluidsim.base.dedalus` and Make sure you read the [installation guide](install) carefully. -:::{admonition} Permission denied while running `pip install fluidsim` from PyPI +:::{admonition} Permission denied while running `pip install fluidsim` from PyPI or `make develop` inside the repository. -or `make develop` inside the repository. This means you are probably using the -Python bundled with the system and as a user you are restricted from installing -packages into it. If this is so, create a [virtual environment]. +This means you are probably using the Python bundled with the system and as a +user you are restricted from installing packages into it. If this is so, create +a [virtual environment]. ::: @@ -31,7 +31,7 @@ following commands should help: ```sh python -m ensurepip -python -m pip install --upgrade pip setuptools wheel +python -m pip install --upgrade pip ``` ::: @@ -42,8 +42,9 @@ By default `pythran` extensions try to use `gcc` and this is a CPU and memory intensive compilation. Instead `pythran` can be configured to use `clang`. See {ref}`pythranrc` for more details. -Additionally to reduce the load during installation is to configure certain -{ref}`build specific environment variables `. +Additionally to reduce the load during installation one can [limit the number of +parallel processes used to build extensions](compile-args-j2). + ::: :::{admonition} `ModuleNotFoundError: No module named 'fluidsim_core. ...` diff --git a/doc/index.md b/doc/index.md index 499906210..c8cd4f869 100644 --- a/doc/index.md +++ b/doc/index.md @@ -114,6 +114,7 @@ maxdepth: 1 install tutorials examples +build-from-source faq ``` diff --git a/doc/install.md b/doc/install.md index 6b5df0327..50083ba43 100644 --- a/doc/install.md +++ b/doc/install.md @@ -1,197 +1,142 @@ -# Installation +# Install and configure -FluidSim is part of the FluidDyn project and requires Python >= 3.6. Some issues -regarding the installation of Python and Python packages are discussed in +First, ensure that you have a recent Python installed, since Fluidsim requires +Python >= 3.9. Some issues regarding the installation of Python and Python +packages are discussed in [the main documentation of the project](http://fluiddyn.readthedocs.org/en/latest/install.html). -We don't upload wheels on PyPI, so the simplest and fastest procedure is to use -`conda` (no compilation needed). Alternatively, one can compile fluidsim and -fluidfft using `pip`. +## Installation methods without compilation -## Installing the conda-forge packages with conda or mamba +Currently, we don't upload wheels on PyPI, implying that installing with `pip` +generally trigger compilation steps. [Building from +source](./build-from-source.md) requires both hardware and software +requirements. Thus, the simplest and fastest installation method is via `conda` +or `mamba`. -The fluidsim packages are in the conda-forge channels, so if it is not already -done, one needs to add it: +### Installing the conda-forge packages with conda or mamba -```sh -conda config --add channels conda-forge -``` +We recommend installing `conda` and `mamba` (using the [conda-forge] channel) with +the [miniforge installer](https://github.com/conda-forge/miniforge). If you just want to run sequential simulations and/or analyze the results of simulations, you can just install the fluidsim package: ```sh -conda install fluidsim +mamba install fluidsim ``` For parallel simulations using MPI, let's create a dedicated environment: ```sh -conda create -n env_fluidsim ipython fluidsim "fluidfft[build=mpi*]" "h5py[build=mpi*]" +mamba create -n env_fluidsim ipython fluidsim "fluidfft[build=mpi*]" "h5py[build=mpi*]" ``` The environment can then be activated with `conda activate env_fluidsim`. -## Build/install using pip +### Install using pip without compilation -We recommend installing with a recent version of pip so you might want to run -`pip install pip -U` before anything (if you use conda, `conda update pip`). Then, -fluidsim can be installed with: - -```sh -pip install fluidsim -``` +```{warning} -However, one have to note that (i) fluidsim builds are sensible to environment -variables (see below) and (ii) fluidsim can optionally use -[fluidfft](http://fluidfft.readthedocs.io) for pseudospectral solvers. Fluidsim -and fluidfft can be both installed with the command: +We mention here how to install Fluidsim with `pip` without compilation, but +this of course leads to a slow version of Fluidsim. Most of the time, this is +not what you want. -```sh -pip install fluidsim[fft] ``` -Moreover, fluidfft builds can also be tweaked so you could have a look at -[fluidfft documentation](http://fluidfft.readthedocs.io/en/latest/install.html). +```{todo} -(env-vars)= +Upload wheels on PyPI (for fluidsim and fluidfft) so that we can simplify this +section a lot. -## Environment variables and build configuration +``` -### Build time configuration +Fluidsim can be installed without compilation with `pip`: -- `FLUIDDYN_NUM_PROCS_BUILD` +```sh +pip install pip -U +pip install fluidsim --config-settings=setup-args=-Dtransonic-backend=python +``` -Fluidsim binaries are builds in parallel. This speedups the build process a lot on -most computers. However, it can be a very bad idea on computers with not enough -memory. If you encounter problems, you can force the number of processes used -during the build using the environment variable `FLUIDDYN_NUM_PROCS_BUILD`: +However, fluidsim can optionally use [fluidfft](http://fluidfft.readthedocs.io) +for pseudospectral solvers. Fluidsim and fluidfft can be both installed without +compilation with the command: ```sh -export FLUIDDYN_NUM_PROCS_BUILD=2 +export FLUIDFFT_TRANSONIC_BACKEND="python" +pip install fluidsim[fft] --config-settings=setup-args=-Dtransonic-backend=python ``` -- Customize compilers to build extensions, if the defaults do not work for you, - either using the environment variables: +Moreover, fluidfft builds can be tweaked so you could have a look at +[fluidfft documentation](http://fluidfft.readthedocs.io/en/latest/install.html). - - `MPICXX`: for Cython extensions in `fluidfft` (default: `mpicxx`) - - `CC`: command to generate object files in `fluidsim` - - `LDSHARED`: command to link and generate shared libraries in `fluidsim` - - `CARCH`: to cross compile (default: `native`) +### Optional dependencies -- `DISABLE_PYTHRAN` disables compilation with Pythran at build time. +Fluidsim has 4 sets of optional dependencies, which can be installed with commands +like `pip install fluidsim[fft]` or `pip install fluidsim[fft, mpi]`: -- `FLUIDSIM_TRANSONIC_BACKEND` +- `fft`: mainly for pseudo spectral solvers using the Fourier basis. - "pythran" by default, it can be set to "python", "numba" or "cython". +- `mpi`: for parallel computing using [MPI]. `pip install fluidsim[mpi]` installs + [mpi4py], which requires a local compilation. -You can also configure the installation of fluidsim by creating the file -`~/.fluidsim-site.cfg` (or `site.cfg` in the repo directory) and modify it to fit -your requirements before the installation with pip: +- `test`: for testing Fluidsim (can be done without the repository). -```sh -wget https://foss.heptapod.net/fluiddyn/fluidsim/-/raw/branch/default/site.cfg.default -O ~/.fluidsim-site.cfg -``` +- `scipy` -### Runtime configuration +## Environment variables and runtime configuration -Fluidsim is also sensitive to the environment variables: +Fluidsim is sensitive to environment variables: - `FLUIDSIM_PATH`: path where the simulation results are saved. In Unix systems, you can for example put this line in your `~/.bashrc`: ```sh - export FLUIDSIM_PATH=$HOME/Data + export FLUIDSIM_PATH=$HOME/data_fluidsim ``` - `FLUIDDYN_PATH_SCRATCH`: working directory (can be useful on some clusters). -## Warning about re-installing fluidsim and fluidfft with new build options - -If fluidsim has already been installed and you want to recompile with new -configuration values, you need to really recompile fluidsim and not just reinstall -an already produced wheel. To do this, use: +## Dependencies with different flavours -```sh -pip install fluidsim --no-binary fluidsim -v -``` +## Fluidfft -`-v` toggles the verbose mode of pip so that we see the compilation log and can -check that everything goes well. - -(pythranrc)= - -## About using Pythran to compile functions - -We choose to use the Python compiler -[Pythran](https://github.com/serge-sans-paille/pythran) for some functions of the -operators. Our microbenchmarks show that the performances are as good as what we -are able to get with Fortran or C++! - -But it implies that building Fluidsim requires a C++11 compiler (for example GCC -4.9 or clang). - -:::{note} - -If you don't want to use Pythran and C++ to speedup Fluidsim, you can -use the environment variable `DISABLE_PYTHRAN`. - -::: - -:::{warning} - -To reach good performance, we advice to try to put in the file -`~/.pythranrc` the lines (it seems to work well on Linux, see the -[Pythran documentation](https://pythran.readthedocs.io)): - -```bash -[pythran] -complex_hook = True -``` - -::: +Fluidsim uses Fluidfft to compute Fourier transforms. Fluidfft can be installed in +different modes. Have a look at the +[fluidfft documentation](http://fluidfft.readthedocs.io/en/latest/install.html). -:::{warning} +```{todo} -The compilation of C++ files produced by Pythran can be long and can -consume a lot of memory. If you encounter any problems, you can try to use clang -(for example with `conda install clangdev`) and to enable its use in the file -`~/.pythranrc` with: +Fluidfft should be fully reorganized so that we can write here something simple +and nice. -```bash -[compiler] -CXX=clang++ -CC=clang ``` -::: +## MPI simulations and mpi4py -## MPI simulations and mpi4py! +Fluidsim can use [mpi4py] (which depends on a [MPI] implementation) for parallel +simulations. -Fluidsim can use [mpi4py](http://mpi4py.scipy.org) (which depends on a MPI -implementation) for MPI simulations. +````{warning} -:::{warning} - -If the system has multiple MPI libraries, it is adviced to -explicitly mention the MPI command. For instance to use Intel MPI: +If the system has multiple MPI libraries, it is adviced to explicitly mention the +MPI command. For instance to use Intel MPI: ```sh CC=mpiicc pip install mpi4py --no-binary mpi4py ``` -::: +```` ## About h5py and HDF5_MPI FluidSim is able to use h5py built with MPI support. -:::{warning} +````{warning} -Prebuilt installations (for e.g. via h5py wheels) lacks MPI support. -Most of the time, this is what you want. However, you can install h5py from source -and link it to a hdf5 built with MPI support, as follows: +Prebuilt installations (for e.g. via h5py wheels) lacks MPI support. Most of the +time, this is what you want. However, you can install h5py from source and link it +to a hdf5 built with MPI support, as follows: ```bash CC="mpicc" HDF5_MPI="ON" HDF5_DIR=/path/to/parallel-hdf5 pip install --no-deps --no-binary=h5py h5py @@ -209,66 +154,8 @@ CC="mpicc" HDF5_MPI="ON" HDF5_DIR=/path/to/parallel-hdf5 pip install --no-deps - See the [h5py documentation](http://docs.h5py.org/en/latest/build.html) for more details. -::: - -## Installing from the repository - -:::{note} - -A good base to install Fluidsim from source can be to create and -activate a conda environment with: - -```sh -conda create -y -n env-fluidsim -c conda-forge "fluidfft=*=*openmpi*" pythran clangdev mako -conda activate env-fluidsim -``` - -::: - -For fluidsim, we use the revision control software Mercurial and the main -repository is hosted [here](https://foss.heptapod.net/fluiddyn/fluidsim) in -Heptapod. Download the source with something like: - -```sh -hg clone https://foss.heptapod.net/fluiddyn/fluidsim -``` - -If you are new with Mercurial and Heptapod, you can also read -[this short tutorial](http://fluiddyn.readthedocs.org/en/latest/mercurial_heptapod.html). - -For particular installation setup, copy the default configuration file to -`site.cfg`: - -```sh -cp site.cfg.default site.cfg -``` - -and modify it to fit your requirements. - -Build/install in development mode, by running from the top-level directory: - -```sh -cd lib && pip install -e .; cd .. -pip install -e . -``` - -:::{note} - -To install from source in a conda environment, it is actually necessary -to disable the isolated build by running the command -`pip install -e . --no-build-isolation`. - -::: - -To install Fluidsim with all optional dependencies and all capacities: - -```sh -pip install -e .[full] -``` - -### Run the tests +```` -You can run some unit tests by running `make tests` (shortcut for -`fluidsim-test -v`) or `make tests_mpi` (shortcut for -`mpirun -np 2 fluidsim-test -v`). Alternatively, you can also run `pytest` from -the root directory or from any of the source directories. +[conda-forge]: https://conda-forge.org/ +[mpi]: https://fr.wikipedia.org/wiki/Message_Passing_Interface +[mpi4py]: https://mpi4py.readthedocs.io/ diff --git a/site.cfg.default b/site.cfg.default deleted file mode 100644 index be6e724c6..000000000 --- a/site.cfg.default +++ /dev/null @@ -1,21 +0,0 @@ -[environ] - - -## Uncomment and specify the following options to modify compilation of -## extensions. - -## To modify compiler used to build Cython extensions: -# MPICXX = -# CC = -# LDSHARED = - -## To modify compiler used to build Pythran extensions (or alternatively, -## set ~/.pythranrc. A word of caution --- the pythranrc approach may result in -## race condition for setting and unsetting compilers for pythran > 0.8.6): -# CXX = - -## To modify target architecture while building Pythran extensions -## Useful when cross-compiling. See whether it is required by comparing: -## gcc -march=native -Q --help=target -## gcc -march=$CARCH -Q --help=target -# CARCH =