Create environment.yml files on the fly in GitHub Actions workflows

[seisman]

Currently, we have six workflows (e.g., ci_dos.yml) that use the mamba-org/setup-micromamba action to create a conda environment. Each workflow has different purposes so they install different packages and sometimes different pinned versions. It's impossible to create a single environment.yml that can be used in different workflows. Thus, we have to pass a list of packages to the creat-args option of the mamba-org/setup-micromamba action. E.g.,

pygmt/.github/workflows/ci_tests.yaml

Lines 106 to 122 in b873c36

	create-args: >-
	python=${{ matrix.python-version }}${{ matrix.optional-packages }}
	gmt=6.4.0
	ghostscript=9.54.0
	numpy=${{ matrix.numpy-version }}
	pandas
	xarray
	netCDF4
	packaging
	build
	dvc
	make
	pip
	pytest
	pytest-cov
	pytest-doctestplus
	pytest-mpl

It means we have to update several workflow files if we want to pin a package, add a new required/optional package, or add a new dev package, which is tedious.

To avoid maintaining the package list in so many places, we can write a bash script that can create environment.yml files on the fly for different purposes. Then we can use the generated environment.yml file in the mamba-org/setup-micromamba action.

Here is an initial/incompleted version of the script:

required="numpy pandas xarray netCDF4 packaging"

cat << EOF
name: pygmt
channels:
  - conda-forge
  - nodefaults
dependencies:
EOF

for package in $required; do
    if [[ -v NUMPY_VERSION && "$package" == "numpy" ]]; then
        echo "  - $package=${NUMPY_VERSION}"
    else
        echo "  - $package"
    fi
done

By default, it produces:

name: pygmt
channels:
  - conda-forge
  - nodefaults
dependencies:
  - numpy
  - pandas
  - xarray
  - netCDF4
  - packaging

If we define an environment variable export NUMPY_VERSION=1.26, it produces:

name: pygmt
channels:
  - conda-forge
  - nodefaults
dependencies:
  - numpy=1.26
  - pandas
  - xarray
  - netCDF4
  - packaging

• Pros: fewer files to maintain when we add/remove/pin packages
• Cons: need to write and maintain a bash script in the repository, which may be not as simple as I thought

Comments?

[seisman]

Much easier than I initially thought. A working script is submitted in #2817.

[weiji14]

Not so sure about this. I feel like having the dependency list in each YAML file is more explicit/readable, and we don't pin or add new dependencies regularly, so the bash script seems to be overcomplicating things a bit. Can we maybe vote yes 👍 or no 👎 on the top post to see if we should go ahead on this?

[seisman]

the bash script seems to be overcomplicating things a bit.

I have the same feeling when working on the script. Closing the issue.