From fda4717c42a76fc6b0faa02e54e1b321d6ff7b5a Mon Sep 17 00:00:00 2001 From: Bart Schilperoort Date: Tue, 18 Apr 2023 09:18:00 +0200 Subject: [PATCH] Add developer documentation to the new docs (#144) * Add vector source of logos * Add developer documentation * Update contribute.md, add CoC. Fix mkdocs page gen * Slight change to contributing page, changelog. * Move CONTRIBUTE.md content to docs. --- CONTRIBUTING.md | 70 +-------- docs/CHANGELOG.md | 3 +- docs/CODE_OF_CONDUCT.md | 128 +++++++++++++++ docs/assets/era5cli_logo_white.svg | 244 +++++++++++++++++++++++++++++ docs/contribute.md | 61 +++----- docs/dev.md | 1 - docs/development_era5cli.md | 24 +++ docs/gen_reference_pages.py | 2 +- docs/general_development.md | 151 ++++++++++++++++++ mkdocs.yml | 3 +- 10 files changed, 574 insertions(+), 113 deletions(-) create mode 100644 docs/CODE_OF_CONDUCT.md create mode 100644 docs/assets/era5cli_logo_white.svg delete mode 100644 docs/dev.md create mode 100644 docs/development_era5cli.md create mode 100644 docs/general_development.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 95e01e0..1ed031a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,71 +6,7 @@ way to propose new features. # Readthedocs For technical assistance with your contribution, please check the [contributing guidelines on -readthedocs](https://era5cli.readthedocs.io/en/latest/contribute.html). +readthedocs](https://era5cli.readthedocs.io/en/latest/contribute/), and possibly the +[developer documentation](https://era5cli.readthedocs.io/en/latest/general_development/) as well. -# Making a release - -## Author information -Ensure all authors are present in: - -- `CITATION.cff` -- `era5cli/__version__.py` - -## Confirm release info -Ensure the right date and upcoming version number (including release candidate -tag, if applicable) is set in: - -- `CITATION.cff` -- `era5cli/__version__.py` -- `docs/source/conf.py` - -## Update the changelog -Update `CHANGELOG.rst` with new features and fixes in the upcoming version. -Confirm that `README.rst` is up to date with new features as well. - -## Release on GitHub -Open [releases](https://github.com/eWaterCycle/era5cli/releases) and draft a new -release. Copy the changelog for this version into the description (though note -that the description is in Markdown, so reformat from Rst if necessary). - -Tag the release according to [semantic versioning -guidelines](https://semver.org/), preceded with a `v` (e.g.: v1.0.0). The -release title is the tag and the release date together (e.g.: v1.0.0 -(2019-07-25)). - -### Release candidate -When releasing a release candidate on Github, tick the pre-release box, and -amend the version tag with `-rc` and the candidate number. Ensure the release -candidate version is accurate in `CITATION.cff` and `era5cli/__version__.py`. -If the version number in these files is not updated, Zenodo release -workflows will fail. - -Releasing a release candidate is not required, but can help detect bugs early. - -## PyPI release workflow -Publishing a new release in github triggers the github Action workflow that -builds and publishes the package to test.PyPI or PyPI. Versions with "rc" -(release candidate) in their version tag will only be published to test.PyPI. -Other version tags will trigger a PyPI release. Inspect -`.github/workflows/publish-to-pypi.yml` for more information. - -Confirm a release candidate on test.PyPI with: -``` -pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ era5cli -``` - -## Release on Zenodo -Confirm the new release on [Zenodo](https://doi.org/10.5281/zenodo.3252665). - -## Release on the Research Software Directory -Wait a few hours, then confirm the addition of a new release on the -[RSD](https://www.research-software.nl/software/era5cli). - -If any contributors have been added, or the description of the software has -changed, this can be edited (by eScience Center employees) via the [RSD admin -interface](https://www.research-software.nl/admin/). More information about -this process (e.g. how to add a new contributor or new affiliation) can be -found in the [RSD -documentation](https://github.com/research-software-directory/research-software-directory/blob/master/docs/entering-data.md) -or in [this -blogpost](https://blog.esciencecenter.nl/the-research-software-directory-and-how-it-promotes-software-citation-4bd2137a6b8). +Note that if readthedocs is down or otherwise unavailable, the instructions are also available in [`docs/contribute.md`](./docs/contribute.md) and [`docs/general_development.md`](./docs/general_development.md) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 71d3e3c..0f61273 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Add validator for `era5cli.txt` keys. This should provide better feedback to users and reduce user error. - Added --splitmonths argument for `era5cli hourly`. This allows users to avoid a Request Too Large error. + - Added --dashed-varname argument, to produce names where the variable name is separated using hyphens. For example: `soil-type` vs. `soil_type`. For the ongoing discussion, see [#53](https://github.com/eWaterCycle/era5cli/issues/53). **Changed:** @@ -19,11 +20,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - When a file already exists and would be overwritten, the user is prompted for confirmation. This should prevent accidental overwriting of files. This check can be skipped with the `--overwrite` flag. - The earliest valid start year of requests has been updated to 1950. - Usage of `--prelimbe` now raises a deprecation warning. It will be deprecated in a future release, as all the back extension years are now included in the main products. + - The documentation has been fully overhauled, and now uses Markdown files & MkDocs. **Dev changes:** - `cli.py` has been refactored to make the structure more clear. Seperate argument builders are now in their own modules. - - The documentation has been overhauled, and now uses Markdown files & MkDocs. ## 1.3.2 - 2021-12-13 **Changed:** diff --git a/docs/CODE_OF_CONDUCT.md b/docs/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..40bb9b9 --- /dev/null +++ b/docs/CODE_OF_CONDUCT.md @@ -0,0 +1,128 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +s.verhoeven@esciencecenter.nl. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct +enforcement ladder](https://github.com/mozilla/diversity). + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see the FAQ at +. Translations are available at +. diff --git a/docs/assets/era5cli_logo_white.svg b/docs/assets/era5cli_logo_white.svg new file mode 100644 index 0000000..f7cf4c3 --- /dev/null +++ b/docs/assets/era5cli_logo_white.svg @@ -0,0 +1,244 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + diff --git a/docs/contribute.md b/docs/contribute.md index 3d2b380..78ccce7 100644 --- a/docs/contribute.md +++ b/docs/contribute.md @@ -1,53 +1,30 @@ -# Contributing to era5cli - ## Bug reports -File bug reports or feature requests, and make contributions (e.g. code -patches), by [opening a new issue on GitHub](https://github.com/ewatercycle/era5cli/issues). +If you think you have found a bug, please [open a new issue on GitHub](https://github.com/ewatercycle/era5cli/issues). -Please give as much information as you can in the issue. It is very useful if +Try give as much information as you can in the issue. It is very useful if you can supply the command or a small self-contained code snippet that reproduces the problem. Please also specify the era5cli version that you are -using and the version of the cdsapi library installed. - -## Contribute to the tool - -Make sure `pip` and `hatch` are up to date: -``` -python3 -m pip install pip hatch --upgrade +using and the version of the cdsapi library installed. You can find these with: +```sh +pip show era5cli cdsapi ``` -Create and activate the development environment: - -``` -hatch shell # Or: python3 -m hatch shell -``` - -This will start a virtual environment with the required dependencies to allow for -development. Run this command before trying out any changes made to the code. - -Before pushing a new addition, some checks are required to confirm that the code -is up to standard. - -To run the test suite: -``` -hatch run test -``` - -To format the code, and check the code styling: -``` -hatch run format -``` +## Contribute to the tool -Exit the the environment with: -``` -exit -``` +We welcome contributions form everyone. Please use our [GitHub issue +tracker](https://github.com/ewatercycle/era5cli/issues) for questions, ideas, or feature requests. -### Contribute to the documentation +If you want to make a pull request: -When updating the documentation, build the documentation with: +1. discuss your idea first, before putting in a lot of effort +2. refer to the [developer documentation](./general_development.md) for information on how to setup your environment, testing, and more. +3. if needed, fork the repository to your own Github profile +4. work on your own feature branch +5. make sure the existing tests still work and add new tests (if necessary) +6. update or expand the documentation +7. make sure your code follows the style guidelines +8. don't be afraid to ask help with any of the above steps. We're happy to help! -``` -hatch run docs:build -``` +By participating in this project, you agree to abide by the [code of +conduct](CODE_OF_CONDUCT.md). diff --git a/docs/dev.md b/docs/dev.md deleted file mode 100644 index 1333ed7..0000000 --- a/docs/dev.md +++ /dev/null @@ -1 +0,0 @@ -TODO diff --git a/docs/development_era5cli.md b/docs/development_era5cli.md new file mode 100644 index 0000000..5ebcffc --- /dev/null +++ b/docs/development_era5cli.md @@ -0,0 +1,24 @@ +# era5cli specifics + +## era5cli code structure + +The typical flow of a CLI call through era5cli is as follows: + +1. The call is parsed by the argparser in `cli.py`, leading to an `argparse.Namespace` + - the specific subparsers and arguments are defined in `args/*.py` +2. If the command is `info` or `config`, those specific routines are called, and the program ends. +3. Otherwise, the "Fetch" is built and executed. +4. The Fetch object is defined in `fetch.py`, and: + - asserts that the user has valid CDS login info + - gathers the request parameters + - splits up the request over variables, years (and optionally months). + - sends the requests to a thread Pool +5. The requests are made to the CDS using the (Python) CDS API. + +## Updating the variable reference +Occasionally changes are made to the ERA5 output. + +An overview of the different ERA5 variables is available [here](https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#heading-Parameterlistings). +No changelog for name changes or additions/removals exists. + +The reference file (`inputref.py`) contains the names of all available variables and pressure levels. diff --git a/docs/gen_reference_pages.py b/docs/gen_reference_pages.py index a01343e..727f7dd 100644 --- a/docs/gen_reference_pages.py +++ b/docs/gen_reference_pages.py @@ -76,7 +76,7 @@ def add_padding(multiline_string: List[str]): ) as process: stdout, stderr = process.communicate() helpstr = stdout.decode("utf-8") - helpstr = helpstr[helpstr.index("optional arguments") :] + helpstr = helpstr[helpstr.index("option") :] # Split out \r\n (for Windows only?) split_str = helpstr.splitlines() diff --git a/docs/general_development.md b/docs/general_development.md new file mode 100644 index 0000000..2edcfd7 --- /dev/null +++ b/docs/general_development.md @@ -0,0 +1,151 @@ +# General development + +## Installation + +For a full development enviroment run the commands below. + +!!! note + This is optional: if you already have [`hatch`](https://hatch.pypa.io/) in your main environment, this setup is not needed, as you can use the `hatch` environments to run all commands. + +=== "Unix" + ```sh + # Create a virtual environment, e.g. with + python3 -m venv env_name + + # activate virtual environment + source env_name/bin/activate + + # make sure to have a recent version of pip and hatch + python3 -m pip install --upgrade pip hatch + + # (from the project root directory:) + # install era5cli as an editable package, along with the dev dependencies + python3 -m pip install --no-cache-dir --editable .[dev] + ``` + +=== "Windows" + ```sh + # Create a virtual environment, e.g. with + python -m venv env_name + + # activate virtual environment + env_name/Scripts/Activate.ps1 + + # make sure to have a recent version of pip and hatch + python -m pip install --upgrade pip hatch + + # (from the project root directory:) + # install era5cli as an editable package, along with the dev dependencies + pip install --no-cache-dir --editable .[dev] + ``` + +## Testing commands + +As the project is set up with `hatch`, you can test changes made to the code by running, for example: + +```sh +hatch run era5cli info 2Dvars +``` + +This will run the *local* era5cli code in a virtual environment. + +## Running unit tests + +era5cli uses `pytest` for unit testing. Running tests has been configured using `hatch`, and can be started by running: + +```sh +hatch run test +``` + +In addition to just running the tests to see if they pass, they can be used for coverage statistics, i.e. to determine how much of the package’s code is actually executed during tests. Inside the package directory, run: + +```sh +hatch run coverage +``` + +This runs tests and prints the results to the command line, as well as storing the result in a `coverage.xml` file (for analysis by, e.g. CodeCov or SonarCloud). + +## Running formatters and linters +For linting and code style we use `flake8`, `black` and `isort`. All tools can simply be run by doing: + +```sh +hatch run lint +``` + +To easily comply with `black` and `isort`, you can also run: + +```sh +hatch run format +``` + +This will apply the `black` and `isort` formatting, and then check the code style. + +## Generating the documentation + +To view the documentation locally, simply run the following command: + +```sh +hatch run docs:serve +``` + +The docs can also be built using `hatch run docs:build`. + +## Versioning + +Bumping the version across all files is done with bumpversion, e.g. + +```sh +bumpversion major +bumpversion minor +bumpversion patch +``` + +## Making a release + +This section describes how to make a release in 3 parts: preparation, release and validation. + +### Preparation +1. Update the `docs/CHANGELOG.md` file. +2. Verify that the information in CITATION.cff is correct, and that the `era5cli/__version__.py` is up to date as well. +3. Make sure the version has been updated. +4. Run the unit tests with `hatch run test`. + +### Making the GitHub release +Make a release and tag on GitHub.com. +Open [releases](https://github.com/eWaterCycle/era5cli/releases) and draft a new release. +Copy the changelog for this version into the description. +Tag the release according to [semantic versioning guidelines](https://semver.org/), preceded with a `v` (e.g.: v1.0.0). +The release title is the tag and the release date together (e.g.: v1.0.0 (2019-07-25)). + +??? note "Release candidates" + When releasing a release candidate on Github, tick the pre-release box, and amend the version tag with `-rc` and the candidate number. + Ensure the release candidate version is accurate in `CITATION.cff` and `era5cli/__version__.py`. + If the version number in these files is not updated, Zenodo release workflows will fail. + + Publishing a new release in github triggers the github Action workflow that builds and publishes the package to test.PyPI or PyPI. + Versions with "rc" (release candidate) in their version tag will only be published to test.PyPI. + Other version tags will trigger a PyPI release. + Inspect `.github/workflows/publish-to-pypi.yml` for more information. + + Confirm a release candidate on test.PyPI with: + ``` + pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ era5cli + ``` + +The release will: + + - trigger Zenodo into making a snapshot of your repository and sticking a DOI on it. + - start a GitHub action that builds and uploads the new version to [PyPI](https://pypi.org/project/era5cli/). + - Which should trigger [conda-forge](https://github.com/conda-forge/era5cli-feedstock) to update the package as well. + +### Validation +After making the release, you should check that: + +1. The [Zenodo page](https://doi.org/10.5281/zenodo.3252665) is updated +2. The publishing action ran successfully, and that `pip install era5cli` installs the new version. +3. The [conda-forge package](https://anaconda.org/conda-forge/era5cli) is updated, and can be installed using conda. +4. Wait a few hours, then confirm the addition of a new release on the [RSD](https://www.research-software.nl/software/era5cli). + +??? note "Adding contributors to the Research Software Directory (RSD)" + If any contributors have been added, or the description of the software has changed, this can be edited (by eScience Center employees) via the [RSD admin interface](https://www.research-software.nl/admin/). + More information about this process (e.g. how to add a new contributor or new affiliation) can be found in the [RSD documentation](https://github.com/research-software-directory/research-software-directory/blob/master/docs/entering-data.md) or in [this blogpost](https://blog.esciencecenter.nl/the-research-software-directory-and-how-it-promotes-software-citation-4bd2137a6b8). diff --git a/mkdocs.yml b/mkdocs.yml index 9a0e92b..5f78c4d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -17,7 +17,8 @@ nav: - "Contributing": - contribute.md - "Developer Documentation": - - dev.md + - general_development.md + - development_era5cli.md - "Changelog": - CHANGELOG.md