Move to new CDS, prep for 2.0.0b1 release (#166)

* Make backend+CLI compatible with new CDS * Make tests comply with beta-cds * Pin cdsapi and new cads-api-client versions * Add note & warning to readme and docs * Bump version to 2.0 beta 1 * Bump codecov action to new version * Update codecov config * Make codecov action use oidc authentication * Change back codecov config, make it use token * Use new api for authentication of user-entered keys * Remove defect pre-commit hook * Remove deprecated "orography" variable, make splitting hourly req. by months the default * Remove the long deprecated "prelimbe" argument * Remove the last references to orography * Test detection of old configs
eWaterCycle · Aug 30, 2024 · 52a6a2d · 52a6a2d
1 parent f211c70
commit 52a6a2d
Show file tree

Hide file tree

Showing 25 changed files with 200 additions and 415 deletions.
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.4.2
+current_version = 2.0.0b1
 
 [comment]
 comment = The contents of this file cannot be merged with that of pyproject.toml until https://github.com/c4urself/bump2version/issues/42 is resolved

diff --git a/.github/.codecov.yml b/.github/.codecov.yml
@@ -1,19 +1,15 @@
-#
-# This codecov.yml is the default configuration for
-# all repositories on Codecov. You may adjust the settings
-# below in your own codecov.yml in your repository.
-#
+comment:
+  behavior: default
+  layout: header, diff
 coverage:
   precision: 2
+  range:
+  - 70.0
+  - 100.0
   round: down
-  range: 70...100
   status:
-    # Learn more at https://codecov.io/docs#yaml_default_commit_status
-    project: true
-    patch: true
     changes: false
+    patch: true
+    project: true
 ignore:
-  - "tests"
-comment:
-  layout: header, diff
-  behavior: default  # update if exists else create new
+- ^tests.*
diff --git a/.github/workflows/test_codecov.yml b/.github/workflows/test_codecov.yml
@@ -22,10 +22,11 @@ jobs:
     - name: Generate test coverage
       run: hatch run coverage
     - name: Upload coverage to Codecov
-      uses: codecov/codecov-action@v3
+      uses: codecov/codecov-action@v4
       with:
         directory: ./coverage/reports/
         files: ./cov.xml
         env_vars: python-version
         fail_ci_if_error: true
         verbose: true
+        token: ${{ secrets.CODECOV_TOKEN }} # required
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
diff --git a/CITATION.cff b/CITATION.cff
@@ -97,4 +97,4 @@ license: Apache-2.0
 message: "If you use this software, please cite it using these metadata."
 repository-code: "https://github.com/ewatercycle/era5cli"
 title: era5cli
-version: "1.4.2"
+version: "2.0.0b1"
diff --git a/README.md b/README.md
@@ -1,5 +1,21 @@
 <img align="right" width="150" alt="Logo" src="docs/assets/era5cli_logo_colors_border.png">
 
+> [!IMPORTANT]
+> The old Climate Data Store (CDS) will be shut down on 3 September 2024.
+> All era5cli versions up to v1.4.2 will no longer work.
+>
+> For more information see: https://forum.ecmwf.int/t/the-new-climate-data-store-beta-cds-beta-is-now-live/3315
+>
+> 
+> To continue using era5cli, you will need to re-register at ECMWF and get a new API key,
+> and transition to the era5cli v2 beta. This can be installed with:
+> `pip install era5cli==2.0.0b1` 
+
+> [!WARNING]
+> netCDF files from the new Climate Data Store Beta are not formatted the same as the
+> old CDS. Some variables might be missing.
+> See the open issue [here](https://github.com/eWaterCycle/era5cli/issues/165), as well as the [ECMWF discussion forum](https://forum.ecmwf.int/).
+
 
 [![github license badge](https://img.shields.io/github/license/eWaterCycle/era5cli)](https://github.com/eWaterCycle/era5cli)
 [![rsd badge](https://img.shields.io/badge/RSD-era5cli-blue)](https://research-software-directory.org/software/era5cli)

diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
@@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+## 2.0.0b1 - 2024-08-30
+
+**Changed:**
+
+ - The `splitmonths` argument now defaults to `True` for hourly requests. To not split requests by year, add `--splitmonths False`.
+
+**Fixed:**
+
+ - Added support for the beta-CDS
+ - For authentication, the new `cads-api-client` is used, instead of a dummy request. This should avoid the dummy requests appearing in the user's queue.
+
+**Removed:**
+
+ - the deprecated `orography` variable. Use `geopotential` instead.
+ - the deprecated `--prelimbe` argument. This one has not been required anymore, as the back-extension is part of the normal dataset now.
+
+**Dev changes:**
+
+ - The pre-commit hook has been removed. Pre-commit does not play well with hatch: it would need to be installed system-wide. No hatch-specific hooks are available.
+
+
 ## 1.4.2 - 2023-12-12
 
 **Fixed:**

diff --git a/docs/formulating_requests.md b/docs/formulating_requests.md
@@ -20,13 +20,12 @@ era5cli hourly \
     --variables 2m_temperature 2m_dewpoint_temperature \
     --startyear 2000 \
     --endyear 2020 \
-    --splitmonths True \
     --area 53.6 3.3 50.7 7.5
 ```
 
 This request asks for *hourly* data of the ERA5-*Land* dataset, more specifically the *2m_temperature* and *2m_dewpoint_temperature* variables.
 
-Additionally, data from the year *2000* up to (and including) *2020* is requested, with the final files being *split up by months*.
+Additionally, data from the year *2000* up to (and including) *2020* is requested.
 Lastly, an *area* is extracted from the dataset (in this case only the Netherlands).
 
 ### Using the info command

diff --git a/docs/general_development.md b/docs/general_development.md
@@ -80,17 +80,6 @@ hatch run format
 
 This will apply the `black` and `isort` formatting, and then check the code style.
 
-??? tip "Using pre-commit"
-    For pre-commit users, a pre-commit configuration has been added. This hook will execute the `hatch run format` command. 
-
-    After installing pre-commit in your python environment (`pip install pre-commit`), you can do
-    ```
-    pre-commit install
-    ```
-    to set up the git hook scripts.
-
-    For more information, see the [pre-commit website](https://pre-commit.com/).
-
 ## Generating the documentation
 
 To view the documentation locally, simply run the following command:

diff --git a/docs/getting_started.md b/docs/getting_started.md
@@ -43,10 +43,10 @@ After activating your account, **login** on the CDS website, go to your **profil
 To configure era5cli to use these keys, open up the environment you installed era5cli in, and do:
 
 ```sh
-era5cli config --uid ID_NUMBER --key "KEY"
+era5cli config --key "KEY"
 ```
 
-*Where ID_NUMBER is your user ID (e.g. 123456) and "KEY" is your API key, inside double quotes (e.g. "4s215sgs-2dfa-6h34-62h2-1615ad163414").*
+*Where "KEY" is your API key, inside double quotes (e.g. "4s215sgs-2dfa-6h34-62h2-1615ad163414").*
 
 After running this command your ID and key are validated and stored inside your home folder, under `.config/era5cli/cds_key.txt`.
 

diff --git a/docs/index.md b/docs/index.md
@@ -2,6 +2,22 @@
 
 A command line interface to download ERA5 data from the [Copernicus Climate Data Store](https://climate.copernicus.eu/).
 
+???+ note
+    The old Climate Data Store (CDS) will be shut down on 3 September 2024.
+    All era5cli versions up to v1.4.2 will no longer work.
+
+    For more information see: https://forum.ecmwf.int/t/the-new-climate-data-store-beta-cds-beta-is-now-live/3315
+
+    To continue using era5cli, you will need to re-register at ECMWF and get a new API key,
+    and transition to the era5cli v2 beta. This can be installed with:
+    `pip install era5cli==2.0.0b1` 
+
+???+ warning
+    netCDF files from the new Climate Data Store Beta are not formatted the same as the
+    old CDS. Some variables might be missing.
+
+    See the open issue [here](https://github.com/eWaterCycle/era5cli/issues/165), as well as the [ECMWF discussion forum](https://forum.ecmwf.int/).
+
 With era5cli you can:
 
  - Download meteorological data in GRIB/NetCDF, including ERA5 data from the preliminary back extension, and ERA5-Land data.

diff --git a/era5cli/__version__.py b/era5cli/__version__.py
@@ -26,4 +26,4 @@
     "Bart Schilperoort",
 )
 __email__ = "[email protected]"
-__version__ = "1.4.2"
+__version__ = "2.0.0b1"
diff --git a/era5cli/args/common.py b/era5cli/args/common.py
@@ -24,7 +24,6 @@ def add_common_args(argument_parser: ArgumentParser) -> None:
         --threads,
         --ensemble,
         --dryrun,
-        --prelimbe,
         --land,
         --area,
         --overwrite
@@ -188,23 +187,6 @@ def add_common_args(argument_parser: ArgumentParser) -> None:
         ),
     )
 
-    argument_parser.add_argument(
-        "--prelimbe",
-        action="store_true",
-        default=False,
-        help=textwrap.dedent(
-            """
-            Whether to download the preliminary back extension
-            (1950-1978). Note that when `--prelimbe` is used,
-            `--startyear` and `--endyear` should be set
-            between 1950 and 1978. Please, be aware that
-            ERA5 data is available from 1940.
-            `--prelimbe` is incompatible with `--land`
-
-            """
-        ),
-    )
-
     argument_parser.add_argument(
         "--land",
         action="store_true",
@@ -215,7 +197,7 @@ def add_common_args(argument_parser: ArgumentParser) -> None:
             dataset. Note that the ERA5-Land dataset starts in
             1950.
             `--land` is incompatible with the use of
-            `--prelimbe` and `--ensemble`
+            `--ensemble`
 
             """
         ),
@@ -287,9 +269,7 @@ def construct_year_list(args):
 
     # check whether correct years have been entered
     for year in (args.startyear, endyear):
-        if args.prelimbe:
-            assert 1950 <= year <= 1978, "year should be between 1950 and 1978"
-        elif args.land:
+        if args.land:
             assert (
                 1950 <= year <= datetime.now().year
             ), "for ERA5-Land, year should be between 1950 and present"

diff --git a/era5cli/args/config.py b/era5cli/args/config.py
@@ -10,7 +10,6 @@ def add_config_args(subparsers: argparse._SubParsersAction) -> None:
 
     Adds the 'config' parser with the following arguments:
         --show
-        --uid
         --key
         --url
 
@@ -27,7 +26,7 @@ def add_config_args(subparsers: argparse._SubParsersAction) -> None:
             This will create a config file in your home directory, in folder named
             ".config". The CDS URL, your UID and the CDS keys will be stored here.
 
-            To find your key and UID, go to https://cds.climate.copernicus.eu/ and
+            To find your key, go to https://beta-cds.climate.copernicus.eu/ and
             login with your email and password. Then go to your user profile (top
             right).
 
@@ -54,16 +53,6 @@ def add_config_args(subparsers: argparse._SubParsersAction) -> None:
         ),
     )
 
-    config.add_argument(
-        "--uid",
-        type=str,
-        help=textwrap.dedent(
-            """
-            Your CDS User ID, e.g.: 123456
-            """
-        ),
-    )
-
     config.add_argument(
         "--key",
         type=str,
@@ -87,6 +76,18 @@ def add_config_args(subparsers: argparse._SubParsersAction) -> None:
         ),
     )
 
+    config.add_argument(
+        "--uid",
+        type=str,
+        required=False,
+        default="",
+        help=textwrap.dedent(
+            """
+            DO NOT USE: deprecated due to changes in the CDS API"
+            """
+        ),
+    )
+
 
 class InputError(Exception):
     "Raised when a user inputs an invalid combination of arguments."
@@ -101,27 +102,22 @@ def run_config(args):
 
     Args:
         args: Arguments collected by argparse
-
-    Returns:
-        True
     """
-    if args.show and any((args.uid, args.key)):
+    if len(args.uid) > 0:
+        msg = (
+            "The `uid` argument is deprecated.\n"
+            "The new CDS API does not use UIDs anymore."
+        )
+        raise InputError(msg)
+
+    if args.show and args.key is not None:
         raise InputError("Either call `show` or set the key. Not both.")
-    if not args.show and (args.uid is None or args.key is None):
-        raise InputError("Both the UID and the key are required inputs.")
+    if not args.show and args.key is None:
+        raise InputError("Your CDS API key is a required input.")
     if args.show:
-        url, fullkey = key_management.load_era5cli_config()
-        uid, key = fullkey.split(":")
+        url, key = key_management.load_era5cli_config()
         print(
-            "Contents of .config/era5cli.txt:\n"
-            f"    uid: {uid}\n"
-            f"    key: {key}\n"
-            f"    url: {url}\n"
+            "Contents of .config/era5cli.txt:\n" f"    key: {key}\n" f"    url: {url}\n"
         )
-        return True
-
-    return key_management.set_config(
-        url=args.url,
-        uid=args.uid,
-        key=args.key,
-    )
+    else:
+        key_management.set_config(args.url, args.key)
diff --git a/era5cli/args/periods.py b/era5cli/args/periods.py
@@ -1,5 +1,4 @@
 import argparse
-import logging
 import textwrap
 from era5cli import utils
 
@@ -73,15 +72,13 @@ def add_period_args(subparsers, common):
     splitmonths.add_argument(
         "--splitmonths",
         type=lambda x: bool(utils.strtobool(x)),  # type=bool doesn't work.
-        default=None,  # To be set to True in the future
+        default=True,
         help=textwrap.dedent(
             """
-            When downloading hourly data, use:
-            `--splitmonths True` to split requests and files
-            by month, and add the month to the filename.
-
-            Defaults to `False`, but will default to `True` in
-            a future release.
+            By default when downloading hourly data requests are split
+            by months.
+            To suppress this behavior, use: `--splitmonths False` to have yearly
+            files.
             """
         ),
     )
@@ -175,17 +172,7 @@ def set_period_args(args):
             hours = args.synoptic
     elif args.command == "hourly":
         synoptic = None
-        if args.splitmonths is None:
-            splitmonths = False
-            logging.warning(
-                "\n  The argument --splitmonths was not used. However, in a future "
-                "\n  version this flag will default to `True`. To avoid this, either"
-                "\n  use `--splitmonths True` and update your workflow accordingly,"
-                "\n  or set --splitmonths to False."
-            )
-        else:
-            splitmonths: bool = args.splitmonths
-
+        splitmonths: bool = args.splitmonths
         statistics: bool = args.statistics
         if statistics:
             assert args.ensemble, (

diff --git a/era5cli/cli.py b/era5cli/cli.py
@@ -67,7 +67,6 @@ def _execute(input_args: argparse.Namespace) -> True:
         threads=input_args.threads,
         splitmonths=splitmonths,
         merge=input_args.merge,
-        prelimbe=input_args.prelimbe,
         land=input_args.land,
         overwrite=input_args.overwrite,
         dashed_vars=input_args.dashed_varname,