Release 0.14.4 backports (#1177)

* Add `cmake/` to MANIFEST.in

Forgot to add in #1128, which is then missing in Python sdist files
for pip from-source builds.

* Add failing test

* Read gridSpacing also from long doubles

* Python: 3.10

Add support for Python 3.10.

* CI: Python 3.10 on Alpine Linux

musllinux uses the musl libc on an Alpine stack, especially for
docker-ish containers.

Let's add coverage while also covering CPython 3.10

* Fix: Python Iteration Repr Typo

Fix a typo in the `__repr__` of `Iteration` in Python.

* Remove invalid records from our data structure entirely

Until now, the half-parsed records were not deleted.
Our sanity checks in `Series::~Series()` caught up on that and threw
errors.

* Fix doxygen?

No idea why this PR triggers that error

* Don't apply compression operators multipe times

Happened so far in variable-based iteration encoding

* ICC 19.1.2: CXX17 Work-Arounds (Variant)

The `Attribute` constructors with implicit variant conversion
sometimes do not work in ICC 19.1.2.

```
openPMD-api/src/RecordComponent.cpp(226): error: no instance of constructor "openPMD::Attribute::Attribute" matches the argument list
            argument types are: (openPMD::Extent)
                  Attribute a(getExtent());
                              ^
openPMD-api/include/openPMD/backend/Attribute.hpp(50): note: this candidate was rejected because arguments do not match
  class Attribute :
        ^
openPMD-api/include/openPMD/backend/Attribute.hpp(50): note: this candidate was rejected because arguments do not match
  class Attribute :
        ^
openPMD-api/include/openPMD/backend/Attribute.hpp(79): note: this candidate was rejected because arguments do not match
      Attribute(resource r) : Variant(std::move(r))
      ^
```

Same work-around as for NVCC in  #1103

* CMake 3.22+: Policy CMP0127

Fix a warning with CMake 3.22+.

We use simple syntax in cmake_dependent_option, so we are compatible
with the extended syntax in CMake 3.22+:
  https://cmake.org/cmake/help/v3.22/policy/CMP0127.html

* Docs: Minor Warnings

Fix minor warnings in the docs:
- double reference (make 2nd ref annonymous)
- misaligned reference

* Docs: Fix HDF5 JSON File

Fix misnamed filename and misnamed prefix in text.

* Docs: Update Citation & Add BibTeX

Add `.bib` blocks for easier citation of standard and API.

* Docs: Fix CLI Highlighting

Fix a highlighting issue for `openPMD-ls` in the docs.
Explicitly use `python3` for module calls.

* CMake: Warning Flags First in CXXFLAGS

If we append them, then we overwrite flags like `-Wno-...` from
environment variables.

* ADIOS2 fixes: Incompatibilites w/ `master`

Referes to post-`2.7.1` `master` branch of ADIOS2.

Catch new ADIOS2 error type: string

getCompressionOperator

Dirtily catch ADIOS2 exception in automatically_deactivate_span

Cleanly destruct a Series that errored during flushing

Problem was that IO tasks that do not return cleanly but throw an
exception must be removed from the IO queue, otherwise the task will be
tried again during destruction. Usually, this means that anything will
happen.

SerialIOTest: Remove Comments (resolved)

Print warning when removing task from IO queue

std::string and fix warning

* Automatically deactivate span-based API when operators are present

* Add .pre-commit-config.yaml

* Release: 0.14.4

Co-authored-by: Franz Pöschel <[email protected]>

.github/workflows/linux.yml

@@ -229,6 +229,32 @@ jobs:
         cmake --build build --parallel 2
         ctest --test-dir build --output-on-failure
 
+  musllinux_py10:
+    runs-on: ubuntu-20.04
+    if: github.event.pull_request.draft == false
+    container:
+      image: quay.io/pypa/musllinux_1_1_x86_64
+    steps:
+    - uses: actions/checkout@v2
+    - name: Install
+      run: |
+        apk update
+        apk add hdf5-dev
+        python3.10 -m pip install numpy
+    - name: Build
+      env: {CXXFLAGS: -Werror -Wno-deprecated-declarations}
+      run: |
+        share/openPMD/download_samples.sh build
+        chmod u-w build/samples/git-sample/*.h5
+        cmake -S . -B build \
+          -DopenPMD_USE_PYTHON=ON \
+          -DopenPMD_USE_MPI=OFF   \
+          -DopenPMD_USE_HDF5=ON   \
+          -DopenPMD_USE_INVASIVE_TESTS=ON \
+          -DPython_EXECUTABLE=$(which python3.10)
+        cmake --build build --parallel 2
+        ctest --test-dir build --output-on-failure
+
   conda_ompi_all:
     runs-on: ubuntu-20.04
     steps:

.pre-commit-config.yaml

@@ -0,0 +1,27 @@
+# To use:
+#
+#     pre-commit run -a
+#
+# Or:
+#
+#     pre-commit install  # (runs every time you commit in git)
+#
+# To update this file:
+#
+#     pre-commit autoupdate
+#
+# See https://pre-commit.com for more information
+
+# Do not check/format anything from third parties
+exclude: '^share/openPMD/thirdParty'
+
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.1.0
+    hooks:
+    - id: mixed-line-ending
+    - id: check-json
+    - id: check-toml
+    - id: check-yaml
+    - id: check-added-large-files

CHANGELOG.rst

@@ -3,7 +3,57 @@
 Changelog
 =========
 
-0.14.2
+0.14.4
+------
+**Date:** 2022-01-21
+
+Increased Compatibility & Python Install Bug
+
+This release fixes various read/parsing bugs and increases compatibility with upcoming versions of ADIOS and old releases of Intel ``icpc``.
+An installation issue for pip-based installs from source in the last release was fixed and Python 3.10 support added.
+Various documentation and installation warnings have been fixed.
+
+Changes to "0.14.3"
+^^^^^^^^^^^^^^^^^^^
+
+Bug Fixes
+"""""""""
+
+- ADIOS2:
+
+  - automatically deactivate ``span`` based ``Put`` API when operators are present #1155
+  - solve incompatibilities w/ post-``2.7.1`` ``master``-branch #1166
+- ICC 19.1.2: C++17 work-arounds (``variant``) #1157
+- Don't apply compression operators multiple times in variable-based iteration encoding #1152
+- Reading/parsing:
+
+  - remove invalid records from data structures entirely #1150
+  - fix grid spacing with type long double #1137
+- Python:
+
+  - fix ``Iteration`` ``__repr__`` typo #1149
+  - add ``cmake/`` to ``MANIFEST.in`` #1140
+
+Other
+"""""
+
+- add simple ``.pre-commit-config.yaml``
+- Python:
+
+  - support Python 3.10 #1139
+- CMake:
+
+  - warning flags first in ``CXXFLAGS`` #1172
+  - add policy CMP0127 (v3.22+) #1165
+- Docs:
+
+  - fix CLI highlighting #1171
+  - update citation & add BibTeX #1168
+  - fix HDF5 JSON File #1169
+  - minor warnings #1170
+
+
+0.14.3
 ------
 **Date:** 2021-11-03
 

CITATION.cff

@@ -25,7 +25,7 @@ contact:
   orcid: https://orcid.org/0000-0003-1943-7141
   email: [email protected]
 title: "openPMD-api: C++ & Python API for Scientific I/O with openPMD"
-version: 0.14.3
+version: 0.14.4
 repository-code: https://github.com/openPMD/openPMD-api
 doi: 10.14278/rodare.27
 license: LGPL-3.0-or-later

CMakeLists.txt

@@ -2,7 +2,7 @@
 #
 cmake_minimum_required(VERSION 3.15.0)
 
-project(openPMD VERSION 0.14.3) # LANGUAGES CXX
+project(openPMD VERSION 0.14.4) # LANGUAGES CXX
 
 # the openPMD "markup"/"schema" standard version
 set(openPMD_STANDARD_VERSION 1.1.0)
@@ -18,6 +18,13 @@ if(POLICY CMP0074)
     cmake_policy(SET CMP0074 NEW)
 endif()
 
+# We use simple syntax in cmake_dependent_option, so we are compatible with the
+# extended syntax in CMake 3.22+
+# https://cmake.org/cmake/help/v3.22/policy/CMP0127.html
+if(POLICY CMP0127)
+    cmake_policy(SET CMP0127 NEW)
+endif()
+
 
 # Project structure ###########################################################
 #
@@ -883,23 +890,22 @@ if(CMAKE_SOURCE_DIR STREQUAL PROJECT_SOURCE_DIR)
         #                     /usr/lib/llvm-6.0/lib/clang/6.0.0/lib/linux/libclang_rt.ubsan_minimal-x86_64.so
         # at runtime when used with symbol-hidden code (e.g. pybind11 module)
 
-        #set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Weverything")
-        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code")
+        set(CMAKE_CXX_FLAGS "-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code ${CMAKE_CXX_FLAGS}")
     elseif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Intel")
-        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w3 -wd193,383,1572")
+        set(CMAKE_CXX_FLAGS "-w3 -wd193,383,1572 ${CMAKE_CXX_FLAGS}")
     elseif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
-        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wunreachable-code")
+        set(CMAKE_CXX_FLAGS "-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wunreachable-code ${CMAKE_CXX_FLAGS}")
     elseif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC")
         # Warning C4503: "decorated name length exceeded, name was truncated"
         # Symbols longer than 4096 chars are truncated (and hashed instead)
-        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -wd4503")
+        set(CMAKE_CXX_FLAGS "-wd4503 ${CMAKE_CXX_FLAGS}")
         # Warning C4244: "conversion from 'X' to 'Y', possible loss of data"
-        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -wd4244")
+        set(CMAKE_CXX_FLAGS "-wd4244 ${CMAKE_CXX_FLAGS}")
         # Yes, you should build against the same C++ runtime and with same
         # configuration (Debug/Release). MSVC does inconvenient choices for their
         # developers, so be it. (Our Windows-users use conda-forge builds, which
         # are consistent.)
-        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -wd4251")
+        set(CMAKE_CXX_FLAGS "-wd4251 ${CMAKE_CXX_FLAGS}")
     endif()
 endif()
 

Dockerfile

@@ -5,8 +5,8 @@ FROM       quay.io/pypa/manylinux2010_x86_64 as build-env
 # FROM       quay.io/pypa/manylinux1_x86_64 as build-env
 ENV        DEBIAN_FRONTEND noninteractive
 
-# Python 3.6-3.9 via "36m 37m 38 39"
-ARG        PY_VERSIONS="36m 37m 38 39"
+# Python 3.6-3.10 via "36m 37m 38 39 310"
+ARG        PY_VERSIONS="36m 37m 38 39 310"
 
 # static libs need relocatable symbols for linking to shared python lib
 ENV        CFLAGS="-fPIC ${CFLAGS}"
@@ -188,6 +188,21 @@ RUN        python3.9 -c "import openpmd_api as io; print(io.__version__); print(
 RUN        python3.9 -m openpmd_api.ls --help
 RUN        openpmd-ls --help
 
+# test in fresh env: Debian:Bullseye + Python 3.10
+FROM       debian:bullseye
+ENV        DEBIAN_FRONTEND noninteractive
+COPY --from=build-env /wheelhouse/openPMD_api-*-cp310-cp310-manylinux2010_x86_64.whl .
+RUN        apt-get update \
+           && apt-get install -y --no-install-recommends python3.10 python3-distutils ca-certificates curl \
+           && rm -rf /var/lib/apt/lists/*
+RUN        python3.10 --version \
+           && curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py \
+           && python3.10 get-pip.py \
+           && python3.10 -m pip install openPMD_api-*-cp310-cp310-manylinux2010_x86_64.whl
+RUN        python3.10 -c "import openpmd_api as io; print(io.__version__); print(io.variants)"
+RUN        python3.10 -m openpmd_api.ls --help
+RUN        openpmd-ls --help
+
 
 # copy binary artifacts (wheels)
 FROM       quay.io/pypa/manylinux2010_x86_64

MANIFEST.in

@@ -2,6 +2,7 @@ include README.md COPYING COPYING.LESSER
 include pyproject.toml
 include requirements.txt
 global-include CMakeLists.txt *.cmake *.in
+recursive-include cmake *
 recursive-include include *
 recursive-include src *
 recursive-include share *

NEWS.rst

@@ -3,6 +3,12 @@
 Upgrade Guide
 =============
 
+0.15.0
+------
+
+Python 3.10 is now supported.
+
+
 0.14.0
 ------
 

README.md

@@ -119,7 +119,7 @@ while those can be built either with or without:
 
 Optional language bindings:
 * Python:
-  * Python 3.6 - 3.9
+  * Python 3.6 - 3.10
   * pybind11 2.6.2+
   * numpy 1.15+
   * mpi4py 2.1+ (optional, for MPI)

docs/source/citation.rst

@@ -18,6 +18,29 @@ The most general reference to openPMD is:
    *"openPMD: A meta data standard for particle and mesh based data,"*
    `DOI:10.5281/zenodo.591699 <https://doi.org/10.5281/zenodo.591699>`_ (2015)
 
+The equivalent BibTeX code is:
+
+.. code-block:: bibtex
+
+   @misc{openPMDstandard,
+     author       = {Huebl, Axel and
+                     Lehe, R{\'e}mi and
+                     Vay, Jean-Luc and
+                     Grote, David P. and
+                     Sbalzarini, Ivo and
+                     Kuschel, Stephan and
+                     Sagan, David and
+                     P{\'e}rez, Fr{\'e}d{\'e}ric and
+                     Koller, Fabian and
+                     Bussmann, Michael},
+     title        = {{openPMD: A meta data standard for particle and mesh based data}},
+     year         = 2015,
+     publisher    = {Zenodo},
+     doi          = {10.5281/zenodo.591699},
+     url          = {https://www.openPMD.org},
+     howpublished = {https://github.com/openPMD}
+   }
+
 Since the openPMD-standard is an actively evolving meta data schema, a specific version of the openPMD standard might be used in your work.
 You can select a version-specific DOI from the `release page <https://github.com/openPMD/openPMD-standard/releases>`_ and add the version number to the cited title, e.g.
 
@@ -34,13 +57,13 @@ openPMD-api is a **software library** that provides a reference implementation o
 It targets both desktop as well as high-performance computing environments.
 
 It is good scientific practice to document all used software, including transient dependencies, with versions in, e.g. a methods section of a publication.
-As a software citation, you almost always want to refer to a *specific version* of openPMD-api in your work, as illustrated for version 0.10.3:
+As a software citation, you almost always want to refer to a *specific version* of openPMD-api in your work, as illustrated for version 0.14.3:
 
 .. tip::
 
-   Fabian Koller, Franz Poeschel, Junmin Gu, and Axel Huebl.
-   *"openPMD-api 0.10.3: C++ & Python API for Scientific I/O with openPMD,"*
-   `DOI:10.14278/rodare.209 <https://doi.org/10.14278/rodare.209>`_ (2019)
+   Axel Huebl, Franz Poeschel, Fabian Koller, and Junmin Gu.
+   *"openPMD-api 0.14.3: C++ & Python API for Scientific I/O with openPMD,"*
+   `DOI:10.14278/rodare.1234 <https://doi.org/10.14278/rodare.1234>`_ (2021)
 
 A list of all releases and DOIs can be found `on the release page <https://github.com/openPMD/openPMD-api/releases>`_.
 
@@ -52,6 +75,23 @@ We also provide a DOI that refers to all releases of openPMD-api:
    *"openPMD-api: C++ & Python API for Scientific I/O with openPMD"*
    `DOI:10.14278/rodare.27 <https://doi.org/10.14278/rodare.27>`_ (2018)
 
+The equivalent BibTeX code is:
+
+.. code-block:: bibtex
+
+   @misc{openPMDapi,
+     author       = {Huebl, Axel and
+                     Poeschel, Franz and
+                     Koller, Fabian and
+                     Gu, Junmin},
+     title        = {{openPMD-api: C++ \& Python API for Scientific I/O with openPMD}},
+     month        = june,
+     year         = 2018,
+     doi          = {10.14278/rodare.27},
+     url          = {https://github.com/openPMD/openPMD-api}
+   }
+
+
 Dependent Software
 ~~~~~~~~~~~~~~~~~~
 

docs/source/conf.py

@@ -83,9 +83,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = u'0.14.3'
+version = u'0.14.4'
 # The full version, including alpha/beta/rc tags.
-release = u'0.14.3'
+release = u'0.14.4'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/source/details/backendconfig.rst

@@ -77,6 +77,11 @@ Explanation of the single keys:
 
   * ``type`` supported ADIOS operator type, e.g. zfp, sz
   * ``parameters`` is an associative map of string parameters for the operator (e.g. compression levels)
+* ``adios2.use_span_based_put``: The openPMD-api exposes the `span-based Put() API <https://adios2.readthedocs.io/en/latest/components/components.html#put-modes-and-memory-contracts>`_ of ADIOS2 via an overload of ``RecordComponent::storeChunk()``.
+  This API is incompatible with compression operators as described above.
+  The openPMD-api will automatically use a fallback implementation for the span-based Put() API if any operator is added to a dataset.
+  This workaround is enabled on a per-dataset level.
+  The workaround can be completely deactivated by specifying ``{"adios2": {"use_span_based_put": true}}`` or it can alternatively be activated indiscriminately for all datasets by specifying ``{"adios2": {"use_span_based_put": false}}``.
 
 Any setting specified under ``adios2.dataset`` is applicable globally as well as on a per-dataset level.
 Any setting under ``adios2.engine`` is applicable globally only.
@@ -94,7 +99,7 @@ A full configuration of the HDF5 backend:
 All keys found under ``hdf5.dataset`` are applicable globally (future: as well as per dataset).
 Explanation of the single keys:
 
-* ``adios2.dataset.chunks``: This key contains options for data chunking via `H5Pset_chunk <https://support.hdfgroup.org/HDF5/doc/RM/H5P/H5Pset_chunk.htm>`__.
+* ``hdf5.dataset.chunks``: This key contains options for data chunking via `H5Pset_chunk <https://support.hdfgroup.org/HDF5/doc/RM/H5P/H5Pset_chunk.htm>`__.
   The default is ``"auto"`` for a heuristic.
   ``"none"`` can be used to disable chunking.
   Chunking generally improves performance and only needs to be disabled in corner-cases, e.g. when heavily relying on independent, parallel I/O that non-collectively declares data records.

docs/source/details/mpi.rst

@@ -43,7 +43,7 @@ Functionality                Behavior           Description
        Note that openPMD represents constant record components with attributes, thus inheriting this for ``::makeConstant``.
 
 .. [4] We usually open iterations delayed on first access. This first access is usually the ``flush()`` call after a ``storeChunk``/``loadChunk`` operation. If the first access is non-collective, an explicit, collective ``Iteration::open()`` can be used to have the files already open.
-Alternatively, iterations might be accessed for the first time by immediate operations such as ``::availableChunks()``.
+       Alternatively, iterations might be accessed for the first time by immediate operations such as ``::availableChunks()``.
 
 .. tip::
 

docs/source/dev/dependencies.rst

@@ -39,7 +39,7 @@ Optional: language bindings
 
 * Python:
 
-  * Python 3.6 - 3.9
+  * Python 3.6 - 3.10
   * pybind11 2.6.2+
   * numpy 1.15+
   * mpi4py 2.1+ (optional, for MPI)

docs/source/index.rst

@@ -42,7 +42,7 @@ openPMD-api version      supported openPMD standard versions
 ======================== ===================================
 ``2.0.0+``               ``2.0.0+``      (not released yet)
 ``1.0.0+``               ``1.0.1-1.1.0`` (not released yet)
-``0.13.1-0.14.3`` (beta) ``1.0.0-1.1.0``
+``0.13.1-0.14.4`` (beta) ``1.0.0-1.1.0``
 ``0.1.0-0.12.0`` (alpha) ``1.0.0-1.1.0``
 ======================== ===================================
 

docs/source/utilities/cli.rst

@@ -19,9 +19,9 @@ The syntax of the command line tool is printed via:
 
 With some ``pip``-based python installations, you might have to run this as a module:
 
-.. code-block:: python3
+.. code-block:: bash
 
-   python -m openpmd_api.ls --help
+   python3 -m openpmd_api.ls --help
 
 ``openpmd-pipe``
 ----------------
@@ -48,4 +48,4 @@ With some ``pip``-based python installations, you might have to run this as a mo
 
 .. code-block:: bash
 
-   python -m openpmd_api.pipe --help
+   python3 -m openpmd_api.pipe --help