Drop python 3.8, use ruff

[mdbenito]

Description

This PR closes #630, closes #548, addresses some of #617, closes #634

Ruff is not without quirks and this PR has made me think hard again about the time-tested principle "if it works, don't fix it" :/

Changes

• Pins the base runner image to Ubuntu 22.04
• Drops 3.8 and sets the base env for tox to 3.9 (this is a PITA for some debian-based distros that don't include distutils for python 3.9 so one must use conda for the environment)
• Drops pylint and black in favour of ruff, updating pre-commit config too
• Fixes a bunch of warnings and deactivates several in ruff's config for convenience (but we should pay attention to line lenghts)
• Updates some package versions, like joblib which released generator_unordered behaviour in 1.4.0
• Pins zarr to be < 3 to avoid some breaking changes
• Makes necessary changes to run with numpy 2.0. However, dask seems not to work with this because it internally references e.g. np.float_ which raises AttributeErrors under numpy 2.0

Checklist

• Wrote Unit tests (if necessary)
• Updated Documentation (if necessary)
• Updated Changelog
• If notebooks were added/changed, added boilerplate cells are tagged with "tags": ["hide"] or "tags": ["hide-input"]