Modernize PUDL packaging

[zaneselvans]

Installation using setup.py has been deprecated in favor of PEP 517 / PEP 518 / PEP 621 / PEP 660 based systems, and we are getting warnings about it, so we need to overhaul the packaging setup. Instead we should probably configure the packaging entirely using pyproject.toml

In addition, the pudl.metadata.templates directory is a "package" with no code (just jinja templates) and it needs to be specifically called out for inclusion in the package -- otherwise it'll be left out in future packaging setups.

We may also want to look at using another build system like poetry, and explicitly lock dependencies to particular versions. A blog post from last year about the changes.

[jdangerx]

Looks like we can use conda-lock to pin our transitive dependencies, which will help with our dev env reproducibility/upgradability. In theory we won't have to recreate our environments when switching branches :)

There's a blog post about it here, too: https://pythonspeed.com/articles/conda-dependency-management/

[zaneselvans]

How does using a lockfile keep us from needing to recreate environments when we switch branches? If the branches have different dependencies wouldn't that be reflected in the different lockfiles, and wouldn't the python environment need to be rebuilt to match?

While we use conda for python environment isolation and installation of some finnicky binary / system dependencies, almost all of the software installation is actually done by pip since the overwhelming majority of the packages in the conda environment are installed by the pip section the file, that installs the actual catalystcoop.pudl package (from the repo, with extras, using --editable). Would those pip installed packages end up being captured by the conda environment? I guess they show up as installed when you do conda list but they come from the pypi "channel"

[jdangerx]

The python environment would be updated to match the lockfile, yeah. Though it'd be an incremental rebuild, not a full one.

In theory different branches will only have different library versions if we've intentionally changed direct dependency versions. Do we anticipate adding/removing/updating direct dependencies frequently? I just took a look at git log setup.py and it seems like we're bumping versions a couple times a week over the last few months, which seems kind of annoying but maybe not terrible.

I'm not sure how conservative the solver is re: trying to change as few library versions as possible when re-solving, but I'd assume that it is still faster than "re-do everything every time."

conda-lock does appear to support pip dependencies within pyproject.toml, see this section of the docs - though I think we'll probably have to try moving everything to pyproject.toml first (looks like conda-lock also allows for the non-Python dependencies to be defined in pyproject.toml, but I wonder about how to propagate that stuff to downstream users...)

[zaneselvans]

This should have been closed by #2479

Opening a new issue to deal with lockfiles in particular: #2896