Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Consolidate optional dependencies; test nuke; parallelize tests
Browse files Browse the repository at this point in the history
* Move previous dev, test, and datasette optional dependencies into
  the required dependencies to simplify application installation.
* Test make nuke; parallelize --live-dbs tests
* Move prettier into conda-only dependencies
* Update conda-lock.yml and rendered conda environment files.
* Remove action test file hashlog
* Remove merge markers.
* Remove transitive astroid dependency that's now correctly included in solve.
* Use the real immature library version of dagster-postgres (0.21.6)
  rather than the accidentally packaged 1.5.6 version found in conda.
  We'll need to keep an eye out for when dagster-postgres graduates
  to the stable versioning and update it. This is a bit of a mess
  because of some broken automation in the conda packaging for dagster
  which has now been fixed.
* Update "make pudl" to remove the old PUDL DB and reinitialize with
  alembic, rather than writing to the DB that already exists.
* Fixed some groupby.agg() deprecation warnings.
* Fix dagster-postgres version (again).
* Update username in path to settings file
* Avoid bugs in ferc_to_sqlite --clobber; don't use cache_dir for pip install.
* Make FERC extraction output removal more specific.
* Bump numpy and numba minimum versions.
* Bump black version in pre-commit
* Bump ruff pre-commit hook version
* Update conda-lock.yml and rendered conda environment files.
zaneselvans committed Nov 13, 2023
1 parent 0196898 commit 7defd4d
Showing 16 changed files with 2,658 additions and 1,841 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/build-deploy-pudl.yml
Original file line number Diff line number Diff line change
@@ -117,8 +117,8 @@ jobs:
--container-env DAGSTER_PG_PASSWORD="$DAGSTER_PG_PASSWORD" \
--container-env DAGSTER_PG_HOST="104.154.182.24" \
--container-env DAGSTER_PG_DB="dagster-storage" \
--container-env PUDL_SETTINGS_YML="/home/catalyst/src/pudl/package_data/settings/etl_full.yml" \
--container-env FLY_ACCESS_TOKEN=${{ secrets.FLY_ACCESS_TOKEN }} \
--container-env PUDL_SETTINGS_YML="/home/mambauser/src/pudl/package_data/settings/etl_full.yml" \
# Start the VM
- name: Start the deploy-pudl-vm
3 changes: 0 additions & 3 deletions .github/workflows/tox-pytest.yml
Original file line number Diff line number Diff line change
@@ -34,7 +34,6 @@ jobs:
environment-file: environments/conda-lock.yml
environment-name: pudl-dev
cache-environment: true
create-args: --category main dev docs test datasette

- name: Log environment details
run: |
@@ -74,7 +73,6 @@ jobs:
environment-file: environments/conda-lock.yml
environment-name: pudl-dev
cache-environment: true
create-args: --category main dev docs test datasette

- name: Log environment details
run: |
@@ -125,7 +123,6 @@ jobs:
environment-file: environments/conda-lock.yml
environment-name: pudl-dev
cache-environment: true
create-args: --category main dev docs test datasette

- name: Log environment details
run: |
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -26,14 +26,14 @@ repos:
# Formatters: hooks that re-write Python & documentation files
####################################################################################
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.1.4
rev: v0.1.5
hooks:
- id: ruff
args: [--fix, --exit-non-zero-on-fix]

# Format the code
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 23.10.1
rev: 23.11.0
hooks:
- id: black
language_version: python3.11
2 changes: 1 addition & 1 deletion .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@ version: 2
build:
os: ubuntu-22.04
tools:
python: mambaforge-4.10
python: mambaforge-22.9

# Define the python environment using conda / mamba
conda:
41 changes: 18 additions & 23 deletions Makefile
Original file line number Diff line number Diff line change
@@ -5,7 +5,6 @@ coverage_report := coverage report --sort=cover
pytest_args := --durations 20 ${pytest_covargs} ${gcs_cache_path}
etl_fast_yml := src/pudl/package_data/settings/etl_fast.yml
etl_full_yml := src/pudl/package_data/settings/etl_full.yml
pip_install_pudl := pip install --no-deps --editable ./

# We use mamba locally, but micromamba in CI, so choose the right binary:
ifdef GITHUB_ACTION
@@ -47,25 +46,17 @@ conda-lock.yml: pyproject.toml
cd environments && conda-lock render \
--kind env \
--dev-dependencies \
--extras docs \
--extras datasette \
conda-lock.yml
prettier --write environments/*.yml

# Create the pudl-dev conda environment based on the universal lockfile
.PHONY: pudl-dev
pudl-dev: conda-lock.yml
conda-lock install \
--name pudl-dev \
--${mamba} \
--dev \
--extras docs \
--extras datasette \
environments/conda-lock.yml
conda-lock install --name pudl-dev --${mamba} --dev environments/conda-lock.yml

.PHONY: install-pudl
install-pudl: pudl-dev
${mamba} run --name pudl-dev pip install --no-deps --editable .
${mamba} run --name pudl-dev pip install --no-cache-dir --no-deps --editable .

########################################################################################
# Build documentation for local use or testing
@@ -92,19 +83,24 @@ docs-build: docs-clean
########################################################################################

# Extract all FERC DBF and XBRL data to SQLite.
ferc1.sqlite ferc1_xbrl.sqlite:
.PHONY: ferc
ferc:
rm -f ${PUDL_OUTPUT}/ferc*.sqlite
rm -f ${PUDL_OUTPUT}/ferc*_xbrl_datapackage.json
rm -f ${PUDL_OUTPUT}/ferc*_xbrl_taxonomy_metadata.json
coverage run ${covargs} -- \
src/pudl/ferc_to_sqlite/cli.py \
--clobber \
${gcs_cache_path} \
${etl_full_yml}

# Run the full PUDL ETL
pudl.sqlite:
coverage run ${covargs} -- \
src/pudl/cli/etl.py \
${gcs_cache_path} \
${etl_full_yml}
# Remove the existing PUDL DB if it exists.
# Create a new empty DB using alembic.
# Run the full PUDL ETL.
.PHONY: pudl
pudl:
rm -f ${PUDL_OUTPUT}/pudl.sqlite
alembic upgrade head
coverage run ${covargs} -- src/pudl/cli/etl.py ${gcs_cache_path} ${etl_full_yml}

########################################################################################
# pytest
@@ -140,11 +136,10 @@ pytest-validate:
# Backgrounding the data validation and integration tests and using wait allows them to
# run in parallel.
.PHONY: nuke
nuke: coverage-erase docs-build pytest-unit ferc1.sqlite ferc1_xbrl.sqlite pudl.sqlite
nuke: coverage-erase docs-build pytest-unit ferc pudl
pudl_check_fks
pytest ${pytest_args} --live-dbs --etl-settings ${etl_full_yml} test/integration & \
pytest ${pytest_args} --live-dbs test/validate & \
wait
pytest ${pytest_args} -n auto --live-dbs --etl-settings ${etl_full_yml} test/integration
pytest ${pytest_args} -n auto --live-dbs test/validate
${coverage_report}

# Check that designated Jupyter notebooks can be run against the current DB
2 changes: 1 addition & 1 deletion docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -46,7 +46,7 @@ COPY docker/dagster.yaml ${DAGSTER_HOME}/dagster.yaml

# Create a conda environment based on the specification in the repo
COPY environments/conda-lock.yml environments/conda-lock.yml
RUN micromamba create --prefix ${CONDA_PREFIX} --yes --category main dev docs test datasette --file environments/conda-lock.yml && \
RUN micromamba create --prefix ${CONDA_PREFIX} --yes --file environments/conda-lock.yml && \
micromamba clean -afy
# Copy the cloned pudl repository into the user's home directory
COPY --chown=${MAMBA_USER}:${MAMBA_USER} . ${CONTAINER_HOME}
16 changes: 9 additions & 7 deletions docker/gcp_pudl_etl.sh
Original file line number Diff line number Diff line change
@@ -24,21 +24,23 @@ function run_pudl_etl() {
alembic upgrade head && \
pudl_setup && \
ferc_to_sqlite \
--loglevel=DEBUG \
--gcs-cache-path=gs://internal-zenodo-cache.catalyst.coop \
--workers=8 \
--loglevel DEBUG \
--gcs-cache-path gs://internal-zenodo-cache.catalyst.coop \
--workers 8 \
$PUDL_SETTINGS_YML && \
pudl_etl \
--loglevel DEBUG \
--gcs-cache-path gs://internal-zenodo-cache.catalyst.coop \
$PUDL_SETTINGS_YML && \
pytest \
--gcs-cache-path=gs://internal-zenodo-cache.catalyst.coop \
--etl-settings=$PUDL_SETTINGS_YML \
-n auto \
--gcs-cache-path gs://internal-zenodo-cache.catalyst.coop \
--etl-settings $PUDL_SETTINGS_YML \
--live-dbs test/integration test/unit && \
pytest \
--gcs-cache-path=gs://internal-zenodo-cache.catalyst.coop \
--etl-settings=$PUDL_SETTINGS_YML \
-n auto \
--gcs-cache-path gs://internal-zenodo-cache.catalyst.coop \
--etl-settings $PUDL_SETTINGS_YML \
--live-dbs test/validate
}

96 changes: 56 additions & 40 deletions environments/conda-linux-64.lock.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Generated by conda-lock.
# platform: linux-64
# input_hash: 874b20e543d8ad29e094c126a85321cefaf15e80f7f713bc52572ed16a1facf3
# input_hash: 62c018f04dcda6922fd5078faad77556aec1dd1f03c6a2262f899eff6feb9116

channels:
- conda-forge
@@ -25,13 +25,15 @@ dependencies:
- fonts-conda-ecosystem=1=0
- libgcc-ng=13.2.0=h807b86a_2
- aws-c-common=0.9.3=hd590300_0
- bzip2=1.0.8=h7f98852_4
- bzip2=1.0.8=hd590300_5
- c-ares=1.21.0=hd590300_0
- fribidi=1.0.10=h36c2ea0_0
- geos=3.12.0=h59595ed_0
- gettext=0.21.1=h27087fc_0
- gflags=2.2.2=he1b5a44_1004
- giflib=5.2.1=h0b41bf4_3
- gmp=6.2.1=h58526e2_0
- gmp=6.3.0=h59595ed_0
- graphite2=1.3.13=h58526e2_1001
- icu=73.2=h59595ed_0
- json-c=0.17=h7ab15ed_0
- keyutils=1.6.1=h166bdaf_0
@@ -51,6 +53,7 @@ dependencies:
- libnuma=2.0.16=h0b41bf4_1
- libsodium=1.0.18=h36c2ea0_1
- libspatialindex=1.9.3=h9c3ff4c_4
- libtool=2.4.7=h27087fc_0
- libutf8proc=2.8.0=h166bdaf_0
- libuuid=2.38.1=h0b41bf4_0
- libuv=1.46.0=hd590300_0
@@ -91,7 +94,7 @@ dependencies:
- libgfortran-ng=13.2.0=h69a702a_2
- libkml=1.3.0=h01aab08_1018
- libllvm14=14.0.6=hcd5def8_4
- libnghttp2=1.55.1=h47da74e_0
- libnghttp2=1.58.0=h47da74e_0
- libpng=1.6.39=h753d276_0
- libprotobuf=4.23.4=hf27288f_6
- librttopo=1.1.0=hb58d41b_14
@@ -115,7 +118,7 @@ dependencies:
- freetype=2.12.1=h267a509_2
- krb5=1.21.2=h659d440_0
- libarchive=3.7.2=h039dbb9_0
- libglib=2.78.0=hebfc3b9_0
- libglib=2.78.1=hebfc3b9_0
- libgrpc=1.57.0=ha4d0f93_2
- libopenblas=0.3.24=pthreads_h413a1c8_0
- libthrift=0.19.0=hb90f79a_1
@@ -134,6 +137,7 @@ dependencies:
- anyascii=0.3.2=pyhd8ed1ab_0
- appdirs=1.4.4=pyh9f0ad1d_0
- astroid=3.0.1=py311h38be061_0
- atk-1.0=2.38.0=hd4edc92_1
- attrs=23.1.0=pyh71513ae_1
- aws-c-event-stream=0.3.2=h6fea174_2
- aws-c-http=0.7.13=hb59894b_2
@@ -157,7 +161,7 @@ dependencies:
- colorama=0.4.6=pyhd8ed1ab_0
- crashtest=0.4.1=pyhd8ed1ab_0
- cycler=0.12.1=pyhd8ed1ab_0
- dagster-pipes=1.5.6=pyhd8ed1ab_1
- dagster-pipes=1.5.7=pyhd8ed1ab_0
- dataclasses=0.8=pyhc8e2a94_3
- dbus=1.13.6=h5008d03_3
- debugpy=1.8.0=py311hb755f60_1
@@ -169,15 +173,18 @@ dependencies:
- entrypoints=0.4=pyhd8ed1ab_0
- et_xmlfile=1.1.0=pyhd8ed1ab_0
- exceptiongroup=1.1.3=pyhd8ed1ab_0
- execnet=2.0.2=pyhd8ed1ab_0
- executing=2.0.1=pyhd8ed1ab_0
- filelock=3.13.1=pyhd8ed1ab_0
- fontconfig=2.14.2=h14ed4e7_0
- freexl=2.0.0=h743c826_0
- frozenlist=1.4.0=py311h459d7ec_1
- fsspec=2023.10.0=pyhca7485f_0
- google-cloud-sdk=453.0.0=py311h38be061_0
- gdk-pixbuf=2.42.10=h829c605_4
- google-cloud-sdk=454.0.0=py311h38be061_0
- greenlet=3.0.1=py311hb755f60_0
- grpcio=1.57.0=py311ha6695c7_2
- gts=0.7.6=h977cf35_4
- hpack=4.0.0=pyh9f0ad1d_0
- httptools=0.6.1=py311h459d7ec_0
- humanfriendly=10.0=pyhd8ed1ab_6
@@ -198,8 +205,9 @@ dependencies:
- lcms2=2.15=hb7c19ff_3
- libblas=3.9.0=19_linux64_openblas
- libcurl=8.4.0=hca28451_0
- libpq=16.0=hfc447b1_1
- llvmlite=0.40.1=py311ha6695c7_0
- libpq=16.1=hfc447b1_0
- libwebp=1.3.2=h658648e_1
- llvmlite=0.41.1=py311ha6695c7_0
- locket=1.0.0=pyhd8ed1ab_0
- lxml=4.9.3=py311h1a07684_1
- marko=1.3.1=pyhd8ed1ab_0
@@ -256,7 +264,7 @@ dependencies:
- rpds-py=0.12.0=py311h46250e7_0
- rtree=1.1.0=py311h3bb2b0f_0
- ruamel.yaml.clib=0.2.7=py311h459d7ec_2
- ruff=0.1.4=py311h7145743_0
- ruff=0.1.5=py311h7145743_0
- send2trash=1.8.2=pyh41d4057_0
- setuptools=68.2.2=pyhd8ed1ab_0
- shellingham=1.5.4=pyhd8ed1ab_0
@@ -293,7 +301,7 @@ dependencies:
- websockets=10.4=py311hd4cff14_1
- wheel=0.41.3=pyhd8ed1ab_0
- widgetsnbextension=4.0.9=pyhd8ed1ab_0
- wrapt=1.15.0=py311h459d7ec_1
- wrapt=1.16.0=py311h459d7ec_0
- xlrd=2.0.1=pyhd8ed1ab_3
- xlsxwriter=3.1.9=pyhd8ed1ab_0
- xorg-libxext=1.3.4=h0b41bf4_2
@@ -333,7 +341,7 @@ dependencies:
- html5lib=1.1=pyh9f0ad1d_0
- hypothesis=6.88.3=pyha770c72_0
- importlib-metadata=6.8.0=pyha770c72_0
- importlib_resources=6.1.0=pyhd8ed1ab_0
- importlib_resources=6.1.1=pyhd8ed1ab_0
- isodate=0.6.1=pyhd8ed1ab_0
- janus=1.0.0=pyhd8ed1ab_0
- jaraco.classes=3.3.0=pyhd8ed1ab_0
@@ -344,6 +352,7 @@ dependencies:
- jupyterlab_pygments=0.2.2=pyhd8ed1ab_0
- latexcodec=2.0.1=pyh9f0ad1d_0
- libcblas=3.9.0=19_linux64_openblas
- libgd=2.3.3=h119a65a_9
- libgoogle-cloud=2.12.0=h8d7e28b_2
- liblapack=3.9.0=19_linux64_openblas
- linear-tsv=1.1.0=py_1
@@ -357,7 +366,7 @@ dependencies:
- pillow=10.1.0=py311ha6c5da5_0
- pint=0.22=pyhd8ed1ab_1
- pip=23.3.1=pyhd8ed1ab_0
- postgresql=16.0=h8972f4a_1
- postgresql=16.1=h8972f4a_0
- proj=9.3.0=h1d62c97_2
- protobuf=4.23.4=py311h46cbc50_3
- psycopg2=2.9.7=py311h03dec38_1
@@ -390,8 +399,8 @@ dependencies:
- arrow=1.3.0=pyhd8ed1ab_0
- async-timeout=4.0.3=pyhd8ed1ab_0
- aws-c-s3=0.3.17=hfb4bb88_4
- botocore=1.31.78=pyhd8ed1ab_0
- branca=0.6.0=pyhd8ed1ab_0
- botocore=1.31.83=pyhd8ed1ab_0
- branca=0.7.0=pyhd8ed1ab_1
- croniter=2.0.1=pyhd8ed1ab_0
- cryptography=41.0.5=py311h63ff55d_0
- fqdn=1.5.1=pyhd8ed1ab_0
@@ -402,31 +411,33 @@ dependencies:
- gql=3.4.1=pyhd8ed1ab_0
- graphql-relay=3.2.0=pyhd8ed1ab_0
- grpcio-health-checking=1.57.0=pyhd8ed1ab_0
- httpcore=1.0.1=pyhd8ed1ab_0
- harfbuzz=8.2.1=h3d44ed6_0
- httpcore=1.0.2=pyhd8ed1ab_0
- importlib_metadata=6.8.0=hd8ed1ab_0
- jsonschema-specifications=2023.7.1=pyhd8ed1ab_0
- jupyter_server_terminals=0.4.4=pyhd8ed1ab_1
- kealib=1.5.2=hcd42e92_1
- libnetcdf=4.9.2=nompi_h80fb2b6_112
- libspatialite=5.1.0=h090f1da_0
- mako=1.2.4=pyhd8ed1ab_0
- numpy=1.24.4=py311h64a7726_0
- pbr=5.11.1=pyhd8ed1ab_0
- libspatialite=5.1.0=h090f1da_1
- mako=1.3.0=pyhd8ed1ab_0
- numpy=1.26.0=py311h64a7726_0
- pbr=6.0.0=pyhd8ed1ab_0
- pendulum=2.1.2=py311h459d7ec_6
- platformdirs=3.11.0=pyhd8ed1ab_0
- poppler=23.10.0=h590f24d_0
- psycopg2-binary=2.9.7=pyhd8ed1ab_1
- pybtex=0.24.0=pyhd8ed1ab_2
- pydantic=1.10.13=py311h459d7ec_1
- pyproj=3.6.1=py311h1facc83_3
- pyproj=3.6.1=py311h1facc83_4
- pytest-console-scripts=1.4.1=pyhd8ed1ab_0
- pytest-cov=4.1.0=pyhd8ed1ab_0
- pytest-mock=3.12.0=pyhd8ed1ab_0
- pytest-xdist=3.3.1=pyhd8ed1ab_0
- python-build=1.0.3=pyhd8ed1ab_0
- requests=2.31.0=pyhd8ed1ab_0
- rich=13.6.0=pyhd8ed1ab_0
- stack_data=0.6.2=pyhd8ed1ab_0
- starlette=0.31.1=pyhd8ed1ab_0
- starlette=0.32.0.post1=pyhd8ed1ab_0
- tiledb=2.16.3=h8c794c1_3
- ukkonen=1.0.1=py311h9547e67_4
- uvicorn=0.24.0=py311h38be061_0
@@ -444,7 +455,7 @@ dependencies:
- dask-core=2023.10.1=pyhd8ed1ab_0
- dnspython=2.4.2=pyhd8ed1ab_1
- ensureconda=1.4.3=pyhd8ed1ab_0
- folium=0.14.0=pyhd8ed1ab_0
- folium=0.15.0=pyhd8ed1ab_0
- google-resumable-media=2.6.0=pyhd8ed1ab_0
- graphene=3.3=pyhd8ed1ab_0
- grpcio-status=1.57.0=pyhd8ed1ab_0
@@ -455,11 +466,12 @@ dependencies:
- jsonschema=4.19.2=pyhd8ed1ab_0
- jupyter_core=5.5.0=py311h38be061_0
- libgdal=3.7.3=h6f3d308_2
- numba=0.57.1=py311h96b013e_0
- numba=0.58.1=py311h96b013e_0
- numexpr=2.8.7=py311h039bad6_104
- oauthlib=3.2.2=pyhd8ed1ab_0
- pandas=2.1.2=py311h320fe9a_0
- prompt-toolkit=3.0.39=pyha770c72_0
- pango=1.50.14=ha41ecd1_2
- prompt-toolkit=3.0.40=pyha770c72_0
- pybtex-docutils=1.0.3=py311h38be061_1
- pyopenssl=23.3.0=pyhd8ed1ab_0
- readthedocs-sphinx-ext=2.2.3=pyhd8ed1ab_0
@@ -475,9 +487,9 @@ dependencies:
- uvicorn-standard=0.24.0=h38be061_0
- virtualenv=20.24.6=pyhd8ed1ab_0
- aws-sdk-cpp=1.11.156=h314d761_4
- boto3=1.28.78=pyhd8ed1ab_0
- boto3=1.28.83=pyhd8ed1ab_0
- cachecontrol-with-filecache=0.13.1=pyhd8ed1ab_0
- dagster=1.5.6=pyhd8ed1ab_1
- dagster=1.5.7=pyhd8ed1ab_0
- datasette=0.64.4=pyhd8ed1ab_1
- doc8=1.1.1=pyhd8ed1ab_0
- email-validator=2.1.0.post1=pyhd8ed1ab_0
@@ -486,50 +498,54 @@ dependencies:
- geopandas-base=0.14.0=pyha770c72_1
- google-auth=2.23.4=pyhca7485f_0
- gql-with-requests=3.4.1=pyhd8ed1ab_0
- gtk2=2.24.33=h90689f9_2
- jsonschema-with-format-nongpl=4.19.2=pyhd8ed1ab_0
- jupyter_client=8.5.0=pyhd8ed1ab_0
- jupyter_client=8.6.0=pyhd8ed1ab_0
- keyring=24.2.0=py311h38be061_1
- librsvg=2.56.3=h98fae49_0
- matplotlib-base=3.8.1=py311h54ef318_0
- nbformat=5.9.2=pyhd8ed1ab_0
- pandera-core=0.17.2=pyhd8ed1ab_0
- pandera-core=0.17.2=pyhd8ed1ab_1
- pre-commit=3.5.0=pyha770c72_0
- prompt_toolkit=3.0.39=hd8ed1ab_0
- prompt_toolkit=3.0.40=hd8ed1ab_0
- requests-oauthlib=1.3.1=pyhd8ed1ab_0
- scikit-learn=1.3.2=py311hc009520_1
- timezonefinder=6.2.0=py311h459d7ec_2
- catalystcoop.ferc_xbrl_extractor=1.2.1=pyhd8ed1ab_0
- conda-lock=2.4.2=pyhd8ed1ab_0
- dagster-graphql=1.5.6=pyhd8ed1ab_1
- dagster-postgres=1.5.6=pyhd8ed1ab_1
- dagster-graphql=1.5.7=pyhd8ed1ab_0
- dagster-postgres=0.21.7=pyhd8ed1ab_0
- fiona=1.9.5=py311hbac4ec9_0
- google-api-core=2.12.0=pyhd8ed1ab_0
- google-api-core=2.13.0=pyhd8ed1ab_0
- google-auth-oauthlib=1.1.0=pyhd8ed1ab_0
- graphviz=8.1.0=h28d9a01_0
- ipython=8.17.2=pyh41d4057_0
- jupyter_events=0.8.0=pyhd8ed1ab_0
- jupyter_events=0.9.0=pyhd8ed1ab_0
- libarrow=13.0.0=h0f80be4_7_cpu
- mapclassify=2.6.1=pyhd8ed1ab_0
- nbclient=0.8.0=pyhd8ed1ab_0
- recordlinkage=0.16=pyhd8ed1ab_0
- tabulator=1.53.5=pyhd8ed1ab_0
- dagster-webserver=1.5.6=pyhd8ed1ab_1
- dagster-webserver=1.5.7=pyhd8ed1ab_0
- geopandas=0.14.0=pyhd8ed1ab_1
- google-cloud-core=2.3.3=pyhd8ed1ab_0
- ipykernel=6.26.0=pyhf8b6a83_0
- ipywidgets=8.1.1=pyhd8ed1ab_0
- nbconvert-core=7.10.0=pyhd8ed1ab_0
- nbconvert-core=7.11.0=pyhd8ed1ab_0
- pyarrow=13.0.0=py311h39c9aba_7_cpu
- pygraphviz=1.11=py311h72a77b7_1
- tableschema=1.19.3=pyh9f0ad1d_0
- datapackage=1.15.2=pyh44b312d_0
- google-cloud-storage=2.13.0=pyhca7485f_0
- jupyter_console=6.6.3=pyhd8ed1ab_0
- jupyter_server=2.9.1=pyhd8ed1ab_0
- nbconvert-pandoc=7.10.0=pyhd8ed1ab_0
- jupyter_server=2.10.0=pyhd8ed1ab_0
- nbconvert-pandoc=7.11.0=pyhd8ed1ab_0
- qtconsole-base=5.5.0=pyha770c72_0
- gcsfs=2023.10.0=pyhd8ed1ab_0
- jupyter-lsp=2.2.0=pyhd8ed1ab_0
- jupyter-resource-usage=1.0.1=pyhd8ed1ab_0
- jupyterlab_server=2.25.0=pyhd8ed1ab_0
- nbconvert=7.10.0=pyhd8ed1ab_0
- jupyterlab_server=2.25.1=pyhd8ed1ab_0
- nbconvert=7.11.0=pyhd8ed1ab_0
- notebook-shim=0.2.3=pyhd8ed1ab_0
- jupyterlab=4.0.8=pyhd8ed1ab_0
- notebook=7.0.6=pyhd8ed1ab_0
4,073 changes: 2,427 additions & 1,646 deletions environments/conda-lock.yml

Large diffs are not rendered by default.

94 changes: 55 additions & 39 deletions environments/conda-osx-64.lock.yml
Original file line number Diff line number Diff line change
@@ -1,19 +1,20 @@
# Generated by conda-lock.
# platform: osx-64
# input_hash: 5d3e47cd81a10d602352a0a75370c4c94de2e376fd3d63a6ffe1838c9c2c3d97
# input_hash: 7ade47c1ca071082a3818606e76b6768241ac2410e31d8f902abbb59ecaba073

channels:
- conda-forge
- defaults
dependencies:
- aws-c-common=0.9.3=h0dc2134_0
- bzip2=1.0.8=h0d85af4_4
- bzip2=1.0.8=h10d778d_5
- c-ares=1.21.0=h10d778d_0
- ca-certificates=2023.7.22=h8857fd0_0
- font-ttf-dejavu-sans-mono=2.37=hab24e00_0
- font-ttf-inconsolata=3.000=h77eed37_0
- font-ttf-source-code-pro=2.038=h77eed37_0
- font-ttf-ubuntu=0.83=hab24e00_0
- fribidi=1.0.10=hbcb3906_0
- giflib=5.2.1=hb7f2c08_3
- icu=73.2=hf5e326d_0
- json-c=0.17=h8e11ae5_0
@@ -27,6 +28,7 @@ dependencies:
- libiconv=1.17=hac89ed1_0
- libjpeg-turbo=3.0.0=h0dc2134_1
- libsodium=1.0.18=hbcb3906_1
- libtool=2.4.7=hf0c8a7f_0
- libutf8proc=2.8.0=hb7f2c08_0
- libuv=1.46.0=h0c2f820_0
- libwebp-base=1.3.2=h0dc2134_0
@@ -51,6 +53,7 @@ dependencies:
- geos=3.12.0=he965462_0
- gettext=0.21.1=h8a4c099_0
- gflags=2.2.2=hb1e8313_1004
- graphite2=1.3.13=h2e338ed_1001
- hdf4=4.2.15=h8138101_7
- lerc=4.0.0=hb486fe8_0
- libabseil=20230802.1=cxx17_h048a20a_0
@@ -89,9 +92,9 @@ dependencies:
- libedit=3.1.20191231=h0678c8f_2
- libevent=2.1.12=ha90c15b_1
- libgfortran=5.0.0=13_2_0_h97931a8_1
- libglib=2.78.0=hc62aa5d_0
- libglib=2.78.1=h6d9ecee_0
- libkml=1.3.0=hab3ca0e_1018
- libnghttp2=1.55.1=hc0a10c5_0
- libnghttp2=1.58.0=h64cf6d3_0
- libprotobuf=4.23.4=he0c2237_6
- librttopo=1.1.0=h23f359d_14
- libssh2=1.11.0=hd019ec5_0
@@ -102,16 +105,20 @@ dependencies:
- nodejs=20.8.1=h9adec40_0
- nss=3.94=hd6ac835_0
- readline=8.2=h9e318b2_1
- atk-1.0=2.38.0=h1d18e73_1
- aws-c-event-stream=0.3.2=he52df9d_2
- aws-c-http=0.7.13=hb43ddd7_2
- brotli=1.1.0=h0dc2134_1
- fontconfig=2.14.2=h5bb23bf_0
- freexl=2.0.0=h3ec172f_0
- gdk-pixbuf=2.42.10=hbb5a27d_4
- gts=0.7.6=h53e17e3_4
- krb5=1.21.2=hb884880_0
- lcms2=2.15=hd6ba6f3_3
- libgrpc=1.57.0=h7cc5909_2
- libopenblas=0.3.24=openmp_h48a4ad5_0
- libthrift=0.19.0=h064b379_1
- libwebp=1.3.2=h44782d1_1
- openjpeg=2.5.0=ha4da562_3
- orc=1.9.0=ha4ae40d_2
- prettier=3.0.3=hbd11d21_0
@@ -146,7 +153,7 @@ dependencies:
- colorama=0.4.6=pyhd8ed1ab_0
- crashtest=0.4.1=pyhd8ed1ab_0
- cycler=0.12.1=pyhd8ed1ab_0
- dagster-pipes=1.5.6=pyhd8ed1ab_1
- dagster-pipes=1.5.7=pyhd8ed1ab_0
- dataclasses=0.8=pyhc8e2a94_3
- debugpy=1.8.0=py311hdf8f085_1
- decorator=5.1.1=pyhd8ed1ab_0
@@ -157,11 +164,12 @@ dependencies:
- entrypoints=0.4=pyhd8ed1ab_0
- et_xmlfile=1.1.0=pyhd8ed1ab_0
- exceptiongroup=1.1.3=pyhd8ed1ab_0
- execnet=2.0.2=pyhd8ed1ab_0
- executing=2.0.1=pyhd8ed1ab_0
- filelock=3.13.1=pyhd8ed1ab_0
- frozenlist=1.4.0=py311h2725bcf_1
- fsspec=2023.10.0=pyhca7485f_0
- google-cloud-sdk=453.0.0=py311h6eed73b_0
- google-cloud-sdk=454.0.0=py311h6eed73b_0
- greenlet=3.0.1=py311hd39e593_0
- grpcio=1.57.0=py311h78ff076_2
- hpack=4.0.0=pyh9f0ad1d_0
@@ -182,8 +190,9 @@ dependencies:
- kiwisolver=1.4.5=py311h5fe6e05_1
- libblas=3.9.0=19_osx64_openblas
- libcurl=8.4.0=h726d00d_0
- libpq=16.0=h3df487d_1
- llvmlite=0.40.1=py311hcbb5c6d_0
- libgd=2.3.3=h0dceb68_9
- libpq=16.1=h6dd4ff7_0
- llvmlite=0.41.1=py311hb5c2e0a_0
- locket=1.0.0=pyhd8ed1ab_0
- lxml=4.9.3=py311h19a211c_1
- marko=1.3.1=pyhd8ed1ab_0
@@ -239,7 +248,7 @@ dependencies:
- rpds-py=0.12.0=py311h5e0f0e4_0
- rtree=1.1.0=py311hbc1f44b_0
- ruamel.yaml.clib=0.2.7=py311h2725bcf_2
- ruff=0.1.4=py311hec6fdf1_0
- ruff=0.1.5=py311hec6fdf1_0
- setuptools=68.2.2=pyhd8ed1ab_0
- shellingham=1.5.4=pyhd8ed1ab_0
- simpleeval=0.9.13=pyhd8ed1ab_1
@@ -275,7 +284,7 @@ dependencies:
- websockets=10.4=py311h5547dcb_1
- wheel=0.41.3=pyhd8ed1ab_0
- widgetsnbextension=4.0.9=pyhd8ed1ab_0
- wrapt=1.15.0=py311h2725bcf_1
- wrapt=1.16.0=py311he705e18_0
- xlrd=2.0.1=pyhd8ed1ab_3
- xlsxwriter=3.1.9=pyhd8ed1ab_0
- xyzservices=2023.10.1=pyhd8ed1ab_0
@@ -307,11 +316,12 @@ dependencies:
- graphql-core=3.2.3=pyhd8ed1ab_0
- h11=0.14.0=pyhd8ed1ab_0
- h2=4.1.0=pyhd8ed1ab_0
- harfbuzz=8.2.1=h7666e2a_0
- hdf5=1.14.2=nompi_hedada53_100
- html5lib=1.1=pyh9f0ad1d_0
- hypothesis=6.88.3=pyha770c72_0
- importlib-metadata=6.8.0=pyha770c72_0
- importlib_resources=6.1.0=pyhd8ed1ab_0
- importlib_resources=6.1.1=pyhd8ed1ab_0
- isodate=0.6.1=pyhd8ed1ab_0
- janus=1.0.0=pyhd8ed1ab_0
- jaraco.classes=3.3.0=pyhd8ed1ab_0
@@ -335,7 +345,7 @@ dependencies:
- pint=0.22=pyhd8ed1ab_1
- pip=23.3.1=pyhd8ed1ab_0
- poppler=23.10.0=hdd5a5e8_0
- postgresql=16.0=hc940a54_1
- postgresql=16.1=h413614c_0
- proj=9.3.0=h23b96cc_2
- protobuf=4.23.4=py311h01dfb71_3
- psycopg2=2.9.7=py311h187f0af_1
@@ -369,8 +379,8 @@ dependencies:
- arrow=1.3.0=pyhd8ed1ab_0
- async-timeout=4.0.3=pyhd8ed1ab_0
- aws-crt-cpp=0.24.2=hf47b73b_2
- botocore=1.31.78=pyhd8ed1ab_0
- branca=0.6.0=pyhd8ed1ab_0
- botocore=1.31.83=pyhd8ed1ab_0
- branca=0.7.0=pyhd8ed1ab_1
- croniter=2.0.1=pyhd8ed1ab_0
- cryptography=41.0.5=py311hd51016d_0
- fqdn=1.5.1=pyhd8ed1ab_0
@@ -381,31 +391,33 @@ dependencies:
- gql=3.4.1=pyhd8ed1ab_0
- graphql-relay=3.2.0=pyhd8ed1ab_0
- grpcio-health-checking=1.57.0=pyhd8ed1ab_0
- httpcore=1.0.1=pyhd8ed1ab_0
- httpcore=1.0.2=pyhd8ed1ab_0
- importlib_metadata=6.8.0=hd8ed1ab_0
- jsonschema-specifications=2023.7.1=pyhd8ed1ab_0
- jupyter_server_terminals=0.4.4=pyhd8ed1ab_1
- kealib=1.5.2=h052fcf7_1
- libnetcdf=4.9.2=nompi_h6a32802_112
- libspatialite=5.1.0=h231fb02_0
- mako=1.2.4=pyhd8ed1ab_0
- numpy=1.24.4=py311hc44ba51_0
- pbr=5.11.1=pyhd8ed1ab_0
- libspatialite=5.1.0=h4a59d8e_1
- mako=1.3.0=pyhd8ed1ab_0
- numpy=1.26.0=py311hc44ba51_0
- pango=1.50.14=h19c1c8a_2
- pbr=6.0.0=pyhd8ed1ab_0
- pendulum=2.1.2=py311h2725bcf_6
- platformdirs=3.11.0=pyhd8ed1ab_0
- psycopg2-binary=2.9.7=pyhd8ed1ab_1
- pybtex=0.24.0=pyhd8ed1ab_2
- pydantic=1.10.13=py311he705e18_1
- pyobjc-framework-cocoa=10.0=py311hf110eff_1
- pyproj=3.6.1=py311he36daed_3
- pyproj=3.6.1=py311he36daed_4
- pytest-console-scripts=1.4.1=pyhd8ed1ab_0
- pytest-cov=4.1.0=pyhd8ed1ab_0
- pytest-mock=3.12.0=pyhd8ed1ab_0
- pytest-xdist=3.3.1=pyhd8ed1ab_0
- python-build=1.0.3=pyhd8ed1ab_0
- requests=2.31.0=pyhd8ed1ab_0
- rich=13.6.0=pyhd8ed1ab_0
- stack_data=0.6.2=pyhd8ed1ab_0
- starlette=0.31.1=pyhd8ed1ab_0
- starlette=0.32.0.post1=pyhd8ed1ab_0
- tiledb=2.16.3=hd3a41d5_3
- ukkonen=1.0.1=py311h5fe6e05_4
- uvicorn=0.24.0=py311h6eed73b_0
@@ -423,10 +435,11 @@ dependencies:
- dask-core=2023.10.1=pyhd8ed1ab_0
- dnspython=2.4.2=pyhd8ed1ab_1
- ensureconda=1.4.3=pyhd8ed1ab_0
- folium=0.14.0=pyhd8ed1ab_0
- folium=0.15.0=pyhd8ed1ab_0
- google-resumable-media=2.6.0=pyhd8ed1ab_0
- graphene=3.3=pyhd8ed1ab_0
- grpcio-status=1.57.0=pyhd8ed1ab_0
- gtk2=2.24.33=h7c1209e_2
- h3-py=3.7.6=py311hdf8f085_1
- httpx=0.25.1=pyhd8ed1ab_0
- identify=2.5.31=pyhd8ed1ab_0
@@ -435,11 +448,12 @@ dependencies:
- jupyter_core=5.5.0=py311h6eed73b_0
- keyring=24.2.0=py311h6eed73b_1
- libgdal=3.7.3=h926149b_2
- numba=0.57.1=py311h5a8220d_0
- librsvg=2.56.3=hec3db73_0
- numba=0.58.1=py311h32f2313_0
- numexpr=2.8.7=py311h1eadf79_4
- oauthlib=3.2.2=pyhd8ed1ab_0
- pandas=2.1.2=py311h1eadf79_0
- prompt-toolkit=3.0.39=pyha770c72_0
- prompt-toolkit=3.0.40=pyha770c72_0
- pybtex-docutils=1.0.3=py311h6eed73b_1
- pyopenssl=23.3.0=pyhd8ed1ab_0
- readthedocs-sphinx-ext=2.2.3=pyhd8ed1ab_0
@@ -454,9 +468,9 @@ dependencies:
- typer=0.9.0=pyhd8ed1ab_0
- uvicorn-standard=0.24.0=h6eed73b_0
- virtualenv=20.24.6=pyhd8ed1ab_0
- boto3=1.28.78=pyhd8ed1ab_0
- boto3=1.28.83=pyhd8ed1ab_0
- cachecontrol-with-filecache=0.13.1=pyhd8ed1ab_0
- dagster=1.5.6=pyhd8ed1ab_1
- dagster=1.5.7=pyhd8ed1ab_0
- datasette=0.64.4=pyhd8ed1ab_1
- doc8=1.1.1=pyhd8ed1ab_0
- email-validator=2.1.0.post1=pyhd8ed1ab_0
@@ -465,49 +479,51 @@ dependencies:
- geopandas-base=0.14.0=pyha770c72_1
- google-auth=2.23.4=pyhca7485f_0
- gql-with-requests=3.4.1=pyhd8ed1ab_0
- graphviz=8.1.0=hc7f41f9_0
- jsonschema-with-format-nongpl=4.19.2=pyhd8ed1ab_0
- jupyter_client=8.5.0=pyhd8ed1ab_0
- jupyter_client=8.6.0=pyhd8ed1ab_0
- libarrow=13.0.0=h66f826e_7_cpu
- matplotlib-base=3.8.1=py311hd316c10_0
- nbformat=5.9.2=pyhd8ed1ab_0
- pandera-core=0.17.2=pyhd8ed1ab_0
- pandera-core=0.17.2=pyhd8ed1ab_1
- pre-commit=3.5.0=pyha770c72_0
- prompt_toolkit=3.0.39=hd8ed1ab_0
- prompt_toolkit=3.0.40=hd8ed1ab_0
- requests-oauthlib=1.3.1=pyhd8ed1ab_0
- scikit-learn=1.3.2=py311h66081b9_1
- timezonefinder=6.2.0=py311he705e18_2
- catalystcoop.ferc_xbrl_extractor=1.2.1=pyhd8ed1ab_0
- conda-lock=2.4.2=pyhd8ed1ab_0
- dagster-graphql=1.5.6=pyhd8ed1ab_1
- dagster-postgres=1.5.6=pyhd8ed1ab_1
- dagster-graphql=1.5.7=pyhd8ed1ab_0
- dagster-postgres=0.21.7=pyhd8ed1ab_0
- fiona=1.9.5=py311hf14a637_0
- google-api-core=2.12.0=pyhd8ed1ab_0
- google-api-core=2.13.0=pyhd8ed1ab_0
- google-auth-oauthlib=1.1.0=pyhd8ed1ab_0
- ipython=8.17.2=pyh31c8845_0
- jupyter_events=0.8.0=pyhd8ed1ab_0
- jupyter_events=0.9.0=pyhd8ed1ab_0
- mapclassify=2.6.1=pyhd8ed1ab_0
- nbclient=0.8.0=pyhd8ed1ab_0
- pyarrow=13.0.0=py311h54e7ce8_7_cpu
- pygraphviz=1.11=py311hc6eba27_1
- recordlinkage=0.16=pyhd8ed1ab_0
- tabulator=1.53.5=pyhd8ed1ab_0
- dagster-webserver=1.5.6=pyhd8ed1ab_1
- dagster-webserver=1.5.7=pyhd8ed1ab_0
- geopandas=0.14.0=pyhd8ed1ab_1
- google-cloud-core=2.3.3=pyhd8ed1ab_0
- ipykernel=6.26.0=pyh3cd1d5f_0
- ipywidgets=8.1.1=pyhd8ed1ab_0
- nbconvert-core=7.10.0=pyhd8ed1ab_0
- nbconvert-core=7.11.0=pyhd8ed1ab_0
- tableschema=1.19.3=pyh9f0ad1d_0
- datapackage=1.15.2=pyh44b312d_0
- google-cloud-storage=2.13.0=pyhca7485f_0
- jupyter_console=6.6.3=pyhd8ed1ab_0
- jupyter_server=2.9.1=pyhd8ed1ab_0
- nbconvert-pandoc=7.10.0=pyhd8ed1ab_0
- jupyter_server=2.10.0=pyhd8ed1ab_0
- nbconvert-pandoc=7.11.0=pyhd8ed1ab_0
- qtconsole-base=5.5.0=pyha770c72_0
- gcsfs=2023.10.0=pyhd8ed1ab_0
- jupyter-lsp=2.2.0=pyhd8ed1ab_0
- jupyter-resource-usage=1.0.1=pyhd8ed1ab_0
- jupyterlab_server=2.25.0=pyhd8ed1ab_0
- nbconvert=7.10.0=pyhd8ed1ab_0
- jupyterlab_server=2.25.1=pyhd8ed1ab_0
- nbconvert=7.11.0=pyhd8ed1ab_0
- notebook-shim=0.2.3=pyhd8ed1ab_0
- jupyterlab=4.0.8=pyhd8ed1ab_0
- notebook=7.0.6=pyhd8ed1ab_0
94 changes: 55 additions & 39 deletions environments/conda-osx-arm64.lock.yml
Original file line number Diff line number Diff line change
@@ -1,19 +1,20 @@
# Generated by conda-lock.
# platform: osx-arm64
# input_hash: 52fe45bcd4eea0ac8640e1014a127bb299336b8ffd184abfc484483db5adb08d
# input_hash: 1678be4da0075835c5823bcfd307ec83872e5ae2ab716ed9f2b6d9f176a89e0d

channels:
- conda-forge
- defaults
dependencies:
- aws-c-common=0.9.3=hb547adb_0
- bzip2=1.0.8=h3422bc3_4
- bzip2=1.0.8=h93a5062_5
- c-ares=1.21.0=h93a5062_0
- ca-certificates=2023.7.22=hf0a4a13_0
- font-ttf-dejavu-sans-mono=2.37=hab24e00_0
- font-ttf-inconsolata=3.000=h77eed37_0
- font-ttf-source-code-pro=2.038=h77eed37_0
- font-ttf-ubuntu=0.83=hab24e00_0
- fribidi=1.0.10=h27ca646_0
- giflib=5.2.1=h1a8c8d9_3
- icu=73.2=hc8870d7_0
- json-c=0.17=h40ed0f5_0
@@ -27,6 +28,7 @@ dependencies:
- libiconv=1.17=he4db4b2_0
- libjpeg-turbo=3.0.0=hb547adb_1
- libsodium=1.0.18=h27ca646_1
- libtool=2.4.7=hb7217d7_0
- libutf8proc=2.8.0=h1a8c8d9_0
- libuv=1.46.0=hb547adb_0
- libwebp-base=1.3.2=hb547adb_0
@@ -52,6 +54,7 @@ dependencies:
- geos=3.12.0=h13dd4ca_0
- gettext=0.21.1=h0186832_0
- gflags=2.2.2=hc88da5d_1004
- graphite2=1.3.13=h9f76cd9_1001
- hdf4=4.2.15=h2ee6834_7
- lerc=4.0.0=h9a09cb3_0
- libabseil=20230802.1=cxx17_h13dd4ca_0
@@ -89,9 +92,9 @@ dependencies:
- libedit=3.1.20191231=hc8eb9b7_2
- libevent=2.1.12=h2757513_1
- libgfortran=5.0.0=13_2_0_hd922786_1
- libglib=2.78.0=h24e9cb9_0
- libglib=2.78.1=hd9b11f9_0
- libkml=1.3.0=h1eb4d9f_1018
- libnghttp2=1.55.1=h2b02ca0_0
- libnghttp2=1.58.0=ha4dd798_0
- libprotobuf=4.23.4=hf590ac1_6
- librttopo=1.1.0=h667cd51_14
- libssh2=1.11.0=h7a5bd25_0
@@ -102,16 +105,20 @@ dependencies:
- nodejs=20.8.1=h0950e01_0
- nss=3.94=hc6b9969_0
- readline=8.2=h92ec313_1
- atk-1.0=2.38.0=hcb7b3dd_1
- aws-c-event-stream=0.3.2=h8f67f9a_2
- aws-c-http=0.7.13=h7dae168_2
- brotli=1.1.0=hb547adb_1
- fontconfig=2.14.2=h82840c6_0
- freexl=2.0.0=hfbad9fb_0
- gdk-pixbuf=2.42.10=h15fa40c_4
- gts=0.7.6=he42f4ea_4
- krb5=1.21.2=h92f50d5_0
- lcms2=2.15=hf2736f0_3
- libgrpc=1.57.0=hbf2b313_2
- libopenblas=0.3.24=openmp_hd76b1f2_0
- libthrift=0.19.0=h026a170_1
- libwebp=1.3.2=hf30222e_1
- openjpeg=2.5.0=h4c1507b_3
- orc=1.9.0=hbfdecac_2
- prettier=3.0.3=hb67532b_0
@@ -146,7 +153,7 @@ dependencies:
- colorama=0.4.6=pyhd8ed1ab_0
- crashtest=0.4.1=pyhd8ed1ab_0
- cycler=0.12.1=pyhd8ed1ab_0
- dagster-pipes=1.5.6=pyhd8ed1ab_1
- dagster-pipes=1.5.7=pyhd8ed1ab_0
- dataclasses=0.8=pyhc8e2a94_3
- debugpy=1.8.0=py311ha891d26_1
- decorator=5.1.1=pyhd8ed1ab_0
@@ -157,11 +164,12 @@ dependencies:
- entrypoints=0.4=pyhd8ed1ab_0
- et_xmlfile=1.1.0=pyhd8ed1ab_0
- exceptiongroup=1.1.3=pyhd8ed1ab_0
- execnet=2.0.2=pyhd8ed1ab_0
- executing=2.0.1=pyhd8ed1ab_0
- filelock=3.13.1=pyhd8ed1ab_0
- frozenlist=1.4.0=py311heffc1b2_1
- fsspec=2023.10.0=pyhca7485f_0
- google-cloud-sdk=453.0.0=py311h267d04e_0
- google-cloud-sdk=454.0.0=py311h267d04e_0
- greenlet=3.0.1=py311hbaf5611_0
- grpcio=1.57.0=py311h79dd126_2
- hpack=4.0.0=pyh9f0ad1d_0
@@ -182,8 +190,9 @@ dependencies:
- kiwisolver=1.4.5=py311he4fd1f5_1
- libblas=3.9.0=19_osxarm64_openblas
- libcurl=8.4.0=h2d989ff_0
- libpq=16.0=hcea71ed_1
- llvmlite=0.40.1=py311hea943cd_0
- libgd=2.3.3=hfdf3952_9
- libpq=16.1=hd435d45_0
- llvmlite=0.41.1=py311hf5d242d_0
- locket=1.0.0=pyhd8ed1ab_0
- lxml=4.9.3=py311hbafe683_1
- marko=1.3.1=pyhd8ed1ab_0
@@ -239,7 +248,7 @@ dependencies:
- rpds-py=0.12.0=py311h94f323b_0
- rtree=1.1.0=py311hd698ff7_0
- ruamel.yaml.clib=0.2.7=py311heffc1b2_2
- ruff=0.1.4=py311h6fc163c_0
- ruff=0.1.5=py311h6fc163c_0
- setuptools=68.2.2=pyhd8ed1ab_0
- shellingham=1.5.4=pyhd8ed1ab_0
- simpleeval=0.9.13=pyhd8ed1ab_1
@@ -275,7 +284,7 @@ dependencies:
- websockets=10.4=py311he2be06e_1
- wheel=0.41.3=pyhd8ed1ab_0
- widgetsnbextension=4.0.9=pyhd8ed1ab_0
- wrapt=1.15.0=py311heffc1b2_1
- wrapt=1.16.0=py311h05b510d_0
- xlrd=2.0.1=pyhd8ed1ab_3
- xlsxwriter=3.1.9=pyhd8ed1ab_0
- xyzservices=2023.10.1=pyhd8ed1ab_0
@@ -307,11 +316,12 @@ dependencies:
- graphql-core=3.2.3=pyhd8ed1ab_0
- h11=0.14.0=pyhd8ed1ab_0
- h2=4.1.0=pyhd8ed1ab_0
- harfbuzz=8.2.1=hf1a6348_0
- hdf5=1.14.2=nompi_h3aba7b3_100
- html5lib=1.1=pyh9f0ad1d_0
- hypothesis=6.88.3=pyha770c72_0
- importlib-metadata=6.8.0=pyha770c72_0
- importlib_resources=6.1.0=pyhd8ed1ab_0
- importlib_resources=6.1.1=pyhd8ed1ab_0
- isodate=0.6.1=pyhd8ed1ab_0
- janus=1.0.0=pyhd8ed1ab_0
- jaraco.classes=3.3.0=pyhd8ed1ab_0
@@ -335,7 +345,7 @@ dependencies:
- pint=0.22=pyhd8ed1ab_1
- pip=23.3.1=pyhd8ed1ab_0
- poppler=23.10.0=hcdd998b_0
- postgresql=16.0=h00cd704_1
- postgresql=16.1=hc6ab77f_0
- proj=9.3.0=h52fb9d0_2
- protobuf=4.23.4=py311h590876e_3
- psycopg2=2.9.7=py311h589e011_1
@@ -369,8 +379,8 @@ dependencies:
- arrow=1.3.0=pyhd8ed1ab_0
- async-timeout=4.0.3=pyhd8ed1ab_0
- aws-crt-cpp=0.24.2=hbe5f050_2
- botocore=1.31.78=pyhd8ed1ab_0
- branca=0.6.0=pyhd8ed1ab_0
- botocore=1.31.83=pyhd8ed1ab_0
- branca=0.7.0=pyhd8ed1ab_1
- croniter=2.0.1=pyhd8ed1ab_0
- cryptography=41.0.5=py311h71175c2_0
- fqdn=1.5.1=pyhd8ed1ab_0
@@ -381,31 +391,33 @@ dependencies:
- gql=3.4.1=pyhd8ed1ab_0
- graphql-relay=3.2.0=pyhd8ed1ab_0
- grpcio-health-checking=1.57.0=pyhd8ed1ab_0
- httpcore=1.0.1=pyhd8ed1ab_0
- httpcore=1.0.2=pyhd8ed1ab_0
- importlib_metadata=6.8.0=hd8ed1ab_0
- jsonschema-specifications=2023.7.1=pyhd8ed1ab_0
- jupyter_server_terminals=0.4.4=pyhd8ed1ab_1
- kealib=1.5.2=h47b5e36_1
- libnetcdf=4.9.2=nompi_hb2fb864_112
- libspatialite=5.1.0=h32510b6_0
- mako=1.2.4=pyhd8ed1ab_0
- numpy=1.24.4=py311hb8f3215_0
- pbr=5.11.1=pyhd8ed1ab_0
- libspatialite=5.1.0=h5b80306_1
- mako=1.3.0=pyhd8ed1ab_0
- numpy=1.26.0=py311hb8f3215_0
- pango=1.50.14=hcf40dda_2
- pbr=6.0.0=pyhd8ed1ab_0
- pendulum=2.1.2=py311heffc1b2_6
- platformdirs=3.11.0=pyhd8ed1ab_0
- psycopg2-binary=2.9.7=pyhd8ed1ab_1
- pybtex=0.24.0=pyhd8ed1ab_2
- pydantic=1.10.13=py311h05b510d_1
- pyobjc-framework-cocoa=10.0=py311hb702dc4_1
- pyproj=3.6.1=py311h20a9b75_3
- pyproj=3.6.1=py311h20a9b75_4
- pytest-console-scripts=1.4.1=pyhd8ed1ab_0
- pytest-cov=4.1.0=pyhd8ed1ab_0
- pytest-mock=3.12.0=pyhd8ed1ab_0
- pytest-xdist=3.3.1=pyhd8ed1ab_0
- python-build=1.0.3=pyhd8ed1ab_0
- requests=2.31.0=pyhd8ed1ab_0
- rich=13.6.0=pyhd8ed1ab_0
- stack_data=0.6.2=pyhd8ed1ab_0
- starlette=0.31.1=pyhd8ed1ab_0
- starlette=0.32.0.post1=pyhd8ed1ab_0
- tiledb=2.16.3=he15c4da_3
- ukkonen=1.0.1=py311he4fd1f5_4
- uvicorn=0.24.0=py311h267d04e_0
@@ -423,10 +435,11 @@ dependencies:
- dask-core=2023.10.1=pyhd8ed1ab_0
- dnspython=2.4.2=pyhd8ed1ab_1
- ensureconda=1.4.3=pyhd8ed1ab_0
- folium=0.14.0=pyhd8ed1ab_0
- folium=0.15.0=pyhd8ed1ab_0
- google-resumable-media=2.6.0=pyhd8ed1ab_0
- graphene=3.3=pyhd8ed1ab_0
- grpcio-status=1.57.0=pyhd8ed1ab_0
- gtk2=2.24.33=h57013de_2
- h3-py=3.7.6=py311ha891d26_1
- httpx=0.25.1=pyhd8ed1ab_0
- identify=2.5.31=pyhd8ed1ab_0
@@ -435,11 +448,12 @@ dependencies:
- jupyter_core=5.5.0=py311h267d04e_0
- keyring=24.2.0=py311h267d04e_1
- libgdal=3.7.3=h116f65a_2
- numba=0.57.1=py311hbf3c4e2_0
- librsvg=2.56.3=h0db3404_0
- numba=0.58.1=py311h9ec4793_0
- numexpr=2.8.7=py311h6e08293_4
- oauthlib=3.2.2=pyhd8ed1ab_0
- pandas=2.1.2=py311h6e08293_0
- prompt-toolkit=3.0.39=pyha770c72_0
- prompt-toolkit=3.0.40=pyha770c72_0
- pybtex-docutils=1.0.3=py311h267d04e_1
- pyopenssl=23.3.0=pyhd8ed1ab_0
- readthedocs-sphinx-ext=2.2.3=pyhd8ed1ab_0
@@ -454,9 +468,9 @@ dependencies:
- typer=0.9.0=pyhd8ed1ab_0
- uvicorn-standard=0.24.0=ha1ab1f8_0
- virtualenv=20.24.6=pyhd8ed1ab_0
- boto3=1.28.78=pyhd8ed1ab_0
- boto3=1.28.83=pyhd8ed1ab_0
- cachecontrol-with-filecache=0.13.1=pyhd8ed1ab_0
- dagster=1.5.6=pyhd8ed1ab_1
- dagster=1.5.7=pyhd8ed1ab_0
- datasette=0.64.4=pyhd8ed1ab_1
- doc8=1.1.1=pyhd8ed1ab_0
- email-validator=2.1.0.post1=pyhd8ed1ab_0
@@ -465,49 +479,51 @@ dependencies:
- geopandas-base=0.14.0=pyha770c72_1
- google-auth=2.23.4=pyhca7485f_0
- gql-with-requests=3.4.1=pyhd8ed1ab_0
- graphviz=8.1.0=h10878c0_0
- jsonschema-with-format-nongpl=4.19.2=pyhd8ed1ab_0
- jupyter_client=8.5.0=pyhd8ed1ab_0
- jupyter_client=8.6.0=pyhd8ed1ab_0
- libarrow=13.0.0=hf4fadc0_7_cpu
- matplotlib-base=3.8.1=py311hfdba5f6_0
- nbformat=5.9.2=pyhd8ed1ab_0
- pandera-core=0.17.2=pyhd8ed1ab_0
- pandera-core=0.17.2=pyhd8ed1ab_1
- pre-commit=3.5.0=pyha770c72_0
- prompt_toolkit=3.0.39=hd8ed1ab_0
- prompt_toolkit=3.0.40=hd8ed1ab_0
- requests-oauthlib=1.3.1=pyhd8ed1ab_0
- scikit-learn=1.3.2=py311ha25ca4d_1
- timezonefinder=6.2.0=py311h05b510d_2
- catalystcoop.ferc_xbrl_extractor=1.2.1=pyhd8ed1ab_0
- conda-lock=2.4.2=pyhd8ed1ab_0
- dagster-graphql=1.5.6=pyhd8ed1ab_1
- dagster-postgres=1.5.6=pyhd8ed1ab_1
- dagster-graphql=1.5.7=pyhd8ed1ab_0
- dagster-postgres=0.21.7=pyhd8ed1ab_0
- fiona=1.9.5=py311h45231e3_0
- google-api-core=2.12.0=pyhd8ed1ab_0
- google-api-core=2.13.0=pyhd8ed1ab_0
- google-auth-oauthlib=1.1.0=pyhd8ed1ab_0
- ipython=8.17.2=pyh31c8845_0
- jupyter_events=0.8.0=pyhd8ed1ab_0
- jupyter_events=0.9.0=pyhd8ed1ab_0
- mapclassify=2.6.1=pyhd8ed1ab_0
- nbclient=0.8.0=pyhd8ed1ab_0
- pyarrow=13.0.0=py311hd7bc329_7_cpu
- pygraphviz=1.11=py311h5178850_1
- recordlinkage=0.16=pyhd8ed1ab_0
- tabulator=1.53.5=pyhd8ed1ab_0
- dagster-webserver=1.5.6=pyhd8ed1ab_1
- dagster-webserver=1.5.7=pyhd8ed1ab_0
- geopandas=0.14.0=pyhd8ed1ab_1
- google-cloud-core=2.3.3=pyhd8ed1ab_0
- ipykernel=6.26.0=pyh3cd1d5f_0
- ipywidgets=8.1.1=pyhd8ed1ab_0
- nbconvert-core=7.10.0=pyhd8ed1ab_0
- nbconvert-core=7.11.0=pyhd8ed1ab_0
- tableschema=1.19.3=pyh9f0ad1d_0
- datapackage=1.15.2=pyh44b312d_0
- google-cloud-storage=2.13.0=pyhca7485f_0
- jupyter_console=6.6.3=pyhd8ed1ab_0
- jupyter_server=2.9.1=pyhd8ed1ab_0
- nbconvert-pandoc=7.10.0=pyhd8ed1ab_0
- jupyter_server=2.10.0=pyhd8ed1ab_0
- nbconvert-pandoc=7.11.0=pyhd8ed1ab_0
- qtconsole-base=5.5.0=pyha770c72_0
- gcsfs=2023.10.0=pyhd8ed1ab_0
- jupyter-lsp=2.2.0=pyhd8ed1ab_0
- jupyter-resource-usage=1.0.1=pyhd8ed1ab_0
- jupyterlab_server=2.25.0=pyhd8ed1ab_0
- nbconvert=7.10.0=pyhd8ed1ab_0
- jupyterlab_server=2.25.1=pyhd8ed1ab_0
- nbconvert=7.11.0=pyhd8ed1ab_0
- notebook-shim=0.2.3=pyhd8ed1ab_0
- jupyterlab=4.0.8=pyhd8ed1ab_0
- notebook=7.0.6=pyhd8ed1ab_0
1 change: 0 additions & 1 deletion hashlog

This file was deleted.

66 changes: 31 additions & 35 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -13,45 +13,68 @@ license = { file = "LICENSE.txt" }
dependencies = [
"addfips>=0.4",
"alembic>=1.12",
"black>=23",
"boto3>=1.28.55",
"bottleneck>=1.3.4", # pandas[performance]
"build>=1.0",
"catalystcoop.dbfread>=3.0,<3.1",
"catalystcoop.ferc-xbrl-extractor>=1.2.0,<2",
"coloredlogs>=14.0", # Dagster requires 14.0
"dagster-postgres",
"dagster-webserver>=1.5.6",
"conda-lock>=2.4",
"coverage>=7",
"dagster>=1.5.6",
"dagster-postgres>=0.21.6,<1", # Update when dagster-postgres graduates to 1.x
"dask>=2023",
"datapackage>=1.15,<2", # Transition datastore to use frictionless.
"datasette>=0.64",
"doc8>=1.1",
"email-validator>=1.0.3", # pydantic[email]
"fsspec>=2023",
"furo>=2022.4.7",
"gcsfs>=2023",
"geopandas>=0.14",
"grpcio==1.57.0", # Required by dagster, binary dependencies are flaky
"grpcio-health-checking==1.57.0", # Required by dagster, binary dependencies are flaky
"grpcio-status==1.57.0", # Required by dagster, binary dependencies are flaky
"hypothesis>=6.87,<7.0",
"jinja2>=3.1",
"jupyter",
"matplotlib>=3.8",
"nbconvert>=7",
"nbformat>=5.9",
"networkx>=3.2",
"numba>=0.57", # pandas[performance]
"numba>=0.58", # pandas[performance]
"numexpr>=2.8.0", # pandas[performance]
"numpy>=1.24,<2.0a0",
"numpy>=1.26,<2.0a0",
"openpyxl>=3.0.10", # pandas[excel]
"pandas>=2.1",
"pandera>=0.17.2",
"pre-commit>=3",
"pyarrow>=13", # pandas[parquet]
"pydantic>=1.10,<2",
# Required after pandera-core is retired and we switch to pydantic v2
#"pydantic>=2.4",
#"pydantic-settings>=2",
"pytest>=7.4",
"pytest-cov>=4.1",
"pytest-console-scripts>=1.4",
"pytest-mock>=3.11",
"pytest-xdist>=3.3",
"python-dotenv>=1",
"pyxlsb>=1.0.9", # pandas[excel]
"pyyaml>=6",
"readthedocs-sphinx-ext>=2",
"recordlinkage",
"responses>=0.23",
"ruff>=0.1",
"scikit-learn>=1.3",
"scipy>=1.11",
"Shapely>=2",
"sphinx-autoapi>=3",
"sphinx-issues>=1.2",
"sphinx-reredirects>=0.1.2",
"sphinx>=7",
"sphinxcontrib_bibtex>=2.6",
"sqlalchemy>=1.4,<2",
"timezonefinder>=6.2",
"xlrd>=2.0.1", # pandas[excel]
@@ -121,40 +144,11 @@ pudl_check_fks = "pudl.etl.check_foreign_keys:main"

[project.optional-dependencies]
dev = [
"black>=23",
"build>=1.0",
"conda-lock>=2.3",
"dagster-webserver>=1.5.6",
"jupyterlab>3.2",
"jupyter-resource-usage",
"nbconvert>=7",
"prettier>=3.0",
"ruff>=0.1",
]
docs = [
"doc8>=1.1",
"furo>=2022.4.7",
"readthedocs-sphinx-ext>=2",
"sphinx-autoapi>=3",
"sphinx-issues>=1.2",
"sphinx-reredirects>=0.1.2",
"sphinx>=7",
"sphinxcontrib_bibtex>=2.6",
]
test = [
"astroid>=3", # sphinx-autoapi dependency
"coverage>=7",
"jupyter",
"nbconvert>=7",
"nbformat>=5.9",
"pre-commit>=3",
"pytest-console-scripts>=1.4",
"pytest-cov>=4.1",
"pytest-mock>=3.11",
"pytest>=7.4",
"responses>=0.23",
"ruff>=0.1",
"pygraphviz",
]
datasette = ["datasette>=0.64"]

[tool.setuptools]
include-package-data = true
@@ -269,6 +263,7 @@ filterwarnings = [
"once:In a future version:FutureWarning:pudl.helpers",
"once:open_binary is deprecated:DeprecationWarning:pudl.glue.ferc1_eia",
"once:open_text is deprecated:DeprecationWarning:pudl.glue.ferc1_eia",
"once:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated.:FutureWarning",
]

addopts = "--verbose --pdbcls=IPython.terminal.debugger:TerminalPdb"
@@ -299,5 +294,6 @@ google-cloud-sdk = ">=452"
nodejs = ">=20"
pandoc = ">=2"
pip = ">=23"
prettier = ">=3.0"
python = ">=3.11,<3.12"
sqlite = ">=3.43"
2 changes: 1 addition & 1 deletion src/pudl/etl/glue_assets.py
Original file line number Diff line number Diff line change
@@ -591,7 +591,7 @@ def update_subplant_ids(subplant_crosswalk: pd.DataFrame) -> pd.DataFrame:
x.subplant_id_connected
+ x.groupby(
["plant_id_eia"], dropna=False
).unit_id_pudl_connected.transform(max)
).unit_id_pudl_connected.transform("max")
)
),
# create a new unique subplant_id based on the connected subplant ids and the
2 changes: 1 addition & 1 deletion src/pudl/ferc_to_sqlite/cli.py
Original file line number Diff line number Diff line change
@@ -48,7 +48,7 @@ def parse_command_line(argv):
"--clobber",
action="store_true",
help="""Clobber existing sqlite database if it exists. If clobber is
not included but the sqlite databse already exists the _build will
not included but the sqlite databse already exists the build will
fail.""",
default=False,
)
1 change: 0 additions & 1 deletion src/pudl/output/ferc714.py
Original file line number Diff line number Diff line change
@@ -631,7 +631,6 @@ def georeferenced_respondents_ferc714(
def summarized_demand_ferc714(
annualized_respondents_ferc714: pd.DataFrame,
demand_hourly_pa_ferc714: pd.DataFrame,
fipsified_respondents_ferc714: pd.DataFrame,
categorized_respondents_ferc714: pd.DataFrame,
georeferenced_counties_ferc714: gpd.GeoDataFrame,
) -> pd.DataFrame:

0 comments on commit 7defd4d

Please sign in to comment.