Code

Merge dev into main for 2023-11-09#3031

Code

Merged

jdangerx merged 63 commits intomaincatalyst-cooperative/pudl:mainfrom nightly-build-2023-11-09catalyst-cooperative/pudl:nightly-build-2023-11-09Copy head branch name to clipboardNov 9, 2023

+758-445

Commits

Commits on Oct 12, 2023

• First pass of integrating the monthly EIA923 data into the rest of the EIA data. This includes updating the package data to account for the 2023 year and updating the way to assign data maturities …

  Show description for 6cd7712
  aesharpe

  committed
  6cd7712

  Copy full SHA for 6cd7712

• remove breakpoint

  aesharpe

  committed
  681b72d

  Copy full SHA for 681b72d

• Merge branch 'dev' into add-data-maturity-for-923m

  aesharpe

  committed
  5899fc0

  Copy full SHA for 5899fc0

• Add function to drop ytd records for annual tables

  aesharpe

  committed
  2bff97d

  Copy full SHA for 2bff97d

• Adjust monthly row expectations for gf and frc tables after dropping ytd values for annual tables

  aesharpe

  committed
  796b5da

  Copy full SHA for 796b5da

Commits on Oct 16, 2023

• Tweak the way we add data maturity to the eia923 monthly files and remove double returns from the drop_ytd_for_annual_tables function

  aesharpe

  committed
  77e052e

  Copy full SHA for 77e052e

Commits on Oct 17, 2023

• Merge branch 'dev' into add-data-maturity-for-923m

  aesharpe

  committed
  44b4d44

  Copy full SHA for 44b4d44

Commits on Oct 24, 2023

• Merge with dev

  aesharpe

  committed
  63dee8e

  Copy full SHA for 63dee8e

Commits on Oct 25, 2023

• Litle updates:

  Show description for b50ea1d
  aesharpe

  committed
  b50ea1d

  Copy full SHA for b50ea1d

• For now, comment out the checks that make sure we have the same years of EIA923 and EIA860 data. This is causing issues for the monthly EIA923 data that gets integrated ahead of any available 860 d…

  Show description for 572147c
  aesharpe

  committed
  572147c

  Copy full SHA for 572147c

• Update min max rows

  aesharpe

  committed
  aa42b28

  Copy full SHA for aa42b28

• Add data_maturity field to harvested EIA tables so that we can drop ytd records from annual EIA tables

  aesharpe

  committed
  a588420

  Copy full SHA for a588420

Commits on Oct 27, 2023

• Address PR comments:

  Show description for 1d267ab
  aesharpe

  committed
  1d267ab

  Copy full SHA for 1d267ab

• Merge branch 'dev' into add-data-maturity-for-923m

  aesharpe

  committed
  8ccc44c

  Copy full SHA for 8ccc44c

Commits on Oct 30, 2023

• Fix release note trailing whitespace error

  aesharpe

  committed
  b3c11c2

  Copy full SHA for b3c11c2

• Update test_eia923_dependency function to make sure some 860 and 923 years overlap but don't need to be the same

  aesharpe

  committed
  3b35892

  Copy full SHA for 3b35892

Commits on Oct 31, 2023

• Only generate alphanumeric entity IDs in test - non-printable characters seem to break groupby. (#2993)

  jdangerx

  authored
  47304b3

  Copy full SHA for 47304b3

• Set up Cloud SQL postgres database for dagster storage

  bendnorman

  committed
  dfea5ef

  Copy full SHA for dfea5ef

• Copy dagster.yaml after DAGSTER_HOME is created

  bendnorman

  committed
  05901ab

  Copy full SHA for 05901ab

• Add proper quoting rules to DAGSTER_PG_PASSWORD secret

  bendnorman

  committed
  02c52f6

  Copy full SHA for 02c52f6

Commits on Nov 1, 2023

• Use max cpus for nightly builds and spin dagster-storage SQL instance up and down

  bendnorman

  committed
  dcfb8de

  Copy full SHA for dcfb8de

• Create and delete Cloud SQL db during nightly builds

  bendnorman

  committed
  ad8442d

  Copy full SHA for ad8442d

• Set PUDL_SETTINGS_YML to etl_full.yml and add git sha to Cloud SQL database name

  bendnorman

  committed
  ab9425a

  Copy full SHA for ab9425a

• Add short github ref to database name

  bendnorman

  committed
  6592fad

  Copy full SHA for 6592fad

• Update DAGSTER_PG_DB with short git sha

  bendnorman

  committed
  2f85315

  Copy full SHA for 2f85315

• Update date range for nightly build links to include 2022

  zaneselvans

  committed
  8c60657

  Copy full SHA for 8c60657

• Update 923 settings files to accomodate 2023 data and update settings tests so that they aren't dependent on having the same years of EIA923 and EIA860 data

  aesharpe

  committed
  24f6b8d

  Copy full SHA for 24f6b8d

• Fix calculating the report_date in demand_hourly_pa_ferc714

  rousik

  committed
  45f2d02

  Copy full SHA for 45f2d02

• Require non-null report_date in FERC 714 hourly demand table.

  zaneselvans

  committed
  bf77d1a

  Copy full SHA for bf77d1a

• Update date validation function to only look at instances where data_maturity is not ytd_incremental

  aesharpe

  committed
  b376141

  Copy full SHA for b376141

• Merge pull request #2936 from catalyst-cooperative/add-data-maturity-for-923m

  Show description for dbf686f
  aesharpe

  authored
  dbf686f

  Copy full SHA for dbf686f

• Fix calculating the report_date in demand_hourly_pa_ferc714

  Show description for e848d64
  rousik

  authored
  e848d64

  Copy full SHA for e848d64

• Merge branch 'dev' into setup-dagster-postgres

  bendnorman

  committed
  1c6dd24

  Copy full SHA for 1c6dd24

• Remove Cloud SQL lifecycle management from gcp_pudl_etl.sh script

  bendnorman

  committed
  5b33500

  Copy full SHA for 5b33500

Commits on Nov 2, 2023

• Merge branch 'dev' into setup-dagster-postgres

  bendnorman

  authored
  c15e063

  Copy full SHA for c15e063

• Update data contributors, add zenodo role and doi field, update US copyright link

  e-belfer

  committed
  633b63d

  Copy full SHA for 633b63d

• Update to ZenodoDoi class, update to https

  e-belfer

  committed
  4b028d2

  Copy full SHA for 4b028d2

• Remove leftover string

  e-belfer

  committed
  6e6bf5e

  Copy full SHA for 6e6bf5e

• Merge pull request #3004 from catalyst-cooperative/datapackage-update

  Show description for e7aedda
  e-belfer

  authored
  e7aedda

  Copy full SHA for e7aedda

• Switch regex strategy to sampling strategy to improve performance (#2998)

  Show description for c11b1a4
  jdangerx

  authored
  c11b1a4

  Copy full SHA for c11b1a4

• add alembic schema changes for the recent constraint.

  rousik

  committed
  a79edeb

  Copy full SHA for a79edeb

• Merge pull request #3012 from catalyst-cooperative/fix-alembic-schema

  Show description for 0fa0c85
  rousik

  authored
  0fa0c85

  Copy full SHA for 0fa0c85

Commits on Nov 3, 2023

• Merge pull request #2996 from catalyst-cooperative/setup-dagster-postgres

  Show description for 18bb60f
  bendnorman

  authored
  18bb60f

  Copy full SHA for 18bb60f

• only fix a reporting_frequency_code when the column exists

  cmgosnell

  committed
  ef20339

  Copy full SHA for ef20339

Commits on Nov 6, 2023

• Update responses requirement from <0.24,>=0.14 to >=0.14,<0.25

  Show description for 2848e01
  dependabot[bot]

  authored
  2848e01

  Copy full SHA for 2848e01

• Update pyarrow requirement from <14,>=13 to >=13,<15

  Show description for 51ed7e5
  dependabot[bot]

  authored
  51ed7e5

  Copy full SHA for 51ed7e5

• Update dagster-postgres requirement

  Show description for c2e7889
  dependabot[bot]

  authored
  c2e7889

  Copy full SHA for c2e7889

• Merge pull request #3015 from catalyst-cooperative/dependabot/pip/dev/responses-gte-0.14-and-lt-0.25

  Show description for 1ed333c
  zaneselvans

  authored
  1ed333c

  Copy full SHA for 1ed333c

• Merge pull request #3016 from catalyst-cooperative/dependabot/pip/dev/pyarrow-gte-13-and-lt-15

  Show description for 039116a
  zaneselvans

  authored
  039116a

  Copy full SHA for 039116a

• Merge pull request #3017 from catalyst-cooperative/dependabot/pip/dev/dagster-postgres-gte-0.21.5-and-lt-0.21.7

  Show description for 71ee2bd
  zaneselvans

  authored
  71ee2bd

  Copy full SHA for 71ee2bd

• [pre-commit.ci] pre-commit autoupdate

  Show description for aea60b9
  pre-commit-ci[bot]

  authored
  aea60b9

  Copy full SHA for aea60b9

• update tox and eia923 rows

  cmgosnell

  committed
  acbf788

  Copy full SHA for acbf788

• Merge pull request #3019 from catalyst-cooperative/pre-commit-ci-update-config

  Show description for 51be942
  zaneselvans

  authored
  51be942

  Copy full SHA for 51be942

• Merge branch 'dev' into fix-reporting_frequency_code

  zaneselvans

  authored
  5a96335

  Copy full SHA for 5a96335

Commits on Nov 7, 2023

• Merge pull request #3013 from catalyst-cooperative/fix-reporting_frequency_code

  Show description for f251def
  zaneselvans

  authored
  f251def

  Copy full SHA for f251def

• update excepted rows for no-fips id-ed respondents but keep annualized demand

  cmgosnell

  committed
  e4c1c46

  Copy full SHA for e4c1c46

• add report year validation test

  cmgosnell

  committed
  1e2857f

  Copy full SHA for 1e2857f

• add minmax rows into validation test for chonky table

  cmgosnell

  committed
  f25e921

  Copy full SHA for f25e921

• Merge pull request #3023 from catalyst-cooperative/ferc714_mystery_date_fix

  Show description for fcf4ccc
  cmgosnell

  authored
  fcf4ccc

  Copy full SHA for fcf4ccc

• idk exactly why the "nan"s began existing but this fixes it

  cmgosnell

  committed
  095d31b

  Copy full SHA for 095d31b

• revert the replace of "nan" by stopping introducing them! plus some light clean up

  cmgosnell

  committed
  14e6bc8

  Copy full SHA for 14e6bc8

Commits on Nov 8, 2023

• REALLY REALLY its a nullable string

  cmgosnell

  committed
  b7533cf

  Copy full SHA for b7533cf

• Merge pull request #3025 from catalyst-cooperative/fix_mystery_fuel_cat_nan

  Show description for a2bdffa
  cmgosnell

  authored
  a2bdffa

  Copy full SHA for a2bdffa