Dbt setup #4011

zschira · 2025-01-13T19:30:47Z

Overview

This PR sets up a dbt project within the PUDL repo that will be used for data testing. Details on setup and usage can all be found in the Readme.md. This PR also includes several data validations that have been converted to dbt tests. The tests currently converted are all vcerare asset_checks and FERC fuel by plant cost per mmbtu range checks.

In scope

Standardize organization and naming conventions
Develop some basic macros that will enable migrating existing tests
- Weighted quantile
- Review validate.py for any other obvious candidates
Auto-generate row count checks
Update readme with directions for migration (how to add new asset, types of test in scope for immediate migration)

Out of scope

Make tests run in asset checks
Plan alternative to complex tests like the existing vs_historical
Migrate tests

closes #3997 #3971

…_setup

… dbt_setup

src/pudl/dbt/.user.yml

src/pudl/dbt/.gitignore

pyproject.toml

zaneselvans · 2025-01-13T21:35:38Z

environments/conda-linux-64.lock.yml

-  - importlib-metadata=8.5.0=pyha770c72_1
+  - importlib-metadata=6.10.0=pyha770c72_0


For some reason dbt-semantic-interfaces is stuck between 6 and 7.

zaneselvans · 2025-01-13T21:38:34Z

environments/conda-linux-64.lock.yml

-  - isodate=0.7.2=pyhd8ed1ab_1
+  - isodate=0.6.1=pyhd8ed1ab_0


dbt-common is stuck between 0.6 and 0.7.

zaneselvans · 2025-01-13T22:16:24Z

src/pudl/dbt/models/schema.yml

+    tables:
+      - name: out_eia923__boiler_fuel
+      - name: out_eia923__monthly_boiler_fuel
+      - name: out_ferc1__yearly_steam_plants_fuel_by_plant_sched402
+      - name: out_vcerare__hourly_available_capacity_factor


In a data warehouse with hundreds of tables, would this file be created and managed by hand? Or would there be some rule-based way to generate it, or parts of it, along the lines of what we're doing with the Pandera schema checks right now? For example, the not_null tests here are a 2nd place that that restriction is being specified -- it's already present in our table metadata, which seems like recipe for them getting out of sync.

Or in the case of row counts, is there a clean, non-manual way to update the row counts to reflect whatever the currently observed counts are? Especially if we're trying to regenerate expected row counts for each individual year, filling it all in manually could be pretty tedious and error prone. We've moved toward specifying per-year row counts on the newer assets so that they work transparently in either the fast or full ETL cases, and the asset checks don't need to be aware of which kind of job they're being run in, which seems both more specific and more robust.

Looks like the "X column is not null" checks are currently defined in fields.py under the field constraints, is that what you're thinking about?

I think it would be nice to have auto-generated tests like the non-null tests & row counts defined alongside manually added tests. Then all the tests will be defined in one place, except for the tests that we need to write custom Python code for.

That seems pretty doable - YAML is easy to work with, and dbt lets us tag tests, so we could easily tag all the auto-generated tests so our generation scripts know to replace them but leave the manually-added tests alone.

In addition to the field specific constraints I think we automatically add NOT NULL check constraints to the PK fields when we construct the SQLite database -- but more generally I'm just saying that we need to get all of these generated tests integrated non-duplicatively into the dbt tests somehow.

It seems totally possible to auto-generate tests, but I think there's also probably many ways to do accomplish this, so we should figure out what we want from it. For example, when we talk about auto-generating row count/not null tests, will these be generated once and committed into the repo, or will some/all of them be dynamically generated at runtime?

It definitely seems tricky to minimize duplication between dbt/our existing python schema info. I also wonder how this plays into any refactoring of our metadata system?

It feels like we may need to clearly define the data tests that are ready to be migrated in a straightforward way, and the things that still need design work, so we can point Margay folks at the stuff that's ready to go and keep thinking about the things that still need some scaffolding?

zaneselvans · 2025-01-13T22:20:20Z

src/pudl/dbt/models/validation/ferc1_fbp_cost_per_mmbtu.sql

If we do end up needing to define these intermediate tables it seems like we would want to have some kind of clear naming convention for them?

Yeah I think that seems like a good idea. Maybe just use a validation_ prefix and otherwise follow existing naming conventions?

zaneselvans · 2025-01-13T22:22:56Z

src/pudl/dbt/models/schema.yml

+          - dbt_expectations.expect_compound_columns_to_be_unique:
+              column_list: ["county_id_fips", "datetime_utc"]


Could be generated based on the PK that's defined for every table?

Should be possible. We can also probably come up with a way to generate foreign key checks so we can actually verify foreign keys for tables only in parquet

zaneselvans · 2025-01-13T22:24:08Z

src/pudl/dbt/models/schema.yml

+          - dbt_expectations.expect_table_row_count_to_equal:
+              value: |
+                {%- if target.name == "etl-fast" -%} 27287400
+                {%- else -%} 136437000
+                {%- endif -%}


Is there a clean way to specify the expected row counts for each year of data (or some other meaningful subset) within a table, as we've started doing for the newer assets in Dagster asset checks, so we don't have to differentiate between fast and full validations, and can identify where the changes are?

We'd probably need to create a custom macro for this, but that seems totally doable. Big question is how we want to generate/store all of those tests.

The row count tests have functionally become regression tests -- we want to know when they change, and verify that the magnitude and nature of the change is expected based on the code or data that we've changed. Given that there are hundreds of tables (and thousands of table-years) it doesn't seem practical to hand-code all of the expected row counts.

It would be nice to have the per table-year row counts stored in (say) YAML somewhere, and be able to generate a new version of that file based on current ETL outputs. Then we could look at the diffs between the old and the new versions of the file when trying to assess changes in the lengths of the outputs.

zaneselvans · 2025-01-13T22:26:04Z

src/pudl/dbt/models/schema.yml

+          - dbt_expectations.expect_column_quantile_values_to_be_between:
+              quantile: 0.05
+              min_value: 1.5


I'm guessing these are not using the weighted quantiles?

Yeah this are just basic quantiles. It's not too hard to get a sql query that can do a version of weighted quantiles, but the existing vs_historical tests are hard because they're computing a bunch of quantiles, then comparing them all

… dbt_setup

zschira and others added 6 commits January 9, 2025 17:11

Add basic dbt setup

bf40ffb

Update to dagster 1.9.7 & grpcio 1.67.1

9aac625

Setup multiple dbt profiles

415a113

Merge remote-tracking branch 'refs/remotes/origin/dbt_setup' into dbt…

ba32bd8

…_setup

Add all vcerare dbt tests

dc51c8f

Add more example dbt tests

590b02a

zschira requested a review from zaneselvans January 13, 2025 19:30

zschira marked this pull request as draft January 13, 2025 19:31

Merge branch 'dbt_setup' of github.com:catalyst-cooperative/pudl into…

63e663a

… dbt_setup

zschira requested a review from e-belfer January 13, 2025 19:44

Merge changes from main and revert to python 3.12

d428b5d

zaneselvans reviewed Jan 13, 2025

View reviewed changes

zaneselvans and others added 9 commits January 14, 2025 10:44

Bump gdal to v3.10.1 bugfix release.

784cf96

Merge branch 'main' into dbt_setup

48a16e1

Merge branch 'main' into dbt_setup

6f45ba5

Update to dagster 1.9.9

ac41a41

Merge branch 'main' into dbt_setup

0ce1648

Merge branch 'main' into dbt_setup

c19cfd8

Reorganize dbt into multiple schema.yml files

6335e94

Merge branch 'dbt_setup' of github.com:catalyst-cooperative/pudl into…

2585eca

… dbt_setup

Move dbt project to top level of repo

e24af8c

jdangerx mentioned this pull request Jan 27, 2025

Prepare first batch of validation tests for migration #4030

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dbt setup #4011

Dbt setup #4011

zschira commented Jan 13, 2025 •

edited

Loading

zaneselvans Jan 13, 2025

zaneselvans Jan 13, 2025

zaneselvans Jan 13, 2025

jdangerx Jan 15, 2025

zaneselvans Jan 15, 2025

zschira Jan 22, 2025

zaneselvans Jan 22, 2025

zaneselvans Jan 13, 2025

zschira Jan 22, 2025

zaneselvans Jan 13, 2025

zschira Jan 22, 2025

zaneselvans Jan 13, 2025

zschira Jan 22, 2025

zaneselvans Jan 22, 2025

zaneselvans Jan 13, 2025

zschira Jan 22, 2025

		- importlib-metadata=8.5.0=pyha770c72_1
		- importlib-metadata=6.10.0=pyha770c72_0

		- dbt_expectations.expect_compound_columns_to_be_unique:
		column_list: ["county_id_fips", "datetime_utc"]

Dbt setup #4011

Are you sure you want to change the base?

Dbt setup #4011

Conversation

zschira commented Jan 13, 2025 • edited Loading

Overview

In scope

Out of scope

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zschira commented Jan 13, 2025 •

edited

Loading