Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dbt setup #4011

Draft
wants to merge 35 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
bf40ffb
Add basic dbt setup
zschira Jan 9, 2025
9aac625
Update to dagster 1.9.7 & grpcio 1.67.1
zaneselvans Jan 10, 2025
415a113
Setup multiple dbt profiles
zschira Jan 10, 2025
ba32bd8
Merge remote-tracking branch 'refs/remotes/origin/dbt_setup' into dbt…
zaneselvans Jan 10, 2025
dc51c8f
Add all vcerare dbt tests
zschira Jan 10, 2025
590b02a
Add more example dbt tests
zschira Jan 13, 2025
63e663a
Merge branch 'dbt_setup' of github.com:catalyst-cooperative/pudl into…
zaneselvans Jan 13, 2025
d428b5d
Merge changes from main and revert to python 3.12
zaneselvans Jan 13, 2025
784cf96
Bump gdal to v3.10.1 bugfix release.
zaneselvans Jan 14, 2025
48a16e1
Merge branch 'main' into dbt_setup
zaneselvans Jan 14, 2025
6f45ba5
Merge branch 'main' into dbt_setup
zaneselvans Jan 15, 2025
ac41a41
Update to dagster 1.9.9
zaneselvans Jan 19, 2025
0ce1648
Merge branch 'main' into dbt_setup
zaneselvans Jan 20, 2025
c19cfd8
Merge branch 'main' into dbt_setup
zaneselvans Jan 20, 2025
6335e94
Reorganize dbt into multiple schema.yml files
zschira Jan 21, 2025
2585eca
Merge branch 'dbt_setup' of github.com:catalyst-cooperative/pudl into…
zschira Jan 21, 2025
e24af8c
Move dbt project to top level of repo
zschira Jan 22, 2025
1ed85b3
Only set parquet path in dbt project once
zschira Jan 30, 2025
e92f5be
Standardize dbt maning scheme
zschira Jan 30, 2025
5de9ebe
Add more detail to README
zschira Jan 30, 2025
da9ae93
Add script to generate dbt scaffolding and row count tests
zschira Feb 5, 2025
7461786
Add documentation for dbt helper script
zschira Feb 5, 2025
0d120c6
Add out_ferc1__yearly_steam_plants_fuel_by_plant_sched402 to yearly r…
zschira Feb 5, 2025
3666360
Add weighted quantile test (broken)
zschira Feb 5, 2025
a3579dc
Change row count test name
zschira Feb 5, 2025
c98219c
Update dbt initialization process
zschira Feb 5, 2025
f9b3fa7
Make dbt helper script work properly with non-yearly partitioned tables
zschira Feb 5, 2025
79e2153
Update dbt readme
zschira Feb 5, 2025
012ba4a
Regenerate ferc dbt schemas
zschira Feb 6, 2025
ff766b3
Merge branch 'main' into dbt_setup
zaneselvans Feb 10, 2025
94267a5
Improve dbt_helper command line usability
zschira Feb 10, 2025
8f660fd
Merge branch 'dbt_setup' of github.com:catalyst-cooperative/pudl into…
zschira Feb 10, 2025
70e6895
Flesh out test migration command
zschira Feb 13, 2025
389c540
Add test migration tutorial
zschira Feb 14, 2025
eb0765a
Merge branch 'main' into dbt_setup
zschira Feb 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge changes from main and revert to python 3.12
zaneselvans committed Jan 13, 2025
commit d428b5dc15c49b68f3cf62b2258026ebbf54b6e8
56 changes: 31 additions & 25 deletions environments/conda-linux-64.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

761 changes: 487 additions & 274 deletions environments/conda-lock.yml

Large diffs are not rendered by default.

56 changes: 31 additions & 25 deletions environments/conda-osx-64.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

56 changes: 31 additions & 25 deletions environments/conda-osx-arm64.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 11 additions & 11 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ name = "catalystcoop.pudl"
description = "An open data processing pipeline for US energy data"
readme = { file = "README.rst", content-type = "text/x-rst" }
authors = [{ name = "Catalyst Cooperative", email = "pudl@catalyst.coop" }]
requires-python = ">=3.12,<3.13"
requires-python = ">=3.12,<3.13.0a0"
zaneselvans marked this conversation as resolved.
Show resolved Hide resolved
dynamic = ["version"]
license = { file = "LICENSE.txt" }
dependencies = [
@@ -22,9 +22,9 @@ dependencies = [
"coloredlogs>=14.0", # Dagster requires 14.0
"conda-lock>=2.5.7",
"coverage>=7.6",
"dagster>=1.9.7",
"dagster-dbt>=0.25.7,<1",
"dagster-postgres>=0.25.7,<1", # Update when dagster-postgres graduates to 1.x
"dagster>=1.9.8",
"dagster-dbt>=0.25.8,<1",
"dagster-postgres>=0.25.8,<1", # Update when dagster-postgres graduates to 1.x
zaneselvans marked this conversation as resolved.
Show resolved Hide resolved
"dask>=2024",
"dask-expr", # Required for dask[dataframe]
"datasette>=0.65",
@@ -41,11 +41,11 @@ dependencies = [
"grpcio==1.67.1", # Required by dagster, binary dependencies are flaky
"grpcio-health-checking==1.67.1", # Required by dagster, binary dependencies are flaky
"grpcio-status==1.67.1", # Required by dagster, binary dependencies are flaky
"hypothesis>=6.110",
"hypothesis>=6.123",
"jellyfish>=1",
"jinja2>=3.1",
"jupyter",
"matplotlib>=3.9",
"matplotlib>=3.10",
"mlflow>=2.19",
"nbconvert>=7",
"nbformat>=5.10",
@@ -55,7 +55,7 @@ dependencies = [
"numpy>=2",
"packaging>=24",
"pandas>=2.2.3",
"pandera>=0.20",
"pandera>=0.22",
"pre-commit>=3",
"pyarrow>=18.1", # pandas[parquet]
"pydantic>=2.10",
@@ -72,7 +72,7 @@ dependencies = [
"readthedocs-sphinx-ext>=2",
"requests>=2.32",
"responses>=0.25",
"ruff>=0.8,<0.9",
"ruff>=0.9",
"scikit-learn>=1.6",
"scipy>=1.15",
"Shapely>=2",
@@ -150,8 +150,8 @@ pudl_service_territories = "pudl.analysis.service_territory:pudl_service_territo

[project.optional-dependencies]
dev = [
"dagster-webserver>=1.9.7",
"jupyterlab>4.1",
"dagster-webserver>=1.9",
"jupyterlab>4.3",
"jupyter-lsp",
"jupyterlab-lsp",
"ruff-lsp",
@@ -346,7 +346,7 @@ nodejs = ">=20"
pandoc = ">=2"
pip = ">=24"
prettier = ">=3.0"
python = ">=3.12,<3.13"
python = ">=3.12,<3.13.0a0"
sqlite = ">=3.47"
zip = ">=3.0"

You are viewing a condensed version of this merge commit. You can view the full changes here.