-
-
Notifications
You must be signed in to change notification settings - Fork 120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dbt setup #4011
base: main
Are you sure you want to change the base?
Dbt setup #4011
Changes from 1 commit
bf40ffb
9aac625
415a113
ba32bd8
dc51c8f
590b02a
63e663a
d428b5d
784cf96
48a16e1
6f45ba5
ac41a41
0ce1648
c19cfd8
6335e94
2585eca
e24af8c
1ed85b3
e92f5be
5de9ebe
da9ae93
7461786
0d120c6
3666360
a3579dc
c98219c
f9b3fa7
79e2153
012ba4a
ff766b3
94267a5
8f660fd
70e6895
389c540
eb0765a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,12 +1,22 @@ | ||
version: 2 | ||
|
||
sources: | ||
- name: pudl_nightly | ||
- name: pudl | ||
meta: | ||
external_location: "https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/{name}.parquet" | ||
external_location: | | ||
{%- if target.name == "nightly" -%} 'https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/{name}.parquet' | ||
{%- else -%} '{{ env_var('PUDL_OUTPUT') }}/parquet/{name}.parquet' | ||
{%- endif -%} | ||
tables: | ||
- name: out_vcerare__hourly_available_capacity_factor | ||
columns: | ||
- name: capacity_factor_solar_pv | ||
tests: | ||
data_tests: | ||
- not_null | ||
- dbt_expectations.expect_column_max_to_be_between: | ||
max_value: 1.02 | ||
- name: capacity_factor_offshore_wind | ||
data_tests: | ||
- not_null | ||
- dbt_expectations.expect_column_max_to_be_between: | ||
max_value: 1.00 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
packages: | ||
- package: calogica/dbt_expectations | ||
version: 0.10.4 | ||
- package: dbt-labs/dbt_utils | ||
version: 1.3.0 | ||
- package: calogica/dbt_date | ||
version: 0.10.1 | ||
sha1_hash: 29571f46f50e6393ca399c3db7361c22657f2d6b |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
packages: | ||
- package: calogica/dbt_expectations | ||
version: [">=0.10.0", "<0.11.0"] | ||
- package: dbt-labs/dbt_utils | ||
version: [">=1.3.0", "<1.4.0"] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see neither of these are available in |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,9 +1,17 @@ | ||
pudl_dbt: | ||
outputs: | ||
dev: | ||
# Define targets for nightly builds, and local ETL full/fast | ||
# See models/schema.yml for further configuration | ||
nightly: | ||
type: duckdb | ||
path: /tmp/pudl.duckdb | ||
filesystems: | ||
- fs: s3 | ||
etl-full: | ||
type: duckdb | ||
path: /tmp/pudl.duckdb | ||
etl-fast: | ||
type: duckdb | ||
path: /tmp/pudl.duckdb | ||
|
||
target: dev | ||
target: nightly |
This file was deleted.
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it also work to point this directly at S3 rather than going through the HTTPS interface?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed this to
s3://pudl.catalyst.coop/nightly/{name}.parquet
and it seems to work. I think going throughs3://
directly will probably be more performant, won't it? E.g. in the case where there are efficiencies to be had in querying only small portions of the larger Parquet files.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interestingly, with the
s3://
URL it didn't give me any error, but it also didn't seem to be making much progress. There was just a ton of data being downloaded. Not sure why.