Switch to new `flusurv` API endpoint #1278

nmdefries · 2023-08-23T17:21:59Z

Summary:

Pull flusurv data from new CDC API endpoint. Ingest previously unlabelled age groupings. Ingest new race/ethnicity and sex breakdowns. Test flusurv.py functions. Add more comments, messaging, and assertions.

Closes #1247
Closes #242

Note:

Some of our CDC contacts are able to provide us with past versions of data from the API. We will want to patch those in. Many of the "new" strata are actually available back to 2009. We'll probably want to patch those in as well.
This also requires updates to the database table, the API server, and the API docs that will be made separately.

Prerequisites:

Unless it is a documentation hotfix it should be merged against the dev branch
Branch is up-to-date with the branch to be merged with, i.e. dev
Build is successful
Code is cleaned up and formatted

- rename input arg to `update` to avoid reassignment later - comment and reuse args_insert - spelling - comment magic constant used in output format - rename location-network/catchmentid map

Previously, age strata were numbered sequentially which allowed us to store rate values by position in a list. With the introduction of the new strata, this system is not robust enough to track all the different groups (e.g. ageids are no longer sequential and there are now race and sex groupings with separate numbering systems).

…piweek

src/acquisition/flusurv/new_grasp_location_result.json

melange396

nice work! i really appreciate the thorough commenting!

youll wanna pull in the changes from the dev branch, PR #1241 added tests for the flusurv endpoint.

src/acquisition/flusurv/flusurv.py

src/acquisition/flusurv/flusurv_update.py

Co-authored-by: melange396 <[email protected]>

sonarqubecloud · 2023-11-07T22:49:02Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
9 Code Smells

No Coverage information
0.0% Duplication

sonarqubecloud · 2023-12-08T19:55:09Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
9 Code Smells

No Coverage information
0.0% Duplication

) * add new columns to flusurv endpoint spec * add new age, sex, race strata, and season to schema * list new signals in documentation * move new season column to end of table * add migration script to add new fields to flusurv table * add field descriptions; reformat table so don't need to match line lengths * point from acquisition script to docs and schema def to deduplicate

…oups" (#1426) This reverts commit 9de7d64.

sonarqubecloud · 2024-05-08T00:21:06Z

Quality Gate passed

Issues
7 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

melange396 · 2024-05-08T00:28:39Z

I wasnt paying attention and accidentally merged #1287 into this PR. I then reverted #1287 in #1426. This is a reminder that #1287 will eventually need to be re-merged into here (which is now probably best done by reverting the revert in #1426). Sorry for the confusion!

aysim319 · 2025-01-16T15:41:20Z

ran the code locally and fails with

found data for 82 epiweeks
[successfully fetched data for network_all]
rows before: 0
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 213, in <module>
    main()
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 204, in main
    update(fetcher, location, args.test)
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 155, in update
    raise Exception(
Exception: network_all 202401 data includes new group(s) {'rate_age_1t4', 'rate_age_gte75', 'rate_age_0tlt1'}

nmdefries · 2025-02-07T22:56:03Z

Exception: network_all 202401 data includes new group(s) {'rate_age_1t4', 'rate_age_gte75', 'rate_age_0tlt1'}

This should be resolved just by adding those signals to the list of those expected.

sonarqubecloud · 2025-02-10T19:58:57Z

Quality Gate passed

Issues
8 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.5% Duplication on New Code

See analysis details on SonarQube Cloud

nmdefries · 2025-02-10T21:37:02Z

This works now. I added a migration .sql script to add the new columns to the schema.

nmdefries · 2025-02-10T21:37:43Z

@melange396 @aysim319 This is ready for review.

nmdefries added 11 commits September 14, 2023 17:47

initial switch to new API endpoint; doesn't account for format change

d99d2d0

separate fetch fn for whole api obj

70c3e16

get_current_issue to use existing json response

3580f3f

cleanup names and comments

9cb1da4

- rename input arg to `update` to avoid reassignment later - comment and reuse args_insert - spelling - comment magic constant used in output format - rename location-network/catchmentid map

define function to convert json obs to dict grouped by location and e…

cea25ab

…piweek

auto-map from valueids to ordinal and label-based group names

24dc088

add new strata to sql insert statement by name, not order

26eff97

pass seasonids around to use in requests for location-specific data

229a96c

include old and new example API responses

1e942a5

flusurv tests

f8a6706

nmdefries force-pushed the ndefries/flusurv-new-endpoint branch from 734dca9 to f8a6706 Compare September 15, 2023 15:50

Merge branch 'dev' into ndefries/flusurv-new-endpoint

d9581ac

nmdefries marked this pull request as ready for review September 15, 2023 16:25

nmdefries commented Sep 15, 2023

View reviewed changes

src/acquisition/flusurv/new_grasp_location_result.json Outdated Show resolved Hide resolved

nmdefries added 2 commits September 15, 2023 16:07

pass metadata around to reduce API calls

9ac2918

add season label as a descriptive column

aa3dd8c

nmdefries requested a review from brookslogan September 15, 2023 20:50

move example API responses to make it clear they are not for prod use

ca7e478

nmdefries removed the request for review from brookslogan September 15, 2023 21:51

nmdefries mentioned this pull request Sep 15, 2023

Update flusurv schema and docs with new age, sex, and race groups #1287

Merged

4 tasks

nmdefries requested a review from melange396 September 18, 2023 15:46

This was referenced Sep 18, 2023

flusurv request is missing columns cmu-delphi/epidatpy#25

Open

flusurv request is missing columns cmu-delphi/epidatr#180

Open

melange396 requested changes Sep 26, 2023

View reviewed changes

nmdefries and others added 5 commits September 27, 2023 13:22

Merge branch 'dev' into ndefries/flusurv-new-endpoint

b979910

review cleanup

c1614a1

capitalize constant max_age

19a5b42

move paren to subtract from # of dates

2f18451

Co-authored-by: melange396 <[email protected]>

move n_exected_groups and big groupid comment to global

d64c4c0

nmdefries added 9 commits November 7, 2023 12:11

map some long race groups to abbreviated names

991fde1

programmatically create sql query to avoid ordering issues

5d00dbb

return empty result with right format when no data available

5cedb30

warning prefix

e7ad06e

add tests for geo-abbr conversion and location fetching

75283d7

don't error for missing groups -- some are expected

276944a

Merge branch 'dev' into ndefries/flusurv-new-endpoint

6c531a6

fn to add empty sex breakdowns to UT

9431489

with UT sex subsets getting filled, error on missing groups again

205c061

nmdefries requested a review from melange396 November 7, 2023 22:52

This was referenced Nov 8, 2023

Fetch new flusurv fields cmu-delphi/epidatr#210

Open

Fetch new flusurv fields cmu-delphi/epidatpy#26

Open

Merge branch 'dev' into ndefries/flusurv-new-endpoint

5248a97

nmdefries and others added 2 commits May 7, 2024 20:15

Revert "Update flusurv schema and docs with new age, sex, and race gr…

40212bb

…oups" (#1426) This reverts commit 9de7d64.

nmdefries mentioned this pull request Dec 19, 2024

Better describe meaning/context of endpoints in documentation cmu-delphi/epidatr#233

Open

7 tasks

aysim319 self-requested a review January 16, 2025 15:41

add new age rate signals

cf2fcec

nmdefries added 3 commits February 10, 2025 13:54

add new columns to flusurv endpoint spec

16634f8

add schema migration script

8be1074

Merge branch 'dev' into ndefries/flusurv-new-endpoint

87ecfdb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to new `flusurv` API endpoint #1278

Switch to new `flusurv` API endpoint #1278

nmdefries commented Aug 23, 2023 •

edited

Loading

melange396 left a comment

sonarqubecloud bot commented Nov 7, 2023

sonarqubecloud bot commented Dec 8, 2023

sonarqubecloud bot commented May 8, 2024

melange396 commented May 8, 2024

aysim319 commented Jan 16, 2025

nmdefries commented Feb 7, 2025

sonarqubecloud bot commented Feb 10, 2025

nmdefries commented Feb 10, 2025

nmdefries commented Feb 10, 2025

Switch to new flusurv API endpoint #1278

Are you sure you want to change the base?

Switch to new flusurv API endpoint #1278

Conversation

nmdefries commented Aug 23, 2023 • edited Loading

Summary:

Prerequisites:

melange396 left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Nov 7, 2023

sonarqubecloud bot commented Dec 8, 2023

sonarqubecloud bot commented May 8, 2024

Quality Gate passed

melange396 commented May 8, 2024

aysim319 commented Jan 16, 2025

nmdefries commented Feb 7, 2025

sonarqubecloud bot commented Feb 10, 2025

Quality Gate passed

nmdefries commented Feb 10, 2025

nmdefries commented Feb 10, 2025

Switch to new `flusurv` API endpoint #1278

Switch to new `flusurv` API endpoint #1278

nmdefries commented Aug 23, 2023 •

edited

Loading