Update Google Docs Meta Data

[github-actions]

Updating Google Docs Meta Data

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[melange396]

the diff is pretty large again, im gonna have to do something like this again: #1546 (comment)

[melange396]

Looks like this change involves some reordering of columns, some new columns ('Member Name', 'Member Short Name', 'Member Description', 'Member Name Dec 04', 'Include in signal app'), and the addition of 4 new rows for the NSSP "*_2023RVR" signals, all of which should be fine.

There are two problematic changes:

• We appear to have lost the 'Short Description' column though, which is used by the webapp.
• Row 440 looks like it lost its source name, which was quidel-flu before (signal is raw_tests_per_device).

Then there were these other changes, which i presume were fixes (2 signal set changes and 11 name changes) :

mismatch in row: 8  [ chng : 7dav_inpatient_covid ]
  Signal Set:
    Change Healthcare COVID-Confirmed Doctor Visits --> Change Healthcare COVID-Confirmed Hospital Admissions
mismatch in row: 13  [ doctor-visits : smoothed_adj_cli ]
  Signal Set:
    COVID-19 Hospital Admissions from Claims --> COVID-Related Doctor Visits

mismatch in row: 367  [ google-symptoms : sum_anosmia_ageusia_raw_search ]
  Name:
    Sum Anosmia Ageusia Searches --> Confirmed COVID-19 Admissions per day
mismatch in row: 368  [ google-symptoms : sum_anosmia_ageusia_smoothed_search ]
  Name:
    Sum Anosmia Ageusia Searches (7-day average) --> Confirmed COVID-19 Admissions per day (per 100k people)
mismatch in row: 369  [ hhs : confirmed_admissions_covid_1d ]
  Name:
    Confirmed COVID-19 Admissions per day --> Confirmed COVID-19 Admissions per day (7-day average)
mismatch in row: 370  [ hhs : confirmed_admissions_covid_1d_prop ]
  Name:
    Confirmed COVID-19 Admissions per day (per 100k people) --> Confirmed COVID-19 Admissions per day (per 100k people, 
7-day average)
mismatch in row: 371  [ hhs : confirmed_admissions_covid_1d_7dav ]
  Name:
    Confirmed COVID-19 Admissions per day (7-day average) --> Confirmed Influenza Admissions per day (7-day average)
mismatch in row: 372  [ hhs : confirmed_admissions_covid_1d_prop_7dav ]
  Name:
    Confirmed COVID-19 Admissions per day (7-day average, per 100k people) --> Confirmed Influenza Admissions per day 
(7-day average)
mismatch in row: 373  [ hhs : confirmed_admissions_influenza_1d ]
  Name:
    Confirmed Influenza Admissions per day --> Confirmed Influenza Admissions per day (per 100k people)
mismatch in row: 374  [ hhs : confirmed_admissions_influenza_1d_7dav ]
  Name:
    Confirmed Influenza Admissions per day (7-day average) --> Confirmed Influenza Admissions per day (per 100k 
population, 7-day average)
mismatch in row: 375  [ hhs : confirmed_admissions_influenza_1d_prop ]
  Name:
    Confirmed Influenza Admissions per day (per 100k people) --> Confirmed and Suspected COVID Admissions per day (per 
100k people)
mismatch in row: 378  [ hhs : sum_confirmed_suspected_admissions_covid_1d_prop ]
  Name:
    Confirmed and Suspected COVID-19 Admissions per day (per 100k people) --> Confirmed and Suspected COVID-19 
Admissions per day (per 100k people, 7-day average)
mismatch in row: 473  [ nssp : smoothed_pct_ed_visits_combined ]
  Name:
    Emergency Department Visits for COVID, Influenza, and RSV Combined (Percent of total ED visits) --> Emergency 
Department Visits for COVID, Influenza, and RSV Combined (Percent of total ED visits, 3 week average)

@carlynvandyke can you comment on the validity of these?

[melange396]

updated comparison code:

import csv

dev = []
# from: https://github.com/cmu-delphi/delphi-epidata/blob/dev/src/server/endpoints/covidcast_utils/db_signals.csv                                                                                                  
with open('dev__db_signals.csv') as f:
    for r in csv.reader(f):
        dev.append(r)

new = []
# from: https://github.com/cmu-delphi/delphi-epidata/blob/bot/update-docs/src/server/endpoints/covidcast_utils/db_signals.csv                                                                                      
with open('new__db_signals.csv') as f:
    for r in csv.reader(f):
        new.append(r)

# column names
dev_cols = set(dev[0])
new_cols = set(new[0])
both_cols = list(dev_cols.intersection(new_cols))

# to get the right column number for each version of the file, based on the column name
dev_col_lookup = {c: i for i,c in enumerate(dev[0])}
new_col_lookup = {c: i for i,c in enumerate(new[0])}

print("added columns:", new_cols-dev_cols)
print("removed columns:", dev_cols-new_cols)
print("# dev rows:", len(dev))
print("# new rows:", len(new))
print("rows difference:", len(new)-len(dev))

for i in range(min(len(dev), len(new))):
    dev_line = [dev[i][dev_col_lookup[col]] for col in both_cols]
    new_line = [new[i][new_col_lookup[col]] for col in both_cols]
    if dev_line != new_line:
        print("mismatch in row:", i+1, " [", new[i][new_col_lookup["Source Subdivision"]], ":", new[i][new_col_lookup["Signal"]], "]")
        print("\n".join(["".join([
                "  ", col, ":\n    ", dev[i][dev_col_lookup[col]], " --> ", new[i][new_col_lookup[col]]])
                for col in both_cols if dev[i][dev_col_lookup[col]]!=new[i][new_col_lookup[col]]
            ]))

based on #1546 (comment)