From 8cfb1914b2926238686bc2e1a1238629d179aed2 Mon Sep 17 00:00:00 2001 From: nmdefries <42820733+nmdefries@users.noreply.github.com> Date: Tue, 28 Jan 2025 10:26:50 -0500 Subject: [PATCH 1/4] move dip warning to limitations section (#1587) --- docs/api/covidcast-signals/google-symptoms.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/api/covidcast-signals/google-symptoms.md b/docs/api/covidcast-signals/google-symptoms.md index a8d403de2..684a429e8 100644 --- a/docs/api/covidcast-signals/google-symptoms.md +++ b/docs/api/covidcast-signals/google-symptoms.md @@ -16,8 +16,6 @@ nav_order: 1 * **Time type:** day (see [date format docs](../covidcast_times.md)) * **License:** To download or use the data, you must agree to the Google [Terms of Service](https://policies.google.com/terms) -
Data issue: Between May 13 2024 and August 6 2024, signals values were 25%-50% lower compared to previous time periods. This affects all signals and symptom sets. Currently there is no explanation for the decrease in search volume, and the issue is under investigation by our data source partners.
- ## Overview This data source is based on the [COVID-19 Search Trends symptoms @@ -25,6 +23,8 @@ dataset](https://console.cloud.google.com/marketplace/product/bigquery-public-da this search data, we estimate the volume of searches mapped to symptom sets related to COVID-19. The resulting daily dataset for each region shows the average relative frequency of searches for each symptom set. The signals are measured in arbitrary units that are normalized for overall search users in the region and scaled by the maximum value of the normalized popularity within a geographic region across a specific time range. **Values are comparable across signals in the same location but NOT across geographic regions**. For example, within a state, we can compare `s01_smoothed_search` and `s02_smoothed_search`. However, we cannot compare `s01_smoothed_search` between states. Larger numbers represent increased relative popularity of symptom-related searches. +Between May 13 2024 and August 6 2024, [signal values were much lower](#limitations) compared to previous time periods due to a data outage. + #### Symptom sets * _s01_: Cough, Phlegm, Sputum, Upper respiratory tract infection @@ -94,7 +94,7 @@ population-weighted averaging. For aggregation purposes only, we assign a value of 0 to source regions that have no data provided due to quality or privacy issues for a certain day (see -Limitations for details). We do not report aggregated regions if none of their +[Limitations](#limitations) for details). We do not report aggregated regions if none of their source regions have data. Because of this censoring behavior, the resulting data for aggregated regions does not fully match the _actual_ search volume for these regions (which is not provided to us). @@ -106,6 +106,13 @@ As a result the delay can range from 3 to 10 days or even more. We check for updates every day and provide the most up-to-date data. ## Limitations + +Between May 13 2024 and August 6 2024, signal values were 25%-50% lower compared to previous time periods. +This affected _all_ signals and symptom sets. +The drop does not reflect actual search term popularity during the affected period. +The apparent decrease in search volume was caused by an outage in the data pipeline on the source side. +The data was unfortunately not recoverable and the dip can not be repaired, but data outside the listed time period is unaffected. + When daily volume in a region does not meet quality or privacy thresholds, set by Google, no daily value is reported. Weekly data may be available from Google in these cases, but we do not yet support importation using weekly data. From 0f1e71b9ddb7e54362c8002cf21829353dca4197 Mon Sep 17 00:00:00 2001 From: aysim319 Date: Wed, 29 Jan 2025 11:44:07 -0500 Subject: [PATCH 2/4] updated min year (#1585) * updated min year w/ comment Co-authored-by: george --------- Co-authored-by: george --- src/acquisition/covidcast/csv_importer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/acquisition/covidcast/csv_importer.py b/src/acquisition/covidcast/csv_importer.py index e9893c0da..33df578a9 100644 --- a/src/acquisition/covidcast/csv_importer.py +++ b/src/acquisition/covidcast/csv_importer.py @@ -68,7 +68,7 @@ class CsvImporter: REQUIRED_COLUMNS = {'geo_id', 'val', 'se', 'sample_size'} # reasonable time bounds for sanity checking time values - MIN_YEAR = 2019 + MIN_YEAR = 2017 # `google-symptoms` has 2017 data MAX_YEAR = 2030 # The datatypes expected by pandas.read_csv. Int64 is like float in that it can handle both numbers and nans. From f5da64aeab84472d770ffaed241c6ba2c73e6d16 Mon Sep 17 00:00:00 2001 From: george Date: Wed, 29 Jan 2025 16:04:37 -0500 Subject: [PATCH 3/4] NHSN documentation fixes for page title and heading (#1588) * new title & header for NHSN page --------- Co-authored-by: nmdefries <42820733+nmdefries@users.noreply.github.com> --- docs/api/covidcast-signals/nhsn.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/api/covidcast-signals/nhsn.md b/docs/api/covidcast-signals/nhsn.md index 30ea5e9da..f5ebb829c 100644 --- a/docs/api/covidcast-signals/nhsn.md +++ b/docs/api/covidcast-signals/nhsn.md @@ -1,10 +1,10 @@ --- -title: NHSN ED Visits +title: NHSN Respiratory Hospitalizations parent: Data Sources and Signals grand_parent: Main Endpoint (COVIDcast) nav_order: 1 --- -# National Syndromic Surveillance Program Emergency Department Visits +# National Healthcare Safety Network Respiratory Hospitalizations {: .no_toc} * **Source name:** `nhsn` From b0b49a85a4ac53b6d4aacb6aa0e18166a270bc87 Mon Sep 17 00:00:00 2001 From: melange396 Date: Wed, 29 Jan 2025 21:08:49 +0000 Subject: [PATCH 4/4] chore: release delphi-epidata 4.1.29 --- .bumpversion.cfg | 2 +- dev/local/setup.cfg | 2 +- src/client/delphi_epidata.R | 2 +- src/client/delphi_epidata.js | 2 +- src/client/packaging/npm/package.json | 2 +- src/server/_config.py | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.bumpversion.cfg b/.bumpversion.cfg index 406cc8fe9..a50b8cd7b 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 4.1.28 +current_version = 4.1.29 commit = False tag = False diff --git a/dev/local/setup.cfg b/dev/local/setup.cfg index d8dc0147d..66c7961d9 100644 --- a/dev/local/setup.cfg +++ b/dev/local/setup.cfg @@ -1,6 +1,6 @@ [metadata] name = Delphi Development -version = 4.1.28 +version = 4.1.29 [options] packages = diff --git a/src/client/delphi_epidata.R b/src/client/delphi_epidata.R index 3fd8a8b09..5f07e84fe 100644 --- a/src/client/delphi_epidata.R +++ b/src/client/delphi_epidata.R @@ -15,7 +15,7 @@ Epidata <- (function() { # API base url BASE_URL <- getOption('epidata.url', default = 'https://api.delphi.cmu.edu/epidata/') - client_version <- '4.1.28' + client_version <- '4.1.29' auth <- getOption("epidata.auth", default = NA) diff --git a/src/client/delphi_epidata.js b/src/client/delphi_epidata.js index 87752339c..f38e1137d 100644 --- a/src/client/delphi_epidata.js +++ b/src/client/delphi_epidata.js @@ -22,7 +22,7 @@ } })(this, function (exports, fetchImpl, jQuery) { const BASE_URL = "https://api.delphi.cmu.edu/epidata/"; - const client_version = "4.1.28"; + const client_version = "4.1.29"; // Helper function to cast values and/or ranges to strings function _listitem(value) { diff --git a/src/client/packaging/npm/package.json b/src/client/packaging/npm/package.json index 4271cbbbb..67d4c73db 100644 --- a/src/client/packaging/npm/package.json +++ b/src/client/packaging/npm/package.json @@ -2,7 +2,7 @@ "name": "delphi_epidata", "description": "Delphi Epidata API Client", "authors": "Delphi Group", - "version": "4.1.28", + "version": "4.1.29", "license": "MIT", "homepage": "https://github.com/cmu-delphi/delphi-epidata", "bugs": { diff --git a/src/server/_config.py b/src/server/_config.py index b5b71d5e4..8774a792b 100644 --- a/src/server/_config.py +++ b/src/server/_config.py @@ -7,7 +7,7 @@ load_dotenv() -VERSION = "4.1.28" +VERSION = "4.1.29" MAX_RESULTS = int(10e6) MAX_COMPATIBILITY_RESULTS = int(3650)