Fix full build notification logic

[bendnorman]

PR Overview

I noticed my full build 271ffc37d3e7a7253beb9bb74ec2ea6585b92e0a-rename-core-assets passed all of the validation tests but the pudl-deployments channel was received a failure notification. I think this happened because #3018 changed the script to catch any errors that happened during the data distribution and datasette publishing:

pudl/docker/gcp_pudl_etl.sh

Lines 92 to 116 in 2493454

	run_pudl_etl 2>&1 | tee $LOGFILE
	# if pipeline is successful, distribute + publish datasette
	if [[ ${PIPESTATUS[0]} == 0 ]]; then
	# Dump outputs to s3 bucket if branch is dev or build was triggered by a tag
	if [ $GITHUB_ACTION_TRIGGER = "push" ] || [ $GITHUB_REF = "dev" ]; then
	copy_outputs_to_distribution_bucket
	fi
	# Deploy the updated data to datasette
	if [ $GITHUB_REF = "dev" ]; then
	python ~/devtools/datasette/publish.py 2>&1 | tee -a $LOGFILE
	fi
	fi
	# Notify slack about entire pipeline's success or failure;
	# PIPESTATUS[0] either refers to the failed ETL run or the last distribution
	# task that was run above
	if [[ ${PIPESTATUS[0]} == 0 ]]; then
	notify_slack "success"
	else
	notify_slack "failure"
	fi
	shutdown_vm

This logic does not work if we have a successful build that isn't kicked off from the dev branch. The first if statement will evaluate to true because the build succeeded, the inner two if statements will evaluate to false because the branch is not dev. The final if statement to determine if slack should be notified will also be false because the previous two if statements exit codes are non-zero.

To resolve this issue, I created a ETL_SUCCESS variable to track failures during the distribution and publishing steps.

PR Checklist

• Merge the most recent version of the branch you are merging into (probably dev).
• All CI checks are passing. Run tests locally to debug failures
• Make sure you've included good docstrings.
• For major data coverage & analysis changes, run data validation tests
• Include unit tests for new functions and classes.
• Defensive data quality/sanity checks in analyses & data processing functions.
• Update the release notes and reference reference the PR and related issues.
• Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

[codecov]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (fdd444d) 88.7% compared to head (cc28b48) 88.7%.
Report is 13 commits behind head on dev.

Files	Patch %	Lines
src/pudl/io_managers.py	81.8%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3058   +/-   ##
=====================================
  Coverage   88.7%   88.7%           
=====================================
  Files         90      90           
  Lines      10994   10995    +1     
=====================================
+ Hits        9758    9759    +1     
  Misses      1236    1236           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jdangerx]

Storing that value as a variable makes way more sense than the weird house of cards I was building, thanks!!