GitHub Actions for CI tests

[anthonyfok]

Add some CI tests with GitHub Actions to try to catch potential issues automatically.

Sprint 36

• Get add_data.sh to run in GitHub-hosted ubuntu-20.04 runner with postgis/postgis as a service
  • Read sample.env, but override POSTGRES_HOST from db-opendrr to localhost
  • Populate python/config.ini with our GitHub personal access token
  • Install python3-{pandas,numpy,elasticsearch}
  • Install Jenkspy? (not available as .deb)
• Get docker-compose up --build to run
• Get docker compose up --build to run

Sprint 37

• Ensure each database is populated. (Bug in Pipeline optimization (Sprint 34–36) #105 as of 2021-06-10)
• Ensure each CSV header is legit. (Bug in Pipeline optimization (Sprint 34–36) #105 as of 2021-06-10, merge_csv: "RUN: awk" log seeping into the CSV file...)
• Ensure file name or directory name with spaces are properly handled by fetch_csv_xz and fetch_csv_lfs, e.g. exposure/general-building-stock/1. documentation/collapse_probability.csv
• TODO (I'm sure I had something else to add, but forgot.)

Test cases include (but not limited to):

• add_data.sh: "dry run", "download-only run" etc. should be successful
• add_data.sh: Unit tests
• Data integrity tests, e.g. some simple tests to ensure a table is correctly populated, not missing columns like lat and lon
• Try to catch warnings and non-fatal errors from the output of docker compose logs

Ideas and TODOs:

• Try both running docker compose and without (ubuntu runner + postgis service).
• Add GitHub badges on README.md to show if all tests pass or not.

Thoughts:

• Should all CI tests be done in this OpenDRR/opendrr-api? Or should some live on OpenDRR/model-factory? E.g. for testing upcoming changes with respect to add_data.sh related Python scripts - flexible data loading model-factory#53

Unit testing tool for bash: BATS vs ShellSpec

• BATS: Bash Automated Testing System
  • https://github.com/bats-core/bats-core
  • Testing Bash with BATS https://opensource.com/article/19/2/testing-bash-bats
• ShellSpec: A full-featured BDD unit testing framework for bash, ksh, zsh, dash and all POSIX shells
  • https://github.com/shellspec/shellspec
  • https://shellspec.info/
• I am leaning toward ShellSpec because that's the one I got working first, and very actively maintained.
• but seeing that BATS isn't abandonware any more, and has been actively maintained, may take a look too.