Some random notes

Table of Contents Background and rationale Cool Projects to study TODO? GitHub Actions AsciiDoc Migrate from MS Word (DOCX) to Asciidoctor References / Bibliography / Citations Workarounds "References appearing as one long line" problem Image formats Dynamic images Using `pandoc` with GitHub Actions OpenQuake Installation on Ubuntu 20.04.2 LTS curl -O https://raw.githubusercontent.com/gem/oq-engine/master/install.py Install python3-pip and python3-venv python3 install.py user `server` installation Docker image model-factory Awesome tools One-liners Git LFS Troubleshooting (to be sorted) BFG Repo-Cleaner Great Articles Chrome tabs overload Docker Videos Compression tools comparisons 7z xz zpaq Eventual talk/presentation on GitHub tips Random experiments OpenQuake with Python 3.9 Benchmark of Python inside vs outside Docker

Background and rationale

The Great Quake and the Great Drowning | Hakai Magazine

Cool Projects to study

TODO?

benchmark manuscript · Issue #5 · OpenDRR/documentation

GitHub Actions

See new GitHub Actions wiki page.

AsciiDoc

AsciiDoc Writer’s Guide | Asciidoctor

Migrate from MS Word (DOCX) to Asciidoctor

Migrate from MS Word to Asciidoctor :: Asciidoctor Docs

As of March 9, 2021, use latest version of pandoc (2.11.4) with docker, especially because the version in Debian (hence Ubuntu) is a little bit too old, stuck at 2.9.2.1-1, see https://bugs.debian.org/979124

 $ docker run --rm --volume "`pwd`:/data" --user `id -u`:`id -g` pandoc/core input.docx -f docx \
   -t asciidoc --wrap=none --markdown-headings=atx --extract-media=extracted-media  -o output2.adoc

Some history: "Convert MS Word to AsciiDoc" content discussion · Issue #542 · asciidoctor/asciidoctor.org

References / Bibliography / Citations

The MS Word documents use EndNote (X8?) for reference management
- Hints: ADDIN EN.CITE, w:fldData found in word/document.xml inside the unzipped DOCX file.
but this is not quite supported by pandoc yet:
- Read docx references, and convert into pandoc citations · Issue #4140 · jgm/pandoc at https://github.com/jgm/pandoc/issues/4140 (feature request since Dec 2017; talk of a paid bounty in February 2021)
Bibliography :: Asciidoctor Docs

Some Asciidoctor gems:

(Aside: Open-source Zotero as an alternative to EndNote.)

Workarounds "References appearing as one long line" problem

Sample showing the issue:

 ReferencesAdams, J., T. Allen, S. Halchuk, and M. Kolaj. 2019. 'Canada's 6th Generation Seismic Hazard Model, as Prepared for the 2020 National Building Code of Canada. 12th Can. Conf', _Earthquake Engineering_.Adams, John, Stephen Halchuk, Trevor Allen, and Gary Rogers. 2015a. "Canada’s 5th Generation Seismic Hazard Model, as Prepared for the 2015 National Building Code of Canada." In _11th Canadian Conference on Earthquake Engineering (11CCEE)_, 11. Victoria, BC.———. 2015b. "Fifth Generation (2015) Seismic Hazard Model for Southwest British Columbia." In _11th Canadian Conference on Earthquake Engineering (11CCEE)_, 11. Victoria, BC.Agriculture and Agri-Foods Canada. 2015. "ISO 19131 – Land Use Data Product Specifications." In, 14.

Solution:

 perl -pe 's/(References|\.)([A-Z][a-z]+, [A-Z.]+|———\.)/$1\n* $2/g' input.adoc

Image formats

PNG: OK!
TIFF: Not supported on Google Chrome. (IE and Safari only?) Need to be converted to PNG or JPEG.
JP2: JPEG2000: supported by Safari only. https://caniuse.com/jpeg2000
WebP: Pretty good support, though not by IE 11, and for Mac, only in macOS 11 Big Sur and later. https://caniuse.com/webp
JPEG XL: Looks promising. Maybe in 2022?

See:

Conclusion: Stick with JPEG (*.jpg) and PNG (*.png) in 2021, but keep an eye out for JPEG XL.

Dynamic images

TODO

Using `pandoc` with GitHub Actions

pandoc/pandoc-action-example: using the pandoc document converter on GitHub Actions https://github.com/pandoc/pandoc-action-example

OpenQuake

Get Started | Global Earthquake Model Foundation | Italy
Global Earthquake Model
Global Earthquake Model Foundation | Italy
oq-engine/ubuntu.md Installing the OpenQuake Engine on Ubuntu Linux (at master · gem/oq-engine)
Video tutorials: OpenQuake online training
Installing the OpenQuake Engine on macOS: oq-engine/macos.md at master · gem/oq-engine

Installation on Ubuntu 20.04.2 LTS

curl -O https://raw.githubusercontent.com/gem/oq-engine/master/install.py

ubuntu@openquake:~$ curl -O https://raw.githubusercontent.com/gem/oq-engine/master/install.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12734  100 12734    0     0  37563      0 --:--:-- --:--:-- --:--:-- 37563
ubuntu@openquake:~$ ls -l
total 16
-rw-rw-r-- 1 ubuntu ubuntu 12734 May  6 10:55 install.py

Install python3-pip and python3-venv

ubuntu@openquake:~$ sudo eatmydata apt-get update
...
ubuntu@openquake:~$ sudo eatmydata apt-get install python3-pip python3-venv

python3 install.py user

ubuntu@openquake:~$ python3 install.py user
Created /home/ubuntu/openquake
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1ee83f8280>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
Collecting pip
  Downloading pip-21.1.1-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.6 MB/s 
Collecting wheel
  Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: pip, wheel
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-21.1.1 wheel-0.36.2
Collecting setuptools==56.0.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/setuptools-56.0.0-py3-none-any.whl (784 kB)
     |████████████████████████████████| 784 kB 83 kB/s 
Collecting pytz==2019.1
  Downloading https://wheelhouse.openquake.org/v3/linux/py/pytz-2019.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 113 kB/s 
Collecting h5py==2.10.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/h5py-2.10.0-cp38-cp38-manylinux1_x86_64.whl (2.9 MB)
     |████████████████████████████████| 2.9 MB 313 kB/s 
Collecting numpy==1.18.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/numpy-1.18.2-cp38-cp38-manylinux1_x86_64.whl (16.3 MB)
     |████████████████████████████████| 16.3 MB 3.9 kB/s 
Collecting scipy==1.4.1
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/scipy-1.4.1-cp38-cp38-manylinux1_x86_64.whl (26.0 MB)
     |████████████████████████████████| 26.0 MB 40 kB/s 
Collecting pandas==1.1.5
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/pandas-1.1.5-cp38-cp38-manylinux1_x86_64.whl (9.3 MB)
     |████████████████████████████████| 9.3 MB 383 kB/s 
Collecting pyzmq==19.0.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/pyzmq-19.0.0-cp38-cp38-manylinux1_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 313 kB/s 
Collecting psutil==5.6.7
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/psutil-5.6.7-cp38-cp38-manylinux1_x86_64.whl (255 kB)
     |████████████████████████████████| 255 kB 295 kB/s 
Collecting Shapely==1.7.1
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/Shapely-1.7.1-cp38-cp38-manylinux1_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 317 kB/s 
Collecting docutils==0.14
  Downloading https://wheelhouse.openquake.org/v3/linux/py/docutils-0.14-py3-none-any.whl (543 kB)
     |████████████████████████████████| 543 kB 218 kB/s 
Collecting decorator==4.4.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py/decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting pbr==5.2.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/pbr-5.2.0-py2.py3-none-any.whl (107 kB)
     |████████████████████████████████| 107 kB 300 kB/s 
Collecting six==1.12.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/six-1.12.0-py2.py3-none-any.whl (10 kB)
Collecting Django==2.2.7
  Downloading https://wheelhouse.openquake.org/v3/linux/py/Django-2.2.7-py3-none-any.whl (7.5 MB)
     |████████████████████████████████| 7.5 MB 110 kB/s 
Collecting sqlparse==0.3.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/sqlparse-0.3.0-py2.py3-none-any.whl (39 kB)
Collecting certifi==2019.3.9
  Downloading https://wheelhouse.openquake.org/v3/linux/py/certifi-2019.3.9-py2.py3-none-any.whl (158 kB)
     |████████████████████████████████| 158 kB 544 kB/s 
Collecting chardet==3.0.4
  Downloading https://wheelhouse.openquake.org/v3/linux/py/chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 584 kB/s 
Collecting idna==2.8
  Downloading https://wheelhouse.openquake.org/v3/linux/py/idna-2.8-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 149 kB/s 
Collecting urllib3==1.25.3
  Downloading https://wheelhouse.openquake.org/v3/linux/py/urllib3-1.25.3-py2.py3-none-any.whl (150 kB)
     |████████████████████████████████| 150 kB 557 kB/s 
Collecting requests==2.22.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/requests-2.22.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 137 kB/s 
Collecting pyshp==1.2.3
  Downloading https://wheelhouse.openquake.org/v3/linux/py/pyshp-1.2.3-py3-none-any.whl (19 kB)
Collecting toml==0.10.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/toml-0.10.0-py2.py3-none-any.whl (25 kB)
Collecting vine==1.3.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/vine-1.3.0-py2.py3-none-any.whl (14 kB)
Collecting amqp==2.4.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py/amqp-2.4.2-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 106 kB/s 
Collecting kombu==4.5.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/kombu-4.5.0-py2.py3-none-any.whl (185 kB)
     |████████████████████████████████| 185 kB 297 kB/s 
Collecting billiard==3.6.0.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/billiard-3.6.0.0-py3-none-any.whl (88 kB)
     |████████████████████████████████| 88 kB 176 kB/s 
Collecting celery==4.3.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/celery-4.3.0-py2.py3-none-any.whl (413 kB)
     |████████████████████████████████| 413 kB 35 kB/s 
Collecting python-dateutil==2.8.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/python_dateutil-2.8.0-py2.py3-none-any.whl (226 kB)
     |████████████████████████████████| 226 kB 306 kB/s 
Collecting pyparsing==2.4.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/pyparsing-2.4.0-py2.py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 37 kB/s 
Collecting cycler==0.10.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting kiwisolver==1.1.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/kiwisolver-1.1.0-cp38-cp38-manylinux1_x86_64.whl (91 kB)
     |████████████████████████████████| 91 kB 197 kB/s 
Collecting matplotlib==3.1.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/matplotlib-3.1.2-cp38-cp38-manylinux1_x86_64.whl (13.1 MB)
     |████████████████████████████████| 13.1 MB 4.2 kB/s 
Collecting pyproj==2.5.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/pyproj-2.5.0-cp38-cp38-manylinux1_x86_64.whl (10.4 MB)
     |████████████████████████████████| 10.4 MB 32 kB/s 
Collecting GDAL==3.2.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/GDAL-3.2.2-cp38-cp38-manylinux1_x86_64.whl (103.2 MB)
     |████████████████████████████████| 103.2 MB 7.9 kB/s 
Installing collected packages: vine, six, setuptools, amqp, urllib3, sqlparse, pytz, python-dateutil, pyparsing, numpy, kombu, kiwisolver, idna, cycler, chardet, certifi, billiard, toml, Shapely, scipy, requests, pyzmq, pyshp, pyproj, psutil, pbr, pandas, matplotlib, h5py, GDAL, docutils, Django, decorator, celery
  Attempting uninstall: setuptools
    Found existing installation: setuptools 44.0.0
    Uninstalling setuptools-44.0.0:
      Successfully uninstalled setuptools-44.0.0
Successfully installed Django-2.2.7 GDAL-3.2.2 Shapely-1.7.1 amqp-2.4.2 billiard-3.6.0.0 celery-4.3.0 certifi-2019.3.9 chardet-3.0.4 cycler-0.10.0 decorator-4.4.2 docutils-0.14 h5py-2.10.0 idna-2.8 kiwisolver-1.1.0 kombu-4.5.0 matplotlib-3.1.2 numpy-1.18.2 pandas-1.1.5 pbr-5.2.0 psutil-5.6.7 pyparsing-2.4.0 pyproj-2.5.0 pyshp-1.2.3 python-dateutil-2.8.0 pytz-2019.1 pyzmq-19.0.0 requests-2.22.0 scipy-1.4.1 setuptools-56.0.0 six-1.12.0 sqlparse-0.3.0 toml-0.10.0 urllib3-1.25.3 vine-1.3.0
Collecting openquake.engine
  Downloading openquake.engine-3.11.3-py3-none-any.whl (31.6 MB)
     |████████████████████████████████| 31.6 MB 94 kB/s 
Requirement already satisfied: requests<2.24,>=2.20 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (2.22.0)
Requirement already satisfied: matplotlib<3.2,>=1.5 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (3.1.2)
Requirement already satisfied: django<2.3,>=1.11 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (2.2.7)
Requirement already satisfied: setuptools in ./openquake/lib/python3.8/site-packages (from openquake.engine) (56.0.0)
Requirement already satisfied: pyproj>=1.9 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (2.5.0)
Requirement already satisfied: pandas<1.2,>=0.25 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (1.1.5)
Requirement already satisfied: pyzmq<20.0 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (19.0.0)
Requirement already satisfied: psutil<5.7,>=2.0 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (5.6.7)
Requirement already satisfied: pyshp==1.2.3 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (1.2.3)
Requirement already satisfied: docutils<0.15,>=0.11 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (0.14)
Requirement already satisfied: toml in ./openquake/lib/python3.8/site-packages (from openquake.engine) (0.10.0)
Requirement already satisfied: scipy<1.5,>=1.3 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (1.4.1)
Requirement already satisfied: numpy<1.19,>=1.16 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (1.18.2)
Requirement already satisfied: shapely<1.8,>=1.3 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (1.7.1)
Requirement already satisfied: decorator>=4.3 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (4.4.2)
Requirement already satisfied: h5py<2.11,>=2.10 in ./openquake/lib/python3.8/site-packages (from openquake.engine) (2.10.0)
Requirement already satisfied: sqlparse in ./openquake/lib/python3.8/site-packages (from django<2.3,>=1.11->openquake.engine) (0.3.0)
Requirement already satisfied: pytz in ./openquake/lib/python3.8/site-packages (from django<2.3,>=1.11->openquake.engine) (2019.1)
Requirement already satisfied: six in ./openquake/lib/python3.8/site-packages (from h5py<2.11,>=2.10->openquake.engine) (1.12.0)
Requirement already satisfied: kiwisolver>=1.0.1 in ./openquake/lib/python3.8/site-packages (from matplotlib<3.2,>=1.5->openquake.engine) (1.1.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in ./openquake/lib/python3.8/site-packages (from matplotlib<3.2,>=1.5->openquake.engine) (2.4.0)
Requirement already satisfied: python-dateutil>=2.1 in ./openquake/lib/python3.8/site-packages (from matplotlib<3.2,>=1.5->openquake.engine) (2.8.0)
Requirement already satisfied: cycler>=0.10 in ./openquake/lib/python3.8/site-packages (from matplotlib<3.2,>=1.5->openquake.engine) (0.10.0)
Requirement already satisfied: certifi>=2017.4.17 in ./openquake/lib/python3.8/site-packages (from requests<2.24,>=2.20->openquake.engine) (2019.3.9)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./openquake/lib/python3.8/site-packages (from requests<2.24,>=2.20->openquake.engine) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./openquake/lib/python3.8/site-packages (from requests<2.24,>=2.20->openquake.engine) (1.25.3)
Requirement already satisfied: idna<2.9,>=2.5 in ./openquake/lib/python3.8/site-packages (from requests<2.24,>=2.20->openquake.engine) (2.8)
Installing collected packages: openquake.engine
Successfully installed openquake.engine-3.11.3
The standalone applications are not installed yet
Please add an alias oq=/home/ubuntu/openquake/bin/oq in your .bashrc or similar
The engine was installed successfully.
You can run a test calculation with the command
/home/ubuntu/openquake/bin/oq engine --run /home/ubuntu/openquake/demos/hazard/AreaSourceClassicalPSHA/job.ini
ubuntu@openquake:~$

`server` installation

ubuntu@openquake:~$ sudo -H /usr/bin/python3 install.py server
Created user openquake
Created /opt/openquake
Collecting pip
  Downloading pip-21.1.1-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.9 MB/s 
Collecting wheel
  Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: pip, wheel
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-21.1.1 wheel-0.36.2
Collecting setuptools==56.0.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/setuptools-56.0.0-py3-none-any.whl (784 kB)
     |████████████████████████████████| 784 kB 304 kB/s 
Collecting pytz==2019.1
  Downloading https://wheelhouse.openquake.org/v3/linux/py/pytz-2019.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 17.6 MB/s 
Collecting h5py==2.10.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/h5py-2.10.0-cp38-cp38-manylinux1_x86_64.whl (2.9 MB)
     |████████████████████████████████| 2.9 MB 733 kB/s 
Collecting numpy==1.18.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/numpy-1.18.2-cp38-cp38-manylinux1_x86_64.whl (16.3 MB)
     |████████████████████████████████| 16.3 MB 2.3 kB/s 
Collecting scipy==1.4.1
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/scipy-1.4.1-cp38-cp38-manylinux1_x86_64.whl (26.0 MB)
     |████████████████████████████████| 26.0 MB 69 kB/s 
Collecting pandas==1.1.5
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/pandas-1.1.5-cp38-cp38-manylinux1_x86_64.whl (9.3 MB)
     |████████████████████████████████| 9.3 MB 738 kB/s 
Collecting pyzmq==19.0.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/pyzmq-19.0.0-cp38-cp38-manylinux1_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 303 kB/s 
Collecting psutil==5.6.7
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/psutil-5.6.7-cp38-cp38-manylinux1_x86_64.whl (255 kB)
     |████████████████████████████████| 255 kB 589 kB/s 
Collecting Shapely==1.7.1
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/Shapely-1.7.1-cp38-cp38-manylinux1_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 316 kB/s 
Collecting docutils==0.14
  Downloading https://wheelhouse.openquake.org/v3/linux/py/docutils-0.14-py3-none-any.whl (543 kB)
     |████████████████████████████████| 543 kB 592 kB/s 
Collecting decorator==4.4.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py/decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting pbr==5.2.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/pbr-5.2.0-py2.py3-none-any.whl (107 kB)
     |████████████████████████████████| 107 kB 589 kB/s 
Collecting six==1.12.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/six-1.12.0-py2.py3-none-any.whl (10 kB)
Collecting Django==2.2.7
  Downloading https://wheelhouse.openquake.org/v3/linux/py/Django-2.2.7-py3-none-any.whl (7.5 MB)
     |████████████████████████████████| 7.5 MB 833 kB/s 
Collecting sqlparse==0.3.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/sqlparse-0.3.0-py2.py3-none-any.whl (39 kB)
Collecting certifi==2019.3.9
  Downloading https://wheelhouse.openquake.org/v3/linux/py/certifi-2019.3.9-py2.py3-none-any.whl (158 kB)
     |████████████████████████████████| 158 kB 9.3 MB/s 
Collecting chardet==3.0.4
  Downloading https://wheelhouse.openquake.org/v3/linux/py/chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 9.8 MB/s 
Collecting idna==2.8
  Downloading https://wheelhouse.openquake.org/v3/linux/py/idna-2.8-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 843 kB/s 
Collecting urllib3==1.25.3
  Downloading https://wheelhouse.openquake.org/v3/linux/py/urllib3-1.25.3-py2.py3-none-any.whl (150 kB)
     |████████████████████████████████| 150 kB 583 kB/s 
Collecting requests==2.22.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/requests-2.22.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 653 kB/s 
Collecting pyshp==1.2.3
  Downloading https://wheelhouse.openquake.org/v3/linux/py/pyshp-1.2.3-py3-none-any.whl (19 kB)
Collecting toml==0.10.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/toml-0.10.0-py2.py3-none-any.whl (25 kB)
Collecting vine==1.3.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/vine-1.3.0-py2.py3-none-any.whl (14 kB)
Collecting amqp==2.4.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py/amqp-2.4.2-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 1.0 MB/s 
Collecting kombu==4.5.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/kombu-4.5.0-py2.py3-none-any.whl (185 kB)
     |████████████████████████████████| 185 kB 21.5 MB/s 
Collecting billiard==3.6.0.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/billiard-3.6.0.0-py3-none-any.whl (88 kB)
     |████████████████████████████████| 88 kB 1.4 MB/s 
Collecting celery==4.3.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/celery-4.3.0-py2.py3-none-any.whl (413 kB)
     |████████████████████████████████| 413 kB 9.2 MB/s 
Collecting python-dateutil==2.8.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/python_dateutil-2.8.0-py2.py3-none-any.whl (226 kB)
     |████████████████████████████████| 226 kB 12.2 MB/s 
Collecting pyparsing==2.4.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/pyparsing-2.4.0-py2.py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 56 kB/s 
Collecting cycler==0.10.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py/cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting kiwisolver==1.1.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/kiwisolver-1.1.0-cp38-cp38-manylinux1_x86_64.whl (91 kB)
     |████████████████████████████████| 91 kB 198 kB/s 
Collecting matplotlib==3.1.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/matplotlib-3.1.2-cp38-cp38-manylinux1_x86_64.whl (13.1 MB)
     |████████████████████████████████| 13.1 MB 940 kB/s 
Collecting pyproj==2.5.0
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/pyproj-2.5.0-cp38-cp38-manylinux1_x86_64.whl (10.4 MB)
     |████████████████████████████████| 10.4 MB 14 kB/s 
Collecting GDAL==3.2.2
  Downloading https://wheelhouse.openquake.org/v3/linux/py38/GDAL-3.2.2-cp38-cp38-manylinux1_x86_64.whl (103.2 MB)
     |████████████████████████████████| 103.2 MB 8.2 kB/s 
Installing collected packages: vine, six, setuptools, amqp, urllib3, sqlparse, pytz, python-dateutil, pyparsing, numpy, kombu, kiwisolver, idna, cycler, chardet, certifi, billiard, toml, Shapely, scipy, requests, pyzmq, pyshp, pyproj, psutil, pbr, pandas, matplotlib, h5py, GDAL, docutils, Django, decorator, celery
  Attempting uninstall: setuptools
    Found existing installation: setuptools 44.0.0
    Uninstalling setuptools-44.0.0:
      Successfully uninstalled setuptools-44.0.0
Successfully installed Django-2.2.7 GDAL-3.2.2 Shapely-1.7.1 amqp-2.4.2 billiard-3.6.0.0 celery-4.3.0 certifi-2019.3.9 chardet-3.0.4 cycler-0.10.0 decorator-4.4.2 docutils-0.14 h5py-2.10.0 idna-2.8 kiwisolver-1.1.0 kombu-4.5.0 matplotlib-3.1.2 numpy-1.18.2 pandas-1.1.5 pbr-5.2.0 psutil-5.6.7 pyparsing-2.4.0 pyproj-2.5.0 pyshp-1.2.3 python-dateutil-2.8.0 pytz-2019.1 pyzmq-19.0.0 requests-2.22.0 scipy-1.4.1 setuptools-56.0.0 six-1.12.0 sqlparse-0.3.0 toml-0.10.0 urllib3-1.25.3 vine-1.3.0
Collecting openquake.engine
  Downloading openquake.engine-3.11.3-py3-none-any.whl (31.6 MB)
     |████████████████████████████████| 31.6 MB 17.6 MB/s 
Requirement already satisfied: shapely<1.8,>=1.3 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (1.7.1)
Requirement already satisfied: django<2.3,>=1.11 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (2.2.7)
Requirement already satisfied: h5py<2.11,>=2.10 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (2.10.0)
Requirement already satisfied: numpy<1.19,>=1.16 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (1.18.2)
Requirement already satisfied: pyproj>=1.9 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (2.5.0)
Requirement already satisfied: scipy<1.5,>=1.3 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (1.4.1)
Requirement already satisfied: docutils<0.15,>=0.11 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (0.14)
Requirement already satisfied: matplotlib<3.2,>=1.5 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (3.1.2)
Requirement already satisfied: pyshp==1.2.3 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (1.2.3)
Requirement already satisfied: pyzmq<20.0 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (19.0.0)
Requirement already satisfied: psutil<5.7,>=2.0 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (5.6.7)
Requirement already satisfied: setuptools in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (56.0.0)
Requirement already satisfied: decorator>=4.3 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (4.4.2)
Requirement already satisfied: toml in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (0.10.0)
Requirement already satisfied: requests<2.24,>=2.20 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (2.22.0)
Requirement already satisfied: pandas<1.2,>=0.25 in /opt/openquake/lib/python3.8/site-packages (from openquake.engine) (1.1.5)
Requirement already satisfied: pytz in /opt/openquake/lib/python3.8/site-packages (from django<2.3,>=1.11->openquake.engine) (2019.1)
Requirement already satisfied: sqlparse in /opt/openquake/lib/python3.8/site-packages (from django<2.3,>=1.11->openquake.engine) (0.3.0)
Requirement already satisfied: six in /opt/openquake/lib/python3.8/site-packages (from h5py<2.11,>=2.10->openquake.engine) (1.12.0)
Requirement already satisfied: python-dateutil>=2.1 in /opt/openquake/lib/python3.8/site-packages (from matplotlib<3.2,>=1.5->openquake.engine) (2.8.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/openquake/lib/python3.8/site-packages (from matplotlib<3.2,>=1.5->openquake.engine) (1.1.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/openquake/lib/python3.8/site-packages (from matplotlib<3.2,>=1.5->openquake.engine) (2.4.0)
Requirement already satisfied: cycler>=0.10 in /opt/openquake/lib/python3.8/site-packages (from matplotlib<3.2,>=1.5->openquake.engine) (0.10.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/openquake/lib/python3.8/site-packages (from requests<2.24,>=2.20->openquake.engine) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/openquake/lib/python3.8/site-packages (from requests<2.24,>=2.20->openquake.engine) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/openquake/lib/python3.8/site-packages (from requests<2.24,>=2.20->openquake.engine) (1.25.3)
Requirement already satisfied: certifi>=2017.4.17 in /opt/openquake/lib/python3.8/site-packages (from requests<2.24,>=2.20->openquake.engine) (2019.3.9)
Installing collected packages: openquake.engine
Successfully installed openquake.engine-3.11.3
The standalone applications are not installed yet
Created /opt/openquake/openquake.cfg
Created symlink /etc/systemd/system/multi-user.target.wants/openquake-dbserver.service → /lib/systemd/system/openquake-dbserver.service.
Created symlink /etc/systemd/system/multi-user.target.wants/openquake-webui.service → /lib/systemd/system/openquake-webui.service.
The engine was installed successfully.
You can run a test calculation with the command
/opt/openquake/bin/oq engine --run /opt/openquake/demos/hazard/AreaSourceClassicalPSHA/job.ini

Docker image

docker run -it --rm -p 8800:8800 \
  -v /home/foka/OpenDRR/thobbs:/home/openquake/thobbs \
  -v /home/foka/OpenDRR/git/scenario-catalogue:/home/openquake/scenario-catalogue \
  -v /home/foka/OpenDRR/git/openquake-inputs:/home/openquake/openquake-inputs \
  -v /home/foka/OpenDRR/git/CanadaSHM6:/home/openquake/CanadaSHM6 \
  openquake/engine:3.11

Note: openquake/engine:3.11 and openquake/engine:nightly are rather different with respect to OpenQuake files organization. https://github.com/gem/oq-engine/tree/master/docker ?

model-factory

model-factory/Create_table_shakemap_update_ste.sql at master · OpenDRR/model-factory

Awesome tools

QuicKey – The quick tab switcher - Chrome Web Store

One-liners

for i in */.git; do ( cd `dirname $i` && pwd && git config user.name 'Anthony Fok' && git config user.email '[email protected]' ); done

Git LFS Troubleshooting (to be sorted)

BFG Repo-Cleaner

Great Articles

Chrome tabs overload

Docker

Docker pull limits coming Nov 2, 2020 · Issue #669 · containrrr/watchtower
- Increase Rate Limits | Docker
- Download rate limit | Docker Documentation

Videos

5 Ways to DevOps-ify your App - Github Actions Tutorial - YouTube

Compression tools comparisons

Related to https://github.com/OpenDRR/data/issues/40

7z

7za a -r -t7z -m0=lzma2 -mx=9 -mfb=273 -md=29 -ms=8g -mmt=off -mmtf=off -mqs=on -bt -bb3 \
    nhsl_hazard_threat_all_indicators_b.7z nhsl_hazard_threat_all_indicators_b.gpkg

xz

zpaq

Incremental Journaling Backup Utility and Archiver http://mattmahoney.net/dc/zpaq.html

Pros: Cross-platform; impressive compression ratio (the very best?).

Cons: Very long compression time, and decompression takes just as long.

foka@debian-e431:/mnt/easystore/OpenDRR/git/data/assets/data/national_human_settlement_layers$ time zpaq a nhsl_hazard_threat_all_indicators_b.zpaq  nhsl_hazard_threat_all_indicators_b.gpkg -m5
zpaq v7.15 journaling archiver, compiled Jan  5 2021
Creating nhsl_hazard_threat_all_indicators_b.zpaq at offset 0 + 0
Adding 1243.942912 MB in 1 files -method 56 -threads 4 at 2021-05-11 18:14:11.
 5.41% 0:00:17 [1..612] 67098701 -method 56,84,0
10.79% 0:00:17 [613..1409] 67092373 -method 56,78,0
16.19% 0:00:17 [1410..2263] 66966409 -method 56,84,0
21.59% 0:00:18 [2264..3071] 67035145 -method 56,88,0
26.97% 0:00:20 [3072..4020] 67058822 -method 56,90,0
32.35% 0:00:19 [4021..4802] 67098440 -method 56,82,0
37.74% 0:00:19 [4803..5588] 67068112 -method 56,85,0
43.14% 0:00:18 [5589..6379] 66986584 -method 56,86,0
48.53% 0:10:16 [6380..7095] 67006453 -method 56,87,0
53.91% 0:08:28 [7096..7866] 67099092 -method 56,85,0
59.30% 0:06:49 [7867..8602] 66979716 -method 56,84,0
64.68% 0:05:26 [8603..9542] 67092869 -method 56,81,0
70.07% 0:07:42 [9543..10370] 67029836 -method 56,83,0
75.46% 0:05:56 [10371..11142] 66686795 -method 56,86,0
80.82% 0:04:21 [11143..11937] 67030623 -method 56,85,0
86.22% 0:02:56 [11938..12561] 66977559 -method 56,86,0
91.60% 0:02:24 [12562..13263] 67098369 -method 56,106,0
97.00% 0:00:49 [13264..14138] 67036835 -method 56,185,0
100.00% 0:00:00 + nhsl_hazard_threat_all_indicators_b.gpkg 1243942912
100.00% 0:00:00 [14139..14607] 37558759 -method 56,177,0
1 +added, 0 -removed.

0.000000 + (1243.942912 -> 1243.942912 -> 50.452521) = 50.452521 MB
2623.651 seconds (all OK)

real	43m43.659s
user	139m9.675s
sys	0m52.396s

foka@debian-e431:/mnt/easystore/OpenDRR/git/data/assets/data/national_human_settlement_layers$ time zpaq add nhsl_hazard_threat_all_indicators_b_-m4.zpaq  nhsl_hazard_threat_all_indicators_b.gpkg -m4
zpaq v7.15 journaling archiver, compiled Jan  5 2021
Creating nhsl_hazard_threat_all_indicators_b_-m4.zpaq at offset 0 + 0
Adding 1243.942912 MB in 1 files -method 46 -threads 4 at 2021-05-11 19:53:50.
 5.41% 0:00:16 [1..612] 67098701 -method 46,84,0
10.79% 0:00:15 [613..1409] 67092373 -method 46,78,0
16.19% 0:00:15 [1410..2263] 66966409 -method 46,84,0
21.59% 0:00:16 [2264..3071] 67035145 -method 46,88,0
26.97% 0:00:17 [3072..4020] 67058822 -method 46,90,0
32.35% 0:00:16 [4021..4802] 67098440 -method 46,82,0
37.74% 0:00:15 [4803..5588] 67068112 -method 46,85,0
43.14% 0:00:14 [5589..6379] 66986584 -method 46,86,0
48.53% 0:02:50 [6380..7095] 67006453 -method 46,87,0
53.91% 0:02:19 [7096..7866] 67099092 -method 46,85,0
59.30% 0:01:53 [7867..8602] 66979716 -method 46,84,0
64.68% 0:01:30 [8603..9542] 67092869 -method 46,81,0
70.07% 0:02:10 [9543..10370] 67029836 -method 46,83,0
75.46% 0:01:39 [10371..11142] 66686795 -method 46,86,0
80.82% 0:01:13 [11143..11937] 67030623 -method 46,85,0
86.22% 0:00:49 [11938..12561] 66977559 -method 46,86,0
91.60% 0:00:41 [12562..13263] 67098369 -method 46,106,0
97.00% 0:00:14 [13264..14138] 67036835 -method 46,185,0
100.00% 0:00:00 + nhsl_hazard_threat_all_indicators_b.gpkg 1243942912
100.00% 0:00:00 [14139..14607] 37558759 -method 46,177,0
1 +added, 0 -removed.

0.000000 + (1243.942912 -> 1243.942912 -> 62.679518) = 62.679518 MB
761.780 seconds (all OK)

real	12m41.790s
user	38m38.066s
sys	0m18.715s

$ time zpaq x nhsl_hazard_threat_all_indicators_b_-m4.zpaq -test
zpaq v7.15 journaling archiver, compiled Jan  5 2021
nhsl_hazard_threat_all_indicators_b_-m4.zpaq: 1 versions, 1 files, 14607 fragments, 62.679518 MB
Extracting 1243.942912 MB in 1 files -threads 4
[1..612] -> 67098701
> nhsl_hazard_threat_all_indicators_b.gpkg
 5.39% 0:44:21 [1410..2263] -> 66966409
10.78% 0:21:01 [2264..3071] -> 67035145
16.17% 0:13:35 [613..1409] -> 67092373
21.56% 0:19:04 [3072..4020] -> 67058822
26.95% 0:14:29 [4021..4802] -> 67098440
32.34% 0:11:14 [5589..6379] -> 66986584
37.73% 0:08:57 [4803..5588] -> 67068112
43.12% 0:10:31 [7867..8602] -> 66977792
48.50% 0:08:31 [7096..7866] -> 67099092
53.90% 0:06:52 [6380..7095] -> 67006453
59.28% 0:05:38 [8603..9542] -> 67092480
64.68% 0:05:58 [9543..10370] -> 67026944
70.07% 0:04:40 [10371..11142] -> 66686795
75.43% 0:03:35 [11143..11937] -> 67030623
80.81% 0:02:38 [11938..12561] -> 66977559
86.20% 0:01:56 [14139..14607] -> 37558759
89.22% 0:01:32 [12562..13263] -> 67098369
94.61% 0:00:43 [13264..14138] -> 67036835
764.662 seconds (all OK)

real	12m44.669s
user	39m10.000s
sys	0m17.122s

Eventual talk/presentation on GitHub tips

git config --global user.name 'FirstName LastName'
git config --global user.email '[email protected]'

Git LFS

Random experiments

OpenQuake with Python 3.9

Seems to work

PYTHONPATH=${PYTHONPATH}:~/gem/oq-engine PATH=~/gem/oq-engine/bin:$PATH run-demos.sh demos/ 2>&1 | tee ~/OpenDRR/tmp/oq/oq-run-demos-2.log

Benchmark of Python inside vs outside Docker

Performance version: 1.0.2
Report on Linux-5.10.0-7-amd64-x86_64-with-glibc2.31
Number of logical CPUs: 4
Start date: 2021-06-27 01:45:29.552710
End date: 2021-06-27 01:46:07.370620

### 2to3 ###
Mean +- std dev: 661 ms +- 39 ms

### django_template ###
Mean +- std dev: 116 ms +- 10 ms

### unpickle_pure_python ###
Mean +- std dev: 547 us +- 27 us

Performance version: 1.0.2
Report on Linux-5.10.0-7-amd64-x86_64-with-glibc2.28
Number of logical CPUs: 4
Start date: 2021-06-27 08:18:00.770937
End date: 2021-06-27 08:18:49.560909

### 2to3 ###
Mean +- std dev: 932 ms +- 32 ms

### django_template ###
Mean +- std dev: 172 ms +- 5 ms

### unpickle_pure_python ###
Mean +- std dev: 1.10 ms +- 0.05 ms

### 2to3 ###
Mean +- std dev: 868 ms +- 38 ms

### django_template ###
Mean +- std dev: 148 ms +- 6 ms

### unpickle_pure_python ###
Mean +- std dev: 964 us +- 30 us