Nice work. Would be interested to see some performance comparisons. LIMIT/OFFSET is also used extensively in many of the *_postgres2es.py scripts.
Would like to make sure that changing the pagination method is at least:

• more efficient than LIMIT/OFFSET
• as robust in making sure that all records and no duplicates are copied from PostGIS to ES

Thank you @drotheram for the great ideas! You have hit the nail on the head, and indeed, my first stack run has already failed — related to #137, used a wrong search-and-replace regex and changing python3 to pypy3 in too many places, and Debian's pypy3 and python3-numpy not working together. Easy to fix, but does show how fragile these changes can be, especially when hurriedly committed in a somewhat sleepy state. 😅

But yeah, there are three main changes in this series of pull requests:

1. New base OS image for python-env, and slightly updated Python libraries
2. The switch from CPython to PyPy for some scripts
3. LIMIT/OFFSET vs server-side cursor

and each of them can introduce unintended changes to the data. I have been wondering e.g. how to verify that the actual data exported to Elasticsearch are identical: Maybe dumping the GeoJSON data for each 10000 rows as individual files to compare? Maybe using a separate script to query the Elasticsearch server?

Ideally, probably need to run 5 test cases:

1. v1.2.0 as control
2. with the new debian:sid-20201012-slim-based python-env OS image, but nothing else changed. (CPython + LIMIT/OFFSET)
3. No. 2 + PyPy + LIMIT/OFFSET
4. No. 2 + CPython + server-side cursor
5. No. 2 + PyPy + server-side cursor

Ditto for the benchmark, and in particular, I am interested to find out whether the LIMIT/OFFSET method does gets slower and slower when OFFSET gets really big. Yes, let's record the timestamps and plot them! And maybe have these tests and benchmarks as part of the Python scripts so we can monitor data integrity and performance over time.

Sounds a bit scary but at the same time exciting to me, but yeah, this would be something that let us monitor the health and performance of our stack. Probably as something that CI/CD with GitHub Actions could do.

And before I forget, I am also starting to think that in the future we could have the Docker Compose logs, test and benchmark results semi-automatically uploaded to an S3 bucket for record and analysis. (Privacy alert!) With the end user's consent of course. And also collect e.g. CPU, RAM, hard disk information too... kind of like CPU-ID or Passmark... Or maybe not that good of an idea. 😅