Abort and restart curl download if speed too slow and/or fails

[anthonyfok]

My most recent test run of docker-composer up --build got stuck when curl tries to fetch social-vulnerability/social-vulnerability-census_2021.csv. The download became extremely slow at around 3.5MB of the 248MB file, with download speed of 9559 B/s and an estimated total download time of up to whopping 24 hours?!

$ docker container attach opendrr-api_python-opendrr_1 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  1  248M    1 3552k    0     0   3027      0 23:51:50  0:20:01 23:31:49  9559
  1  248M    1 3648k    0     0   3030      0 23:50:25  0:20:32 23:29:53  9596
  1  248M    1 3648k    0     0   3023      0 23:53:43  0:20:35 23:33:08  9596
  1  248M    1 3648k    0     0   3013      0 23:58:29  0:20:39 23:37:50     0
  1  248M    1 3648k    0     0   3008      0 24:00:52  0:20:41 23:40:11     0
  1  248M    1 3936k    0     0   3024      0 23:53:15  0:22:12 23:31:03   1
...
  6  248M    6 16.6M    0     0   5713      0 12:38:39  0:50:58 11:47:41 2335k
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

But it eventually timed out, supposedly because the token used for download expired. Very strange indeed.

The download command at the time was:

curl -o social-vulnerability-census_2021.csv -L \
    https://media.githubusercontent.com/media/OpenDRR/model-inputs/develop/social-vulnerability/social-vulnerability-census_2021.csv?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
    --retry 999 --retry-max-time 0

I was able to run the same command manually elsewhere, and had the file downloaded rather quickly, multiple times even, until the token expired supposedly. The curl instance inside the Docker container never sped up though and eventually died.

So, may need to do too things:

1. Abort curl if download speed is too slow with the -Y and -y options. (?) See https://stackoverflow.com/questions/3756614/curl-abort-transfer-if-upload-speed-is-slower-than and https://daniel.haxx.se/blog/2020/05/11/curl-ootw-y-speed-limit/
2. Make a loop such that if curl fails, query GitHub API to get a new download_url, and fetch the LFS file again.

Perhaps look at --connect-timeout and -C too?

       --connect-timeout <seconds>
              Maximum  time  in  seconds  that  you allow curl's connection to
              take.  This only limits the connection phase, so  if  curl  con‐
              nects  within the given period it will continue - if not it will
              exit.  Since version 7.32.0, this option accepts decimal values.

              If this option is used several times, the last one will be used.

              See also -m, --max-time.

       -C, --continue-at <offset>
              Continue/Resume a previous file transfer at  the  given  offset.
              The  given  offset  is  the  exact  number of bytes that will be
              skipped, counting from the beginning of the source  file  before
              it is transferred to the destination.  If used with uploads, the
              FTP server command SIZE will not be used by curl.

              Use "-C -" to tell curl to automatically find out  where/how  to
              resume  the  transfer. It then uses the given output/input files
              to figure that out.

              If this option is used several times, the last one will be used.

              See also -r, --range.

       -Y, --speed-limit <speed>
              If a download is slower than this given speed (in bytes per sec‐
              ond) for speed-time seconds it gets aborted. speed-time  is  set
              with -y, --speed-time and is 30 if not set.

              If this option is used several times, the last one will be used.

       -y, --speed-time <seconds>
              If a download is slower than speed-limit bytes per second during
              a speed-time period, the download gets aborted. If speed-time is
              used,  the  default  speed-limit  will  be 1 unless set with -Y,
              --speed-limit.

              This option controls transfers and thus  will  not  affect  slow
              connects  etc.  If this is a concern for you, try the --connect-
              timeout option.

              If this option is used several times, the last one will be used.

See Meta Issue #76, and related issues #66 (retries), #83 (verify checksum).

[anthonyfok]

Another error:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 97 16.1M   97 15.7M    0     0  44375      0  0:06:20  0:06:12  0:00:08 3799k 
curl: (18) transfer closed with 347510 bytes remaining to read

where "18" is an exit code meaning:

        18     Partial file. Only a part of the file was transferred.