Impact of Maxmind R2 presigned URLs

[vinay01tech]

Hi Team,

As mentioned in the below url:

https://dev.maxmind.com/geoip/release-notes/2024#presigned-urls-for-database-downloads

Maxmind will start using presigned URL and will be applied by 1st May.

If we see the code of K8s Nginx Ingress Controller, url is hard-coded as https://download.maxmind.com/app/geoip_download?license_key=%v&edition_id=%v&suffix=tar.gz

https://github.com/kubernetes/ingress-nginx/blob/main/internal/nginx/maxmind.go#L62

Have few questions ?

1. Are we impacted with this change ?
2. If yes,
   • Do we need to wait for new version upgrade ?
   • Or there is a workaround in-place to unblock older version

Thanks
Vinay

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ufou]

Maxmind have also recently introduced a download limit of 30 times per 24 hours (for the free editions), this means if you are running several ingresses + nginx instances requiring the database you can hit this limit quite quickly, I tried to work around this by setting an extra arg to the ingress controller for maxmind mirror, but this does not seem to work, I have uploaded the databases to a custom location and they are in tar.gz format, which is what the code in maxmind.go suggests, but I then get:

I0325 12:55:13.424947       7 flags.go:387] "downloading maxmind GeoIP2 databases"
E0325 12:55:13.770916       7 maxmind.go:74] GeoLite2-City.mmdb not found
E0325 12:55:13.794877       7 maxmind.go:74] GeoLite2-City.mmdb not found

[strongjz]

Looks like it does download the db in the flag processing, like your logs say

https://github.com/kubernetes/ingress-nginx/blob/main/pkg/flags/flags.go#L400

It should error out @ufou if there is an issue with the download.

Can you exec into the container and see if the file does exist?

[NeckBeardPrince]

Maxmind have also recently introduced a download limit of 30 times per 24 hours (for the free editions), this means if you are running several ingresses + nginx instances requiring the database you can hit this limit quite quickly, I tried to work around this by setting an extra arg to the ingress controller for maxmind mirror, but this does not seem to work, I have uploaded the databases to a custom location and they are in tar.gz format, which is what the code in maxmind.go suggests, but I then get:

I0325 12:55:13.424947       7 flags.go:387] "downloading maxmind GeoIP2 databases"
E0325 12:55:13.770916       7 maxmind.go:74] GeoLite2-City.mmdb not found
E0325 12:55:13.794877       7 maxmind.go:74] GeoLite2-City.mmdb not found

Can you post a snippet of your deployment manifest?

[NeckBeardPrince]

Looks like it does download the db in the flag processing, like your logs say

https://github.com/kubernetes/ingress-nginx/blob/main/pkg/flags/flags.go#L400

It should error out @ufou if there is an issue with the download.

Can you exec into the container and see if the file does exist?

What is the expected format for --maxmind-mirror?
Is it a URL pointing to the database in a .tar.gz format?
Or do you have to use https://www.npmjs.com/package/maxmind-geolite-mirror?

[ufou]

I set a plain URL for the maxmind mirror parameter and had both required databases named like GeoLite2-City.tar.gz GeoLite2-ASN.tar.gz but perhaps the requirement was to have both databases in a single tar.gz file named in a similar convention to the real maxmind download servers instead? I don't think I tried that, we have worked around this anyway by signing up for a 2nd maxmind account, if I get round to testing this I will report back

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[ufou]

Looks like it does download the db in the flag processing, like your logs say

https://github.com/kubernetes/ingress-nginx/blob/main/pkg/flags/flags.go#L400

It should error out @ufou if there is an issue with the download.

Can you exec into the container and see if the file does exist?

Sorry for the delayed response, we thought we'd found a work around for this but actually we haven't - we just didn't hit our limit of daily downloads in a while, except today we were doing some rolling node updates, so consequently all the ingress pods got re-homed and then we hit the maxmind license limit.

So, when I override the mirror using --maxmind-mirror= as an arg to the controller, there are no complaints about the download, so I can only assume we dowloaded and verfied the db file exists as I see no error being surfaced from this function

ingress-nginx/internal/nginx/maxmind.go

Line 141 in cee3fb3

func downloadDatabase(dbName string) error {

But the logs say the db cannot be found:

I0711 14:16:13.388180       6 flags.go:387] "downloading maxmind GeoIP2 databases"
E0711 14:16:13.693421       6 maxmind.go:74] GeoLite2-City.mmdb not found
W0711 14:16:13.693440       6 store.go:1214] The GeoIP2 feature is enabled but the databases are missing. Disabling
E0711 14:16:13.702215       6 maxmind.go:74] GeoLite2-City.mmdb not found

and indeed they are not present, there is no /etc/nginx/geoip directory, there is this, but also empty:

ls -la /etc/ingress-controller/geoip/
total 12
drwxr-xr-x    2 www-data www-data      4096 Apr 10 18:08 .
drwxr-xr-x    1 root     root          4096 Apr 10 18:08 ..

I enabled debug logging for the controller but it gave no clues, I'd like to help resolve this, are there some hints about how to print more info on the controller bootup of GeoIP?

[ufou]

I rebuilt the docker image and enabled some debug lines, and the download appears to work now, I think this was down to 2 things:

1. I had a trailing slash on the maxmind URL
2. each mmdb file should be in a tar.gz file of it's own and not in a sub directory within the tar