Issues pulling images from embedded registry mirror

[vngah]

Hi all,

I have a single node k3s cluster in an air-gapped environment setup using the documentation (https://docs.k3s.io/installation/airgap). The node is running Ubuntu 22.04.5 LTS with ufw enabled. The k3s cluster has an embedded registry mirror enabled (thru embedded-registry: true in the config file)

I did import images using

ctr -n k8s.io image import <images.tar>

Registry configuration (/etc/rancher/k3s/registries.yaml) looks like

mirrors:
  "*":
    endpoint:
      - "http://<registrydomain>:5001/“

When trying to deploy nginx ingress am getting (net/http: HTTP/1.x transport connection broken: malformed HTTP response)

  Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.0@sha256:aaafd456bda110628b2d4ca6296f38731a3aaf0bf7581efae824a41c770a8fc4": failed to pull and unpack image "registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:aaafd456bda110628b2d4ca6296f38731a3aaf0bf7581efae824a41c770a8fc4": failed to resolve reference "registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:aaafd456bda110628b2d4ca6296f38731a3aaf0bf7581efae824a41c770a8fc4": failed to do request: Head "http://<registrydomain>:5001/v2/ingress-nginx/kube-webhook-certgen/manifests/sha256:aaafd456bda110628b2d4ca6296f38731a3aaf0bf7581efae824a41c770a8fc4?ns=registry.k8s.io": net/http: HTTP/1.x transport connection broken: malformed HTTP response "f߃W>y)\x89ct\x84\x83\xaa\x80E\xc4`$c#⼛\xe6\xc2@\xf0\x90\x9d\r\xf5\xf9+R.\x7f\xfa\xd4N\xfe\x93,K\x16"

I tried deploying the Nginx Ingress with ufw enabled and disabled and was still getting the same error. Is anyone familiar with this issues in combination with the embedded mirror? Suggestions are much appreciated

[brandond]

malformed HTTP response "f߃W>y)\x89ct\x84\x83\xaa\x80E\xc4`$c#⼛\xe6\xc2@\xf0\x90\x9d\r\xf5\xf9+R.\x7f\xfa\xd4N\xfe\x93,K\x16"

That sure looks like a malformed response to me. I don't see how K3s or containerd is doing anything wrong here though.

Are you sure you didn't want to use https? Can you successfully curl -vk http://<registrydomain>:5001/v2/“ from this node?

Also, are you trying to use the embedded registry with airgap tarballs, or your local registry mirror? Or both? What you're doing here isn't using the embedded registry at all. It's failing to pull from the mirror you specified in the config.

[vngah]

Also, are you trying to use the embedded registry with airgap tarballs, or your local registry mirror? Or both? What you're doing here isn't using the embedded registry at all. It's failing to pull from the mirror you specified in the config.

Am trying to use the embedded registry with airgap tarballs. Right now the tarballs are downloaded, scp to the server and imported using ctr. Am new to this so am not sure what is the best approach to use here!

The curl output is

root@***:/home/ubuntu# curl -vk http://registrydomain:5001/v2/
*   Trying *.*.*.*:5001...
* Connected to registrydomain (*.*.*.*) port 5001 (#0)
> GET /v2/ HTTP/1.1
> Host: registrydomain:5001
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Received HTTP/0.9 when not allowed
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed
root@***:/home/ubuntu# 

[brandond]

If you only want to use the embedded registry then why are you configuring http://registrydomain:5001/v2/ as a registry mirror? Is this a service under your control or are you just copy-pasting from ChatGPT or something? Whatever it is, it appears to be broken or not actually a functional HTTP server.

Please read the docs at https://docs.k3s.io/installation/registry-mirror