Kind with podman fails to create cluster on Ubuntu 20.04.6 (kernel 5.15.0-1056-azure and likely more versions)

[tnqn]

What happened:

Podman (ipv4, singleNode) and Podman (ipv4, multiNode) keeps failing with the following error for PR #3483 which has nothing to do with podman.

RROR: failed to create cluster: command "podman run --name kind-control-plane --hostname kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume c53bc695772fc5446770f8cf711cf7c92571a7aad2139f3cb90def947d5d3d8e:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --cgroupns=private --publish=0.0.0.0:80:80/tcp --publish=0.0.0.0:443:443/tcp --publish=127.0.0.1:37201:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf docker.io/kindest/node@sha256:51a1434a5397193442f0be2a297b488b6c919ce8a3931be0ce822606ea5ca245" failed with error: exit status 126

Command Output: WARNING: The same type, major and minor should not be used for multiple devices.
Error: OCI runtime error: chmod `run/shm`: Operation not supported

Stack Trace: 
sigs.k8s.io/kind/pkg/errors.WithStack
	sigs.k8s.io/kind/pkg/errors/errors.go:59
sigs.k8s.io/kind/pkg/exec.(*LocalCmd).Run
	sigs.k8s.io/kind/pkg/exec/local.go:124
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman.createContainerWithWaitUntilSystemdReachesMultiUserSystem
	sigs.k8s.io/kind/pkg/cluster/internal/providers/podman/provision.go:431
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman.planCreation.func2
	sigs.k8s.io/kind/pkg/cluster/internal/providers/podman/provision.go:111
sigs.k8s.io/kind/pkg/errors.UntilErrorConcurrent.func1
	sigs.k8s.io/kind/pkg/errors/concurrent.go:30
runtime.goexit
	runtime/asm_amd64.s:1598

I compared succeeded jobs and failed jobs and found the difference is Github runner image versions:

• The last succeeded job's runner image version is 20240212.2.0.
• The failed job's runner image version is 20240218.1.0.

Comparing 20240212.2.0 and 20240218.1.0, the kernel version was upgraded from 5.15.0-1054-azure to 5.15.0-1056-azure.

According to changelog of linux-azure-5.15_5.15.0-1056.64, it includes "attr: block mode changes of symlinks", which causes the error. The issue has been fixed in crun by containers/crun#1309 but needs crun >=v1.9.1. However, Ubuntu 20.04 only packages "crun/focal 0.12.1+dfsg-1", and neither Jammy (packaged with 0.17+dfsg-1.1) nor Mantic (packaged with 1.8.5-1) meets the requirement, so even upgrading runner image to ubuntu-latest won't fix this.

@BenTheElder @aojea Currently none of new PRs can succeed. Could we download crun from other source like assets from https://github.com/containers/crun/releases/tag/1.14.3 when running the Github actions to fix it? or it should be fixed in another way?

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Any PR running the two validations can produce it.

Anything else we need to know?:

Environment:

• kind version: (use kind version):
• Runtime info: (use docker info or podman info):
• OS (e.g. from /etc/os-release):
• Kubernetes version: (use kubectl version):
• Any proxies or other special environment settings?:

[tnqn]

Could we download crun from other source like assets from https://github.com/containers/crun/releases/tag/1.14.3 when running the Github actions to fix it?

#3527 implements this and it works.

[BenTheElder]

Thank you!