[WIP]Add ipv6 error message check to support the nerdctl

[yankay]

Do not merge

HI dears.

I'm one of the maintainers of nerdctl, and I'm working on the nerdctl to support the kind recently.

• nerdctl doesn't work with kind containerd/nerdctl#349
• containerd / nerdctl provider support #2317

The cluster has been successfully created by kind with nerdctl:

root@kay201:~/oss/kind# ln -s /usr/local/bin/nerdctl /usr/local/bin/docker ## Using the nerdctl instead of docker
root@kay201:~/oss/kind# export KUBE_VERSION="1.28.0"
root@kay201:~/oss/kind#  ./bin/kind create cluster --config /root/kind.yml --image release-ci.daocloud.io/docker.m.daocloud.io/kindest/node:v${KUBE_VERSION}

Creating cluster "kind" ...
 ✓ Ensuring node image (release-ci.daocloud.io/docker.m.daocloud.io/kindest/node:v1.28.0) 🖼
 ✓ Preparing nodes 📦 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind"

root@kay201:~/oss/kind# ./kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   76s   v1.28.0
kind-worker          Ready    <none>          51s   v1.28.0
kind-worker2         Ready    <none>          53s   v1.28.0
kind-worker3         Ready    <none>          52s   v1.28.0

The issues of nerdctl has been almost fixed, only 2 PR is pending:

• Fix nerdctl ps --format {{.Names}} behaviour containerd/nerdctl#2594
• Support create network with com.docker.network.bridge.enable_ip_masquerade option containerd/nerdctl#2592

And there needs a change of kind.

Because the nerdctl do not support the ipv6 network now, so it needs kind to skip the ipv6 config.
If the kind can read the error message of kind, and skip it , the cluster can be create successully.

Thanks :-)

[yankay]

/retest

[aojea]

the problem I see is that once nerdctl implement the --ipv6 flag we'll have to remove the error, right?

what is blocking nerdctl to implement the ipv6 flag? is there any link or place where I can look more?

[BenTheElder]

the problem I see is that once nerdctl implement the --ipv6 flag we'll have to remove the error, right

I think it depends on how long it takes to ship IPv6, right now released nerdctl won't work even with this patch AIUI but there might be intermediate versions worth supporting and this path could be safely left in place.

I do wonder if we should fork to a nerdctl provider though, which would also let us avoid requiring the user to put a fake docker symlink in their path. We've had a bad experience with this with podman in the past versus detecting that it's actually podman and having dedicated provider / code paths.

[aojea]

I do wonder if we should fork to a nerdctl provider though, which would also let us avoid requiring the user to put a fake docker symlink in their path. We've had a bad experience with this with podman in the past versus detecting that it's actually podman and having dedicated provider / code paths.

@AkihiroSuda has some list somewhere with the missing things, his opinion will be interesting

[AkihiroSuda]

what is blocking nerdctl to implement the ipv6 flag? is there any link or place where I can look more?

The lack of the CI for IPv6 is considered to be a blocker for this PR:

• Support IPv6 for nerdctl network containerd/nerdctl#1558

I think this is just a soft blocker, as Docker doesn't have IPv6 CI either (IIUC).

[AkihiroSuda]

I do wonder if we should fork to a nerdctl provider though, which would also let us avoid requiring the user to put a fake docker symlink in their path.

SGTM, but I think the implementation should try to avoid the amount of the code clones (unlike podman driver).

[aojea]

The lack of the CI for IPv6 is considered to be a blocker for this PR:

we run ipv6 in github actions, what is exactly nlockling ... I'll try to take a look later

[BenTheElder]

SGTM, but I think the implementation should try to avoid the amount of the code clones (unlike podman driver).

FWIW It's not too much code (~1200 loc for the entire "provider") and when we review changes to the existing drivers we ask that they be copied between them when PRs aren't obviously specific to that driver.

We can start with a copy paste and just swap docker => nerdctl, if enough code stays common and/or they get larger we can refactor out more common code now that we have N=3 examples to derive APIs from.

$ echo $(pwd) && cloc .
/usr/local/google/home/bentheelder/go/src/sigs.k8s.io/kind/pkg/cluster/internal/providers/docker
      10 text files.
       9 unique files.                              
       1 file ignored.

github.com/AlDanial/cloc v 1.96  T=0.01 s (648.8 files/s, 124063.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Go                               9            170            354           1197
-------------------------------------------------------------------------------
SUM:                             9            170            354           1197
-------------------------------------------------------------------------------

I don't think IPv6 support should be a blocker, kind already works on hosts without IPv6 and will need to continue to for the forseeable future. IPv6 is also somewhat broken in docker, kind.

I think we'll also want to get some minimal CI up, at least testing one configuration of nerdctl

[yankay]

Thanks @aojea @BenTheElder @AkihiroSuda for the PR review

The ipv6 has been supported by the nerdctl by containerd/nerdctl#1558
The kind cluster can be created successfully without the change :-)

Thank you all very much. :-)

---

And what's the next step to do to make kind support nerdctl ? for example:

• [docs] add experimetal support for nerdctl
• e2e test (minimal CI up)
• find bug and fix, eg:
  • ./bin/kind get clusters error , nerdctl ps -a --filter label=io.x-k8s.kind.cluster --format '{{.Label "io.x-k8s.kind.cluster"}}' behaviour is inconsistent with docker containerd/nerdctl#2598

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: yankay
Once this PR has been reviewed and has the lgtm label, please ask for approval from aojea. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@yankay: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kind-e2e-kubernetes-1-24	239bc7d	link	true	/test pull-kind-e2e-kubernetes-1-24

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.