CSI plugins do not work out of the box

[mhutter]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 5.14.0-162.6.1.el9_1.0.1.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Nov 28 18:44:09 UTC 2022 x86_64 GNU/Linux
NAME="Rocky Linux"
VERSION="9.1 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.1 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.1"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"

Version

v1.25.4+k0s.0

Sysinfo

`k0s sysinfo`

```text Machine ID: "633c16e45de5eb29f0558739e10d3ce2f455891e53e8d01bd356203e1aea66cd" (from machine) (pass) Total memory: 3.7 GiB (pass) Disk space available for /var/lib/k0s: 33.0 GiB (pass) Operating system: Linux (pass) Linux kernel release: 5.14.0-162.6.1.el9_1.0.1.x86_64 (pass) Max. file descriptors per process: current: 524288 / max: 524288 (pass) Executable in path: modprobe: /sbin/modprobe (pass) /proc file system: mounted (0x9fa0) (pass) Control Groups: version 2 (pass) cgroup controller "cpu": available (pass) cgroup controller "cpuacct": available (via cpu in version 2) (pass) cgroup controller "cpuset": available (pass) cgroup controller "memory": available (pass) cgroup controller "devices": available (assumed) (pass) cgroup controller "freezer": available (assumed) (pass) cgroup controller "pids": available (pass) cgroup controller "hugetlb": available (pass) cgroup controller "blkio": available (via io in version 2) (pass) CONFIG_CGROUPS: Control Group support: built-in (pass) CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass) CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass) CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass) CONFIG_CPUSETS: Cpuset support: built-in (pass) CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass) CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass) CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass) CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass) CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass) CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass) CONFIG_BLK_CGROUP: Block IO controller: built-in (pass) CONFIG_NAMESPACES: Namespaces support: built-in (pass) CONFIG_UTS_NS: UTS namespace: built-in (pass) CONFIG_IPC_NS: IPC namespace: built-in (pass) CONFIG_PID_NS: PID namespace: built-in (pass) CONFIG_NET_NS: Network namespace: built-in (pass) CONFIG_NET: Networking support: built-in (pass) CONFIG_INET: TCP/IP networking: built-in (pass) CONFIG_IPV6: The IPv6 protocol: built-in (pass) CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass) CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass) CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: built-in (pass) CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass) CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass) CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass) CONFIG_NETFILTER_XT_SET: set target and match support: module (pass) CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass) CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass) CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass) CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass) CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass) CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass) CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass) CONFIG_NETFILTER_NETLINK: module (pass) CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass) CONFIG_NF_NAT: module (pass) CONFIG_IP_SET: IP set support: module (pass) CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass) CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass) CONFIG_IP_VS: IP virtual server support: module (pass) CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass) CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning) CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass) CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning) CONFIG_IP_NF_IPTABLES: IP tables support: module (pass) CONFIG_IP_NF_FILTER: Packet filtering: module (pass) CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass) CONFIG_IP_NF_NAT: iptables NAT support: module (pass) CONFIG_IP_NF_MANGLE: Packet mangling: module (pass) CONFIG_NF_DEFRAG_IPV4: module (pass) CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning) CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning) CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass) CONFIG_IP6_NF_FILTER: Packet filtering: module (pass) CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass) CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass) CONFIG_NF_DEFRAG_IPV6: module (pass) CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass) CONFIG_LLC: module (pass) CONFIG_STP: module (pass) CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass) CONFIG_PROC_FS: /proc file system support: built-in (pass) ```

What happened?

3rd party CSI drivers do not come up because they have to make assumptions about the location of kubelet runtime directories (/var/lib/kubelet).

Those assumptions do not hold true for K0s, as K0s uses non-standard directories (/var/lib/k0s/kubelet). This derivation from the norm is not documented at the relevant places (eg. https://docs.k0sproject.io/v1.25.4+k0s.0/storage/). On the contrary, the documentation starting page says "Certified and 100% upstream Kubernetes", which (at least to me) raises the expectation that it also behaves like upstream Kubernetes.

Steps to reproduce

Install any 3rd party CSI driver, eg. https://github.com/hetznercloud/csi-driver

1. kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.1.0/deploy/kubernetes/hcloud-csi.yml

Expected behavior

The CSI driver starts up and works correctly

Actual behavior

The node pods are stuck in ContainerCreating

Screenshots and logs

Warning FailedMount 14s (x8 over 77s) kubelet MountVolume.SetUp failed for volume "registration-dir" : hostPath type check failed: /var/lib/kubelet/plugins_registry/ is not a directory

Additional context

The issue was already reported in 2021 #810 and 2022 #1842 but dismissed both time, apparently expecting everyone who wishes to use K0s to discover this on their own.

Also see

• HCloud-CSI-Node stuck at "Container creating" using k0s hetznercloud/csi-driver#260
• hcloud-csi-node stuck at "ContainerCreating" on k0s hetznercloud/csi-driver#357
• Undocumented folder requirements during setup (k0s) rook/rook#7516
• OSD pod permissions broken, unable to open OSD superblock after node restart rook/rook#7519
• PVs not mounted due to kubernetes.io/csi/pv/*/globalmount is missing, with possibly related omap errors rook/rook#7520
• Can't launch on K0s distribution democratic-csi/democratic-csi#257

[jnummelin]

This derivation from the norm is not documented at the relevant places

This is something that needs to get fixed!

documentation starting page says "Certified and 100% upstream Kubernetes", which (at least to me) raises the expectation that it also behaves like upstream Kubernetes.

Well, the data location is configurable in kubelet and k0s setting it to something other than the default value for the flag does not make it non-certified or anything. :)

The issue was already reported in 2021 #810 and 2022 #1842 but dismissed both time, apparently expecting everyone who wishes to use K0s to discover this on their own.

Both of those issues do have some resolution in them, hence they were closed at the time. Like I menationed in one of the earlier issues on the topic:

We do get that it's bit inconvenient in some cases, but as kubelet itself has a config option for the data-dir it uses, it's really pretty much always an upstream issue if those are not configurable.

So in this case, is there a reason why the relevant path(s) cannot be made configurable in hcloud/csi-driver Helm charts?

[mhutter]

No, I think I figured out all the dials I have to adjust in hcloud-csi :)

Where would be a proper place to document derivations? "Common Pitfalls"? Are there other things that might be worth documenting there?

[jnummelin]

No, I think I figured out all the dials I have to adjust in hcloud-csi :)

Good to hear.

Where would be a proper place to document derivations? "Common Pitfalls"? Are there other things that might be worth documenting there?

I think the relevant place for CSI related notes is in the docs/storage.md file. Would not hurt to also add into common FAQ/pitfalls page too.

[jnummelin]

@mhutter Are you planning to open a PR for this?

[mhutter]

Almost forgot, thanks for reminding me @jnummelin :) PR is linked