Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MountVolume.SetUp failed for volume "kube-api-access-wfmh2" no space left on device #694

Open
Jeffwan opened this issue Feb 18, 2025 · 2 comments
Labels
area/kv-cache kind/bug Something isn't working priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Milestone

Comments

@Jeffwan
Copy link
Collaborator

Jeffwan commented Feb 18, 2025

🐛 Describe the bug

Image

 Warning  FailedMount  25s (x19 over 37m)  kubelet            (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-wfmh2" : write /var/lib/kubelet/pods/26af5e70-f9b9-4c12-a9de-dcf2d4e93689/volumes/kubernetes.io~projected/kube-api-access-wfmh2/..2025_02_18_00_11_37.99831085/namespace: no space left on device
Warning  FailedMount        2m1s (x11 over 22m)  kubelet             (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-tzrpb" : write /var/lib/kubelet/pods/a951d40b-518a-4b29-a3c9-36ef96b4f09c/volumes/kubernetes.io~projected/kube-api-access-tzrpb/..2025_02_17_23_18_11.2933250311/ca.crt: no space left on device

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       2.0G  1.2G  775M  61% /
devtmpfs         24G     0   24G   0% /dev
tmpfs            24G     0   24G   0% /dev/shm
tmpfs           9.5G  6.2M  9.5G   1% /run
tmpfs            24G  248K   24G   1% /etc/machine-id
tmpfs           256K     0  256K   0% /mnt/disks
efivarfs         56K   24K   27K  48% /sys/firmware/efi/efivars
tmpfs            24G     0   24G   0% /tmp
overlayfs        24G  248K   24G   1% /etc
/dev/nvme0n1p8   11M   24K   11M   1% /usr/share/oem
/dev/nvme0n1p1   95G   57G   38G  60% /mnt/stateful_partition
tmpfs           2.0M  224K  1.8M  11% /var/lib/cloud
none             24G   38M   24G   1% /dev/kube_shm
tmpfs            24G   12K   24G   1% /var/lib/kubelet/pki

If has enough room but failed

Steps to Reproduce

latest version

kubectl apply -f samples/kvcache/kvcache.yaml
kubectl apply -f samples/kvcache/deployment.yaml

Expected behavior

it should not through any errors

Environment

nightly & GCP

@Jeffwan
Copy link
Collaborator Author

Jeffwan commented Feb 18, 2025

look at this issue and notice the default CPU & Memory are wrong, it all use value 2 instead of 2Gi. Update the kubebuilder default value

@Jeffwan Jeffwan added this to the v0.2.0 milestone Feb 18, 2025
@Jeffwan
Copy link
Collaborator Author

Jeffwan commented Feb 18, 2025

Image

@Jeffwan Jeffwan added kind/bug Something isn't working priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. area/kv-cache labels Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kv-cache kind/bug Something isn't working priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Development

No branches or pull requests

1 participant