Recover workloads using local PVs from data loss in the cloud

[cofyc]

This is the track issue for some related problems in the cloud when the node is permanently deleted.

I'm thinking about writing a cloud controller to automate the recovery process. Here is the proposal: https://docs.google.com/document/d/1SA9epEwA3jPwibRV0ccQwJ2UfZXoeUYKyNxNegt0vn4

related issues:

• statefulset should delete LocalPersistentVolume when node was deleted kubernetes/kubernetes#78756
• Dealing with pvs and pvcs on deleted nodes #201
• Rotating cloud instances with PVCs in a StatefulSet #181

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Bessonov]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Bessonov]

/remove-lifecycle stale

[cdenneen]

@cofyc any further work past the initial proposal?
/help
/kind feature

[k8s-ci-robot]

@cdenneen:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

@cofyc any further work past the initial proposal?
/help
/kind feature

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[arianvp]

Wouldn't it already be enough as a first approximation to simply respect --pod-eviction-timeout as pased to kube-controller-manager (Which is 5 minutes by default)?

Just like pods get evicted after --pod-eviction-timeout after a node goes to Ready=Unknown or Ready=False, we can evict the PVC if the node is down that long.

If that timeout is good enough to try and evict the pod; then it's good enough to try and evict the PVC; is it not?

It does not handle the cases of node reboots losing data (e.g. in GKE and EKS) but it does handle the case of "Node
permanently gone" both on on-prem and on managed services where each new spawned node gets a new name (E.g. EKS)

Edit: Ah wait this will not work; as kubernetes doesn't delete pods if the node isn't reachable. Only does so when the node manually gets deleted or is reachable

In that case; I guess a first approximation that works both on cloud and on-prem is to auto-delete PVs that are bound to nodes that do not exist anymore

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[Bessonov]

/remove-lifecycle stale

[cdenneen]

Edit: Ah wait this will not work; as kubernetes doesn't delete pods if the node isn't reachable. Only does so when the node manually gets deleted or is reachable

In that case; I guess a first approximation that works both on cloud and on-prem is to auto-delete PVs that are bound to nodes that do not exist anymore

Exactly!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Bessonov]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Bessonov]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Bessonov]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Bessonov]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Bessonov]

/remove-lifecycle stale

[mattcary]

Update! We will be having a summer intern doing some work on this. We'll post more updates here as the project comes together.

We may not organize kubernetes-sigs membership for the intern so the actual PR upstream may be indirected through another member of my team. Just fair warning.