Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend MaintenanceWindows parameter usage #2810

Merged
merged 10 commits into from
Jan 15, 2025

Conversation

hughcapet
Copy link
Member

@hughcapet hughcapet commented Dec 4, 2024

Consider maintenance window when migrating master pods and replacing pods (rolling update)

@hughcapet hughcapet added the major Major feature changes or updates, e.g. feature rollout to a new country, new API calls. label Dec 4, 2024
@hughcapet hughcapet marked this pull request as ready for review December 23, 2024 12:09
pkg/cluster/util.go Outdated Show resolved Hide resolved
@hughcapet hughcapet force-pushed the maintwindow-master-migration branch from 7fade35 to 5cab328 Compare January 3, 2025 16:39
@idanovinda
Copy link
Member

👍

@FxKu
Copy link
Member

FxKu commented Jan 6, 2025

If I recall correctly moving the primary is a once or never operation. Removing the node readiness label triggers an node event which the operator is listening on to perform the migration. If the migration does not work the operator will not try again, because no other node event is send. Hence the primary stays on the node and blocks further operators by the K8s admins. We have seen it in the past where we were asked to trigger a switchover to unblock K8s updates.

Therefore, I think we greatly increase the possibility that pod migrations will not happen at all if they are omitted due to maintenance window restrictions. However, we could instead mark the primary pod to be rotated so another sync will trigger a switchover so that the old instance can be removed by K8s drainage.

@hughcapet hughcapet marked this pull request as draft January 7, 2025 08:14
@hughcapet hughcapet force-pushed the maintwindow-master-migration branch from 04a488a to f488bee Compare January 8, 2025 07:56
@hughcapet hughcapet force-pushed the maintwindow-master-migration branch from f488bee to 9689f25 Compare January 8, 2025 13:36
@hughcapet hughcapet marked this pull request as ready for review January 8, 2025 15:16
@hughcapet hughcapet marked this pull request as draft January 12, 2025 19:16
@hughcapet hughcapet force-pushed the maintwindow-master-migration branch from af7057b to 9e926d8 Compare January 12, 2025 20:40
@hughcapet hughcapet marked this pull request as ready for review January 13, 2025 06:00
@hughcapet hughcapet changed the title Consider maintenance window when migrating master pods Extend MaintenanceWindows parameter usage Jan 13, 2025
@hughcapet hughcapet force-pushed the maintwindow-master-migration branch from 9e926d8 to 3265113 Compare January 13, 2025 06:04
pkg/cluster/sync.go Outdated Show resolved Hide resolved
pkg/cluster/cluster.go Outdated Show resolved Hide resolved
pkg/cluster/cluster.go Outdated Show resolved Hide resolved
@FxKu
Copy link
Member

FxKu commented Jan 14, 2025

We have to update reference docs and also sections that talk about rolling updates like in admin docs. About user docs I'm not sure. So far they do not mention maintenance windows at all.

@FxKu FxKu added this to the 1.15.0 milestone Jan 14, 2025
@hughcapet hughcapet force-pushed the maintwindow-master-migration branch from 88b9845 to 88b1931 Compare January 14, 2025 13:27
docs/administrator.md Outdated Show resolved Hide resolved
@FxKu
Copy link
Member

FxKu commented Jan 15, 2025

👍

1 similar comment
@hughcapet
Copy link
Member Author

👍

@hughcapet hughcapet merged commit 8522331 into master Jan 15, 2025
10 checks passed
@hughcapet hughcapet deleted the maintwindow-master-migration branch January 15, 2025 17:04
hughcapet added a commit that referenced this pull request Jan 23, 2025
Only schedule switchover for pod migration, consider mainWindow for PGVERSION env change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
major Major feature changes or updates, e.g. feature rollout to a new country, new API calls.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants