diff --git a/keps/sig-api-machinery/4977-dedicated-priority-level-for-event-requests/README.md b/keps/sig-api-machinery/4977-dedicated-priority-level-for-event-requests/README.md new file mode 100644 index 00000000000..53bf0dab060 --- /dev/null +++ b/keps/sig-api-machinery/4977-dedicated-priority-level-for-event-requests/README.md @@ -0,0 +1,982 @@ + + +# KEP-4977: Dedicated Priority Level For Event Requests + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Beta](#beta) + - [GA](#ga) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / +release*. + +- [ ](R) Enhancement issue in release milestone, which links to KEP dir in + [kubernetes/enhancements](not the initial KEP PR) +- [ ](R) KEP approvers have approved the KEP status as `implementable` +- [ ](R) Design details are appropriately documented +- [ ](R) Test plan is in place, giving consideration to SIG Architecture and + SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ](R) Ensure GA e2e tests meet requirements for + [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ](R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ](R) Graduation criteria is in place + - [ ](R) + [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) + must be hit by + [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ](R) Production readiness review completed +- [ ](R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for + publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to + mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +This KEP proposes creating a new priority level for Event requests in default +APF configuration for API Server to have better performance and reliability when +under pressure. + +## Motivation + + + +Many features have been developed and use cases surfaced since the introduction +of [KEP-1040]. These prompt us to revise APF default configuration to better +make use of APF for better performance and reliability of API Server. One thing +we have learned is that Event requests are plentiful in upscaling but they are +resilient to unavailability of API Server. So we can deprioritize them when API +Server is under stress. + +[KEP-1040]: https://github.com/kubernetes/enhancements/blob/97713189b3107b41c4c19505d04aa7ef22df063b/keps/sig-api-machinery/1040-priority-and-fairness/README.md + +### Goals + + + +- Better overload protection in perspective of whole Kubernetes control plane + instead of just API Server alone. +- More performant in cluster upscaling (e.g. increasing node count, deploying + pods) + +### Non-Goals + + + +- There will be no fundamental changes in APF's own implementation + +## Proposal + + + +The APF feature provides flow control to API Server by introducing a limit on +how much a priority level can consume the concurrency. In addition to that, +borrowing is also instroduced to better utilize spare concurrency. The limits on +borrowing are two-sided: a given priority level has a limit on how much it may +borrow and a limit on how much may be borrowed from it. Worth mentioning, APF +controller periodically populates current concurrency shares for each priority +level (i.e. concurrency share that can be borrowed or need to be borrowed) by +taking past usage into consideration. So in combination of those two aspects, a +priority level's Nomial Share is only an initial value and the actual share +might increase or decrease according to actual usage with borrowing. However, +the owner priority level should have full Nominal Share as its current +concurrency share if in full utilization. + +This KEP proposes the following settings: + +| Name | Proposed Nominal Shares | Proposed Lendable | Proposed Borrowing Limit | Proposed Guaranteed Shares | +| ----- | -----------------------: | ----------------: | -----------------------: | -------------------------: | +| event | 5 | 0% | 100% | 5 | + +Note: The event priority level is specified with Borrowing Limit to prevent it +from borrowing all spare shares from other priority levels. We may change it to +none once we have [weighted borrowing] implemented. + +[weighted borrowing]: https://github.com/kubernetes/enhancements/pull/4733 + +In addition to above, a new priority level and flow schema will be created for +Events: + +``` +apiVersion: flowcontrol.apiserver.k8s.io/v1beta3 +kind: PriorityLevelConfiguration +metadata: + name: catch-all-events +spec: + limited: + nominalConcurrencyShares: 5 + limitResponse: + type: Reject + type: Limited +--- +apiVersion: flowcontrol.apiserver.k8s.io/v1beta3 +kind: FlowSchema +metadata: + name: events +spec: + distinguisherMethod: + type: ByUser + matchingPrecedence: 450 + priorityLevelConfiguration: + name: catch-all-events + rules: + - resourceRules: + - apiGroups: + - "" + - events.k8s.io + namespaces: + - '*' + resources: + - events + verbs: + - create + - update + - delete + - patch + subjects: + - group: + name: system:authenticated + kind: Group +``` + +Read Verbs are excluded from this new flow schema because Administrators or +Monitoring may need to read them for troubleshooting/monitoring while the +cluster is under extreme stress. + +### User Stories (Optional) + + + +#### Story 1 + +#### Story 2 + +### Notes/Constraints/Caveats (Optional) + + + +### Risks and Mitigations + + + +## Design Details + + + +### Test Plan + + + +[X] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes +necessary to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +- ``: `` - `` + +##### Integration tests + + + + + +- : + +##### e2e tests + + + +- : + +### Graduation Criteria + + + +#### Alpha + +- Feature implemented behind a feature flag +- Manual upscaling tests to verify improvements +- Ensure no negative impact in feature enable/disable + +#### Beta + +- Ensure no negative impact in upgrade/downgrade + +#### GA + +- Allowing time for feedback + +### Upgrade / Downgrade Strategy + + + +### Version Skew Strategy + + + +As API Server is responsible to apply default APF settings, this may bring in +problems: + +- reconciling churns when multiple API Server instances are in process of + upgrade. Old instances will constantly try to apply old settings, as do the + new instances. +- the newly added priority level won't be removed after rollback. + +The first problem will be resolved once all API Server instances are upgraded to +the new version or downgraded to old version. The churn in the progress should +be acceptable given the change only becomes effective every minute due to +periodic adjustment. + +We need to either add a logic to remove priority levels that annonated with +`auto-update` and is not in default settings before making this KEP Beta, or +well document it to users to remove it manually after rollback. + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [X] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: ReviseAPFDefaultConfiguration + - Components depending on the feature gate: kube-apiserver +- [ ] Other + - Describe the mechanism: + - Will enabling / disabling the feature require downtime of the control + plane? + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? + +###### Does enabling the feature change any default behavior? + + + +Users may see increased/decreased throughputs in different priority levels when +API Server is under stress. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +Yes. + +###### What happens if we reenable the feature if it was previously rolled back? + +The feature will be restored. + +###### Are there any tests for feature enablement/disablement? + + + +No. Manual tests were run before switching feature gate to beta. + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +A misconfiguration could cause apiserver requests to be rejected, which could +have widespread impact such as: (1) rejecting controller requests, thereby +bringing a lot of things to a halt, (2) dropping node heartbeats, which may +result in overloading other nodes, (3) rejecting kube-proxy requests to +apiserver, thereby breaking existing workloads, (4) dropping leader election +requests, resulting in HA failure, or any combination of the above. + +###### What specific metrics should inform a rollback? + + + +Probably abnormal number of rejected requests in metric +`apiserver_rejected_requests` + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +Manual tests will be run. + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +No + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + +Use `kubectl get prioritylevelconfiguration` to check if configuratiom matches +the proposed. + + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [X] Other (treat as last resort) + - Details: + +Users can check `api_priority_and_fairness/dump_priority_levels` or similar +metrics prefixed `apiserver_flowcontrol_` to see differences when testing with +similar workloads. Ideally, there are zeros in `RejectedRequests`, +`TimedoutRequests` and `CancelledRequests`. + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +None have been identified. + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +No + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + +No. + + + +###### Will enabling / using this feature result in introducing new API types? + +No. + + + +###### Will enabling / using this feature result in any new calls to the cloud provider? + +No. + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + +No. + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + +No. + + + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + +No. + + + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + +No. + + + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +This feature is entirely in the apiserver, so will be unavailable when the +apiserver is unavailable. + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +No SLOs are proposed. + +## Implementation History + + + +## Drawbacks + + + +Introduction of user visible changes. + +## Alternatives + + + +Instead of an separate priority level for Event, squeeze it into workload-low +priority level. Given the characteristic of Event, requests of workload-low will +be likely competed by Event requests and get starved, which can be a severe +impact to user workloads. + +## Infrastructure Needed (Optional) + + diff --git a/keps/sig-api-machinery/4977-dedicated-priority-level-for-event-requests/kep.yaml b/keps/sig-api-machinery/4977-dedicated-priority-level-for-event-requests/kep.yaml new file mode 100644 index 00000000000..dff46588420 --- /dev/null +++ b/keps/sig-api-machinery/4977-dedicated-priority-level-for-event-requests/kep.yaml @@ -0,0 +1,26 @@ +title: Dedicated Priority Level For Event Requests +kep-number: 4793 +authors: + - "@linxiulei" +owning-sig: sig-api-machinery +participating-sigs: + - sig-scalability +status: provisional +creation-date: 2024-11-22 +reviewers: + - "@MikeSpreitzer" + - "@wojtek-t" +approvers: + - TBD + +see-also: + - "/keps/sig-api-machinery/1040-priority-and-fairness" +stage: alpha +latest-milestone: "v1.33" +milestone: + alpha: "v1.33" +feature-gates: + - name: DedicatedPriorityLevelForEventRequests + components: + - kube-apiserver +disable-supported: true