Security hardening guide for scheduler configuration

Signed-off-by: Anshuman Tripathi <[email protected]> [WIP] Security hardening guide for scheduler configurations Signed-off-by: Anshuman Tripathi <[email protected]> Updates after passing through hemmingway.app Signed-off-by: Anshuman Tripathi <[email protected]> Update scheduling configurations Signed-off-by: Anshuman Tripathi <[email protected]> Apply suggestions from code review Co-authored-by: Tim Bannister <[email protected]> Co-authored-by: Daniel Register <[email protected]> Updates based on PR feedback Signed-off-by: Anshuman Tripathi <[email protected]> Update bind-address definition Signed-off-by: Anshuman Tripathi <[email protected]> Update phrasing of permit-address-sharing Signed-off-by: Anshuman Tripathi <[email protected]> Add -- to args Signed-off-by: Anshuman Tripathi <[email protected]> Sentence case in table title Signed-off-by: Anshuman Tripathi <[email protected]> Reword and correct grammer based on feedback Signed-off-by: Anshuman Tripathi <[email protected]> Remove verbatim argument description Signed-off-by: Anshuman Tripathi <[email protected]> More updates Signed-off-by: Anshuman Tripathi <[email protected]> Update custom scheduler heading and description Signed-off-by: Anshuman Tripathi <[email protected]> Remove dashes on args Signed-off-by: Anshuman Tripathi <[email protected]>
kubernetes · May 5, 2024 · e3ece46 · e3ece46
1 parent f83793a
commit e3ece46
Showing 1 changed file with 90 additions and 0 deletions.
diff --git a/content/en/docs/concepts/security/hardening-guide/scheduler.md b/content/en/docs/concepts/security/hardening-guide/scheduler.md
@@ -0,0 +1,90 @@
+---
+title: "Hardening Guide - Scheduler Configuration"
+description: >
+    Information about how to make the Kubernetes scheduler more secure.
+content_type: concept
+weight: 90
+---
+
+<!-- overview -->
+The Kubernetes {{< glossary_tooltip text="scheduler" term_id="kube-scheduler" >}} is
+one of the critical components of the
+{{< glossary_tooltip text="control plane" term_id="control-plane" >}}.
+
+This document covers how to improve the security posture of the Schduler.
+
+A misconfigured scheduler can have security implications. Such a scheduler can target specific nodes and evict the workloads or applications that are sharing the node and its resources. This can aid an attacker with a [Yo-Yo attack](https://arxiv.org/abs/2105.00542): an attack on a vulnerable autoscaler.
+<!-- body -->
+## kube-scheduler configurations
+
+### Scheduler Authentication & Authorization configuration options
+{{<table caption="Authentication and Authorization Configurations">}}
+| Configuration                            | Security hardening advice                                                                                                           |
+| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| `authentication-kubeconfig`              | Make sure to provide a proper kubeconfig so that the server calls are secure. This kubeconfig file should also maintained securely. |
+| `authentication-tolerate-lookup-failure` | Set to `false` to make sure invalid authentication configurations do not lead to requests passing off as anonymous                  |
+| `authentication-skip-lookup`             | This should be set to `false` to make sure all missing authentication configuration falls back to the authentication kubeconfig.    |
+| `authorization-always-allow-paths`       | These paths should respond with data that is appropriate for anonymous authorization. Defaults to `/healthz,/readyz,/livez`.        |
+{{</table>}}
+
+### Address configuration options
+{{<table caption="Address Configurations">}}
+| Configuration            | Security hardening advice                                                                                                                                                          |
+| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `bind-address`           | In most cases, the kube-scheduler does not need to be externally accessible. Setting the bind address to `localhost` is a secure practice.                                         |
+| `permit-address-sharing` | Set this configuration to `false` to  disable connection sharing through `SO_REUSEADDR`. `SO_REUSEADDR` can lead to reuse of terminated connections that are in `TIME_WAIT` state. |
+| `permit-port-sharing`    | Default `false`. Use the default unless you are confident you understand the security implications.                                                                                |
+{{</table>}}
+
+### TLS configuration options
+{{<table caption="Address Configurations">}}
+| Configuration                  | Security hardening advice                                                                                                                                                                                                          |
+| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `requestheader-client-ca-file` | Generally do not depend on authorization being already done for incoming requests. Always provide the root certificate bundle. This allows authorization to happen on each incoming request through `requestheader-allowed-names`. |
+| `tls-cipher-suites`            | Always provide a list of preferred cipher suites. This ensures encryption never happens with insecure cipher suites.                                                                                                               |
+{{</table>}}
+
+## Scheduling configurations for custom schedulers
+This section covers security hardening when using custom schedulers.
+The cluster administrator needs be careful with the plugins that use the `queueSort`, `filter`, or permit [extension points](https://github.com/docs/reference/scheduling/config/#extension-points).
+Scheduling happens in a series of stages that are exposed through the extension points.
+As a cluster administrator, you can enable plugins that define their own extension points.
+Doing so can affect the defined scheduling behaviors of the kube-scheduler in your cluster.
+
+When using a custom scheduler plugin extension points such as `queueSort`, `filter` and `permit` should be used with care.
+
+Exactly one plugin that uses the `queueSort` extension point can be enabled at a time. Any plugins that use `queueSort` should be scrutinized.
+
+Plugins that implement the `filter` extension point can potentially mark all nodes as unschedulable. This can bring scheduling of new pods to a halt.
+
+Plugins that implement the `permit` extension point  can prevent or delay the binding of a Pod. Such plugins should be thoroughly reviewed by the cluster administrator.
+
+When using a plugin that is not one of the [default plugins](https://kubernetes.io/docs/reference/scheduling/config/#scheduling-plugins), consider disabling the `queueSort`, `filter` and `permit` extension points as follows:
+
+```yaml
+apiVersion: kubescheduler.config.k8s.io/v1
+kind: KubeSchedulerConfiguration
+profiles:
+  - schedulerName: default-scheduler
+  - schedulerName: my-custom-scheduler
+    plugins:
+      queueSort:
+        disabled:
+        - name: '*'
+      filter:
+        disabled:
+        - name: '*'
+      permit:
+        disabled:
+        - name: '*'
+```
+This creates two scheduler profiles: `default-scheduler` and ` my-custom-scheduler`.
+Whenever the `.spec` of a Pod does not have a value for `.spec.schedulerName`,
+the kube-scheduler runs for that Pod, using its main configuration, and default plugin.
+If you define a Pod with `.spec.schedulerName` set to `my-custom-scheduler`, the
+kube-scheduler also runs but with a custom configuration; in that custom configuration,
+the  `queueSort`, `filter` and `permit` extension points are disabled.
+If you use this kube-scheduler configuration, and don't run any custom scheduler, and
+you then define a Pod with  `.spec.schedulerName` set to `nonexistent-scheduler` (or
+any other scheduler name that doesn't exist in your cluster), then the Pod stays pending scheduling. You would see a `FailedScheduling` event for the Pod and could manually
+remove that Pod using `kubectl delete`.