Security hardening guide for scheduler configuration

Signed-off-by: Anshuman Tripathi <[email protected]>

[WIP] Security hardening guide for scheduler configurations

Signed-off-by: Anshuman Tripathi <[email protected]>

Updates after passing through hemmingway.app

Signed-off-by: Anshuman Tripathi <[email protected]>

Update scheduling configurations

Signed-off-by: Anshuman Tripathi <[email protected]>

Apply suggestions from code review

Co-authored-by: Tim Bannister <[email protected]>
Co-authored-by: Daniel Register <[email protected]>

Updates based on PR feedback

Signed-off-by: Anshuman Tripathi <[email protected]>

Update bind-address definition

Signed-off-by: Anshuman Tripathi <[email protected]>

Update phrasing of permit-address-sharing

Signed-off-by: Anshuman Tripathi <[email protected]>

Add -- to args

Signed-off-by: Anshuman Tripathi <[email protected]>

Sentence case in table title

Signed-off-by: Anshuman Tripathi <[email protected]>

Reword and correct grammer based on feedback

Signed-off-by: Anshuman Tripathi <[email protected]>

Remove verbatim argument description

Signed-off-by: Anshuman Tripathi <[email protected]>

More updates

Signed-off-by: Anshuman Tripathi <[email protected]>

Update custom scheduler heading and description

Signed-off-by: Anshuman Tripathi <[email protected]>

Remove dashes on args

Signed-off-by: Anshuman Tripathi <[email protected]>

Apply suggestions from code review

Co-authored-by: Tim Bannister <[email protected]>
Signed-off-by: Anshuman Tripathi <[email protected]>

Update table title

Signed-off-by: Anshuman Tripathi <[email protected]>

Update based on feedback

Signed-off-by: Anshuman Tripathi <[email protected]>

node selector

Signed-off-by: Anshuman Tripathi <[email protected]>

Feedback

Signed-off-by: Anshuman Tripathi <[email protected]>

Update authentication and TLS configuration

Signed-off-by: Anshuman Tripathi <[email protected]>

profiling

Signed-off-by: Anshuman Tripathi <[email protected]>

Replace tables with bullets

Signed-off-by: Anshuman Tripathi <[email protected]>

content/en/docs/concepts/security/hardening-guide/scheduler.md

@@ -0,0 +1,85 @@
+---
+title: "Hardening Guide - Scheduler Configuration"
+description: >
+    Information about how to make the Kubernetes scheduler more secure.
+content_type: concept
+weight: 90
+---
+
+<!-- overview -->
+The Kubernetes {{< glossary_tooltip text="scheduler" term_id="kube-scheduler" >}} is
+one of the critical components of the
+{{< glossary_tooltip text="control plane" term_id="control-plane" >}}.
+
+This document covers how to improve the security posture of the Scheduler.
+
+A misconfigured scheduler can have security implications. Such a scheduler can target specific nodes and evict the workloads or applications that are sharing the node and its resources. 
+This can aid an attacker with a [Yo-Yo attack](https://arxiv.org/abs/2105.00542): an attack on a vulnerable autoscaler.
+
+<!-- body -->
+## kube-scheduler configuration
+
+### Scheduler authentication & authorization command line options
+When setting up authentication configuration, it should be made sure that kube-scheduler's authentication remains consistent with kube-api-server's authentication. If any request has missing authentication headers, the authentication should happen through the kube-api-server allowing all authentication to be consistent in the cluster.
+
+
+- `authentication-kubeconfig`: Make sure to provide a proper kubeconfig so that the scheduler can retrieve authentication configuration options from the API Server. This kubeconfig file should be protected with strict file permissions.
+- `authentication-tolerate-lookup-failure`: Set this to `false` to make sure the scheduler _always_ looks up its authentication configuration from the API server.
+- `authentication-skip-lookup`: Set this to `false` to make sure the scheduler _always_ looks up its authentication configuration from the API server.
+- `authorization-always-allow-paths`: These paths should respond with data that is appropriate for anonymous authorization. Defaults to `/healthz,/readyz,/livez`.
+- `profiling`: Set to `false` to disable the profiling endpoints which are provide debugging information but which should not be enabled on production clusters as they present a risk of denial of service or information leakage. The `--profiling` argument is deprecated and can now be provided through the [KubeScheduler DebuggingConfiguration](https://kubernetes.io/docs/reference/config-api/kube-scheduler-config.v1/#DebuggingConfiguration). Profiling can be disabled through the kube-scheduler config by setting `enableProfiling` to `false`.                                                                                     
+
+
+### Scheduler networking command line options
+
+- `bind-address`: In most cases, the kube-scheduler does not need to be externally accessible. Setting the bind address to `localhost` is a secure practice.
+- `permit-address-sharing`: Set this to `false` to  disable connection sharing through `SO_REUSEADDR`. `SO_REUSEADDR` can lead to reuse of terminated connections that are in `TIME_WAIT` state.
+- `permit-port-sharing`: Default `false`. Use the default unless you are confident you understand the security implications.
+
+
+### Scheduler TLS command line options
+
+- `requestheader-client-ca-file`: Avoid passing this argument and instead rely on `requestheader-allowed-names` instead. [This configuration makes sure that authentication configuration consistent across the cluster.](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/#original-request-username-and-group)
+- `tls-cipher-suites`: Always provide a list of preferred cipher suites. This ensures encryption never happens with insecure cipher suites. 
+
+
+## Scheduling configurations for custom schedulers
+This section covers security hardening when using custom schedulers based on the Kubernetes
+scheduling code.
+The cluster administrator needs be careful with the plugins that use the `queueSort`, `prefilter`, `filter`, or permit [extension points](https://github.com/docs/reference/scheduling/config/#extension-points).
+Scheduling happens in a series of stages that are exposed through the extension points.
+As a cluster administrator, you can enable plugins that define their own extension points.
+Doing so can affect the defined scheduling behaviors of the kube-scheduler in your cluster.
+
+When using a custom scheduler, plugin extension points such as `queueSort`, `prefilter`, `filter` and `permit` should be used with care.
+
+- Exactly one plugin that uses the `queueSort` extension point can be enabled at a time. Any plugins that use `queueSort` should be scrutinized.
+- Plugins that implement the `prefilter` or `filter` extension point can potentially mark all nodes as unschedulable. This can bring scheduling of new pods to a halt.
+- Plugins that implement the `permit` extension point  can prevent or delay the binding of a Pod. Such plugins should be thoroughly reviewed by the cluster administrator.
+
+When using a plugin that is not one of the [default plugins](/docs/reference/scheduling/config/#scheduling-plugins), consider disabling the `queueSort`, `filter` and `permit` extension points as follows:
+
+```yaml
+apiVersion: kubescheduler.config.k8s.io/v1
+kind: KubeSchedulerConfiguration
+profiles:
+  - schedulerName: default-scheduler
+  - schedulerName: my-custom-scheduler
+    plugins:
+      queueSort:
+        disabled:
+        # Globally only one queueSort can be specified.
+        - name: my-custom-scheduler
+      # - name: default-scheduler
+      permit:
+        disabled:
+        - name: '*'
+```
+This creates two scheduler profiles: `default-scheduler` and ` my-custom-scheduler`.
+Whenever the `.spec` of a Pod does not have a value for `.spec.schedulerName`, the kube-scheduler runs for that Pod, using its main configuration, and default plugins.
+If you define a Pod with `.spec.schedulerName` set to `my-custom-scheduler`, the kube-scheduler also runs but with a custom configuration; in that custom configuration,
+the  `queueSort`, `filter` and `permit` extension points are disabled.
+If you use this kube-scheduler configuration, and don't run any custom scheduler, and you then define a Pod with  `.spec.schedulerName` set to `nonexistent-scheduler` (or any other scheduler name that doesn't exist in your cluster), no events would be generated for a pod.
+
+## Disallow labeling nodes
+A cluster administrator should ensure that cluster users cannot label the nodes. A malicious actor can use `nodeSelector` to schedule workloads on nodes where those workloads should not be present.