New Docs page for API Server Bypass Risks #35908

raesene · 2022-08-12T12:24:50Z

This is a new documentation page for the Security Concepts section, looking at the risks of attackers bypassing the Kubernetes API server.

We've been working on this in Kubernetes SIG-Security docs (issue here)

cc @savitharaghunathan @sftim

netlify · 2022-08-12T12:29:23Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`49bc9b3`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/6307a2a1e3ceb70009b1c82a
😎 Deploy Preview	https://deploy-preview-35908--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

kbhawkey · 2022-08-12T14:11:41Z

Page preview:
https://deploy-preview-35908--kubernetes-io-main-staging.netlify.app/docs/concepts/security/api-server-bypass-risks/

content/en/docs/concepts/security/api-server-bypass-risks.md

tengqm · 2022-08-13T09:00:36Z

Most of the contents in this PR are valuable know-hows for cluster operators. However, I'm wondering whether/how these information can be combined with #33992 in which a checklist is being proposed. In other words, I'd like to propose an outline for security hardening, some can be concepts, others can be tasks. Once we kick both PRs in, we may need to re-organize the contents. So ... it would be desirable to have an upfront framework for these insights/recommendations.

raesene · 2022-08-13T10:11:26Z

@tengqm Ultimately we could combine the longer form documents like this and the RBAC good practices guide into a full hardening guide that would then correlate to the checklist, but at the moment they're quite different documentation styles.

We started a hardening guide idea, but I think the top-down approach looked a bit too daunting, so instead this is a more bottom-up approach of filling gaps in Kubernetes security knowledge for specific areas, that are not well served by other guides (e.g CIS benchmarks and the NSA Kubernetes hardening guide).

shannonxtreme · 2022-08-13T10:40:24Z

I do understand where Qiming is coming from as well. It's good to have a plan or a larger "good" end state into which PRs that add value, like this one, feed so that we consolidate and focus our efforts.

@tengqm do you know if we already have an umbrella type issue that tracks this type of work on hardening docs? If not I'll dig for it or make one

raesene · 2022-08-13T13:18:19Z

@shannonxtreme There's an Issue and attached skeleton doc from last year here kubernetes/sig-security#30

FWIW I wouldn't really describe this page as a hardening guide though, it's more security architecture. Hardening guides tend to be focused on settings that can be changed in a product to improve security posture. This page isn't that, it's higher level about the security architecture choices made be Kubernetes that operators should be aware of, and some ideas to mitigate them, if it fits an organization's threat model.

shannonxtreme · 2022-08-13T13:51:36Z

Yeah I wouldn't describe this as a hardening guide either. What I'd like to see as we build out our collection of docs like this is to some day have a top level page that acts as a one-step location that links you through each stage of the "Secure your cluster" journey. That page could have, for example, the "Discover" phase, linking to docs like the one in this PR, docs about the security measures built into k8s, etc. Then a "Harden your nodes" phase, with links to guides for taking hardening steps on nodes. So on and so forth. A reader could use it to wayfind the complex animal that is k8s

tengqm · 2022-08-13T15:05:27Z

@shannonxtreme It might be desirable to have such an umbrella issue as kubernetes security best practices. We can link to existing topics such authz&authn, RBAC, Pod Security Standard/Policy, or maybe even consolidate them to make the whole subject better organized than what we have today (https://kubernetes.io/docs/concepts/security/). My understanding is that, security, is a cross-cutting concern for all serious deployments. You don't care that much if you don't think it is an issue. But when you do care, you want to see a comprehensive list of topics in one place and then dive into a specific topic at will.
@raesene I didn't mean to block this PR. I was just thinking loudly how we can get this whole subject better covered/presented. Maybe we can meet somewhere in the middle way if we have both a top-down plan and a collection of bottom-up proposals.

shannonxtreme · 2022-08-13T15:08:18Z

@tengqm yes exactly. We want our readers to find these docs in a way that makes sense as they go through Kubernetes. My previous comment about a single page that logically consolidates and groups all our security topics across our docs would be a great way to improve the Security concept page I think.

I'm not at my computer to open a tracking issue but if you do, please cc me as well ☺️

sftim · 2022-08-14T17:26:29Z

/sig security
/sig auth

sftim

A couple of tweaks.

Also: How about an exec into a container that's part of an API server Pod? Some less common cluster setups make this possible.

content/en/docs/concepts/security/api-server-bypass-risks.md

shannonxtreme

Overall LGTM @raesene thank you.

content/en/docs/concepts/security/api-server-bypass-risks.md

shannonxtreme · 2022-08-17T18:19:39Z

content/en/docs/concepts/security/api-server-bypass-risks.md

+   mechanisms such as Linux kernel namespaces.
+- Ensure that you restrict or forbid the use of [`hostPath` mounts](/docs/concepts/storage/volumes/#hostpath)
+   that include the container runtime socket.
+- Restrict user access to nodes, and especially restrict superuser access to nodes.


Maybe a ## {% heading "whatsnext" %} with some useful next step links?

shannonxtreme · 2022-08-17T18:20:29Z

content/en/docs/concepts/security/api-server-bypass-risks.md

+title: Kubernetes API Server Bypass Risks
+description: >
+  Security architecture information relating to the API server and other components
+content_type: concept


nit: if you can explicitly define a weight here for ordering the pages, that would be nice

raesene · 2022-08-18T12:25:17Z

Thanks very much all, think I've got most of the formatting/wording changes sorted and I've squashed up the commits :)

raesene · 2022-08-19T10:02:32Z

/assign @savitharaghunathan (I think this is the right process as we've got an LGTM now :) )

sftim · 2022-08-19T10:04:36Z

This PR doesn't yet have an LGTM @raesene (at least, not one that Prow counts).

@shannonxtreme I'm guessing you were reviewing for SIG Docs, and now this should have a SIG Security technical review?

content/en/docs/concepts/security/api-server-bypass-risks.md

raesene · 2022-08-19T10:08:02Z

@sftim ahh interesting, how do I tell which LGTMs count?

sftim · 2022-08-19T10:28:46Z

Prow adds an LGTM label to the issue when it sees one that counts.

content/en/docs/concepts/security/api-server-bypass-risks.md

liggitt · 2022-08-23T15:29:30Z

overall, this is a positive addition that looks accurate... I noted a few things we might want to clarify and a couple things to expand on / correct

sftim · 2022-08-24T09:24:58Z

@raesene I recommend that you rebase this against main; previews will work more quickly, and the deploy preview will be seen in a v1.25 context (Netlify doesn't do the merge for you when previewing).

raesene · 2022-08-25T16:19:18Z

Updated to address feedback from the later reviews and re-squashed.

content/en/docs/concepts/security/api-server-bypass-risks.md

liggitt

updates lgtm, one non-blocking suggestion to strengthen the hostPath readOnly guidance

New Docs page for API Server Bypass Risks This is a new documentation page for the Security Concepts section, looking at the risks of attackers bypassing the Kubernetes API server. We've been working on this in Kubernetes SIG-Security docs (issue [here](kubernetes/sig-security#42)) Co-Authored-By: Shannon Kularathna <[email protected]> Co-Authored-By: Qiming Teng <[email protected]> Co-Authored-By: Tim Bannister <[email protected]> Co-Authored-By: Jordan Liggitt <[email protected]>

mtardy

I really love this addition to the security doc, especially the static pod and kubelet thing that are as not as widely known as the rest. They were a bit in a "secret knowledge" state at the moment, only discussed in some issues or talks!

Thanks, @raesene for the work, this is well organized and pleasant to read! I would love to add a reference to it from the security checklist!

mtardy · 2022-09-02T13:24:22Z

content/en/docs/concepts/security/api-server-bypass-risks.md

+  authentication to that service.
+- Control access to the private key for the etcd server certificate, and to the API server's
+  client certificate and key.
+- Consider restricting access to the the etcd port at a network level, to only allow access


Suggested change

- Consider restricting access to the the etcd port at a network level, to only allow access

- Consider restricting access to the etcd port at a network level, to only allow access

mtardy · 2022-09-02T13:26:34Z

content/en/docs/concepts/security/api-server-bypass-risks.md

+
+If a static Pod fails admission control, the kubelet won't register the Pod with the
+API server. However, the Pod still runs on the node. For more information, refer to
+[kubeadm issue #1541](https://github.com/kubernetes/kubeadm/issues/1541#issuecomment-487331701)). 


Suggested change

[kubeadm issue #1541](https://github.com/kubernetes/kubeadm/issues/1541#issuecomment-487331701)).

[kubeadm issue #1541](https://github.com/kubernetes/kubeadm/issues/1541#issuecomment-487331701).

mtardy · 2022-09-02T13:26:53Z

content/en/docs/concepts/security/api-server-bypass-risks.md

+access to this location could modify the configuration of static pods loaded from that
+source, or could introduce new static Pods.
+
+Static Pods are restricted from accessing other objects in the Kubernetes API . For example,


Suggested change

Static Pods are restricted from accessing other objects in the Kubernetes API . For example,

Static Pods are restricted from accessing other objects in the Kubernetes API. For example,

mtardy · 2022-09-02T13:37:07Z

content/en/docs/concepts/security/api-server-bypass-risks.md

+bypass controls that detect or prevent certain actions.
+
+The kubelet API can be configured to authenticate requests in a number of ways. 
+By default, the kubelet configuration allows anonymous access. Most Kubernetes providers


From the kubelet doc we can find:

--anonymous-auth Default: true
Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.)

I remember reading that anonymous access is granted by default only with this deprecated flag but if you use the --config flag then the default is different. Have I dreamed it?

mtardy · 2022-09-02T13:37:44Z

content/en/docs/concepts/security/api-server-bypass-risks.md

+## The etcd API
+
+Kubernetes clusters use etcd as a datastore. The `etcd` service listens on TCP port 2379.
+the only clients that need access are the Kubernetes API server and any backup tooling


+1 to shannonxtreme comment for the uppercase.

mtardy · 2022-09-02T13:41:23Z

content/en/docs/concepts/security/api-server-bypass-risks.md

+  from specified and trusted IP address ranges.
+
+
+## Container runtime socket {#runtime-socket}


I would have put maybe this section near the static pods one because these are quite similar bypasses but I don't know if it's super relevant/important.

sftim · 2022-09-02T16:05:30Z

content/en/docs/concepts/security/api-server-bypass-risks.md

+take other security sensitive actions, such as using `hostPath` mounts from the underlying
+node.


This feels reasonable until we have a page about managing risk associated with hostPath volumes.

sftim · 2022-09-02T16:06:55Z

I'd like to get a tech review from SIG Auth or SIG Security (or anyone else who's willing to state they've given this PR appropriate scrutiny).

Even if there are nits, let's look to see if we can merge this as-is and then file issues about any remaining details.

liggitt · 2022-09-02T16:10:06Z

I'd like to get a tech review from SIG Auth or SIG Security (or anyone else who's willing to state they've given this PR appropriate scrutiny).

I left a tech review lgtm for sig-auth at #35908 (review), I don't think there's been significant changes since then

sftim · 2022-09-02T16:13:36Z

Great - thanks @liggitt

/lgtm
for SIG Docs

/approve

k8s-ci-robot · 2022-09-02T16:13:43Z

LGTM label has been added.

Git tree hash: e086206218b07456afbb08f4947e48b80b839658

k8s-ci-robot · 2022-09-02T16:13:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sftim, shannonxtreme

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/OWNERS~~ [sftim]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 12, 2022

k8s-ci-robot requested review from kbhawkey and rajeshdeshpande02 August 12, 2022 12:25

k8s-ci-robot added language/en Issues or PRs related to English language sig/docs Categorizes an issue or PR as relevant to SIG Docs. labels Aug 12, 2022

shannonxtreme reviewed Aug 12, 2022

View reviewed changes

tengqm reviewed Aug 13, 2022

View reviewed changes

content/en/docs/concepts/security/api-server-bypass-risks.md Outdated Show resolved Hide resolved

k8s-ci-robot added sig/security Categorizes an issue or PR as relevant to SIG Security. sig/auth Categorizes an issue or PR as relevant to SIG Auth. labels Aug 14, 2022

sftim reviewed Aug 15, 2022

View reviewed changes

content/en/docs/concepts/security/api-server-bypass-risks.md Outdated Show resolved Hide resolved

content/en/docs/concepts/security/api-server-bypass-risks.md Outdated Show resolved Hide resolved

shannonxtreme mentioned this pull request Aug 16, 2022

Add title case style guideline #35990

Merged

shannonxtreme approved these changes Aug 17, 2022

View reviewed changes

sftim reviewed Aug 19, 2022

View reviewed changes

content/en/docs/concepts/security/api-server-bypass-risks.md Outdated Show resolved Hide resolved

liggitt reviewed Aug 23, 2022

View reviewed changes

liggitt reviewed Aug 25, 2022

View reviewed changes

content/en/docs/concepts/security/api-server-bypass-risks.md Outdated Show resolved Hide resolved

liggitt reviewed Aug 25, 2022

View reviewed changes

mtardy reviewed Sep 2, 2022

View reviewed changes

sftim reviewed Sep 2, 2022

View reviewed changes

k8s-ci-robot assigned sftim Sep 2, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 2, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 2, 2022

k8s-ci-robot merged commit 09707c0 into kubernetes:main Sep 2, 2022

mtardy mentioned this pull request Sep 2, 2022

Fix some mini typos in the API bypass security page #36536

Merged

This was referenced Sep 6, 2022

Kubernetes API Server Bypass Risks kubernetes/sig-security#42

Closed

REQUEST: New membership for raesene kubernetes/org#3667

Closed

	- Consider restricting access to the the etcd port at a network level, to only allow access
	- Consider restricting access to the etcd port at a network level, to only allow access

	[kubeadm issue #1541](https://github.com/kubernetes/kubeadm/issues/1541#issuecomment-487331701)).
	[kubeadm issue #1541](https://github.com/kubernetes/kubeadm/issues/1541#issuecomment-487331701).

	Static Pods are restricted from accessing other objects in the Kubernetes API . For example,
	Static Pods are restricted from accessing other objects in the Kubernetes API. For example,

		from specified and trusted IP address ranges.


		## Container runtime socket {#runtime-socket}

		take other security sensitive actions, such as using `hostPath` mounts from the underlying
		node.

New Docs page for API Server Bypass Risks #35908

New Docs page for API Server Bypass Risks #35908

Conversation

raesene commented Aug 12, 2022

netlify bot commented Aug 12, 2022 • edited Loading

✅ Pull request preview available for checking

kbhawkey commented Aug 12, 2022

tengqm commented Aug 13, 2022

raesene commented Aug 13, 2022

shannonxtreme commented Aug 13, 2022

raesene commented Aug 13, 2022 • edited Loading

shannonxtreme commented Aug 13, 2022

tengqm commented Aug 13, 2022

shannonxtreme commented Aug 13, 2022

sftim commented Aug 14, 2022

sftim left a comment

Choose a reason for hiding this comment

shannonxtreme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raesene commented Aug 18, 2022

raesene commented Aug 19, 2022

sftim commented Aug 19, 2022

raesene commented Aug 19, 2022

sftim commented Aug 19, 2022

liggitt commented Aug 23, 2022 • edited Loading

sftim commented Aug 24, 2022

raesene commented Aug 25, 2022

liggitt left a comment

Choose a reason for hiding this comment

mtardy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtardy Sep 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sftim commented Sep 2, 2022

liggitt commented Sep 2, 2022

sftim commented Sep 2, 2022 • edited Loading

k8s-ci-robot commented Sep 2, 2022

k8s-ci-robot commented Sep 2, 2022

netlify bot commented Aug 12, 2022 •

edited

Loading

raesene commented Aug 13, 2022 •

edited

Loading

liggitt commented Aug 23, 2022 •

edited

Loading

mtardy Sep 2, 2022 •

edited

Loading

sftim commented Sep 2, 2022 •

edited

Loading