Scanning Container images in a Kubernetes Release with Snyk

[PushkarJ]

Background

There are lot of places where images related to Kubernetes can be found:

1. Container image manifests and yamls: https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io
2. Base Image exception list: https://github.com/kubernetes/sig-release/blob/master/release-engineering/baseimage-exception-list.md
3. SBOM: (Software Bill of Materials) @puerco: any pointers on how to parse the SBOM files and if they are a good source to get a list of images?
   • https://sbom.k8s.io/v1.22.0/release
   • https://sbom.k8s.io/v1.22.0/source

Goal

• Find out an authoritative source for list of container images under Kubernetes org (i.e. release, k8s.io, kubernetes) including but not limited to the above resources
• Filter the list to only point to the most recent image or in case of core k8s images most recent n, n-1 and n-2 release images
• Convert this into a JSON doc that can be consumed as input for container image scanner automation

xref: #3
/sig security release

Side note: This issue might be transferred to kubernetes-sigs/security when it is created

[puerco]

The kubernetes sbom lists all images we push out to the registries as part of a release. You can get all images from the release SBOM like this:

curl -L https://sbom.k8s.io/v1.22.1/release  | grep 'PackageName: k8s.gcr.io/' | awk '{print $2}'

[nehaLohia27]

@PushkarJ I am interested in contributing to this issue.

[PushkarJ]

/assign @nehaLohia27

[nehaLohia27]

@PushkarJ @puerco

Below is the list of container images from curl command. Does this list have all of the container images or not ?

curl -L https://sbom.k8s.io/v1.22.1/release | grep 'PackageName: k8s.gcr.io/' | awk '{print $2}'

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 154 100 154 0 0 442 0 --:--:-- --:--:-- --:--:-- 441
100 184k 100 184k 0 0 230k 0 --:--:-- --:--:-- --:--:-- 230k
k8s.gcr.io/kube-scheduler-amd64:v1.22.1
k8s.gcr.io/kube-apiserver-arm:v1.22.1
k8s.gcr.io/kube-proxy-ppc64le:v1.22.1
k8s.gcr.io/kube-scheduler-ppc64le:v1.22.1
k8s.gcr.io/kube-scheduler-s390x:v1.22.1
k8s.gcr.io/conformance-amd64:v1.22.1
k8s.gcr.io/kube-controller-manager-arm:v1.22.1
k8s.gcr.io/kube-proxy-arm:v1.22.1
k8s.gcr.io/kube-proxy-arm64:v1.22.1
k8s.gcr.io/kube-scheduler-arm64:v1.22.1
k8s.gcr.io/kube-proxy-amd64:v1.22.1
k8s.gcr.io/kube-scheduler-arm:v1.22.1
k8s.gcr.io/conformance-arm64:v1.22.1
k8s.gcr.io/kube-apiserver-arm64:v1.22.1
k8s.gcr.io/kube-apiserver-ppc64le:v1.22.1
k8s.gcr.io/kube-controller-manager-ppc64le:v1.22.1
k8s.gcr.io/conformance-s390x:v1.22.1
k8s.gcr.io/kube-proxy-s390x:v1.22.1
k8s.gcr.io/conformance-arm:v1.22.1
k8s.gcr.io/kube-controller-manager-amd64:v1.22.1
k8s.gcr.io/kube-controller-manager-arm64:v1.22.1
k8s.gcr.io/conformance-ppc64le:v1.22.1
k8s.gcr.io/kube-apiserver-s390x:v1.22.1
k8s.gcr.io/kube-controller-manager-s390x:v1.22.1
k8s.gcr.io/kube-apiserver-amd64:v1.22.1

[puerco]

All that are part of a Kubernetes release, yes. Keep in mind that there are other images (say k8s.gcr.io/pause or k8s.gcr.io/etcd ) which we are published outside of a regular release and are not described by the sbom.

[PushkarJ]

This is great progress @nehaLohia27 . Can you pick one or two of these images and run a snyk scan on them? Here's a useful guide to get started.
The idea is to just ensure if the images from here, can be pulled and scanned with snyk as we already have snyk licenses (Ref: kubernetes/steering#206) You can use the free tier for scanning for this PoC.

PS: (Don't share the scan report here :) )

@puerco those set of images are exactly what I was expecting from SBOM but I agree that we can and should include more images in future. Curious, Is there a way to get a JSON Response from SBOM URL? I suspect converting to JSON would not be hard on client side but if there is some sort of flag to get the results in JSON directly, that would be fantastic :)

On that note, Is there a way for us to get a list of images mentioned here: https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io programmatically? I imagine the images from SBOM are a subset of images from this directory in the same repo

[nehaLohia27]

@PushkarJ I have installed snyk on my local machine and ran the scan on few of the container images from sbom url response.
eg: command to scan the images: snyk container test k8s.gcr.io/kube-scheduler-s390x:v1.22.1
In scan results, no issues and no vulnerable paths were found. let me know the next steps.

Curious to know, do we need to identify all the images from this https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io ?
I want to know the sources which can be used to identify all the container images.

[PushkarJ]

This is fantastic progress @nehaLohia27 you already, knocked out first three items in the umbrella issue for container images 🎉

Next steps could be two fold in IMO, but please share your thoughts as well as I maybe missing something 😄

1. Write a shell script that does something similar to what we did with snyk and build time deps scanning but for container image scanning: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/wg-k8s-infra/trusted/sig-security-trusted.yaml#L29-L51
2. Work with @puerco and @kubernetes/sig-release-leads to figure out how can we extract the list of images that are not part of k8s release but are part of https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io . This may mean some extra work outside of the scope of this initiative on vulnerability mgmt. but that's okay since what matters at the end of the day is the SIG friends we made along the way :) Let me know if I can help or own any items coming out of this collaboration as well!

[nehaLohia27]

Thanks @PushkarJ @puerco for helping on this :). I will work on the shell scripting. Will post the issues if any.

@puerco do you have any idea on how can we extract the list of images that are not part of k8s release but are part of https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io , using any commands or urls ?

[nehaLohia27]

@PushkarJ created the draft PR for shell script code. kubernetes/test-infra#23419

[PushkarJ]

/transfer sig-security

[PushkarJ]

Fixed by kubernetes/test-infra#23419