skopeo delete - multiarch image oci v1 not fully deleted (artifactory)

[ggjulio]

skopeo delete doesn't fully delete oci v1 multiarch images. (at least on artifactory )

It just deletes the "index" list.manifest.json but won't delete any of the referenced manifests corresponding to each architecture & attestation.
Thus the associated layers will never be deleted because they are still referenced by an existing manifest.
In other terms it (almost) don't free any disk space. (a warning in the man page would be nice)

Maybe this issue should be at container/image, let me know.
Any inputs welcomed.

Example

list.manifest.json of tag latest :

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "manifests": [{
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:bafedd5a97eac06124376a79f857e9506d451c9f8f074f79535b986a8fd5a0a7",
            "size": 42,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:dca88a204d74dc532b0a005068c578d46f0de98dc15921c22d16e5f92769a13e",
            "size": 42,
            "annotations": {
                "vnd.docker.reference.digest": "sha256:bafedd5a97eac06124376a79f857e9506d451c9f8f074f79535b986a8fd5a0a7",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            }
        }
    ]
}

Manifest.json of sha256:bafedd5...

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "application/vnd.oci.image.config.v1+json",
        "digest": "sha256:b629ae61a4cc1ec62d644dc80c00021d106e87422dc1a54758f8e7c4bec5651c",
        "size": 42
    },
    "layers": [{
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:6414378b647780fee8fd903ddb9541d134a1947ce092d08bdeb23a54cb3684ac",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:1bba78c12f958ee27df9663b2f00bbd90cdf69c4a454da208df989612b0f4786",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:bb6add37eeb4e50173d21f376f52ebf63f52827df5622b3051107ce242feaed5",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:548e32a73cf97e82e0ca48a63dfec043c48956714b53bb40f04f93b546943926",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:4ae9d15bf174912b3b43a5ac99635a071345e9f14d879951bc6dd89ef2aaf0ca",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:83f4956db9b042c910b898eed255b6311ae7e6c4c9f1def35830f3355a785280",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:e52e7530754c500257f89598edeedd72c5852a13e73652f814bd3b48142cc77f",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:daab72e3a4d7e11397077294df356d3776dc5a8e2d3c2446241159b4fcdd870e",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:5e19ef6847f342f9d73f51a238d985c87eda53c1b012e901caadee24c0305698",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:883c3f33cabb7a1228b3e6ed1015007fad5a47a58cc3d7cf1f33c8d609bac41c",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:ce7ac68c1c4f136e8a1e2e3383f2ba68c720718ce7fde25b8c8d817122d266e7",
            "size": 42
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:ddec4b1052b02c5fff0d05fa7be75587f5532c801d32e120d51cd6031ce5da18",
            "size": 42
        }
    ],
    "annotations": {
        "com.opencontainers.image.authors": "[email protected]",
        "org.opencontainers.image.created": "2042-01-16T00:42:42Z",
        "org.opencontainers.image.description": "...",
        "org.opencontainers.image.revision": "....",
        "org.opencontainers.image.title": "Example",
        "org.opencontainers.image.vendor": "Vendor example",
        "org.opencontainers.image.version": "0.1.0"
    }
}

Before deletion of tag latest

After deletion of latest with skopeo delete docker://..../my-image:latest, the two linked manifests still exists:

Still possible to delete the sub manifests manually with skopeo delete docker://.../my-image@sha256:bafedd65.. :

resources:

• https://github.com/containers/image/blob/c893fddfd544a5f0cc8fad408f76b9547d4906dd/oci/layout/oci_delete.go#L19
• Oci image deletion image#2003
• Multi-arch containers actions/delete-package-versions#90

[mtrmac]

Thanks for reaching out.

In general, explicitly deleting the components is not safe, because those manifests may well be referenced from other images, or from other tags (and deleting a manifest by digest, the way skopeo delete does now, also removes all tags pointing at the manifest). So, this typically needs to be done by the registry, which can enumerate other references in a race-free way. (And, if I remember correctly, some registries prune manifests with no tags pointing to them eventually, but not immediately; such dangling manifests must be allowed to exist for a few minutes for a multi-arch push to be possible, but it’s possible that a registry would delete them after 24 hours or so. This is all, if it exists, a registry-side feature, with perhaps some registry-side configuration.)

[ggjulio]

Unfortunately artifactory registry provides nothing out of the box.
I did a small bash func that prune dandling images safely (at least for our use-cases).

1. Fetch all dandling digests that are older than 10minutes.
2. Extract to a list all digests of each list.manifest.json file found in the repository.
3. Then for each dandling ref we lookup in the list, if ref not found then it's safe to delete.

Example script:

#!/bin/bash

set -euo pipefail

# Requires jf cli, jq, parallel and skopeo

export ART_HOSTNAME='artifactory.example.com'

main(){
   	garbage_collect_dandling_images "my-art-repo-1"
   	garbage_collect_dandling_images "my-art-repo-2"
        # ...
}

# $1: artifactory repository name
garbage_collect_dandling_images(){
	local repo="${1}"
	local query_aql_all_images=; query_aql_all_images=$(cat <<EOF
items.find({
	"repo": "${repo}",
	"modified": {"\$before": "10minutes"},
	"\$and": [
		{"name": {"\$eq": "manifest.json"}},
		{"path": {"\$match": "*/sha256:*"}}
	],
	"type": "file"
}).include("repo", "path", "name", "created", "modified")
EOF
)
	local query_aql_all_indexes=; query_aql_all_indexes=$(cat <<EOF
items.find({
	"repo": "${repo}",
	"\$and": [
		{"name": {"\$eq": "list.manifest.json"}}
	],
	"type": "file"
}).include("repo", "path", "name", "created", "modified")
EOF
)
	# First fetch each images to avoid race condition creation vs deletion (sha256:.* folders only) + added a fixed duration for extra safety
	local images_to_process=;images_to_process="$(jf rt  curl -X POST /api/search/aql -H "Content-Type: text/plain" -d "${query_aql_all_images}" | jq -r '.results[] | "\(.path)"')"
	log_info "Found $(echo "${images_to_process}" |wc -l) images to process in ${1}"
	local index_files=;index_files="$(jf rt curl -X POST /api/search/aql -H "Content-Type: text/plain" -d "${query_aql_all_indexes}" | jq -r '.results[] | "\(.path)/\(.name)"')"
	log_info "Found $(echo "${index_files}" |wc -l) list.manifest.json in ${1}"
	log_info "Fetching all digests images from all list.manifest.json files."
	echo "${index_files}" | parallel --halt now,fail=1 -kj 9 "jf rt 2>/dev/null curl \"${1}/{}\" | jq -r '.manifests[] | .digest'" > /tmp/images_to_keep.txt
	log_info "Processing images..."
	local nb_deleted=0
	while IFS= read -r image; do
		# TODO optimize with parallel requests if too slow...
		image="${image%/*}@${image##*/}"
		if grep -Fxq "${image##*@}" /tmp/images_to_keep.txt; then
			echo "    Keep image still referenced ${image}"
		else
			echo "    Delete dandling image ${image}"
			delete_image "${ART_HOSTNAME}/${repo}/${image}"
			nb_deleted=$((nb_deleted+1))
		fi
	done <<< "${images_to_process}"
	log_info "Garbage collection done for ${1}\n	Deleted ${nb_deleted} images of $(echo "${images_to_process}" |wc -l) images."
}

# $1: Image ref to delete
# One does not simply delete a tag without deleting all tags associated with the same manifest...
# You may want to use regctl to delete a tag without deleting the manifest and all tags associated with it.
#   - https://flightaware.engineering/how-hard-is-it-to-delete-a-docker-tag/
#   - https://stackoverflow.com/questions/71576754/delete-tags-from-a-private-docker-registry
delete_image(){
	if [ "${DRY_RUN}" = 'false' ]; then
		skopeo delete "docker://${1}" || log_warn "Image could not be deleted or does not exist.  ${1}"
	fi
}

main "$@"