ecr: Garbage collection for ECR prints runs indefinitely #32742

BwL1289 · 2025-01-05T17:25:39Z

Describe the bug

Seeing unexpected behavior, specifically in ECR.

In one account I see the following:

...
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[10:33:17] [sdk info] ECR.PutImage({"repositoryName":"cdk-hnb659fds-container-assets-<redacted>-<redacted>","imageDigest":"sha256:<redacted>","imageManifest":"{\n   \"schemaVersion\": 2,\n   \"mediaType\": \"<redacted>\",\n   \"config\": {\n      \"mediaType\": \"<redacted>\",\n      \"size\": 5901,\n      \"digest\": \"sha256:<redacted>\"\n   },\n   \"layers\": [\n      {\n         \"mediaType\": \"<redacted>\",\n         \"size\": 51425992,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 123664308,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 12743115,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 91031203,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 213512017,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 9206109,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 11062104,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 45374,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 29905,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 1361,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 346,\n         \"digest\": \"sha256:<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 220,\n         \"digest\": \"sha256:<redacted>\"\n      }\n   ]\n}","imageTag":"443-aws-cdk.isolated-<redacted>"}) -> OK
[10:33:17] Tagged 444 assets
[57.46%] 1468 files scanned: 1362 assets (2105.36 GiB) tagged, 0 assets (0.00 GiB) deleted.
[10:33:17] Reading cached notices from /Users/benjaminleff/.cdk/cache/notices.json
[57.46%] 1468 files scanned: 1362 assets (2105.36 GiB) tagged, 0 assets (0.00 GiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[57.46%] 1468 files scanned: 1362 assets (2105.36 GiB) tagged, 0 assets (0.00 GiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[57.46%] 1468 files scanned: 1362 assets (2105.36 GiB) tagged, 0 assets (0.00 GiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[57.46%] 1468 files scanned: 1362 assets (2105.36 GiB) tagged, 0 assets (0.00 GiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[57.46%] 1468 files scanned: 1362 assets (2105.36 GiB) tagged, 0 assets (0.00 GiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[57.46%] 1468 files scanned: 1362 assets (2105.36 GiB) tagged, 0 assets (0.00 GiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[57.46%] 1468 files scanned: 1362 assets (2105.36 GiB) tagged, 0 assets (0.00 GiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[57.46%] 1468 files scanned: 1362 assets (2105.36 GiB) tagged, 0 assets (0.00 GiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.
[100.00%] 4762 files scanned: 0 assets (0.00 MiB) tagged, 0 assets (0.00 MiB) deleted.

This continues seemingly indefinitely. It's also strange that the verbose flag stops working after some period of time.

In another account I see the following:

...
[99.73%] 1839 files scanned: 2 assets (0.34 MiB) tagged, 0 assets (0.00 MiB) deleted.
[09:46:49] [sdk info] ECR.PutImage({"repositoryName":"cdk-hnb659fds-container-assets-<redacted>-<redacted>","imageDigest":"sha256:<redacted>
<redacted>","imageManifest":"{\n   \"schemaVersion\": 2,\n   \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n   \"config\": {\n      \"mediaType\": \"application/vnd.docker.container.image.v1+json\",\n      \"size\": 9791,\n      \"digest\": \"sha256:<redacted>
<redacted>\"\n   },\n   \"layers\": [\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 51425992,\n         \"digest\": \"sha256:<redacted>
<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 6857210421,\n         \"digest\": \"sha256:<redacted>
<redacted>\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 236,\n         \"digest\": \"sha256:<redacted>
<redacted>\"\n      }\n   ]\n}","imageTag":"186-aws-cdk.isolated-<redacted>"}) -> OK
[09:46:51] Tagged 187 assets
[21.78%] 294 files scanned: 187 assets (405.60 GiB) tagged, 0 assets (0.00 GiB) deleted.
[99.73%] 1839 files scanned: 2 assets (0.34 MiB) tagged, 0 assets (0.00 MiB) deleted.
[09:46:51] Reading cached notices from /Users/benjaminleff/.cdk/cache/notices.json
[99.73%] 1839 files scanned: 2 assets (0.34 MiB) tagged, 0 assets (0.00 MiB) deleted.
[99.73%] 1839 files scanned: 2 assets (0.34 MiB) tagged, 0 assets (0.00 MiB) deleted.
[99.73%] 1839 files scanned: 2 assets (0.34 MiB) tagged, 0 assets (0.00 MiB) deleted.
[99.73%] 1839 files scanned: 2 assets (0.34 MiB) tagged, 0 assets (0.00 MiB) deleted.
[99.73%] 1839 files scanned: 2 assets (0.34 MiB) tagged, 0 assets (0.00 MiB) deleted.
[99.73%] 1839 files scanned: 2 assets (0.34 MiB) tagged, 0 assets (0.00 MiB) deleted
...

This also continues seemingly indefinitely. Same with verbose flag behavior not working as expected after some period of time.

I've left it running for 6 hours and still the same.

Regression Issue

Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Delete all unused ecr and s3 cdk assets.

Current Behavior

Ecr and s3 assets that are unused are not being deleted and runs indefinitely.

Reproduction Steps

Run cdk gc --unstable='gc' --type='all' -vvvv --rollback-buffer-days=1

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.174.0

Framework Version

No response

Node.js Version

22.12.0

OS

MacOS

Language

Python

Language Version

No response

Other information

No response

The text was updated successfully, but these errors were encountered:

khushail · 2025-01-06T18:18:22Z

Hi @BwL1289 , thanks for reaching out. Looks like a similar issue with garbage collector was reported few days ago and fix was also put in place last week- #32679
Just curious have you recently tried this with the latest version and still facing the issue ?

BwL1289 · 2025-01-06T18:21:26Z

Hi @khushail. I am using v2.174.0 which is the latest version.

khushail · 2025-01-06T19:14:19Z

@BwL1289 , thanks for the confirmation. I tried to repro the scenario in my account with the latest version and here is what I am getting -

Images are being tagged but not deleted. Trying to reproduce it in other region.

BwL1289 · 2025-01-06T19:19:41Z

@khushail I am seeing the same behavior, in addition to what I described originally.

kaizencc · 2025-01-06T20:57:14Z

This is strange @BwL1289, sorry that you are experiencing this. If this is a nextToken issue as we thought before, I believe a repro with 1000+ images in the ecr repo would help. However, before I get to that tomorrow, I just want to make sure -- you are on CDK CLI version 2.174.0, not just aws-cdk-lib 2.174.0 right?

BwL1289 · 2025-01-06T21:00:05Z

@kaizencc all good. Appreciate the support. Yes, on CDK CLI version 2.174.0

BwL1289 · 2025-01-07T05:27:08Z

@kaizencc also just to clarify, I don't believe it's only a nextToken issue (I experienced that before the latest fix). In the previous version in the verbose output I could see the same image ids being iterated over.

Now I'm seeing images also get tagged but not deleted.

What's also interesting is the verbose output stops after a period of time running.

BwL1289 · 2025-01-08T15:41:33Z

@kaizencc were you able to reproduce?

BwL1289 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 5, 2025

github-actions bot added the @aws-cdk/assets Related to the @aws-cdk/assets package label Jan 5, 2025

BwL1289 mentioned this issue Jan 5, 2025

Garbage Collection for Assets aws/aws-cdk-rfcs#64

Closed

11 tasks

khushail added investigating This issue is being investigated and/or work is in progress to resolve the issue. p2 and removed needs-triage This issue or PR still needs to be triaged. labels Jan 6, 2025

khushail self-assigned this Jan 6, 2025

khushail added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Jan 6, 2025

khushail added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Jan 6, 2025

pahud changed the title ~~(aws-cdk/lib/api/garbage-collection): Garbage collection for ECR prints runs indefinitely~~ ecr: Garbage collection for ECR prints runs indefinitely Jan 6, 2025

khushail removed the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Jan 6, 2025

khushail removed their assignment Jan 6, 2025

khushail added the effort/small Small work item – less than a day of effort label Jan 6, 2025

kaizencc self-assigned this Jan 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ecr: Garbage collection for ECR prints runs indefinitely #32742

ecr: Garbage collection for ECR prints runs indefinitely #32742

BwL1289 commented Jan 5, 2025 •

edited

Loading

khushail commented Jan 6, 2025 •

edited

Loading

BwL1289 commented Jan 6, 2025

khushail commented Jan 6, 2025

BwL1289 commented Jan 6, 2025

kaizencc commented Jan 6, 2025

BwL1289 commented Jan 6, 2025

BwL1289 commented Jan 7, 2025

BwL1289 commented Jan 8, 2025

ecr: Garbage collection for ECR prints runs indefinitely #32742

ecr: Garbage collection for ECR prints runs indefinitely #32742

Comments

BwL1289 commented Jan 5, 2025 • edited Loading

Describe the bug

Regression Issue

Last Known Working CDK Version

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

khushail commented Jan 6, 2025 • edited Loading

BwL1289 commented Jan 6, 2025

khushail commented Jan 6, 2025

BwL1289 commented Jan 6, 2025

kaizencc commented Jan 6, 2025

BwL1289 commented Jan 6, 2025

BwL1289 commented Jan 7, 2025

BwL1289 commented Jan 8, 2025

BwL1289 commented Jan 5, 2025 •

edited

Loading

khushail commented Jan 6, 2025 •

edited

Loading