Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow to retag docker images with CalVer string #32

Merged
merged 5 commits into from
Dec 5, 2022
Merged

Workflow to retag docker images with CalVer string #32

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1fbd80e
Workflow to retag docker images with CalVer string
weiji14 Dec 3, 2022
e35c33d
Remove free up disk space step
weiji14 Dec 5, 2022
5b276e7
Remove set up docker buildx step
weiji14 Dec 5, 2022
014714b
Revert "Remove free up disk space step"
weiji14 Dec 5, 2022
f58fac1
Don't retag with latest label
weiji14 Dec 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,6 @@
- binder.yaml - Adds a Binder badge to Pull Requests that are newly opened
- build.yaml - Build and push docker container images to a docker registry
- conda-lock-command.yml - Refresh conda-lock files by writing `/condalock` in a Pull Request comment
- retag.yml - Republish docker images originally tagged with a short hash using a new CalVer string
- slash-command-dispatch.yml - ChatOps that looks for slash commands in Pull Requests to trigger automated scripts
- test.yaml - Test building docker container images in a Pull Request
58 changes: 58 additions & 0 deletions .github/workflows/retag.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Re-tag staging SHA-tagged image with git tag and 'latest'
# tags can be anything, but typically calver string (2022.12.02)
name: Retag
on:
push:
tags:
- '*'

env:
GITHUB_SHA: ${{ github.sha }}
GITHUB_REF: ${{ github.ref }}

permissions: # added using https://github.com/step-security/secure-workflows
contents: read

jobs:
retag-using-calver:
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v3

# https://github.com/actions/runner-images/issues/2840#issuecomment-790492173
- name: Free up disk space
run: |
sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc
df -h
Comment on lines +23 to +27
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could probably be removed too, since the cryo-hub-image is only ~2GB in size. Not expecting it to become too big unless some machine learning libraries are added.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I generally suggest keeping it, because when the disk does become full, it can throw weird errors. And the image size is calculated compressed, while it can definitely become bigger when on disk. And GitHub actions by default only has about 10-12GB of free space, so it can add up fast.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've added it back in 014714b. This step will take ~50 seconds to execute. GitHub Actions runners have 14GB of space according to https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @weiji14!


- name: Set Job Environment Variables
run: |
SHA12="${GITHUB_SHA::12}"
TAG="${GITHUB_REF##*/}"
echo "SHA12=${SHA12}" >> $GITHUB_ENV
echo "TAG=${TAG}" >> $GITHUB_ENV

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

Copy link
Member Author

@weiji14 weiji14 Dec 3, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if we actually need buildx here. Does the default GitHub Actions runner come with docker installed?

Edit: Yes, docker seems to be installed on the Ubuntu GitHub Actions runner, according to https://github.com/actions/runner-images/blob/ac8ae6b0eeeec50305bf54ffdf588d5bbe9fc145/images/linux/Ubuntu2204-Readme.md#tools

- name: Login to Quay.io
uses: docker/login-action@v2
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}

- name: Pull Image for Corresponding GitHub Commit
run: |
docker pull quay.io/cryointhecloud/cryo-hub-image:${SHA12}

- name: Retag Images
run: |
docker tag cryointhecloud/cryo-hub-image:${SHA12} quay.io/cryointhecloud/cryo-hub-image:latest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we explicitly turn off the 'latest' tag here:

hub-image/.github/workflows/build.yaml

Line 43 in f254450

LATEST_TAG_OFF: true
. In our experience, it's mostly caused problems - people use it because it's convenient, but then 'latest' can refer to different images at the same point in time, especially when being pulled across multiple nodes! I'd suggest not having a latest tag.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. I've removed the retagging with 'latest' at f58fac1.

docker tag cryointhecloud/cryo-hub-image:${SHA12} quay.io/cryointhecloud/cryo-hub-image:${TAG}

- name: Push Tags To Quay.io
run: |
docker push quay.io/cryointhecloud/cryo-hub-image:latest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here about 'latest'

docker push quay.io/cryointhecloud/cryo-hub-image:${TAG}