github-aws-commit-mirror (Archived)

This project fork has been archived. Due to GitHub disabling Actions automatically after 60 days on an inactive repo, without notification, this became a bad solution for us. In light of this, we have moved this job to an internal Jenkins server, where it will run until the cows come home.

Credit for the original work goes to @rribeiro1

You can use this project to automate the replication of a source repository in Github to a repository in AWS CodeCommit and S3, and it can be useful for:

• One-off task to migrate all active repositories to AWS CodeCommit
• Continuous backup process to mirror Github repos to AWS CodeCommit
• Backing up GitHub & CodeCommit repositories to AWS S3 buckets

It was inspired on this AWS article however, instead of Jenkins and EC2 we are using Github Actions to create a Cronjob and executing a Python Script which fetches all repositories from an account and for each repository, it creates the same repository in CodeCommit (if it does not exist) and mirrors the repository.

1. Requirements

• Github API Token
• An account on AWS and a user with right permissions to interact with AWS CodeCommit

2. Setup

2.1 Setup AWS account

1. Create a group on AWS e.g Devops
2. Create a user on AWS e.g codecommit_user and add to the group Devops
3. Create a policy e.g AwsCodeCommitMirroring and attach this policy to the group Devops

This is the minimum permission required to make it work

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "codecommit:TagResource",
                "codecommit:GetRepository",
                "codecommit:CreateRepository",
                "codecommit:GitPush",
                "codecommit:UpdateDefaultBranch"
            ],
            "Resource": "*"
        }
    ]
}

2.2 Setup Enviroment Variables

1. Fork this project
2. Configure the environment variables as described below:

• AWS_ACCESS_KEY_ID Access key from the user on AWS
• AWS_SECRET_ACCESS_KEY Secret access key from the user on AWS
• AWS_SSH_KEY_ID AWS SSH key ID from the user on AWS
• AWS_DEFAULT_REGION Region on AWS where you are using CodeCommit
• GH_API_TOKEN Github API Token

2.3 Setup Job scheduler

At the top of the main.yml file you can configure some as aspects of the job, such as the scheduler as well as the target branch to run the pipeline.

Use the cron parameter to configure the schedule, Crontab Guru can help on this task.

'on':
  pull_request:
  push:
    branches:
      - master
  schedule:
    #Every 15 minutes
    - cron: "*/15 * * * *"

2.4 Output

...
> Processing repository: spring-tdd-experiments
--> Cloning repository spring-tdd-experiments to local storage
Cloning into bare repository 'spring-tdd-experiments'...
remote: Enumerating objects: 51, done.
Receiving objects: 100% (51/51), 9.90 KiB | 9.90 MiB/s, done.
Resolving deltas: 100% (4/4), done.
remote: Total 51 (delta 0), reused 0 (delta 0), pack-reused 51
--> Pushing changes from repository spring-tdd-experiments to AWS CodeCommit
Everything up-to-date
--> Deleting repository spring-tdd-experiments from local storage
...

3. Workflow

The GitHub Action:

1. Installs the default enviroment specified in the pipfile.
2. Installs SSH keys for Github and AWS SSH.
3. Clones all GitHub repositories under the account associated with the GitHub Token.
4. Creates (if doesn't already exist) an identical repository on AWS CodeCommit.
5. Syncs the AWS CodeCommit repository if there are differences.
6. Backs up repository to AWS S3 if there were changes since the last run.

4. References

• Using IAM with CodeCommit: Git Credentials, SSH Keys, and AWS Access Keys
• Add CodeCommit to Your SSH Configuration
• Troubleshooting SSH Connections to AWS CodeCommit
• Using Identity-Based Policies (IAM Policies) for CodeCommit