Brainstorming: Build Script CI

[SinTh0r4s]

A loose collection of ideas on how it might be possible to test build scripts automatically. I've already had moments where I broke something with cleanup and fixes

[SinTh0r4s]

Trigger example branch builds on change and inspect jars for expected manifest entries

[SinTh0r4s]

Check version replacement: boilerplate mod + one specific key replaced. Build mod and check build/temporary build files for correct replacement

[SinTh0r4s]

Can we open a GitHub project and run CI project wide? Habe the separate test cases as repositories or as branches or on repo with tons of subfolders/modules?

[TheElan]

Can we open a GitHub project and run CI project wide? Habe the separate test cases as repositories or as branches or on repo with tons of subfolders/modules?

Interesting idea, would need to figure out pros and cons

[TheElan]

Perform tests twice on repo, before and after upgrading (do not attempt upgrade if pre is failing? Or upgrade if failures match?)

[TheElan]

Run upgrade and test on all repos before committing, if any fail - do not commit any (some way to list failed ones with reasons would be nice)

[TheElan]

Perform "dry run" (not commuting anything) upgrade on subset of most simple and advanced cases for PR to make CI fast and allow catching things earlier

[TheElan]

Triggering upgrades via hooks (if all tests pass commit to master)

[TheElan]

Manual upgrade via having all repos listed in a file and script walking all repos and committing when considered safe (e.g. all passed tests vs only ones that passed tests)

[TheElan]

It would be nice to have CI workflows upgrade too. They are evolving and not in their final form. We could just replace the folder each time and if wanted upgrade to trigger it would be a simple matter of expanding if condition to check for changes in .gitHub folder or jitpack.yml

We could even go as far as having a file with a list of files/folders to watch and replace, shouldn't be too hard at that point

[TheElan]

A script/task/workflow listing all repos which are not yet up to date

[TheElan]

A scheduled workflow in each repo which checks new version and if it's there - attempts upgrade. Runs once a day.

A badge in repo readme which shows last upgrade status.

This way auto upgrade will be baked in our starter/migration packs.

[SinTh0r4s]

I think it should be sufficient to iterate over repositories and copy-paste files and commit.
CI does not change that often compared to the build script. Or we have the build script replace .github completely whenever an update of the script is available. Should only show up as change if there was one.

Should we provide .gitignore updates? There are some changes / minimization / additions required.

Updating production repositories should always involve manual selection of a version!

[TheElan]

Use tags instead of hashes in buildscript. Change version on new tag added. Not to spam upgrades on small not critical changes.

Then we might not even need a version line inside, we can just create buildscript update tag each time it successfully upgraded and compare it with newest tag in buildscript repo SemVer is designed to allow easy comparison)

[SinTh0r4s]

Tags would also allow updates of CI without changing the build script

[SinTh0r4s]

Can we add CI badges automatically? Or do we need to do that manually?

[TheElan]

Extract buildscript to separate repository, which contains bare minimum, allowing simple copy over without filtering. Then example project becomes one of test subjects

[TheElan]

Can we add CI badges automatically? Or do we need to do that manually?

Could probably do it semi auto. Insert after first line and open a PR for humans to verify. Or commit and if there any bad formatting - somebody will eventually notice it.

[SinTh0r4s]

Why not add

def BuildScriptVersion = loadedTagFromMasterRepo

Instead of creating a tag in the target repo? A tag will be deleted by some user at some point. Just write that line before the rest of the file is written from remote

[SinTh0r4s]

Can we test commits on the build script on production repositories and compare before and after? Validate them manually before (refmaps for Mixins exist, MANIFEST entries are valid and so on). Then we don't need to create our own test repos. Might be less stable though

[TheElan]

Since people not always use SemVer (either no knowing or forgetting) and API rarely managed correctly it would be good to know that some dependants were broken.

Maybe introduce w step into build which won't fail PR but warn about problems, like getting list of dependants, bumping ver in each and trying to build

[TheElan]

It would be an option to have some scheduled workflow try to build with latest snapshot of everything instead, then at least people will be notified in advance.

[SinTh0r4s]

Print hash at beginning of build script for easier trace of errors