Updating cargo-mutants without introducing new patterns

[sourcefrog]

I've been making about 1 release every month and expect to keep doing so for a while. Each release has some combination of:

1. New mutation patterns, such as Mutate +, -, *, /, %, &, ^, |, <<, >>, and their assignment versions #198, or for example a future release might delete assignment statements or unary operators.
2. Other features, such as Shard mutants #197, Add --test-tool=nextest #223.
3. Bug fixes, such as Error: Patch input contains repeated filename: "/dev/null" #219.
4. Potentially, removal of mutation patterns that turned out to be too noisy or low-value.

When you update, you get all the fixes and features in the new version, and also all the new mutation patterns that it might introduce.

I think many users will and should have CI jobs that always install the current version, which is what cargo install and install-action naturally default to, and they'll want to get fixes and potentially new features.

However as a consequence this will mean that they also get new mutation patterns from time to time, which may surface new missed mutants and so might make CI fail due to discovering pre-existing coverage gaps.

One solution for this, maybe the simplest, would be for users to pin the installation in CI to a known version, and only update it from time to time, when they're ready to fix or skip newly-discovered gaps. This obviously doesn't need any cargo-mutants changes, but we could just document the suggestion.

Another option might be for cargo-mutants to have a concept of "which version introduced a mutation pattern", and a config option to only generate mutants up to a certain version. For example, if you set --pattern-stability=23.12.2 then you could keep installing cargo-mutants from head and it would never generate any new patterns and so, in theory, your build would never break unless you introduced new untested code. (This is very broadly analogous to Rust editions.) Perhaps there's a better name than this.

I think a rough version of this might not be too complicated to introduce.

Perhaps it would be onerous to guarantee that the patterns are always exactly stable, especially when bugs are fixed or when we have to refactor to enable new capabilities. For example, if we start following #[cfg(unix)] annotations, we might drop some previously existing patterns. Or, the changes to mutate smaller-than-function elements caused some changes to the reported spans for existing patterns. We could have tests that assert that the output on a given tree is exactly what it was in a previous release.

Since the main goal is to avoid unexpectedly dumping thousands of new failures on users then it seems it would be OK if the output is only generally stable and not 100% guaranteed to always be precisely the same.

One consequence of this is that users might not remember to update it, and so might never benefit from new mutation patterns. We could emit a message like "1234 mutants were suppressed by --pattern-stability=23.12.2".

Would you want this, or use it if it was available? Do you think you'd remember to increase the version from time to time?

related: #224 about documenting stability