How to deal with CI that machines are changing

[marcosschroh]

Hi,

Thanks for this library. This is not an issue, is more than a question about how to deal when either when a python version changes or a OS machine changes.

We are running during the CI the command pytest tests/test_benchmarks.py --benchmark-compare --benchmark-compare-fail=mean:10% which it compares the result against the previous benchmark, the problem is that sometimes the machines are not the same per PRs, then the command fails due ERROR: --benchmark-compare-fail requires valid --benchmark-compare. Any ideas how to bypass the error as we can not control the machines types during the CI? How the file to compare is selected?