CI/tests: cache reference metrics | PoC w/ text

[Borda]

What does this PR do?

it is a subset of #2335 with setting the CI and validation on Text only
with local runs, it seems to cut 40% times so for example BLUE score went from 2m32 to 1m24

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

---

📚 Documentation preview 📚: https://torchmetrics--2382.org.readthedocs.build/en/2382/

[Borda]

@stancld is it possible that some metrics/references are not deterministic?
also it seem to be happening for metrics which has some list like arguments
cc: @SkafteNicki

FAILED unittests/text/test_mer.py::TestMatchErrorRate::test_mer_class[False-preds0-targets0] - AssertionError
FAILED unittests/text/test_mer.py::TestMatchErrorRate::test_mer_class[False-preds1-targets1] - AssertionError
FAILED unittests/text/test_wil.py::TestWordInfoLost::test_wil_class[False-preds1-targets1] - AssertionError
FAILED unittests/text/test_mer.py::TestMatchErrorRate::test_mer_functional[preds0-targets0] - AssertionError
FAILED unittests/text/test_wil.py::TestWordInfoLost::test_wil_functional[preds1-targets1] - AssertionError
FAILED unittests/text/test_mer.py::TestMatchErrorRate::test_mer_functional[preds1-targets1] - AssertionError
FAILED unittests/text/test_wip.py::TestWordInfoPreserved::test_wip_class[False-preds0-targets0] - AssertionError
FAILED unittests/text/test_wip.py::TestWordInfoPreserved::test_wip_class[False-preds1-targets1] - AssertionError
FAILED unittests/text/test_wip.py::TestWordInfoPreserved::test_wip_functional[preds0-targets0] - AssertionError
FAILED unittests/text/test_wip.py::TestWordInfoPreserved::test_wip_functional[preds1-targets1] - AssertionError

also when I run just the one test suite like wip it passes

[Borda]

seems to be cache issue: python-cachier/cachier#196

[codecov]

Codecov Report

Merging #2382 (184b392) into master (b548fad) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #2382   +/-   ##
======================================
  Coverage      69%     69%           
======================================
  Files         307     307           
  Lines       17352   17352           
======================================
  Hits        11961   11961           
  Misses       5391    5391           

[Borda]

seems like unrelated GPU falling on GPU for timeout

FAILED unittests/image/test_fid.py::test_fid_same_input[768] - Failed: Timeout >240.0s
FAILED unittests/image/test_fid.py::test_fid_same_input[2048] - Failed: Timeout >240.0s
FAILED unittests/image/test_mifid.py::test_fid_same_input[2048] - Failed: Timeout >240.0s

[Borda]

Observed speed-up for standard unittests on GPU is 34min -> 18min