No way to distinguish between mix test --warnings-as-errors test failures (exit 2) or test failures with warnings in test files (exit 2)

[naserca]

In our CI pipeline, we rerun failed tests once to pass flaky tests that fail inconsistently. This means the following scenario is quite common:

1. mix test --warnings-as-errors -> exit 2
   1. flaky test failure in foo_test.exs
   2. compilation warning in bar_test.exs but tests pass
2. mix test --failed --warnings-as-errors -> exit 0
   1. flaky foo_test.exs passes
   2. compiler doesn't require bar_test.exs again so the warning is lost

Happy to help with a solution here if folks think the problem is worth solving.

[marcofognog]

I don't think --warnings-as-errors flag is meant to be understood as "warnings as failures". 🤔

From the Mix.Test.Task module docs:

* `--warnings-as-errors` *(since v1.12.0)* - treats compilation warnings (from loading the
  test suite) as errors and returns a non-zero exit status if the test suite would otherwise
  pass. Note that failures reported by `--warnings-as-errors` cannot be retried with the
  `--failed` flag.
  This option only applies to test files. To treat warnings as errors during compilation and
  during tests, run:
      MIX_ENV=test mix do compile --warnings-as-errors + test --warnings-as-errors

So strictly speaking, the behavior you described works as expected.

Now, trying to rephrase the problem: --warnings-as-errors was introduced to make builds fail when all tests pass and have warnings, like a safety net.
But if your build fails tests with no warnings, you retry with --failed, the warnings-as-errors flag doesn't fulfill its purpose anymore.

Should we think about a warnings-as-failures flag?