Fix "Unrecognized Option" error when using Criterion-specific arguments in benchmarks

[BD103]

Objective

• Commands like cargo bench -- --save-baseline before do not work because the default libtest is intercepting Criterion-specific CLI arguments.
• Fixes Can't save baselines when benchmarking #17200.

Solution

• Disable the default libtest benchmark harness for the library crate, as per the Criterion book.

Testing

• cargo bench -p benches -- --save-baseline before
  • You don't need to run the entire benchmarks, just make sure that they start without any errors. :)

[BenjaminBrienen]

PS C:\Users\BenjaminBrienen\source\bevy> git status          
On branch fix-bench-unrecognized-option
Your branch is up to date with 'BD103/fix-bench-unrecognized-option'.

nothing to commit, working tree clean
PS C:\Users\BenjaminBrienen\source\bevy> cargo bench -- --save-baseline before
warning: unused import: `warn`
 --> crates\bevy_render\src\extract_resource.rs:7:22
  |
7 | use tracing::{error, warn};
  |                      ^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: `bevy_render` (lib) generated 1 warning (run `cargo fix --lib -p bevy_render` to apply 1 suggestion)
    Finished `bench` profile [optimized] target(s) in 0.41s
     Running unittests src\lib.rs (target\release\deps\bevy-d792cd0559babc77.exe)
error: Unrecognized option: 'save-baseline'
error: bench failed, to rerun pass `--lib`

[BD103]

Looks like the warning if because tracing::error! is only used behind #[cfg(debug_assertions)], but the benchmarks are compiled with release mode. I'm working on a fix now!

[BenjaminBrienen]

PS C:\Users\BenjaminBrienen\source\bevy> gh pr checkout 17222
Already on 'fix-bench-unrecognized-option'
Your branch is up to date with 'BD103/fix-bench-unrecognized-option'.
Already up to date.
PS C:\Users\BenjaminBrienen\source\bevy> cargo bench -- --save-baseline before
    Finished `bench` profile [optimized] target(s) in 0.60s
     Running unittests src\lib.rs (target\release\deps\bevy-d792cd0559babc77.exe)
error: Unrecognized option: 'save-baseline'
error: bench failed, to rerun pass `--lib`

@BD103 still not recognizing the option

[BD103]

Ah, I know! You need to cd benches first, since the benchmarks are associated with their own crate. The command that you should run is cargo bench -p benches -- --save-baseline before, for brevity. :)

Edit: I updated the original PR description with the better version of the command.

[BenjaminBrienen]

Oh yeah! Oops, lol. Stand by while I try benching the benches.

[BenjaminBrienen]

PS C:\Users\BenjaminBrienen\source\bevy> cargo bench -p benches -- --save-baseline before
    Finished `bench` profile [optimized] target(s) in 0.44s
     Running benches/bevy_ecs/main.rs (target\release\deps\ecs-0f0bbf4f3c69bf77.exe)
Gnuplot not found, using plotters backend
Benchmarking all_added_detection/5000_entities_ecs::change_detection::Table: Collecting 100 samples in estimated 4.3560 s (30k iterations)

much better!

[hymm]

Tried this, but it's still not saving the baselines. There's no benches/target/criterion folder and running critcmp --baselines doesn't show any either. Are they saving for you? Maybe it's a problem with my setup.

[BD103]

There's no benches/target/criterion folder [...]

The benchmarks were recently added to the Cargo workspace, so it would be target/criterion, not benches/target/criterion.

What error message are you getting?

[hymm]

I didn't know that the benches were added to the workspace. I was running critcmp from the benches folder, so that was the problem. Works for me when I run it from the base folder.