Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve benchmark stability with cpusets, nice, and disabling of hyper threading #354

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

rwstauner
Copy link
Contributor

This adds to run_benchmarks.rb several additional linux features which can help to make the benchmark results more reliable:

  • set scheduler priority (niceness) high
  • setup two cpusets, one for benchmarks and one for everything else to move existing processes off of benchmark cpu
  • disable hyper threading for benchmark cpus
  • move parent process off of cpu that will be used for benchmarks

You can see this improve the results when running the same ruby version twice:
it increases the number of benchmarks that compare at 1.000 on two different computers:

image

Screenshot_2024-12-17_20-48-08

By putting all this logic into a module we can easily reuse it from yjit-metrics.

I think it would make sense to eventually move the existing checks for turbo into this module as well.

@rwstauner
Copy link
Contributor Author

For the curious, it looks like it isn't finding any cpuset files in CI, it's only setting the nice value there.

run_benchmarks.rb Outdated Show resolved Hide resolved
@maximecb maximecb requested review from k0kubun and XrXr December 20, 2024 21:55
@maximecb
Copy link
Contributor

Looks pretty good to me but I will also tag Alan and Kokubun to review since it's very important that we get benchmarking right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants