Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarkables named like tuples are missing from the results table #154

Open
adrhill opened this issue Aug 12, 2024 · 0 comments
Open

Benchmarkables named like tuples are missing from the results table #154

adrhill opened this issue Aug 12, 2024 · 0 comments

Comments

@adrhill
Copy link

adrhill commented Aug 12, 2024

In our benchmark suite, @gdalle and I had several tests named similarly to named-tuples:

N = 2
K = 3
SUITE["Foo"]["(N=$N, K=$K)"] = @benchmarkable foo($x)

In the table generated by PkgTemplate's export_markdown, benchmarks like this get renamed to the ID ["Foo", (:(N = 2), :(K = 3))].
My best guess is that the string "(N=$N, K=$K)" somehow gets parsed as a named tuple.

The real problem is that all benchmarks with named-tuple-like names get excluded from the comparison table, regardless of improvements or regressions.


We ran into this issue here, where none of our ["bench_jogger.jl", "Hessian", ...] benchmarks appeared in the comparison. However, they did appear in the baseline and target sections with noticeable improvements, as shown in these screenshots. Note that all of them had named-tuple-like names.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant