Allow to output the resulting table to a file

[aldanor]

This would be extremely useful if you run this as a part of continuous integration -- currently, the images can be saved (per each benchmark), but the resulting table itself cannot. Grepping through test logs on a build server is not fun -- would be much nicer if the benchmarks could be pulled out.

If it was possible to dump the results into a file (txt, csv or maybe even a nice formatted html, kind of like coverage does), then the benchmarks could be automatically published on each build as test artifacts.

[ionelmc]

There's the JSON saving option right now. We can add a
--benchmark-save-text or something I think ...

On Sunday, September 13, 2015, Ivan Smirnov [email protected]
wrote:

This would be extremely useful if you run this as a part of continuous
integration -- currently, the images can be saved (per each benchmark), but
the resulting table itself cannot. Grepping through test logs on a build
server is not fun -- would be much nicer if the benchmarks could be pulled
out.

If it was possible to dump the results into a file (txt, csv or maybe even
a nice formatted html, kind of like coverage does), then the benchmarks
could be automatically published on each build as test artifacts.

—
Reply to this email directly or view it on GitHub
#22.

Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro

[aldanor]

Yep, JSON is nice but you can't publish it directly, like Python coverage reports which can produce nice HTML that you can then just publish as an artifact and provide a way to view benchmarks for any build in previous history.

Of course, I could implement a script that reads json and outputs results into txt/csv/html but it sounds like something that should be done internally.

[ionelmc]

Or maybe a --benchmar-result-log=path? Pytest has the builtin --result-log=path option so a similar name would be good.

[aldanor]

It's not really a log though (which implies something machine-readable), I primarily meant for more human-readable formats like csv or txt or HTML?

Please see my comment in #20 -- if the data collection and reporting logic are decoupled then it would be simple to add any number of output backends like csv/html/txt/json/terminal/pygal.

[lelit]

A related improvement could be being able to emit the table in a somewhat standard format, for example as a reST table (see texttable).

[ionelmc]

Well, we could have an option to output restructuredtext but it's not as rich as html. Coloring and alignment are tricky there.

[risoms]

@ionelmc Just wanted to check if there's any plan to implement this, or if there's a workaround using the hooks?

[ionelmc]

You can read the data files (that you get when using --benchmark-save) and convert that json to whatever you like.

[risoms]

@ionelmc Ideally I'm hoping to construct a table with both current (NOW) and historic comparison (--benchmark-compare) data. Using the pytest_benchmark_generate_json does not provide comparison data, while pytest_benchmark_group_stats does not provide commit information.

Since the benchmark.json ultimately DOES contain both commit information and benchmark details, is there a way to capture both comparison data and commit information via hook?

[ionelmc]

I think the only way would be to extend the pytest_benchmark_group_stats hook to also pass a commit_info argument...

[risoms]

@ionelmc I do think it would be useful.

What's the procedure for introducing something like this for your repo? (ie fork)

[whyzdev]

See #230 (comment) for an example to use the saved benchmark json to generate html tables in pytest html output file, if it helps.