Skip to content

Commit

Permalink
walkthrough_performance-benchmarks.md: Update results
Browse files Browse the repository at this point in the history
  • Loading branch information
Samuel Kalbfleisch committed Jun 8, 2022
1 parent 4e8e499 commit da5aeb7
Showing 1 changed file with 13 additions and 15 deletions.
28 changes: 13 additions & 15 deletions artifact-evaluation/walkthrough_performance-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,25 +12,25 @@ fs-testing/scripts/script_run-paper-benchmark.sh
The benchmark may take between 30 and 60 minutes to complete and writes its
files to `results_paper-benchmark`.

Its final output should be similar to the following (only placeholder values listed here):
Its final output should be similar to the following:
```
== Figure 8 (pmemtrace.py) results ==
count mean sstdev
total elapsed tracer process time: 20 6.04 4.4494718330153e-19
->boot (only minimal instrumentation): 20 1.5320778638124 2.2247359165076e-19
->trace (from outside): 20 3.8895835317671 0
->->trace (in guest, portion of command): 20 2.36 2.2247359165076e-19
execution in guest with raw PANDA: 20 0.0895 0.015035046776746
total elapsed tracer process time: 20 6.374 0.11495536891379
->boot (only minimal instrumentation): 20 1.6254023430869 0.02828601624545
->trace (from outside): 20 4.152885376662 0.098335081859825
->->trace (in guest, portion of command): 20 3.016 0.051646572210096
execution in guest with raw PANDA: 20 0.0895 0.0022360679774998
== Figure 9 (trace2img.py) results ==
count mean sstdev
total elapsed process time: 20 84.52 7.1191549328245e-18
->boot: 20 1.5242083892226 3.3371038747615e-19
->crash image generator: 20 38.556703411043 0
->->cross-failure tracing (heuristic): 12 2.0293949271242 0.46475635500365
->tester: 20 43.777359969914 0
->->reset to snapshot & load image: 31 0.080034287826669 0.0049169874557684
->->run dumper command (PANDA): 31 1.0457796561382 0.094908553471208
total elapsed process time: 20 83.8215 0.52551002794779
->boot: 20 1.6333558445796 0.033064026122977
->crash image generator: 20 37.874895266071 0.32519190866099
->->cross-failure tracing (heuristic): 12 1.9969639750198 0.28365390070709
->tester: 20 43.645736410469 0.35437410606065
->->reset to snapshot & load image: 31 0.075392498364372 0.0056485017603337
->->run dumper command (PANDA): 31 1.0400925968443 0.11656504327783
The tester processes 31 unique crash images
that stem from 77 origins (§3.3)
Expand All @@ -39,7 +39,5 @@ and result in 7 unique semantic crash states.
success
```

NOTE: The benchmark results listed in the paper PDF are not yet up to date with the latest artifact version. The differences are not big though.

The tables should approximately reproduce the numbers in Figure 8 and Figure 9 (depending on the hardware used).
The paragraph at the end of the output should further contain the same numbers as the corresponding sentence in §6.3.

0 comments on commit da5aeb7

Please sign in to comment.