What kind of experiments are required to compare designs

[fgmccabe]

One of the advantages to having multiple approaches to stack switching is the potential to compare them. The question is, what is the best basis for comparing potential designs?

One natural way to compare alternatives is to design one or more experiments and to examine the results of those experiments. However, designing experiments requires addressing some concerns:

1. What is being measured? This can vary from simple performance benchmarks to cost of implementation. Some desirable qualities are hard to measure; for example, security is typically difficult to measure.
2. What is the cost of the experiment? Some experiments are easier to 'run' than others. For example, a performance benchmark requires implementing a design as well as designing and running benchmark programs. Other experiments are more like Gedankenexperiments and can be performed by thinking (sic).
3. What is the validity of the experiment? If an experiment comparing multiple approaches is intended to be fair, then minimizing the variables is important. For example, implementing the fibers proposal in V8 and the typed continuations proposal in Wasmtime is unlikely to result in meaningful performance comparisons because of the fundamentally different architectures and goals of V8 and Wasmtime.

Some suggested experiments to run:

• Implement the fibers proposal and the type continuations proposal in Wasmtime.
  • Wasmtime is suggested because it is a simpler engine than V8, and the experiment itself should be cheaper to run.
  • Such implementations should not rely on external libraries to function, but should be based on generated code.
• Construct a series of 'benchmarks', implement them in suggested proposals and compare resulting codes
• Construct a toolchain backend for Dart (say) that generates code for all suggested proposals. Compare the 'damage' to the Dart compiler required (in terms of changed loc in the compiler).