[loadgen] Separate into otelbench for benchmarking, otelsoak for soak testing

[carsonip]

Separate loadgen into otelbench and otelsoak.

otelbench

Run go benchmark on top of loadgen collector for a predefined duration -test.benchtime. otelbench accepts go benchmark flags, an otel collector config yaml, and some handy command line options to override the config. It produces go benchmark output.

Run ./otelbench -config ./config.yaml -test.benchtime=1m with ELASTIC_APM_SERVER_URL and ELASTIC_APM_SECRET_TOKEN env vars or flags.

Example output:

logs-otlp       	      16	4521329779 ns/op	         0 failed_logs/s	         0 failed_metric_points/s	         0 failed_requests/s	         0 failed_spans/s	       319.2 logs/s	         0 metric_points/s	       136.5 requests/s	         0 spans/s
logs-otlphttp   	      19	3900502735 ns/op	         0 failed_logs/s	         0 failed_metric_points/s	         0 failed_requests/s	         0 failed_spans/s	       370.0 logs/s	         0 metric_points/s	       158.2 requests/s	         0 spans/s
metrics-otlp    	    3584	  20884237 ns/op	         0 failed_logs/s	         0 failed_metric_points/s	         0 failed_requests/s	         0 failed_spans/s	         0 logs/s	      5746 metric_points/s	       143.6 requests/s	         0 spans/s
metrics-otlphttp	    3506	  19158620 ns/op	         0 failed_logs/s	         0 failed_metric_points/s	         0 failed_requests/s	         0 failed_spans/s	         0 logs/s	      6263 metric_points/s	       156.6 requests/s	         0 spans/s
traces-otlp     	      84	 730912803 ns/op	         0 failed_logs/s	         0 failed_metric_points/s	         0 failed_requests/s	         0 failed_spans/s	         0 logs/s	         0 metric_points/s	       153.2 requests/s	      1034 spans/s
traces-otlphttp 	      88	 716443449 ns/op	         0 failed_logs/s	         0 failed_metric_points/s	         0 failed_requests/s	         0 failed_spans/s	         0 logs/s	         0 metric_points/s	       156.3 requests/s	      1055 spans/s

otelsoak

otelsoak is the Elastic collector components distro with an example collector config file.

[carsonip]

[to reviewer] Printing to stdout as that's what go test -test.bench does. Not sure why apm-server benchtest writes to stderr (code).

[axw]

I don't know either, writing to stdout seems like the right thing to do - so we can easily tee the results to a file without getting any noise from debug logs.

[carsonip]

[to reviewer] originally i started with sharing a doneCh across signals, but since it should be used when the signal's MaxReplay is reached, I've made it per-signal. It doesn't make any difference in practice now, since we benchmark per signal now, but it may be useful later when we mix signals in benchmarks.

[axw]

Beautiful!