Add additional tests for CI Visibility configuration behaviour #6513

andrewlock · 2025-01-08T15:10:47Z

Summary of changes

Adds some more tests for CI Visibility configuration

Reason for change

In #6399 we changed how this all worked. We thought we had a regression, but it seems that we fixed a bug. This adds tests to confirm the CI Vis behaviour is as expected

Implementation details

Adds some unit tests

Test coverage

More

datadog-ddstaging · 2025-01-08T15:42:37Z

Datadog Report

Branch report: andrew/add-ci-visibility-tests
Commit report: 8a5d9ae
Test service: dd-trace-dotnet

✅ 0 Failed, 237009 Passed, 2095 Skipped, 18h 39m 0.08s Total Time

andrewlock · 2025-01-08T15:47:33Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6513) - mean (69ms)  : 65, 72
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6513) - mean (985ms)  : 957, 1013
     .   : milestone, 985,
    master - mean (981ms)  : 954, 1008
     .   : milestone, 981,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6513) - mean (107ms)  : 105, 110
     .   : milestone, 107,
    master - mean (107ms)  : 105, 110
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6513) - mean (680ms)  : 663, 697
     .   : milestone, 680,
    master - mean (678ms)  : 662, 695
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6513) - mean (91ms)  : 89, 94
     .   : milestone, 91,
    master - mean (91ms)  : 90, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6513) - mean (635ms)  : 617, 654
     .   : milestone, 635,
    master - mean (633ms)  : 616, 651
     .   : milestone, 633,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6513) - mean (194ms)  : 188, 200
     .   : milestone, 194,
    master - mean (195ms)  : 189, 202
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6513) - mean (1,101ms)  : 1071, 1132
     .   : milestone, 1101,
    master - mean (1,104ms)  : 1076, 1132
     .   : milestone, 1104,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6513) - mean (279ms)  : 274, 283
     .   : milestone, 279,
    master - mean (278ms)  : 273, 282
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6513) - mean (876ms)  : 848, 904
     .   : milestone, 876,
    master - mean (874ms)  : 847, 901
     .   : milestone, 874,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6513) - mean (267ms)  : 262, 273
     .   : milestone, 267,
    master - mean (268ms)  : 264, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6513) - mean (854ms)  : 822, 885
     .   : milestone, 854,
    master - mean (857ms)  : 829, 885
     .   : milestone, 857,

andrewlock · 2025-01-09T03:08:09Z

Benchmarks Report for tracer 🐌

Benchmarks for #6513 compared to master:

1 benchmarks are faster, with geometric mean 1.172
1 benchmarks are slower, with geometric mean 1.143
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.11μs	45.6ns	329ns	0.0166	0.0083	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	50ns	320ns	0.0157	0.00524	0	5.8 KB
master	`StartStopWithChild`	net472	16.6μs	67.3ns	261ns	1.06	0.331	0.0889	6.21 KB
#6513	`StartStopWithChild`	net6.0	8.08μs	45ns	288ns	0.02	0.008	0	5.61 KB
#6513	`StartStopWithChild`	netcoreapp3.1	10.2μs	52.8ns	248ns	0.0206	0.00514	0	5.81 KB
#6513	`StartStopWithChild`	net472	16.4μs	62.1ns	232ns	1.05	0.306	0.105	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	480μs	324ns	1.26μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	654μs	328ns	1.18μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	855μs	841ns	3.26μs	0.422	3.3 KB
#6513	`WriteAndFlushEnrichedTraces`	net6.0	494μs	451ns	1.56μs	0	2.7 KB
#6513	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	653μs	554ns	2.15μs	0	2.7 KB
#6513	`WriteAndFlushEnrichedTraces`	net472	860μs	592ns	2.21μs	0.428	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	156μs	957ns	9.52μs	0.152	14.47 KB
master	`SendRequest`	netcoreapp3.1	172μs	1.01μs	9.35μs	0.165	17.27 KB
master	`SendRequest`	net472	0.00178ns	0.000725ns	0.00281ns	0	0 b
#6513	`SendRequest`	net6.0	154μs	900ns	8.49μs	0.142	14.47 KB
#6513	`SendRequest`	netcoreapp3.1	169μs	969ns	7.51μs	0.167	17.27 KB
#6513	`SendRequest`	net472	0.00364ns	0.00145ns	0.0056ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	581μs	3.27μs	22μs	0.587	0	0	41.6 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	671μs	3.72μs	22μs	0.322	0	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	867μs	3.9μs	15.1μs	8.19	2.59	0.431	53.33 KB
#6513	`WriteAndFlushEnrichedTraces`	net6.0	564μs	1.5μs	5.81μs	0.563	0	0	41.57 KB
#6513	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	692μs	3.87μs	23.8μs	0.327	0	0	41.91 KB
#6513	`WriteAndFlushEnrichedTraces`	net472	823μs	2.86μs	11.1μs	8.17	2.45	0.408	53.28 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.29μs	1.29ns	4.84ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.8μs	1.33ns	4.97ns	0.0135	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.13μs	2.18ns	8.44ns	0.157	0.00106	987 B
#6513	`ExecuteNonQuery`	net6.0	1.31μs	0.775ns	3ns	0.0144	0	1.02 KB
#6513	`ExecuteNonQuery`	netcoreapp3.1	1.84μs	1.07ns	4.16ns	0.0138	0	1.02 KB
#6513	`ExecuteNonQuery`	net472	2.12μs	3.16ns	12.2ns	0.156	0.00106	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.16μs	0.665ns	2.49ns	0.0139	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.54μs	1.66ns	6.21ns	0.0132	976 B
master	`CallElasticsearch`	net472	2.53μs	2.45ns	9.16ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.23μs	0.618ns	2.31ns	0.013	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	0.65ns	2.43ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.68μs	1.79ns	6.94ns	0.167	1.05 KB
#6513	`CallElasticsearch`	net6.0	1.23μs	0.729ns	2.73ns	0.0135	976 B
#6513	`CallElasticsearch`	netcoreapp3.1	1.53μs	0.887ns	3.2ns	0.013	976 B
#6513	`CallElasticsearch`	net472	2.56μs	2.72ns	10.5ns	0.158	995 B
#6513	`CallElasticsearchAsync`	net6.0	1.22μs	0.668ns	2.59ns	0.0134	952 B
#6513	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	1.11ns	4.15ns	0.0139	1.02 KB
#6513	`CallElasticsearchAsync`	net472	2.55μs	1.69ns	6.54ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.28μs	0.864ns	3.23ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	0.581ns	2.18ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.82μs	1.09ns	3.78ns	0.145	915 B
#6513	`ExecuteAsync`	net6.0	1.37μs	1.14ns	4.41ns	0.013	952 B
#6513	`ExecuteAsync`	netcoreapp3.1	1.62μs	0.786ns	2.94ns	0.0122	952 B
#6513	`ExecuteAsync`	net472	1.77μs	0.635ns	2.46ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.45μs	1.73ns	6.68ns	0.0315	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.29μs	2.82ns	10.9ns	0.0371	2.85 KB
master	`SendAsync`	net472	7.36μs	3.05ns	11ns	0.494	3.12 KB
#6513	`SendAsync`	net6.0	4.4μs	2.33ns	8.72ns	0.033	2.31 KB
#6513	`SendAsync`	netcoreapp3.1	5.34μs	1.94ns	7.27ns	0.0374	2.85 KB
#6513	`SendAsync`	net472	7.36μs	2.31ns	8.96ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.54μs	0.653ns	2.45ns	0.0231	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.09μs	0.997ns	3.73ns	0.0225	1.64 KB
master	`EnrichedLog`	net472	2.69μs	2.29ns	8.57ns	0.249	1.57 KB
#6513	`EnrichedLog`	net6.0	1.5μs	0.808ns	3.02ns	0.0232	1.64 KB
#6513	`EnrichedLog`	netcoreapp3.1	2.25μs	1.59ns	5.95ns	0.0218	1.64 KB
#6513	`EnrichedLog`	net472	2.73μs	2.16ns	8.07ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	118μs	195ns	756ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	164ns	635ns	0.0603	0	4.28 KB
master	`EnrichedLog`	net472	151μs	173ns	648ns	0.671	0.224	4.46 KB
#6513	`EnrichedLog`	net6.0	119μs	112ns	419ns	0	0	4.28 KB
#6513	`EnrichedLog`	netcoreapp3.1	121μs	355ns	1.37μs	0	0	4.28 KB
#6513	`EnrichedLog`	net472	152μs	309ns	1.2μs	0.682	0.227	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.92μs	1.13ns	4.38ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.2μs	1.84ns	6.88ns	0.0294	2.2 KB
master	`EnrichedLog`	net472	4.88μs	1.69ns	6.54ns	0.319	2.02 KB
#6513	`EnrichedLog`	net6.0	2.89μs	0.806ns	3.12ns	0.0303	2.2 KB
#6513	`EnrichedLog`	netcoreapp3.1	4.16μs	1.06ns	4.09ns	0.0293	2.2 KB
#6513	`EnrichedLog`	net472	5.01μs	1.38ns	5.34ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	0.818ns	3.17ns	0.0162	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.76μs	0.633ns	2.28ns	0.0152	1.14 KB
master	`SendReceive`	net472	2.13μs	1.58ns	5.92ns	0.184	1.16 KB
#6513	`SendReceive`	net6.0	1.32μs	0.438ns	1.58ns	0.0159	1.14 KB
#6513	`SendReceive`	netcoreapp3.1	1.75μs	1.24ns	4.81ns	0.0158	1.14 KB
#6513	`SendReceive`	net472	2.12μs	0.918ns	3.44ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.79μs	0.881ns	3.3ns	0.0223	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.95μs	1.6ns	6.19ns	0.0219	1.65 KB
master	`EnrichedLog`	net472	4.24μs	3.28ns	11.8ns	0.322	2.04 KB
#6513	`EnrichedLog`	net6.0	2.62μs	2.62ns	10.2ns	0.0224	1.6 KB
#6513	`EnrichedLog`	netcoreapp3.1	3.85μs	1.54ns	5.75ns	0.0213	1.65 KB
#6513	`EnrichedLog`	net472	4.4μs	2.66ns	10.3ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6513

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.143	538.86	615.75

Faster 🎉 in #6513

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.172	562.66	480.22

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	402ns	0.511ns	1.98ns	0.0081	576 B
master	`StartFinishSpan`	netcoreapp3.1	539ns	0.687ns	2.66ns	0.00777	576 B
master	`StartFinishSpan`	net472	650ns	1.43ns	5.56ns	0.0917	578 B
master	`StartFinishScope`	net6.0	563ns	0.836ns	3.24ns	0.00971	696 B
master	`StartFinishScope`	netcoreapp3.1	713ns	1.11ns	4.31ns	0.00965	696 B
master	`StartFinishScope`	net472	885ns	1.17ns	4.53ns	0.104	658 B
#6513	`StartFinishSpan`	net6.0	404ns	0.569ns	2.2ns	0.00802	576 B
#6513	`StartFinishSpan`	netcoreapp3.1	615ns	0.775ns	2.68ns	0.00785	576 B
#6513	`StartFinishSpan`	net472	713ns	1.42ns	5.51ns	0.0918	578 B
#6513	`StartFinishScope`	net6.0	480ns	0.643ns	2.49ns	0.00977	696 B
#6513	`StartFinishScope`	netcoreapp3.1	769ns	1.59ns	6.16ns	0.00939	696 B
#6513	`StartFinishScope`	net472	822ns	1.84ns	7.13ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	600ns	1.14ns	4.4ns	0.00973	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	896ns	1.28ns	4.96ns	0.00933	696 B
master	`RunOnMethodBegin`	net472	1.09μs	1.93ns	7.46ns	0.104	658 B
#6513	`RunOnMethodBegin`	net6.0	646ns	0.603ns	2.09ns	0.00967	696 B
#6513	`RunOnMethodBegin`	netcoreapp3.1	958ns	2.97ns	11.5ns	0.00918	696 B
#6513	`RunOnMethodBegin`	net472	1.16μs	1.69ns	6.55ns	0.104	658 B

andrewlock added area:tests unit tests, integration tests area:ci-visibility labels Jan 8, 2025

andrewlock requested a review from a team as a code owner January 8, 2025 15:10

tonyredondo approved these changes Jan 8, 2025

View reviewed changes

Add additional tests for CI Visibility test behaviour

8a5d9ae

andrewlock force-pushed the andrew/add-ci-visibility-tests branch from cbe967d to 8a5d9ae Compare January 8, 2025 16:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add additional tests for CI Visibility configuration behaviour #6513

Add additional tests for CI Visibility configuration behaviour #6513

andrewlock commented Jan 8, 2025

datadog-ddstaging bot commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 9, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6513

Faster 🎉 in #6513

Raw results

Raw results

Add additional tests for CI Visibility configuration behaviour #6513

Are you sure you want to change the base?

Add additional tests for CI Visibility configuration behaviour #6513

Conversation

andrewlock commented Jan 8, 2025

Summary of changes

Reason for change

Implementation details

Test coverage

datadog-ddstaging bot commented Jan 8, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 8, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 9, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6513

Faster 🎉 in #6513

Raw results

Raw results

datadog-ddstaging bot commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 8, 2025 •

edited

Loading