Consolidate all integration test targets into a single target #6501

andrewlock · 2025-01-06T10:10:37Z

Summary of changes

Reason for change

We previously had RunWindowsIntegrationTests, RunLinuxIntegrationTests, RunOsxIntegrationTests, each with subtle differences. Overall that adds complexity which isn't really necessary

Implementation details

Combine everything into a single target, and just specialize the filter instead to reduce the code

Test coverage

Tested locally on Windows, and ran a CI test, but OSX side is untested currently.

Other details

Stacked on

Build the test sample projects in a separate CI stage #6498

https://datadoghq.atlassian.net/browse/LANGPLAT-66

andrewlock · 2025-01-06T10:44:58Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (69ms)  : 66, 72
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (981ms)  : 954, 1008
     .   : milestone, 981,
    master - mean (981ms)  : 954, 1008
     .   : milestone, 981,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (107ms)  : 105, 109
     .   : milestone, 107,
    master - mean (107ms)  : 105, 110
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (680ms)  : 664, 697
     .   : milestone, 680,
    master - mean (678ms)  : 662, 695
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (91ms)  : 89, 94
     .   : milestone, 91,
    master - mean (91ms)  : 90, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (635ms)  : 622, 649
     .   : milestone, 635,
    master - mean (633ms)  : 616, 651
     .   : milestone, 633,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (193ms)  : 189, 198
     .   : milestone, 193,
    master - mean (195ms)  : 189, 202
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (1,100ms)  : 1068, 1133
     .   : milestone, 1100,
    master - mean (1,104ms)  : 1076, 1132
     .   : milestone, 1104,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (278ms)  : 272, 284
     .   : milestone, 278,
    master - mean (278ms)  : 273, 282
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (868ms)  : 845, 891
     .   : milestone, 868,
    master - mean (874ms)  : 847, 901
     .   : milestone, 874,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (267ms)  : 260, 274
     .   : milestone, 267,
    master - mean (268ms)  : 264, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (849ms)  : 815, 884
     .   : milestone, 849,
    master - mean (857ms)  : 829, 885
     .   : milestone, 857,

datadog-ddstaging · 2025-01-06T10:55:58Z

Datadog Report

Branch report: andrew/ci/combine-integration-test-targets
Commit report: d813115
Test service: dd-trace-dotnet

❌ 158 Failed (0 Known Flaky), 220916 Passed, 1301 Skipped, 11h 33m 11.05s Total Time
⌛ 1 Performance Regression

❌ Failed Tests (158)

This report shows up to 5 failed tests.

CallTargetBubbleUpExceptionIntegrations - Datadog.Trace.ClrProfiler.IntegrationTests.CallTargetNativeTests - Details
Expand for error
```
 application not found: /project/artifacts/bin/CallTargetNativeTest/release_net8.0/CallTargetNativeTest.dll
```
CallTargetBubbleUpExceptionIntegrations - Datadog.Trace.ClrProfiler.IntegrationTests.CallTargetNativeTests - Details
Expand for error
```
 application not found: /project/artifacts/bin/CallTargetNativeTest/release_net9.0/CallTargetNativeTest.dll
```
CallTargetBubbleUpExceptionIntegrations - Datadog.Trace.ClrProfiler.IntegrationTests.CallTargetNativeTests - Details
Expand for error
```
 application not found: /project/artifacts/bin/CallTargetNativeTest/release_net9.0/CallTargetNativeTest.dll
```
CallTargetBubbleUpExceptionIntegrations - Datadog.Trace.ClrProfiler.IntegrationTests.CallTargetNativeTests - Details
Expand for error
```
 application not found: /project/artifacts/bin/CallTargetNativeTest/release_net8.0/CallTargetNativeTest.dll
```
CallTargetBubbleUpExceptionIntegrations - Datadog.Trace.ClrProfiler.IntegrationTests.CallTargetNativeTests - Details
Expand for error
```
 application not found: /project/artifacts/bin/CallTargetNativeTest/release_netcoreapp3.1/CallTargetNativeTest.dll
```

⌛ Performance Regressions vs Default Branch (1)

ObjectExtractorSimpleBody - Benchmarks.Trace.Asm.AppSecBodyBenchmark 183ns (+41ns, +30%) - Details

andrewlock · 2025-01-06T11:07:13Z

Benchmarks Report for appsec 🐌

Benchmarks for #6501 compared to master:

1 benchmarks are slower, with geometric mean 1.143
2 benchmarks have fewer allocations
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6501

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0	1.143	160.62	183.51

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	202μs	169ns	655ns	2.61	0	188.76 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	299μs	200ns	773ns	2.68	0	196.09 KB
master	`AllCycleSimpleBody`	net472	273μs	118ns	441ns	35.8	2.05	225.79 KB
master	`AllCycleMoreComplexBody`	net6.0	209μs	123ns	459ns	2.72	0	192.26 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	317μs	139ns	520ns	2.68	0	199.5 KB
master	`AllCycleMoreComplexBody`	net472	278μs	174ns	674ns	36.4	2.08	229.3 KB
master	`ObjectExtractorSimpleBody`	net6.0	161ns	0.13ns	0.486ns	0.00393	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	195ns	0.139ns	0.52ns	0.00373	0	272 B
master	`ObjectExtractorSimpleBody`	net472	168ns	0.0874ns	0.338ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	2.91μs	1.13ns	4.07ns	0.0524	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.76μs	3.16ns	11.8ns	0.0488	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.63μs	2.34ns	8.75ns	0.603	0.00722	3.8 KB
#6501	`AllCycleSimpleBody`	net6.0	206μs	76ns	284ns	2.68	0	188.76 KB
#6501	`AllCycleSimpleBody`	netcoreapp3.1	302μs	107ns	415ns	2.71	0	196.09 KB
#6501	`AllCycleSimpleBody`	net472	273μs	277ns	1μs	35.8	2.02	225.79 KB
#6501	`AllCycleMoreComplexBody`	net6.0	213μs	95.5ns	357ns	2.67	0	192.26 KB
#6501	`AllCycleMoreComplexBody`	netcoreapp3.1	320μs	141ns	528ns	2.71	0	199.51 KB
#6501	`AllCycleMoreComplexBody`	net472	275μs	118ns	427ns	36.4	2.06	229.3 KB
#6501	`ObjectExtractorSimpleBody`	net6.0	184ns	0.106ns	0.395ns	0.00393	0	280 B
#6501	`ObjectExtractorSimpleBody`	netcoreapp3.1	193ns	0.21ns	0.785ns	0.00378	0	272 B
#6501	`ObjectExtractorSimpleBody`	net472	163ns	0.148ns	0.572ns	0.0446	0	281 B
#6501	`ObjectExtractorMoreComplexBody`	net6.0	2.91μs	1.65ns	6.17ns	0.0538	0	3.78 KB
#6501	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.81μs	2.43ns	9.08ns	0.0499	0	3.69 KB
#6501	`ObjectExtractorMoreComplexBody`	net472	3.62μs	3.3ns	11.9ns	0.602	0.00541	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EncodeArgs`	net6.0	36.7μs	26.7ns	104ns	0.457	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	55.2μs	33.8ns	131ns	0.412	0	32.4 KB
master	`EncodeArgs`	net472	67.9μs	47.8ns	179ns	5.15	0.0678	32.5 KB
master	`EncodeLegacyArgs`	net6.0	74.4μs	22.5ns	84.3ns	0	0	2.14 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	105μs	155ns	602ns	0	0	2.14 KB
master	`EncodeLegacyArgs`	net472	156μs	56.3ns	218ns	0.312	0	2.15 KB
#6501	`EncodeArgs`	net6.0	35.8μs	21ns	78.7ns	0.45	0	32.4 KB
#6501	`EncodeArgs`	netcoreapp3.1	54.2μs	20.2ns	75.5ns	0.431	0	32.4 KB
#6501	`EncodeArgs`	net472	66.7μs	52.4ns	203ns	5.16	0.0666	32.5 KB
#6501	`EncodeLegacyArgs`	net6.0	71.2μs	36ns	139ns	0.0356	0	2.14 KB
#6501	`EncodeLegacyArgs`	netcoreapp3.1	106μs	95.4ns	370ns	0	0	2.14 KB
#6501	`EncodeLegacyArgs`	net472	155μs	87.8ns	328ns	0.31	0	2.15 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunWafRealisticBenchmark`	net6.0	174μs	90.3ns	338ns	0	2.44 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	187μs	225ns	812ns	0	2.39 KB
master	`RunWafRealisticBenchmark`	net472	199μs	119ns	462ns	0.3	2.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	114μs	65.1ns	252ns	0	1.47 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	124μs	279ns	1.08μs	0	1.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	132μs	51.1ns	198ns	0.198	1.49 KB
#6501	`RunWafRealisticBenchmark`	net6.0	176μs	254ns	951ns	0	2.44 KB
#6501	`RunWafRealisticBenchmark`	netcoreapp3.1	188μs	154ns	596ns	0	2.39 KB
#6501	`RunWafRealisticBenchmark`	net472	200μs	257ns	994ns	0.3	2.46 KB
#6501	`RunWafRealisticBenchmarkWithAttack`	net6.0	116μs	77.2ns	299ns	0	1.47 KB
#6501	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	123μs	101ns	390ns	0	1.46 KB
#6501	`RunWafRealisticBenchmarkWithAttack`	net472	131μs	59.3ns	230ns	0.198	1.49 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6501

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472	276.42 KB	278.53 KB	2.11 KB	0.76%

Fewer allocations 🎉 in #6501

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	61.73 KB	60.49 KB	-1.24 KB	-2.01%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	262.87 KB	256.06 KB	-6.81 KB	-2.59%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	60μs	755ns	7.55μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	53.2μs	243ns	1.09μs	42.64 KB
master	`StringConcatBenchmark`	net472	37.5μs	111ns	416ns	61.73 KB
master	`StringConcatAspectBenchmark`	net6.0	321μs	1.8μs	11.7μs	255.18 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	340μs	1.87μs	11.4μs	262.87 KB
master	`StringConcatAspectBenchmark`	net472	269μs	5.16μs	49.2μs	276.42 KB
#6501	`StringConcatBenchmark`	net6.0	59.6μs	850ns	8.46μs	43.44 KB
#6501	`StringConcatBenchmark`	netcoreapp3.1	53.6μs	230ns	920ns	42.64 KB
#6501	`StringConcatBenchmark`	net472	37.2μs	65.6ns	236ns	60.49 KB
#6501	`StringConcatAspectBenchmark`	net6.0	308μs	1.58μs	7.07μs	256.27 KB
#6501	`StringConcatAspectBenchmark`	netcoreapp3.1	358μs	1.97μs	12.8μs	256.06 KB
#6501	`StringConcatAspectBenchmark`	net472	281μs	6.46μs	62.3μs	278.53 KB

andrewlock · 2025-01-06T11:20:51Z

Benchmarks Report for tracer 🐌

Benchmarks for #6501 compared to master:

1 benchmarks are slower, with geometric mean 1.197
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.11μs	45.6ns	329ns	0.0166	0.0083	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	50ns	320ns	0.0157	0.00524	0	5.8 KB
master	`StartStopWithChild`	net472	16.6μs	67.3ns	261ns	1.06	0.331	0.0889	6.21 KB
#6501	`StartStopWithChild`	net6.0	8.05μs	40.1ns	212ns	0.0191	0.00762	0	5.61 KB
#6501	`StartStopWithChild`	netcoreapp3.1	10.3μs	50.7ns	209ns	0.0202	0.0101	0	5.8 KB
#6501	`StartStopWithChild`	net472	16.4μs	31ns	107ns	1.04	0.299	0.0968	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	480μs	324ns	1.26μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	654μs	328ns	1.18μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	855μs	841ns	3.26μs	0.422	3.3 KB
#6501	`WriteAndFlushEnrichedTraces`	net6.0	473μs	185ns	716ns	0	2.7 KB
#6501	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	661μs	452ns	1.69μs	0	2.7 KB
#6501	`WriteAndFlushEnrichedTraces`	net472	859μs	450ns	1.74μs	0.428	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	156μs	957ns	9.52μs	0.152	14.47 KB
master	`SendRequest`	netcoreapp3.1	172μs	1.01μs	9.35μs	0.165	17.27 KB
master	`SendRequest`	net472	0.00178ns	0.000725ns	0.00281ns	0	0 b
#6501	`SendRequest`	net6.0	156μs	906ns	8.3μs	0.14	14.47 KB
#6501	`SendRequest`	netcoreapp3.1	171μs	982ns	7.55μs	0.162	17.27 KB
#6501	`SendRequest`	net472	0ns	0ns	0ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6501

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.6 KB	41.81 KB	210 B	0.50%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	581μs	3.27μs	22μs	0.587	0	0	41.6 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	671μs	3.72μs	22μs	0.322	0	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	867μs	3.9μs	15.1μs	8.19	2.59	0.431	53.33 KB
#6501	`WriteAndFlushEnrichedTraces`	net6.0	597μs	3.38μs	23.2μs	0.59	0	0	41.81 KB
#6501	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	683μs	3.34μs	14.2μs	0.327	0	0	41.88 KB
#6501	`WriteAndFlushEnrichedTraces`	net472	825μs	3.11μs	12.1μs	8.28	2.48	0.414	53.26 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.29μs	1.29ns	4.84ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.8μs	1.33ns	4.97ns	0.0135	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.13μs	2.18ns	8.44ns	0.157	0.00106	987 B
#6501	`ExecuteNonQuery`	net6.0	1.25μs	1.48ns	5.54ns	0.0143	0	1.02 KB
#6501	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	1.71ns	6.63ns	0.014	0	1.02 KB
#6501	`ExecuteNonQuery`	net472	2.11μs	2.62ns	10.2ns	0.156	0.00105	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.16μs	0.665ns	2.49ns	0.0139	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.54μs	1.66ns	6.21ns	0.0132	976 B
master	`CallElasticsearch`	net472	2.53μs	2.45ns	9.16ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.23μs	0.618ns	2.31ns	0.013	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	0.65ns	2.43ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.68μs	1.79ns	6.94ns	0.167	1.05 KB
#6501	`CallElasticsearch`	net6.0	1.21μs	0.424ns	1.59ns	0.0134	976 B
#6501	`CallElasticsearch`	netcoreapp3.1	1.53μs	0.784ns	3.04ns	0.0131	976 B
#6501	`CallElasticsearch`	net472	2.63μs	1.31ns	4.89ns	0.157	995 B
#6501	`CallElasticsearchAsync`	net6.0	1.32μs	0.766ns	2.87ns	0.0132	952 B
#6501	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	1.86ns	7.19ns	0.0135	1.02 KB
#6501	`CallElasticsearchAsync`	net472	2.6μs	1.03ns	3.73ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.28μs	0.864ns	3.23ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	0.581ns	2.18ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.82μs	1.09ns	3.78ns	0.145	915 B
#6501	`ExecuteAsync`	net6.0	1.24μs	2.45ns	9.49ns	0.0136	952 B
#6501	`ExecuteAsync`	netcoreapp3.1	1.6μs	3.43ns	13.3ns	0.0128	952 B
#6501	`ExecuteAsync`	net472	1.91μs	0.569ns	2.13ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.45μs	1.73ns	6.68ns	0.0315	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.29μs	2.82ns	10.9ns	0.0371	2.85 KB
master	`SendAsync`	net472	7.36μs	3.05ns	11ns	0.494	3.12 KB
#6501	`SendAsync`	net6.0	4.47μs	1.73ns	6.46ns	0.0314	2.31 KB
#6501	`SendAsync`	netcoreapp3.1	5.28μs	3.25ns	12.6ns	0.037	2.85 KB
#6501	`SendAsync`	net472	7.37μs	1.17ns	4.22ns	0.493	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.54μs	0.653ns	2.45ns	0.0231	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.09μs	0.997ns	3.73ns	0.0225	1.64 KB
master	`EnrichedLog`	net472	2.69μs	2.29ns	8.57ns	0.249	1.57 KB
#6501	`EnrichedLog`	net6.0	1.51μs	0.845ns	3.16ns	0.0228	1.64 KB
#6501	`EnrichedLog`	netcoreapp3.1	2.22μs	5.82ns	21.8ns	0.022	1.64 KB
#6501	`EnrichedLog`	net472	2.68μs	0.977ns	3.66ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	118μs	195ns	756ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	164ns	635ns	0.0603	0	4.28 KB
master	`EnrichedLog`	net472	151μs	173ns	648ns	0.671	0.224	4.46 KB
#6501	`EnrichedLog`	net6.0	116μs	187ns	726ns	0.0579	0	4.28 KB
#6501	`EnrichedLog`	netcoreapp3.1	122μs	200ns	720ns	0	0	4.28 KB
#6501	`EnrichedLog`	net472	151μs	154ns	596ns	0.679	0.226	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.92μs	1.13ns	4.38ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.2μs	1.84ns	6.88ns	0.0294	2.2 KB
master	`EnrichedLog`	net472	4.88μs	1.69ns	6.54ns	0.319	2.02 KB
#6501	`EnrichedLog`	net6.0	2.84μs	0.896ns	3.35ns	0.0298	2.2 KB
#6501	`EnrichedLog`	netcoreapp3.1	4.3μs	1.57ns	5.88ns	0.0278	2.2 KB
#6501	`EnrichedLog`	net472	4.99μs	0.93ns	3.35ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	0.818ns	3.17ns	0.0162	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.76μs	0.633ns	2.28ns	0.0152	1.14 KB
master	`SendReceive`	net472	2.13μs	1.58ns	5.92ns	0.184	1.16 KB
#6501	`SendReceive`	net6.0	1.49μs	0.706ns	2.73ns	0.0163	1.14 KB
#6501	`SendReceive`	netcoreapp3.1	1.78μs	1.27ns	4.74ns	0.0154	1.14 KB
#6501	`SendReceive`	net472	2.07μs	1.78ns	6.91ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.79μs	0.881ns	3.3ns	0.0223	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.95μs	1.6ns	6.19ns	0.0219	1.65 KB
master	`EnrichedLog`	net472	4.24μs	3.28ns	11.8ns	0.322	2.04 KB
#6501	`EnrichedLog`	net6.0	2.82μs	1.59ns	6.17ns	0.0225	1.6 KB
#6501	`EnrichedLog`	netcoreapp3.1	3.85μs	2.14ns	8.27ns	0.0211	1.65 KB
#6501	`EnrichedLog`	net472	4.34μs	2.1ns	7.86ns	0.324	2.04 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	402ns	0.511ns	1.98ns	0.0081	576 B
master	`StartFinishSpan`	netcoreapp3.1	539ns	0.687ns	2.66ns	0.00777	576 B
master	`StartFinishSpan`	net472	650ns	1.43ns	5.56ns	0.0917	578 B
master	`StartFinishScope`	net6.0	563ns	0.836ns	3.24ns	0.00971	696 B
master	`StartFinishScope`	netcoreapp3.1	713ns	1.11ns	4.31ns	0.00965	696 B
master	`StartFinishScope`	net472	885ns	1.17ns	4.53ns	0.104	658 B
#6501	`StartFinishSpan`	net6.0	401ns	0.515ns	1.99ns	0.00804	576 B
#6501	`StartFinishSpan`	netcoreapp3.1	568ns	1.05ns	4.05ns	0.00786	576 B
#6501	`StartFinishSpan`	net472	661ns	0.852ns	3.3ns	0.0915	578 B
#6501	`StartFinishScope`	net6.0	542ns	1.34ns	5.18ns	0.00961	696 B
#6501	`StartFinishScope`	netcoreapp3.1	700ns	1.21ns	4.69ns	0.0095	696 B
#6501	`StartFinishScope`	net472	850ns	1.48ns	5.75ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6501

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.197	600.43	718.54

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	600ns	1.14ns	4.4ns	0.00973	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	896ns	1.28ns	4.96ns	0.00933	696 B
master	`RunOnMethodBegin`	net472	1.09μs	1.93ns	7.46ns	0.104	658 B
#6501	`RunOnMethodBegin`	net6.0	718ns	0.617ns	2.22ns	0.00971	696 B
#6501	`RunOnMethodBegin`	netcoreapp3.1	881ns	0.957ns	3.58ns	0.0096	696 B
#6501	`RunOnMethodBegin`	net472	1.16μs	1.33ns	5.16ns	0.104	658 B

bouwkast

LGTM, worked for Windows

If you need/want someone to test the Mac I'll have mine setup later in the week hopefully

… target

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 6, 2025

andrewlock requested a review from a team as a code owner January 6, 2025 10:10

andrewlock force-pushed the andrew/ci/combine-integration-test-targets branch from c9f6d1b to 100ea4c Compare January 7, 2025 13:52

bouwkast approved these changes Jan 7, 2025

View reviewed changes

andrewlock force-pushed the andrew/ci/build-samples-separately branch from 14b7e65 to 86ce608 Compare January 7, 2025 14:16

andrewlock requested review from a team as code owners January 7, 2025 14:16

andrewlock force-pushed the andrew/ci/combine-integration-test-targets branch from 100ea4c to 875b4c5 Compare January 7, 2025 14:16

Try to consolidate all the integration test definitions into a single…

d813115

… target

andrewlock force-pushed the andrew/ci/build-samples-separately branch from 86ce608 to 1af72cc Compare January 8, 2025 16:18

andrewlock force-pushed the andrew/ci/combine-integration-test-targets branch from 875b4c5 to d813115 Compare January 8, 2025 16:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consolidate all integration test targets into a single target #6501

Consolidate all integration test targets into a single target #6501

andrewlock commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading

Slower ⚠️ in #6501

Raw results

Raw results

Raw results

More allocations ⚠️ in #6501

Fewer allocations 🎉 in #6501

Raw results

andrewlock commented Jan 6, 2025 •

edited

Loading

Raw results

Raw results

Raw results

More allocations ⚠️ in #6501

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6501

Raw results

bouwkast left a comment

Consolidate all integration test targets into a single target #6501

Are you sure you want to change the base?

Consolidate all integration test targets into a single target #6501

Conversation

andrewlock commented Jan 6, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Jan 6, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Jan 6, 2025 • edited Loading

Datadog Report

❌ Failed Tests (158)

⌛ Performance Regressions vs Default Branch (1)

andrewlock commented Jan 6, 2025 • edited Loading

Benchmarks Report for appsec 🐌

Benchmark details

Slower ⚠️ in #6501

Raw results

Raw results

Raw results

More allocations ⚠️ in #6501

Fewer allocations 🎉 in #6501

Raw results

andrewlock commented Jan 6, 2025 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

More allocations ⚠️ in #6501

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6501

Raw results

bouwkast left a comment

Choose a reason for hiding this comment

andrewlock commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading