Problem with DateTimeOffset scrubber when running multiple tests

[mci-kmd]

I have a problem with the DateTimeOffset scrubber. There are cases where it produces one set of ids for the scrubbed values when running a test in isolation and another set when running all my tests. It seems to indicate that the scrubbed ids are globally grouped since running all the tests will cause the failing tests to fx. change an id from DateTimeOffset_1 to DateTimeOffset_2.

Does enyone know if there is a way to configure Verify to avoid this, fx. by changing the scrubber to group only within a single result (which is what I would have expected)? If nothing else works, I would prefer the numeric part of the datetimeoffset id to be omitted, since I generally don't really care if they represent the same time or not.

[SimonCropp]

can you share an example of how your tests look?

[mci-kmd]

I think I've run into something very strange here. I've been trying to isolate the behavior to a minimal repro but it has gotten quite weird.

I have a test class and when I run both tests then the problem occurs and if I run the problematic test alone then it works fine. Lets call these methods Test1 and Test2 (it is Test2 which has the problem). In my quest to isolate the issue I have ended up with 2 IDENTICAL tests which do not exhibit the same behavior, lets call them TestA and TestB. I tried looking at the "verified" files of each pair of tests and they are identical as well.

Note that Test1/TestA does not even contain any DateTimeOffset values, which makes this even stranger.

The following code will make Test2 fail when I run both tests. If I rename the tests to TestA and TestB then the test will succeed even when run together. I used a diff tool to verify that the "verified" files for the corresponding tests is identical. The use of the "ScrubLinesReplaceWithRegex" method does not have any impact on the test but is kept for completeness.

So I'm quite baffled because I have no idea what the difference is between the two pairs of test (except their names).

[UsesVerify]
public class LoginInteractive_Post_PostUrls : SamlControllerTestBase
{
    private readonly HttpClient _client;
    private readonly Task<HttpResponseMessage> _request;

    public LoginInteractive_Post_PostUrls(WebApplicationFactory<Program> factory) : base(factory)
    {
        // Arrange
        _client = CreateClient();

        // Act
        _request = _client.PostAsync(
            "/saml/logininteractive",
            PrepareValidSamlLoginInteractiveRequest(urlPrefix: "POST|")
        );
    }

    [Fact]
    public async void Test1()
    {
        var response = await _request;

        var html = await response.Content.ReadAsStringAsync();
        Assert.NotNull(html);
        await Verify(html).ScrubLinesReplaceWithRegex(SamlResponseRegex, "$SAML_RESPONSE$");
    }

    [Fact]
    public async void Test2()
    {
        var response = await _request;

        var html = await response.Content.ReadAsStringAsync();
        Assert.NotNull(html);
        var samlResponse = ExtractMatch(SamlResponseRegex, html);
        Assert.NotNull(samlResponse);
        await
            VerifyXml(Base64Decode(samlResponse))
            .ScrubLinesReplaceWithRegex(DigestValueRegex, "$DIGEST_VALUE$")
            .ScrubLinesReplaceWithRegex(SignatureValueRegex, "$SIGNATURE_VALUE$")
            .ScrubInlineGuids();
    }


    public static SettingsTask ScrubLinesReplaceWithRegex(this SettingsTask task, Regex regex, string placeholder)
    {
        return task.ScrubLinesWithReplace(s =>
        {
            var match = regex.Match(s);

            if (!match.Success)
                return s;

            return s.Replace(match.Groups[1].Value, placeholder);
        });
    }

}

[SimonCropp]

without a repro this will be pretty difficult to look into

[mci-kmd]

Yeah, I get that :) I don't know if VS might cache some stuff related to tests which causes this and makes it impossible to solve only by looking at the code. I think I will just have to solve the cases I see (2 for now) by renaming the tests.

[mci-kmd]

If anyone else runs into this, then you can normalize the generated placeholders with this code:

public static class Extensions
{
    private static readonly Regex DateTimeOffsetPlaceholderRegex = new Regex("DateTimeOffset_(\\d*)");

    /// <summary>
    /// For lines matched by the regex, the matched element is replaced with the placeholder value.
    /// </summary>
    /// <param name="regex">A regex matching an element to scrub</param>
    /// <param name="placeholder">The placeholder value to replace the matched element with in the output</param>
    public static SettingsTask ScrubLinesReplaceWithRegex(this SettingsTask task, Regex regex, string placeholder)
    {
        return task.ScrubLinesWithReplace(s =>
        {
            var matches = regex.Matches(s);

            if (!matches.Any(m => m.Success))
                return s;

            foreach (Match match in matches)
            {
                var updatedValue = match.Groups[0].Value.Replace(match.Groups[1].Value, placeholder);
                s = s.Replace(match.Groups[0].Value, updatedValue);
            }

            return s;
        });
    }

    /// <summary>
    /// This fixes a bug which sometimes causes the DateTimeOffset scrubbing to change the id number
    /// depending on whether a test is run in isolation or not. This method normalizes all instances
    /// to use the placeholder 'DateTimeOffset_X'.
    /// </summary>
    /// <param name="task"></param>
    /// <returns></returns>
    public static SettingsTask NormalizeDateTimeOffsetPlacerholders(this SettingsTask task)
    {
        return task.ScrubLinesReplaceWithRegex(DateTimeOffsetPlaceholderRegex, "X");
    }
}