This is not a complete solution, the JSON exception is simply thrown later:

The above is in generated cdp code (\cdp\v131\Fetch\FetchAdapter.cs in the above screenshot).

Related relevant code:

@nvborisenko You mentioned trying out JavaScriptEncoder.UnsafeRelaxedJsonEscaping. I cannot edit the generated code directly but I added a spoof here, and it did not fix the exception

I modified generator to use JavaScriptEncoder.UnsafeRelaxedJsonEscaping, it didn't help. The worst scenario is that it works with Newtonsoft.Json (Selenium v4.23?).

I do not understand how Newtonsoft accepts this nonsense input. It’s not real JSON, just some raw binary data.

Maybe we should intersept the post data and sanitize it ourselves?

One more observation: in json response I see binary data as is, probably not even encoded! This is why VS Code shows json improperly. CDP specification doesn't say it should be encoded (base64). I think this is why they deprecated it.

PS: But why Newtonsoft.Json could parse it successfully? I think it is issue in this library. I didn't check it yet, but I guess:

1 using old selenium version (v4.23?) when we upload binary file
2 and we download it manually back
3 then the content of "original" and "downloaded" files is different

I tried switching from JsonNode to JsonElement just to see what would happen.

Good news, no exceptions on ToString() like the logging has, but the de-serialization still fails

Maybe we should parse and encode the postData ourselves?

I am stupid for this activity. It is easier to:

• really Newtonsoft.Json parses it correctly?

I narrowed down the issue. It looks like this character U+DB40 appears here for some reason. It is the first part of a UTF-16 surrogate pair, but the second half is missing.

I opened the raw Firefox Installer.exe file and found this segment here:

Note how in this case, it is just a regular I.

What is happening here? The CDP is giving us back broken JSON, this may be a bug on their end.

I will investigate whether it is possible to work around this somehow.

I found a workaround, with a custom converter.

We can add to the postData property:

As well as to PostDataEntry.Bytes:

The following converter:

internal sealed class InvalidUtf16Converter : JsonConverter<string>
{
    public override bool HandleNull => true;

    public override string? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        try
        {
            return reader.GetString();
        }
        catch (InvalidOperationException)
        {
            var bytes = reader.ValueSpan;
            var sb = new StringBuilder(bytes.Length);
            foreach (var b in bytes)
                sb.Append(Convert.ToChar(b));
            return sb.ToString();
        }
    }

    public override void Write(Utf8JsonWriter writer, string value, JsonSerializerOptions options) =>
        writer.WriteStringValue(value);
}

Since we have a good idea of which models may have arbitrary data,

However... it only works with JsonElement.Deserialize<T>. When using a JsonNode.Deserialize<T>, the exception happens way before even the converter is reached. Even JsonNode.ToString() throws an exception!

For this reason, we may need to change various JsonNodes to JsonElements, such as DevToolsEventReceivedEventArgs.EventData (that property is used everywhere in the cdp-generated code, but luckily the line e.EventData.Deserialize(eventData.EventArgsType) works exactly the same (it just hits a different overload). Also in DevToolsCommandData.Result

I am not familiar enough with the cdp code generation process to make generator changes. Is there any docs for this?

Most likely you are interested in https://github.com/SeleniumHQ/selenium/tree/trunk/third_party/dotnet/devtools/src/generator/Templates

There is no docs, generally saying build process invokes generation of CDP classes based on *.cdl (I don't remember exactly, it is a definition of contract) files applying *.hbs templates (https://handlebarsjs.com/).

Let us improve this comment:

• don't mention Newtonsoft
• don't use "sometimes" word - we always should know what is happening

Removing mention to Newtonsoft.

I would also like to know why this is happening! I don't know what is causing this.

Since this solution is so targeted, do you think we should try to target Network.Request.postData specifically here?

No, we cannot manage the source for generator, it is provided by Google.
In this comment we should leave all knowledge we know about why this "strange" workaround lives.

I don't think so, it will add complexity and will lead to even more "strange workaround". As soon as Google will remove deprecated property, we will "easily" remove this workaround.

But from other hand why not to be specific? If we will generate attribute conditionally, then will it be friendly with upcoming AOT?

No, adding a complexity to stuff which will be deprecated is not good idea.

• Chromium removes "bad property" (postData) - we "remove" workaround
• CDP is supposed to be replaced by BiDi

Speaking of removing complexity, wouldn't it be cool to convert the handlebars generator into a modern incremental source generator? That sounds like a fun little project.

comment added