std.http: more proxy support, buffer writes, tls toggle

[truemedian]

std.http.Client Changes

Proxies are now separated, so HTTP and HTTPS requests can be diverted thru different proxies. Connection pool now accepts a Allocator instead of a *Client for allocations. Now passes around a *Connection instead of a *ConnectionPool.Node.

Writes are now buffered, and reads now use iovecs to reduce syscalls.

Adds client.loadDefaultProxies to loads proxy information from common
environment variables (http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY,
all_proxy, ALL_PROXY).

• no_proxy and NO_PROXY are currently unsupported.

Other Associated Fixes

std.Uri has expanded formatting capabilities so to enable more control over what appears in the output.

std.crypto.tls.Client.readv was evidently not tested, it was missing a parameter.

std_options Changes

Removed http_connection_pool_size, Added http_disable_tls

• http_connection_pool_size: replaced with client.connection_pool.resize(client.allocator, size)
• http_disable_tls: When true, all HTTPS requests will error with error.TlsInitializationFailed, and std.http will make no references to std.crypto.tls (which may reduce the size of the overall binary, and potentially reduce compile times; unless referenced elsewhere).

When compiling without TLS: test/standalone/http.zig compiles 10 seconds faster, and is half the size (aprox. -4MB in ReleaseSafe).

Documentation

Adds documentation to public functions that was missing documentation. Adds information about when order-specific functions need to be called.

Revisited old documentation and pulled it in line with what each function is actually doing now.

[binarycraft007]

hi @truemedian, thanks for working on http proxy, does this support socks5? I didn't see any socks5 protocol handling. related: #15048.

[truemedian]

does this support socks5?

No SOCKS support yet, just HTTP proxying (when you send a fully qualified uri to a proxy server and it makes a request for you) and HTTP tunneling (when you make a CONNECT request to a proxy to open a tunnel to the target).

[andrewrk]

How's it going? You looking for a merge yet?

[truemedian]

I think so. I don't have anything else important to fix or add in this batch.

[andrewrk]

Nice work! I only have superficial changes requested, but one of them regards the public API, so I do think it should be changed before merging. After this however I think it is ready to land 👍

[andrewrk]

I forgot to mention, I think this PR solves #17422.

[truemedian]

This at least mostly solves #17422 for std.http.Client, but I don't want to close that yet because std.http.Server likely still suffers this problem.

[andrewrk]

Looks ready to land to me, I think it only needs a zig fmt in order to pass the CI.

[ncnnnnn]

I am a newcomer to Zig and encountered a network link error when trying to compile zsl using Zig

Under Chinese WIN10, loadDefaultProxies cannot use system settings correctly

ZIG: 0.12.0-dev.1270+6c9d34bce
win: WIN10 21H2

The test code is as follows:

const std = @import("std");
const os = std.os;
const parseInt = std.fmt.parseInt;
const mem = std.mem;
const http = std.http;
const Server = http.Server;
const Client = http.Client;

const max_header_size = 8192;

var gpa_server = std.heap.GeneralPurposeAllocator(.{ .stack_trace_frames = 12 }){};
var gpa_client = std.heap.GeneralPurposeAllocator(.{ .stack_trace_frames = 12 }){};

const salloc = gpa_server.allocator();
const calloc = gpa_client.allocator();


test "parse integers" { 
	const key_w = comptime std.unicode.utf8ToUtf16LeStringLiteral("http_proxy");
	const key_slice = mem.sliceTo(key_w, 0);

	 const ptr = os.windows.peb().ProcessParameters.Environment;
	     var i: usize = 0;
    while (ptr[i] != 0) {
        const key_start = i;

        // There are some special environment variables that start with =,
        // so we need a special case to not treat = as a key/value separator
        // if it's the first character.
        // https://devblogs.microsoft.com/oldnewthing/20100506-00/?p=14133
        if (ptr[key_start] == '=') i += 1;

        while (ptr[i] != 0 and ptr[i] != '=') : (i += 1) {}
        const this_key = ptr[key_start..i];

        if (ptr[i] == '=') i += 1;

        const value_start = i;
        while (ptr[i] != 0) : (i += 1) {}
        const this_value = ptr[value_start..i :0];

       if (os.windows.eqlIgnoreCaseWTF16(key_slice, this_key)) {
	   std.debug.print(
            "{u} has a field called {u} \n",
            .{
                this_key,this_value,
            },
			);
           // return ;
        }
		
        i += 1; // skip over null byte
    }
	
	    var client = Client{ .allocator = calloc };
    errdefer client.deinit();
    // defer client.deinit(); handled below

    try client.loadDefaultProxies();
	std.debug.print(
            "https_proxy:: {any}  \n http_proxy:: {any} \n",
            .{
                client.https_proxy,client.http_proxy,
            },
			);
	 var h = http.Headers{ .allocator = calloc };
        defer h.deinit();
		 const uri = try std.Uri.parse("http://baidu.com");
	        var req = try client.open(.GET,uri , h, .{});
        defer req.deinit();

        try req.send(.{});
        try req.wait();	
		const body = try req.reader().readAllAlloc(calloc, 8192);
        defer calloc.free(body);
		 std.debug.print(
            "{s} has a field called {u} \n",
            .{
                body,body,
            },
			);
}

No proxy can be used to access http/https under cmd:

 { H, T, T, P, _, P, R, O, X, Y } has a field called { 1, ,,, }
https_proxy:: null
 http_proxy:: null

powershell Ability to use HTTP_ PROXY access to http, cannot use https_ Proxy access to https:

 { H, T, T, P, S, _, P, R, O, X, Y } has a field called { h, t, t, p, :, /, /, 1, ,,,}
https_proxy:: null
 http_proxy:: http.Client.Proxy{

I hope to receive your help and support thanks.

[truemedian]

#17732 should fix both of these.