http_GET vs http_client.get

[Lucas-C]

Why are there two methods to perform HTTP calls?

• http_client.get() (defined in src/http_client.cc)
• http_GET() (defined in src/http_req.cc)

I wrote some code to test both: https://github.com/sncf-connect-tech/motis/blob/test-http/exe/main.cc#L143

The problem with http_GET() is that it does not support proxies.
It's used for example in https://github.com/motis-project/motis/blob/master/src/rt_update.cc#L76.
We would like to use a proxy in that case, so using http_client would be handier.

http_GET does not seem used anywhere else.

Would you be OK to replace http_GET by http_client.get() in src/rt_update.cc,
and use the proxy in the config file if it's set?

---

Side notes:

Once compiled, the modified motis command in https://github.com/sncf-connect-tech/motis/tree/test-http can be used like this:

build/motis test-http_client
build/motis test-http_GET

The first command seems to "hang" in my environment.

Also, when launching a local Python HTTP server with python -m http.server (tested with Python 3.10 & 3.12),
if I target http://localhost:8000 in test-http_GET instead of https://httpbin.org/get, I get this error:

Starting coroutine...
Starting HTTP GET request...
Coroutine failure: bad version [beast.http:14 at /opt/motis/deps/boost/boost/beast/http/detail/basic_parser.ipp:357:9 in function 'static void boost::beast::http::detail::basic_parser_base::parse_version(const char*&, const char*, int&, boost::beast::error_code&)']

[Lucas-C]

Bonus: maybe http_client should be added to https://github.com/motis-project/utl?

[felixguendling]

Would you be OK to replace http_GET by http_client.get() in src/rt_update.cc,
and use the proxy in the config file if it's set?

I think this makes sense.
Any objections @pablohoch?

Bonus: maybe http_client should be added to https://github.com/motis-project/utl?

It could replace our own HTTP client implementation in motis-project/net.

[felixguendling]

The first command seems to "hang" in my environment.

The http_client is supposed to keep the TCP connection open.
Maybe the channels keep the I/O service running.

---

But: I experienced problems in the past with the number of open connections exceeding 1024. I was not able to find the cause. So for now I just used this SystemD config:

[Service]
LimitNOFILE=65536
...

(you can achieve the same with ulimit)

[pablohoch]

Would you be OK to replace http_GET by http_client.get() in src/rt_update.cc,
and use the proxy in the config file if it's set?

I think this makes sense. Any objections @pablohoch?

Agreed.

Also, when launching a local Python HTTP server with python -m http.server (tested with Python 3.10 & 3.12), if I target http://localhost:8000 in test-http_GET instead of https://httpbin.org/get, I get this error:

Starting coroutine...
Starting HTTP GET request...
Coroutine failure: bad version [beast.http:14 at /opt/motis/deps/boost/boost/beast/http/detail/basic_parser.ipp:357:9 in function 'static void boost::beast::http::detail::basic_parser_base::parse_version(const char*&, const char*, int&, boost::beast::error_code&)']

I've encountered this error after sending multiple requests using HTTP pipelining (with http_client) to some hosts, which led me to believe that it could be a server issue (was using SSL for that, so not that easy to debug). If the error can be reproduced using a single HTTP request, it's probably a client bug.

[Lucas-C]

It could replace our own HTTP client implementation in motis-project/net.

Alright.
This HTTP client in motis-project/net is currently unused?

[felixguendling]

MOTIS currently only uses web_server and query_router from the net library.