SDK seems to create too many HTTP Clients causing EMFILE errors in macOS

[pcolazurdo]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
In MacOs (BigSur 11.5.1 and 11.5.2), the SDK is causing a sample to return many EMFILE (too many open files) errors. I can't trace where this is caused. I've tried changing the HTTP client with the code below, with no much luck.

I've tried the same code, which open many goroutines, in Linux and presented no problem at all. I understand MacOS has very low openfiles limits by default, and I understand this may be the cause, but the problem is so hard to trace that I wonder how many users are having silent issues because of this. The error is detected in other parts of the code because I try to open a few files in rapid succession (less than 50) and sometimes this "file opening" operation produces the error. The SDK itself is not exposing any errors at all.

I would like to know if it is possible to track when a new http client or connection is opened and why. Also, if there are good mechanisms to track/log these metrics.

Also, understanding some safe defaults on different OSes may be really useful.

httpClient, err := NewHTTPClientWithSettings(HTTPClientSettings{
		Connect:          5 * time.Second,
		ExpectContinue:   1 * time.Second,
		IdleConn:         90 * time.Second,
		ConnKeepAlive:    30 * time.Second,
		MaxAllIdleConns:  10, // To avoid issues with EMFILE errors when too many Idle connections are kept in MacOS
		MaxHostIdleConns: 2,
		ResponseHeader:   5 * time.Second,
		TLSHandshake:     5 * time.Second,
	}, obs)
	if err != nil {
		fmt.Println("Got an error creating custom HTTP client:")
		fmt.Println(err)
		panic("Got an error creating custom HTTP client:")
	}

	sess := session.Must(session.NewSessionWithOptions(session.Options{
		Config: aws.Config{
			HTTPClient: httpClient,
		},
		SharedConfigState: session.SharedConfigEnable,
	}))

Version of AWS SDK for Go?
AWS SDK Version: 1.40.16

Version of Go (go version)?
go version go1.16.6 darwin/amd64

To Reproduce (observed behavior)
Relatively large code sample here.

Expected behavior
First, safe defaults in different OS would be expected. Second, a mechanism to know and control how many http clients (I'm assuming this is the problem but I don't have a mechanism to confirm) exist at any given time. Also, more eventing/logging (probably it exists but I don't know how to activate) about when new HTTP clients are created.

Hope this helps,

[rittneje]

@pcolazurdo It sounds like your issue is with the Go standard library's http.Client. You can use the httptrace package to hook into the lifecycle of the client pool's connections. https://pkg.go.dev/net/http/httptrace

[pcolazurdo]

I've tried running the same code on Linux and lower the number of open files (to 256) to the same default than in Mac and the problem doesn't seem to appear. I did add some tracing but can't see anything obvious - the percentage of reused connections among both platforms seems to be the same.

[vudh1]

Hi, can you confirm if this is still persisting with the latest version of SDK?

[github-actions]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[pcolazurdo]

I recently did upgrade to SDK v2 and did some other tests and I'm facing additional problems that seem to be related with this but can't be sure yet. I'm trying to create a clean reproduction but I can't have a solid repro example. My understanding at the moment is that somehow goroutines/CGO threads start piling up at some point and this create a cascade effect that includes Panic / running out of file handles.

[RanVaknin]

Hi @pcolazurdo ,

Its been a while since we heard from you. We didnt get the reproduction steps from you. I have seen this error manifest in other SDKs before. This is likely due to you creating multiple copies of the SDK client, instead of using one client per region, per service.

If this is still an issue with v2, please create a separate issue on the v2 repo so we may better assist you,
Ran~

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.