DDBClient with Gzip enabled occasionally fails to decompress error responses

[drshriveer]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

We construct a client with EnableAcceptEncodingGzip = true and have occasionally been getting the following error:

operation error DynamoDB: UpdateItem, https response error StatusCode: 400, RequestID: <redacted>, deserialization failed, failed to copy error response body, failed to decompress gzip response, gzip: invalid header

"gzip: invalid header" comes from the gzip library when the headers are invalid/unexpected.
The "failed to decompress gzip response" string comes from this part of the SDK.

This pretty clearly indicates a mismatch between what the server is sending and how the SDK is attempting to parse the value. E.g. perhaps the server is sometimes responding without gzipping the response even though the SDK is always expecting it? maybe particular errors / small errors just don't get zipped by default?

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I expected the SDK to be able to parse the response received from AWS/DDB regardless when accepting Gzip encoded data.

Current Behavior

Sometimes the gzip headers from AWS/DDB are invalid (most likely missing), and the SDK cannot handle the response because it unconditionally expects gzip encoding when EnableAcceptEncodingGzip is set to true.

Reproduction Steps

Unfortunately I do not know how to reliably reproduce this. I'm not sure when AWS/DDB might not be sending valid gzip headers. However, you must set EnableAcceptEncodingGzip to true.

Possible Solution

• Supposing the headers are simply missing because the response is not compressed: In this file, gracefully handle gzip.ErrHeader and just read the bytes directly as non-compressed data.
• Fix this on the DDB/Coral side so the response is always valid when gzip encoding is accepted.

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

    github.com/aws/aws-sdk-go-v2 v1.32.7
    github.com/aws/aws-sdk-go-v2/config v1.28.7
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.38.1
    github.com/aws/aws-sdk-go-v2/service/kms v1.37.8
    github.com/aws/aws-sdk-go-v2/credentials v1.17.48
    github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.22 // indirect
    github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.26 // indirect
    github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.26 // indirect
    github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.1 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.10.7 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.7 // indirect
    github.com/aws/aws-sdk-go-v2/service/sqs v1.37.3
    github.com/aws/aws-sdk-go-v2/service/sso v1.24.8 // indirect
    github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.7 // indirect
    github.com/aws/aws-sdk-go-v2/service/sts v1.33.3 // indirect
    github.com/aws/aws-sdk-go-v2/service/sns v1.33.8 // indirect

Compiler and Version used

go1.23.1 darwin/arm64

Operating System and version

macOS 15.1.1

[Madrigal]

E.g. perhaps the server is sometimes responding without gzipping the response even though the SDK is always expecting it? maybe particular errors / small errors just don't get zipped by default?

This sounds like it's worth exploring. The very best way to catch this would be to print the response when this happens, as described here

[github-actions]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.