Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove internal AWS SDK implementation #1030

Merged
merged 2 commits into from
Dec 16, 2024

Conversation

smallhive
Copy link
Contributor

Closes #1029

@evgeniiz321
Copy link

evgeniiz321 commented Nov 29, 2024

@smallhive @roman-khimov if try to add s3 gw from this PR to nexus, it fails like this:

error	api/user_auth.go:39	failed to pass authentication	{"error": "SignatureDoesNotMatch: 403 => The request signature we calculated does not match the signature you provided. Check your key and signing method."}
info	api/router.go:177	call method	{"host": "nexus-bucket-v3.localhost:33057", "request_id": "30deb4e3-35a0-4ebd-8ccf-7b36c0124766", "method": "GetBucketACL", "bucket": "nexus-bucket-v3", "object": ""}
error	api/user_auth.go:39	failed to pass authentication	{"error": "SignatureDoesNotMatch: 403 => The request signature we calculated does not match the signature you provided. Check your key and signing method."}
info	api/router.go:177	call method	{"host": "nexus-bucket-v3.localhost:33057", "request_id": "9b1e449f-cad4-4b69-a68a-33ce814471db", "method": "GetBucketACL", "bucket": "nexus-bucket-v3", "object": ""}
error	api/user_auth.go:39	failed to pass authentication	{"error": "SignatureDoesNotMatch: 403 => The request signature we calculated does not match the signature you provided. Check your key and signing method."}
info	api/router.go:177	call method	{"host": "nexus-bucket-v3.localhost:33057", "request_id": "0ce0f1ae-eaaf-4a3f-bad9-8ff2d042d9c0", "method": "GetBucketACL", "bucket": "nexus-bucket-v3", "object": ""}
error	api/user_auth.go:39	failed to pass authentication	{"error": "SignatureDoesNotMatch: 403 => The request signature we calculated does not match the signature you provided. Check your key and signing method."}
info	api/router.go:177	call method	{"host": "nexus-bucket-v3.localhost:33057", "request_id": "d5a395cf-4a3d-4eab-b34f-b0d579e08b9e", "method": "GetBucketACL", "bucket": "nexus-bucket-v3", "object": ""}
error	api/user_auth.go:39	failed to pass authentication	{"error": "SignatureDoesNotMatch: 403 => The request signature we calculated does not match the signature you provided. Check your key and signing method."}

if use s3 gw from master (same parameters in nexus), it fails like this:

info    api/router.go:182   call method {"status": 200, "host": "nexus-bucket-v2.localhost:53689", "request_id": "a9d203e5-c3bc-4d10-b422-cb0d7843c7c9", "method": "CreateBucket", "bucket": "nexus-bucket-v2", "object": "", "description": "OK"}
error   handler/util.go:34  call method {"status": 501, "request_id": "d45cd4a9-b989-4f33-b9c8-b5c86547d870", "method": "PutBucketLifecycle", "bucket": "nexus-bucket-v2", "object": "", "description": "not implemented", "error": "NotImplemented: 501 => A header you provided implies functionality that is not implemented"}

So, it doesn't work in both cases, but errors are different.

To reproduce:

  1. Run nexus (by default it runs on 8081)
docker run -d --network host --name nexus -v nexus-data:/nexus-data -e INSTALL4J_ADD_VM_PARAMS="-Dcom.amazonaws.sdk.disableCertChecking=true" sonatype/nexus3
  1. Login to UI as admin (password will be here docker exec -it nexus cat /nexus-data/admin.password )
  2. Go to Administration -> Repository -> Blob Stores -> Create Blob Store
  3. Specify all required s3 params and click Save.

@smallhive
Copy link
Contributor Author

This problem with the signature

error	api/user_auth.go:39	failed to pass authentication	{"error": "SignatureDoesNotMatch: 403 => The request signature we calculated does not match the signature you provided. Check your key and signing method."}

is fixed with proper configuration

Administration -> Repository -> Blob Stores -> Create Blob Store -> Advanced Connection Settings (Optional) -> Signature Version -> AWSS3V4SignerType

Then, the problem with PutBucketLifecycle:

error   handler/util.go:34  call method {"status": 501, "request_id": "d45cd4a9-b989-4f33-b9c8-b5c86547d870", "method": "PutBucketLifecycle", "bucket": "nexus-bucket-v2", "object": "", "description": "not implemented", "error": "NotImplemented: 501 => A header you provided implies functionality that is not implemented"}

Is fixed with

Administration -> Repository -> Blob Stores -> Create Blob Store -> Expiration Days -> -1

Then, the problem with GetBucketLifecycle:

Here I hacked the response from not implemented to NoSuchLifecycleConfiguration. It is enough to go through this step.

And here is the last stop sign: Etag. The Nexus uses MD5; we use sha256.

An error occurred saving data. ValidationErrorXO{id='*', message='Unable to verify integrity of data upload. Client calculated content hash (contentMD5: bOJCN+GHNw4hyUURykPbWA== in base 64) didn't match hash (etag: 564f5f204e99644ee6f532accd524086c02dcf9346a2470eb0bd177fbcd25d19 in hex) calculated by Amazon S3. You may need to delete the data stored in Amazon S3. (metadata.contentMD5: null, md5DigestStream: com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream@7fdde751, bucketName: bb2, key: metadata.properties)'}

I failed to find how to configure this moment in Nexus. According to the AWS docs, md5 is a valid behaviour. In the S3 gate, we can't change sha256 to md5 as well

Closes #1029.

Signed-off-by: Evgenii Baidakov <[email protected]>
Having our own implementation for AWS SDK lib looks like overkill.

Signed-off-by: Evgenii Baidakov <[email protected]>
@smallhive smallhive force-pushed the 1029-remove-internal-aws-sdk-implementation branch from 849a272 to 54ff7e6 Compare December 16, 2024 12:26
Copy link
Member

@roman-khimov roman-khimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AWSS3V4SignerType is the key to me here. If it works, we're fine with this change. Other issues are... other issues.

@roman-khimov roman-khimov merged commit b0ac82c into master Dec 16, 2024
18 checks passed
@roman-khimov roman-khimov deleted the 1029-remove-internal-aws-sdk-implementation branch December 16, 2024 15:47
@evgeniiz321 evgeniiz321 mentioned this pull request Dec 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove internal AWS SDK implementation
3 participants