-
Describe the bugwhen attempting to Put a lifecycle configuration onto a bucket, it fails with:
The input looks as follows:
Expected BehaviorExpect it to write the lifecycle configuration with the instantiated input values Current Behaviorfailure with:
Reproduction StepsUse this input:
Possible SolutionNo response Additional Information/ContextNo response AWS Go SDK V2 Module Versions Used"github.com/aws/aws-sdk-go-v2/config" Compiler and Version usedgo version go1.18.3 darwin/arm64 Operating System and versionMac OSX 12.4 |
Beta Was this translation helpful? Give feedback.
Replies: 14 comments 2 replies
-
Could you follow the instructions in the Developer Guide to enable client logging with the |
Beta Was this translation helpful? Give feedback.
-
Here’s the serialized XML on the request body:
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><AbortIncompleteMultipartUpload><DaysAfterInitiation>1</DaysAfterInitiation></AbortIncompleteMultipartUpload><Expiration><ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker></Expiration><ID>cleanup853211</ID><Status>Enabled</Status></Rule></LifecycleConfiguration>
… On Jun 8, 2022, at 12:44 PM, Sean McGrail ***@***.***> wrote:
Could you follow the instructions in the Developer Guide <https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/logging/> to enable client logging with the aws.LogRequestWithBody option enabled? That would help us understand the serialized XML that is being sent to the service for your particular request. Thanks!
—
Reply to this email directly, view it on GitHub <#1722 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB2J6HWUPSKYUINIBMK2VTVODEWNANCNFSM5X7GOUEQ>.
You are receiving this because you authored the thread.
|
Beta Was this translation helpful? Give feedback.
-
Thank you for the logging information @rabarar , I was also able to reproduce this on my end as well in both the V2 and V1 Go SDKs, and the AWS CLI. Will follow-up this investigation with the service team. |
Beta Was this translation helpful? Give feedback.
-
I've root caused this down to the service expecting the filter API member to be serialized to the request regardless if sent or not. If you provide an empty filter at the CLI it will work correctly, same with the V1 SDK (provide a pointer to a structure). Works<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>1</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
<Filter />
<ID>abortMulti</ID>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration> Doesn't Work<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>1</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
<ID>abortMulti</ID>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration> Unfortunately the Go V2 SDK uses a union type, so you can't quite set this in exactly the same way. |
Beta Was this translation helpful? Give feedback.
-
as an aside, attempting to reference the Filter type LifecycleRuleAndOperator in types fails to resolve and is inconsistent with the go-docs.
… On Jun 9, 2022, at 12:52 PM, Sean McGrail ***@***.***> wrote:
I've root caused this down to the service expecting the filter API member to be serialized to the request regardless if sent or not.
Works
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>1</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
<Filter />
<ID>abortMulti</ID>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>
Doesn't Work
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>1</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
<ID>abortMulti</ID>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>
—
Reply to this email directly, view it on GitHub <#1722 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB2J6EXR3D2CN2TO6HCKZLVOIOOPANCNFSM5X7GOUEQ>.
You are receiving this because you were mentioned.
|
Beta Was this translation helpful? Give feedback.
-
This code snippet does appear to work, though the serialization is different. It would provide a temporary work around at least: _, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3.PutBucketLifecycleConfigurationInput{
Bucket: aws.String("mcgrails-test-data"),
LifecycleConfiguration: &types.BucketLifecycleConfiguration{
Rules: []types.LifecycleRule{
{
Status: types.ExpirationStatusEnabled,
AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
DaysAfterInitiation: 1,
},
Expiration: &types.LifecycleExpiration{
ExpiredObjectDeleteMarker: true,
},
Filter: &types.LifecycleRuleFilterMemberPrefix{Value: ""},
ID: aws.String("abortMulti"),
},
},
},
}) This serializes as: <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>1</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
<Filter>
<Prefix></Prefix>
</Filter>
<ID>abortMulti</ID>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration> |
Beta Was this translation helpful? Give feedback.
-
This seems to be broken in the S3 Control model as well, for different, currently unknown, reasons... client := s3control.NewFromConfig(cfg)
_, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3control.PutBucketLifecycleConfigurationInput{
AccountId: aws.String("accountId"),
Bucket: aws.String("mcgrails-test-data"),
LifecycleConfiguration: &types.LifecycleConfiguration{
Rules: []types.LifecycleRule{
{
Status: types.ExpirationStatusEnabled,
Filter: &types.LifecycleRuleFilter{},
AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
DaysAfterInitiation: 1,
},
Expiration: &types.LifecycleExpiration{
ExpiredObjectDeleteMarker: true,
},
ID: aws.String("abortMulti"),
},
},
},
}) Returns
Seeing the same behavior with the Go V1 SDK and CLI. |
Beta Was this translation helpful? Give feedback.
-
Is there a timeline for a fix? Feel free to send me a t-shirt for finding this one ;)
… On Jun 9, 2022, at 4:01 PM, Sean McGrail ***@***.***> wrote:
This seems to be broken in the S3 Control model as well, for different, currently unknown, reasons...
client := s3control.New(sess)
_, err := client.PutBucketLifecycleConfiguration(&s3control.PutBucketLifecycleConfigurationInput{
AccountId: aws.String("<account>"),
Bucket: aws.String("mcgrails-test-data"),
LifecycleConfiguration: &s3control.LifecycleConfiguration{
Rules: []*s3control.LifecycleRule{
{
AbortIncompleteMultipartUpload: &s3control.AbortIncompleteMultipartUpload{
DaysAfterInitiation: aws.Int64(1),
},
Expiration: &s3control.LifecycleExpiration{
ExpiredObjectDeleteMarker: aws.Bool(true),
},
Filter: &s3control.LifecycleRuleFilter{},
ID: aws.String("abortMulti"),
Status: aws.String("Enabled"),
},
},
},
})
Returns
<?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse>
<Error>
<Code>InvalidURI</Code>
<Message>Couldn't parse the specified URI.</Message>
<URI>bucket/mcgrails-test-data/lifecycleconfiguration</URI>
</Error>
<RequestId>Z2AGJDNK1QRRVC4V</RequestId>
<HostId>3pXY5FhtMiP+Bq0w6wEU786mNkWl4ROwekWZ9JOusTZ5UlKkZZictU0+FkpbZlVwY2oa43KmzdQ=</HostId>
</ErrorResponse>
Seeing the same behavior with the Go V1 SDK and CLI.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.
|
Beta Was this translation helpful? Give feedback.
-
Definitely works but has zero relationship to the docs … not sure where types.LifecycleRuleFilterMemberPrefix comes from …
… On Jun 9, 2022, at 2:00 PM, Sean McGrail ***@***.***> wrote:
This code snippet does appear to work, though the serialization is different. It would provide a temporary work around at least:
_, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3.PutBucketLifecycleConfigurationInput{
Bucket: aws.String("mcgrails-test-data"),
LifecycleConfiguration: &types.BucketLifecycleConfiguration{
Rules: []types.LifecycleRule{
{
Status: types.ExpirationStatusEnabled,
AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
DaysAfterInitiation: 1,
},
Expiration: &types.LifecycleExpiration{
ExpiredObjectDeleteMarker: true,
},
Filter: &types.LifecycleRuleFilterMemberPrefix{Value: ""},
ID: aws.String("abortMulti"),
},
},
},
})
This serializes as:
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>1</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
<Filter>
<Prefix></Prefix>
</Filter>
<ID>abortMulti</ID>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>
—
Reply to this email directly, view it on GitHub <#1722 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB2J6D2DV5GSC5ZFGIBCCTVOIWLRANCNFSM5X7GOUEQ>.
You are receiving this because you were mentioned.
|
Beta Was this translation helpful? Give feedback.
-
Hi @rabarar , You have to specify the source:
I think this isn't a bug, just an API design choice from s3. |
Beta Was this translation helpful? Give feedback.
-
Hi -The API doesn’t afford access to the xml - that is internal to the go api and therefore is a bug imho.Best,RobOn Nov 30, 2022, at 6:03 PM, Ran Vaknin ***@***.***> wrote:
Hi @rabarar ,
You have to specify the <Filter> tag unless you have a <Prefix> tag in your xml.
source:
Filter
The Filter is used to identify objects that a Lifecycle Rule applies to. A Filter must have exactly one of Prefix, Tag, or And specified. Filter is required if the LifecycleRule does not contain a Prefix element.
Type: LifecycleRuleFilter data type
I think this isn't a bug, just an API design choice from s3.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
@rabarar , After taking a deeper look into this here is what I have found. The SDK serializes Golang strcuts into XML. When you send a request without that field in the request you're essentially leaving it out of the XML. This is all handled on the service-side, and is documented. To show that its not a GO SDK bug I've tested it on JS SDK V3 with the following code: ❌ Incorrect:import { S3Client, PutBucketLifecycleConfigurationCommand } from '@aws-sdk/client-s3';
const client = new S3Client({ region: 'us-east-1' });
const command = new PutBucketLifecycleConfigurationCommand({
Bucket: 'mybucket123',
LifecycleConfiguration: {
Rules: [
{
Status: 'Enabled',
ID: 'abortMulti',
Expiration: {ExpiredObjectDeleteMarker: true},
AbortIncompleteMultipartUpload: {DaysAfterInitiation: 1},
// <- missing Filter
}
]
}
});
try {
const data = await client.send(command);
console.log(data)
} catch (error) {
console.log(error);
} ❌ Incorrect:const command = new PutBucketLifecycleConfigurationCommand({
Bucket: 'mybucket123',
LifecycleConfiguration: {
Rules: [
{
Status: 'Enabled',
ID: 'abortMulti',
Expiration: {ExpiredObjectDeleteMarker: true},
AbortIncompleteMultipartUpload: {DaysAfterInitiation: 1},
Filter: {} // <- change is here
}
]
}
}); ✅ Correctconst command = new PutBucketLifecycleConfigurationCommand({
Bucket: 'mybucket123',
LifecycleConfiguration: {
Rules: [
{
Status: 'Enabled',
ID: 'abortMulti',
Expiration: {ExpiredObjectDeleteMarker: true},
AbortIncompleteMultipartUpload: {DaysAfterInitiation: 1},
Filter: {Prefix: {}} // <- change is here
}
]
}
}); ========== Golang Example:❌ Wont Compile: LifecycleConfiguration: &types.BucketLifecycleConfiguration{
Rules: []types.LifecycleRule{
{
Status: types.ExpirationStatusEnabled,
Filter: &types.LifecycleRuleFilter{}, // <- this is an interface not a concrete type.
AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
DaysAfterInitiation: 1,
},
Expiration: &types.LifecycleExpiration{
ExpiredObjectDeleteMarker: true,
},
ID: aws.String("abortMulti"),
},
},
},
}) ❌ Incorrect: LifecycleConfiguration: &types.BucketLifecycleConfiguration{
Rules: []types.LifecycleRule{
{
Status: types.ExpirationStatusEnabled,
// <- missing Filter
AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
DaysAfterInitiation: 1,
},
Expiration: &types.LifecycleExpiration{
ExpiredObjectDeleteMarker: true,
},
ID: aws.String("abortMulti"),
},
},
}, ✅ Correct:LifecycleConfiguration: &types.BucketLifecycleConfiguration{
Rules: []types.LifecycleRule{
{
Status: types.ExpirationStatusEnabled,
Filter: &types.LifecycleRuleFilterMemberPrefix{},
AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
DaysAfterInitiation: 1,
},
Expiration: &types.LifecycleExpiration{
ExpiredObjectDeleteMarker: true,
},
ID: aws.String("abortMulti"),
},
},
}, Again quoting documentation:
The only thing I could argue is that the s3 exception is not detailed enough and requires you to dig through documentation to see what is missing in the XML that is being serialized. Conclusion: Not a bug.I will convert this into a discussion and keep it open so I may address more questions if you have them in the future. Thanks again! |
Beta Was this translation helpful? Give feedback.
-
I'm having trouble with creating a rule in Go, I'd like to try the suggested approach but I believe the approach works for the aws-sdk-go-v2 API while I'm using the v1 API which does not provide The closest I got is: deleteMarkers := &s3.LifecycleRule{
ID: aws.String("Delete expired object delete markers"),
Expiration: &s3.LifecycleExpiration{
Days: aws.Int64(1),
ExpiredObjectDeleteMarker: aws.Bool(true),
},
Status: aws.String(s3.ExpirationStatusEnabled),
Filter: &s3.LifecycleRuleFilter{},
} but that does not work, I'm using the aws-sdk-go v1.44.184 (again not aws-sdk-go-v2). Is there a workaround for the v1 API? |
Beta Was this translation helpful? Give feedback.
-
Hello! Reopening this discussion to make it searchable. |
Beta Was this translation helpful? Give feedback.
@rabarar ,
After taking a deeper look into this here is what I have found.
The SDK serializes Golang strcuts into XML. When you send a request without that field in the request you're essentially leaving it out of the XML.
When the service reads the XML body sent by the GO SDK Client and sees that your LifeCycleRule doesn't have a
<Filter>
tag, it looks for<Prefix>
type and if both are missing it throws this exception.This is all handled on the service-side, and is documented.
To show that its not a GO SDK bug I've tested it on JS SDK V3 with the following code:
❌ Incorrect: