amplify push deployed an older version of package in lambda layer

[hisham]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

18

Amplify CLI Version

12.0.3

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

no

Describe the bug

We typically deploy to our prod backend environment through Amplify Console CD.

However, in this scenario I did an amplify push to our prod env on my local machine. I don't do this often, but gitlab was down so I did deployment from my machine. My machine has the latest code from git - the same code Amplify Console would have pulled.

What happened is a new lambda layer was made, however the lambda layer had an older version of a package we reference in the layer's package.json (e.g. v6.10.3). This is despite the actual local package.json (and the one in git referring to a newer version (v7).

This obviously caused an outage in our system because we have functionality that relies on the newer package version (v7) that is not in the old one.

What I think happened is the lambda layer created used the last package version I used when I last deployed to prod in my local machine. Again, I don't push to prod often via my local machine, so the last package pushed via my local machine was old. But I don't understand why it would do that when at the time of this latest push, my layer's package.json did refer to the new version of the package to be included.

Am I supposed to do an amplify pull before amplify push when pushing to an env I have not pushed to in a while (but other people / amplify console pushes to frequently?).

The interesting thing is the layer did not appear as needing any change yet amplify did create a new layer (or made our lambdas reference an older layer version, I'm not 100% sure):

Another thing is we recently updated our Lambdas to Node 18, but perhaps the prod amplify push referenced an old lambda cloudformations with Node 16, so perhaps it used our last layer that was compatible with Node 16? Not sure.

Expected behavior

amplify push should use latest code on machine to deploy and not use code from last deployment.

Reproduction steps

I believe the repro steps would be:

1. amplify push via your local machine
2. Make a change (e.g. update the version of a package inside lambda layer's package.json)
3. amplify push via your machine
4. Make a change (e.g. update the version of a package inside lambda layer's package.json)
   5.. amplify push via Amplify Console
5. Make a change (e.g. update graphql schema)
6. amplify push via your local machine

In step 6, the push will create a layer referencing the package.json in step 2, not step 4

Project Identifier

9aa765f6154c6bafb93f14988a5c3497

Log output

full log of the problematic deployment

# Put your logs below this line
➜  ess-app git:(master-redact) ✗ amplify push -y
⠙ Building resource api/EssGraphQLAPIBe careful when using @auth directives on a field in a root type...
⚠️ 
NodeToNodeEncryption is enabled for this Search Domain, disabling this flag or reverting to Amplify CLI <= 10.5.2 will result in this being disabled, triggering a rebuild of the Search Index. To backfill your search domain see https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table.

⚠️ WARNING: owners may reassign ownership for the following model(s) and role(s): <redacted>. To read more: https://docs.amplify.aws/cli/graphql/authorization-rules/#per-user--owner-based-data-access.
⠸ Building resource api/EssGraphQLAPI✅ GraphQL schema compiled successfully.

Edit your schema at /<redacted>/amplify/backend/api/EssGraphQLAPI/schema.graphql or place .graphql files in a directory at /<redacted>/amplify/backend/api/EssGraphQLAPI/schema
✔ Successfully pulled backend environment prod from the cloud.
⠙ Building resource api/EssGraphQLAPIBe careful when using @auth directives on a field in a root type...
⚠️ 
NodeToNodeEncryption is enabled for this Search Domain, disabling this flag or reverting to Amplify CLI <= 10.5.2 will result in this being disabled, triggering a rebuild of the Search Index. To backfill your search domain see https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table.

⚠️ WARNING: owners may reassign ownership for the following model(s) and role(s): <redacted>. If this is not intentional, you may want to apply field-level authorization rules to these fields. To read more: https://docs.amplify.aws/cli/graphql/authorization-rules/#per-user--owner-based-data-access.
⠸ Building resource api/EssGraphQLAPI✅ GraphQL schema compiled successfully.

Edit your schema at /<redacted>/amplify/backend/api/EssGraphQLAPI/schema.graphql or place .graphql files in a directory at /<redacted>/amplify/backend/api/EssGraphQLAPI/schema

    Current Environment: prod
    
┌───────────┬───────────────────────────────────────────┬───────────┬───────────────────┐
│ Category  │ Resource name                             │ Operation │ Provider plugin   │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Api       │ EssGraphQLAPI                             │ Update    │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Auth      │ essapp0f5644a0                            │ Update    │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ EssRestAPIFn                              │ Update    │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Storage   │ EssUsersS3                                │ Update    │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Analytics │ EssPinpoint                               │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Api       │ EssRestAPI                                │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ EssDailyPatientRecallFn                   │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ EssDynamoStreamFn                         │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ EssGraphQLResolverFn                      │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ EssPtGraphQLResolverFn                    │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ EssS3TriggerFn                            │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ essapp0f5644a0CreateAuthChallenge         │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ essapp0f5644a0DefineAuthChallenge         │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ essapp0f5644a0PostConfirmation            │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ essapp0f5644a0PreTokenGeneration          │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ essapp0f5644a0VerifyAuthChallengeResponse │ No Change │ awscloudformation │
├───────────┼───────────────────────────────────────────┼───────────┼───────────────────┤
│ Function  │ essappCliLambdaLayer                      │ No Change │ awscloudformation │
└───────────┴───────────────────────────────────────────┴───────────┴───────────────────┘
Be careful when using @auth directives on a field in a root type. @auth directives on field definitions use the source object to perform authorization logic and the source will be an empty object for fields on root types. Static group authorization should perform as expected....
NodeToNodeEncryption is enabled for this Search Domain, disabling this flag or reverting to Amplify CLI <= 10.5.2 will result in this being disabled, triggering a rebuild of the Search Index. To backfill your search domain see https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table.

⚠️ WARNING: owners may reassign ownership for the following model(s) and role(s): <redacted>. If this is not intentional, you may want to apply field-level authorization rules to these fields. To read more: https://docs.amplify.aws/cli/graphql/authorization-rules/#per-user--owner-based-data-access.
✅ GraphQL schema compiled successfully.

Edit your schema at /<redcated>/amplify/backend/api/EssGraphQLAPI/schema.graphql or place .graphql files in a directory at /<redacted>/amplify/backend/api/EssGraphQLAPI/schema
⠸ Building resource api/EssGraphQLAPIBe careful when using @auth directives on a field in a root type. @auth directives on field definitions use the source object to perform authorization logic and the source will be an empty object for fields on root types. Static group authorization should perform as expected....
⚠️ 
NodeToNodeEncryption is enabled for this Search Domain, disabling this flag or reverting to Amplify CLI <= 10.5.2 will result in this being disabled, triggering a rebuild of the Search Index. To backfill your search domain see https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table.

⚠️ WARNING: owners may reassign ownership for the following model(s) and role(s): <redcated>. If this is not intentional, you may want to apply field-level authorization rules to these fields. To read more: https://docs.amplify.aws/cli/graphql/authorization-rules/#per-user--owner-based-data-access.
⠴ Building resource api/EssGraphQLAPI✅ GraphQL schema compiled successfully.

Edit your schema at /<redcated>/amplify/backend/api/EssGraphQLAPI/schema.graphql or place .graphql files in a directory at /<redacted>/amplify/backend/api/EssGraphQLAPI/schema

Deployment completed.
Deploying root stack ess [ ======================================-- ] 17/18
	amplify-ess-app-prod-150238    AWS::CloudFormation::Stack     UPDATE_COMPLETE_CLEANUP_IN_PR… Fri Jul 07 2023 12:06:34…     
	functionessapp0f5644a0DefineA… AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:00:05…     
	functionessapp0f5644a0CreateA… AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:00:05…     
	functionessappCliLambdaLayer   AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:00:40…     
	analyticsEssPinpoint           AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:00:06…     
	functionessapp0f5644a0VerifyA… AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:01:17…     
	functionessapp0f5644a0PostCon… AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:01:17…     
	functionessapp0f5644a0PreToke… AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:01:18…     
	authessapp0f5644a0             AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:01:21…     
	apiEssGraphQLAPI               AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:05:26…     
	functionEssS3TriggerFn         AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:05:51…     
	functionEssDailyPatientRecall… AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:06:03…     
	functionEssPtGraphQLResolverFn AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:06:01…     
	storageEssUsersS3              AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:05:54…     
	functionEssGraphQLResolverFn   AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:06:18…     
	functionEssDynamoStreamFn      AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:06:31…     
	functionEssRestAPIFn           AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:06:19…     
	apiEssRestAPI                  AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:06:22…     
Deployed api EssGraphQLAPI [ ======================================== ] 18/18
	AuthRolePolicy02A05DD8DC       AWS::IAM::ManagedPolicy        UPDATE_COMPLETE                Fri Jul 07 2023 12:01:48…     
	GraphQLAPITransformerSchema3C… AWS::AppSync::GraphQLSchema    UPDATE_COMPLETE                Fri Jul 07 2023 12:01:47…     
	Referral                       AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:02:40…     
	FunctionDirectiveStack         AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:02:27…     
	RxPaymentIntent                AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:03:17…     
	MedicalHistory                 AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:03:18…     
	FormSubmission                 AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:03:18…     
	Rx                             AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:04:10…     
	RefundIntent                   AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:03:56…     
	PaymentIntent                  AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:04:10…     
	MedicationFollowup             AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:03:45…     
	RxMedicationFollowup           AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:03:57…     
	SearchableStack                AWS::CloudFormation::Stack     UPDATE_IN_PROGRESS             Fri Jul 07 2023 12:04:14…     
	ConnectionStack                AWS::CloudFormation::Stack     UPDATE_COMPLETE                Fri Jul 07 2023 12:04:36…     
Deployed api EssRestAPI [ ======================================== ] 5/5
Deployed auth essapp0f5644a0 [ ======================================== ] 16/16
Deployed function EssDailyPatientRecallFn [ ======================================== ] 6/6
Deployed function EssDynamoStreamFn [ ======================================== ] 14/14
Deployed function EssGraphQLResolverFn [ ======================================== ] 4/4
Deployed function EssPtGraphQLResolverFn [ ======================================== ] 4/4
Deployed function EssRestAPIFn [ ======================================== ] 4/4
Deployed function EssS3TriggerFn [ ======================================== ] 4/4
Deployed function essapp0f5644a0CreateAuthChallenge [ ======================================== ] 3/3
Deployed function essapp0f5644a0DefineAuthChallenge [ ======================================== ] 3/3
Deployed function essapp0f5644a0PostConfirmation [ ======================================== ] 3/3
Deployed function essapp0f5644a0PreTokenGeneration [ ======================================== ] 4/4
Deployed function essapp0f5644a0VerifyAuthChallengeResponse [ ======================================== ] 3/3
	LambdaFunction                 AWS::Lambda::Function          UPDATE_IN_PROGRESS             Fri Jul 07 2023 12:05:33…     
Deployed function essappCliLambdaLayer [ ======================================== ] 3/3
	LambdaLayerVersiona79f5a05     AWS::Lambda::LayerVersion      UPDATE_COMPLETE                Fri Jul 07 2023 12:00:33…     
	LambdaLayerPermissionAwsAccou… AWS::Lambda::LayerVersionPerm… UPDATE_COMPLETE                Fri Jul 07 2023 12:00:37…     
	LambdaLayerPermissionPrivatea… AWS::Lambda::LayerVersionPerm… UPDATE_COMPLETE                Fri Jul 07 2023 12:00:38…     
Deployed storage EssUsersS3 [ ======================================== ] 11/11

✔ Generated GraphQL operations successfully and saved at frontend/projects/shared/graphql
✔ Code generated successfully and saved in file frontend/projects/shared/graphql/API.service.ts
Deployment state saved successfully.
Be careful when using @auth directives on a field in a root type. @auth directives on field definitions use the source object to perform authorization logic and the source will be an empty object for fields on root types. Static group authorization should perform as expected....

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[josefaidt]

Hey @hisham 👋 thanks for raising this! We've had a few reports of this in the past but they were not reproducible. I will try to reproduce this using a combination of CLI and Hosting, but just to clarify when you mention

amplify push via Amplify Console

Are you referring to pushing to git and allowing Amplify Hosting to build for you? Are you running amplify pull after the build succeeds?

[hisham]

Are you referring to pushing to git and allowing Amplify Hosting to build for you? Are you running amplify pull after the build succeeds?

Yes that's correct. And I am not running amplify pull on my local machine after the build succeeds.

[josefaidt]

Hey @hisham I am definitely noticing some odd behavior. First, I was able to push twice with the CLI: first with an empty package.json and second with just typescript as a devDependency. After I hooked it up to Hosting and pushed to git, it removed layer versions 2 and 3

After adding another dependency and pushing with the CLI we now see version 5 alongside 4 (the one created by Hosting)

However when I download the latest zip of the layer through the AWS Lambda Console I see it only has the two dependencies, not the new, third dep

I was able to reproduce this using the provided reproduction steps, thank you for including those! Marking as a bug

[hisham]

Awesome thanks @josefaidt :).

I believe this should be treated as a high priority bug. Pushing old code caused an outage in some of our systems + bad data in our database that had to be fixed up. Fortunately we caught it early and an AWS Amplify Console build was already underway immediately after the bad push happened.

[josefaidt]

Hey @hisham, agreed! We're going to do a bit of a dive here to identify the root cause, and in the meantime I will go back through this reproduction to find a suitable workaround.

related issues:

#10784
#8088
#7476
#5174
#5569
#5577
#6140
#4928

[cappy123abc]

I don't know if this is of any use but I did notice that a hard reference to the layer being used was inserted into the lambda CF template that used the layer in the file amplify/backend/function/<function-name>/<function-name>-cloudformation-template.json
I replaced that with variable reference and the deploy worked as expected. (via a git push). example of change below:

    -    "Fn::Sub": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:layer:photoslayer-dev:218"
    +   "Ref": "functionphotoslayerArn"

[0618]

might be related to #9386