Long deployment times of ~40 mins for large Amplify project

[mitchconquer]

Is this related to a new or existing framework?

No response

Is this related to a new or existing API?

No response

Is this related to another service?

No response

Describe the feature you'd like to request

Hi! I'm not sure how to categorize this but we are running very long deployment times for our amplify backend. I mentioned this in the Amplify Office Hours and was told I should open an issue so we can discuss it in more detail.

Here's a breakdown of our resources. We are using 100% TypeScript.

• 71 Lambdas
• 1 GraphQL API
• 2 REST API's
• 2 SNS
• 10 SQS

Our deployment times to our dev and main environments are at about 40 minutes now and have been creeping up over the past year so it's quite painful to deploy now.

Describe the solution you'd like

I wonder if there's a way to dramatically improve the deployment times, either through configuration or perhaps a different container for running the Amplify deployment?

Describe alternatives you've considered

It was mentioned in Office Hours that I should consider setting up the Lambda functions as custom resources, I wonder if I would still be able to access the other resources with the backend-config.js file and whether they would work with TypeScript.

Additional context

I'd be happy to send the build logs as well if you let me know where I can do that.

Is this something that you'd be interested in working on?

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

[chrisbonifacio]

Hi @mitchconquer 👋 thanks for raising this issue. Transferring it over the CLI repo for better feedback and support.

[josefaidt]

Hey @mitchconquer 👋 I have a few questions:

• how are the resources set up with Amplify CLI
• what sort of change is causing the long deploy times? (i.e. updating a model name)
• Are the Lambda's attached as resolvers to a GraphQL model?
• What version of the CLI are you currently using?

[mitchconquer]

Hi, @josefaidt! Thanks for getting back to me! Answers to your questions below but generally speaking, we've been using Amplify for over a year and the time has been steadily creeping up so I don't think it's a sudden change, rather just the quanity of resources we are deploying.

1. how are the resources set up with Amplify CLI
   • I'm not sure I understand but the resources were set up using CLI commands such as amplify function add and then some additional configuration to add handling for TypeScript
2. what sort of change is causing the long deploy times? (i.e. updating a model name)
   • No particular change causes long deploy times. It happens for every deployment regardless of changes.
3. Are the Lambda's attached as resolvers to a GraphQL model?
   • No, none are being used as GraphQL resolvers
4. What version of the CLI are you currently using?
   • The build settings in Amplify are using the latest CLI. Locally I am using 7.6.21.

[josefaidt]

Hey @mitchconquer thank you for clarifying! From your last note, are you experiencing long deploy times from amplify push or the CI/CD pipeline builds, or both? With amplify push it should only deploy the changed resources.

[mitchconquer]

CI/CD pipeline builds. I haven't been able to run amplify push for months due to various errors I could never solve.

It always complained about something like the apijoggoGraphQLAPIKeyOutput missing despite the fact that it was in the cloud formation template and backend-config.json.

[mitchconquer]

Hi just following up on this! 😺

[josefaidt]

Hey @mitchconquer 👋 apologies for the delay! As I investigate the long console builds time I would like to further understand the ...APIKeyOutput issue you have been experiencing. Have you continuously experienced this issue when attempting to push from your local machine?

[josefaidt]

Hey @mitchconquer 👋 apologies for the delay here! Can you try downloading the Hosting pipeline's amplifyPush.sh script to a fresh clone of your project and execute with sh amplifyPush.sh --simple and compare the time difference with amplify push -y in your original app's directory after a minor update? I would like to see if there's a difference with the helper script vs local amplify push

[josefaidt]

Hey @mitchconquer 👋 are you still experiencing this issue?

[josefaidt]

Closing due to inactivity

[caridavidson]

Closing due to inactivity

I am. I think anyone with a reasonably-sized environment is. Just because the OP isn't responding, doesn't mean there is no issue.

[roonz11]

Also experiencing incredibly long deployment times with ~50 functions. Simple code changes take 40-50min to deploy.
using amplify-cli v11.1.1

[asashakira]

Hey, I'm also experiencing long deployment time and I have a question regarding it.

Say there are 1000 javascript functions in my project.
Will deployment time improve if we put those functions in 1000 separate Lambda Functions or have one Lambda function with 1000 functions in it?

I'm assuming the latter will improve deployment time since I assume it's taking a load of time building each resource, but it still takes around 20~30 minutes deploying a simple code change with only ~20 functions and ~10 custom resources in our project.

[heinrichcoetzee]

Same here. Getting to the point where I want to move my app to another infrastructure. Not even a lot of functions, Just a pretty bif angular application with a couple of endpoints. It takes 20-30 mins.

Clearly an issue.

[mitchconquer]

I really appreciate the work of the Amplify team but I think it's underfunded considering the scope and usage of the tool. We migrated to SST and I really loved it.

[AleksandarGT]

The CI/CD pipeline is taking 20-30 minutes every time you want to deploy a single change. Every big project that is running on amplify is having this issue and yet they close the issue due to inactivity...