Fetching updates to backend environment: takes a lot of time

[hackmajoris]

Which Category is your question related to?
Amplify Hosting

Amplify CLI Version
9.2.1

What AWS Services are you utilizing?

• Dynamo Database
• AppSync API
• OpenSearch
• Lambda
• Other

Short story: our project takes almost 1 hour to be deployed.
I'm wondering:

• why it takes so long to fetch the environment from the cloud: 13+ minutes. (see the pictures)
• why it takes so long to uploading the files: 10+ minutes. (see the pictures)

Also, the 'assumed role' token which Amplify uses to do the build, expires after 1 hour. We already started to get to that limit, and have no workaround for that.

Any insights are useful.

Thank you!

[ghost]

Hi @hackmajoris can you please provide your App ID so we can investigate why your builds are running for an extended period?

[hackmajoris]

@hloriana The app ID is d3gbyqqyggsz5w
I spent the whole day today to investigate this issue. This is what I found.

We use Amplify hooks with JavaScript, and here are some job examples inside hooks:

  await forceSetCognitoTriggers(data);
  await createTestUsers(data);
  await updateSsmParams(data);
  await addCustomDataSources(data);

Some of them are for out application logic. Some of them are the post-push fixes which Amplify brakes during the deployment, like Cognito Triggers, and data source attachments to the AppSync.

Until the version 10 of Amplify, all the hooks files were pushed and pulled to the S3 deployment bucket. You may guess that we need some node packages, like AWS SDK, in order to build those scripts. Locally, node_modules are ignored in the hooks-ignore file and are not pushed remotely. But remotely, via Amplify Console, the https://github.com/aws-amplify/amplify-hosting/blob/main/scripts/amplifyPush.sh script is used to deploy the app. When it runs, it pushes all the files to the S3 bucket, then does the build. When the build is finished, the hooks are triggered. Because the node_modules at the step where the hooks are triggered, were not in place, we had to install them before calling the amplifyPush --simple script.

The amplify.yml then, will look like this:

build:
    - cd amplify/hooks and yarn
    - amplifyPush --simple 

At this step, the hooks are executed with no issues, but all those node_modules(20k+ files) ~20mb were pushed and pulled from and to S3 bucket again and again, at each build.

This consumed a lot of time. The solution I've done so far is this:

• copied that amplifyPush.sh script in the codebase.
• in the pre-build step, at the amplify.yml, moved the files from amplify/hooks folder in a tmp folder(they are git based)
• modified the amplifyPush.sh script in a way, that after env init, replaced everything from hooks folder(amplify is pulling the files from S3 every time) with that tmp folder and installed the dependencies(the node_modules are ignored in the hooks-ignore file, so they will not be pushed next time)

This could be avoided if this bug aws-amplify/amplify-cli#11021
and this fix aws-amplify/amplify-cli#11179 were already released.

[hackmajoris]

Fixed with #11179

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.