Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to build backend when using Amazon Linux:2023 and default Amplify CLI #13423

Closed
5 tasks done
kachkaev opened this issue Nov 18, 2023 · 12 comments
Closed
5 tasks done
Labels
hosting Issues tied to hosting category pending-response Issue is pending response from the issue author pending-triage Issue is pending triage platform Issues tied to the general CLI platform

Comments

@kachkaev
Copy link

Before opening, please confirm:

  • I have checked to see if my question is addressed in the FAQ.
  • I have searched for duplicate or closed issues.
  • I have read the guide for submitting bug reports.
  • I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
  • I have removed any sensitive information from my code snippets and submission.

App Id

d3bq67l0547585

AWS Region

eu-west-2

Amplify Hosting feature

Build settings

Frontend framework

Next.js

Next.js version

13.5.6

Next.js router

Pages Router

Describe the bug

I am trying to upgrade to Next.js 14. The original issue (aws-amplify/amplify-hosting#3773) got resolved by a new image Amazon Linux 2023. I tried this image and saw v18.18.2 when running node --version. That’s good.

However, the build failed:

Screenshot 2023-11-18 at 13 50 55

Last few lines of the logs:

2023-11-18T01:52:59.157Z [INFO]: # Completed phase: preBuild
                                 # Starting phase: build
2023-11-18T01:52:59.157Z [INFO]: # Executing command: export PATH="$PWD/amplify/bin:$PATH"
                                 # Executing command: export COREPACK_ENABLE_STRICT=0
                                 ## Running amplify pull to generate aws-exports.js file for frontend
2023-11-18T01:53:02.761Z [INFO]: �[0mAmplify AppID found: d3bq67l0547585. Amplify App name is: myapp[0m
2023-11-18T01:53:02.815Z [INFO]: �[0mBackend environment dev found in Amplify Console app: myapp[0m
2023-11-18T01:53:06.970Z [INFO]: 🛑 Failed to pull the backend.
2023-11-18T01:53:06.973Z [INFO]: Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/
2023-11-18T01:53:06.988Z [INFO]: 
2023-11-18T01:53:06.988Z [INFO]: Session Identifier: 4f025418-c3cc-4045-bd4a-a1a240e35809
2023-11-18T01:53:07.009Z [ERROR]: !!! Build failed
2023-11-18T01:53:07.009Z [INFO]: Please check the supported SSR features to find if your build failure is related to an unsupported feature: https://docs.aws.amazon.com/amplify/latest/userguide/ssr-Amplify-support.html#supported-unsupported-features. You may also find this troubleshooting guide useful: https://docs.aws.amazon.com/amplify/latest/userguide/troubleshooting-ssr-deployment.html
2023-11-18T01:53:07.009Z [ERROR]: !!! Non-Zero Exit Code detected
2023-11-18T01:53:07.009Z [INFO]: # Starting environment caching...
2023-11-18T01:53:07.009Z [INFO]: # Uploading environment cache artifact...
2023-11-18T01:53:07.095Z [INFO]: # Uploaded environment cache artifact
2023-11-18T01:53:07.095Z [INFO]: # Environment caching completed
Terminating logging...

Expected behavior

Backend builds without errors.

Reproduction steps

  1. Switch to Amazon Linux:2023
  2. Don't configure Amplify CLI version or set it to latest (12.8.2 at the time of writing)
  3. Run the build
  4. Observe backend build failing

Build Settings

No response

Log output

No response

Additional information

I managed to find a workaround:

Setting Amplify CLI version to anything higher than 12.5.2 causes the failure.

Backend build logs when Amplify CLI version is set to 12.5.2:

2023-11-18T11:22:16.609Z [INFO]: # Completed phase: preBuild
                                 # Starting phase: build
2023-11-18T11:22:16.610Z [INFO]: # Executing command: export PATH="$PWD/amplify/bin:$PATH"
                                 # Executing command: export COREPACK_ENABLE_STRICT=0
                                 ## Running amplify pull to generate aws-exports.js file for frontend
2023-11-18T11:22:18.728Z [INFO]: �[0mAmplify AppID found: d3bq67l0547585. Amplify App name is: myapp�[0m
2023-11-18T11:22:18.776Z [INFO]: �[0mBackend environment dev found in Amplify Console app: myapp�[0m
2023-11-18T11:22:24.874Z [WARNING]: - Fetching updates to backend environment: dev from the cloud.
2023-11-18T11:22:29.289Z [WARNING]: - Building resource api/NeptuneApi
2023-11-18T11:22:30.376Z [INFO]: �[32mGraphQL schema compiled successfully.�[39m
                                 �[32mEdit your schema at /codebuild/output/src3127556268/src/myapp/amplify/backend/api/NeptuneApi/schema.graphql or place .graphql files in a directory at /codebuild/output/src3127556268/src/myapp/amplify/backend/api/NeptuneApi/schema�[39m
2023-11-18T11:22:30.378Z [WARNING]: - Building resource auth/myapp356a8770
2023-11-18T11:22:30.496Z [WARNING]: ✔ Successfully pulled backend environment dev from the cloud.
2023-11-18T11:22:30.767Z [INFO]: ✅
2023-11-18T11:22:32.878Z [INFO]: # Completed phase: build

Frontend build is successful too.

@kachkaev kachkaev added the pending-triage Issue is pending triage label Nov 18, 2023
@Narrator
Copy link

Narrator commented Nov 18, 2023

Hello! Thanks for reaching out to us.

We bumped up the default Amplify CLI version in the new build image. The new version is probably not compatible with your deployment.

The default version of the Amplify CLI in the AL2 image was 10.5.2.

Do you have live package updates for the Amplify CLI? If you don't, then your app was building against 10.5.2 all this while, which is probably the safest version for your build.

However, to confirm whether this issue is related to something else going on in the build image or not, I'd recommend that you use the AL2 image and set the Amplify CLI version to latest via live package update. If the build fails with the same reason, then the Amplify CLI version is the root cause.

@kachkaev
Copy link
Author

kachkaev commented Nov 18, 2023

I was previously using a custom image: aws-amplify/amplify-hosting#3779 (comment). Base AL2 did not work: aws-amplify/amplify-hosting#3109 (comment)

If the build fails with the same reason, then the Amplify CLI version is the root cause.

Not sure I understand how a minor bump (from 12.5.2 to 12.6.0 and above) can cause an issue. The build fails without any extra information, so it’s not clear what to look for. See 🛑 Failed to pull the backend in the logs I have shared above. There don't seem to be any warnings or errors preceding the failure which is quite confusing.

@Narrator
Copy link

@kachkaev I apologize for the inconvenience caused by this issue.

Is there a specific version of the Amplify CLI that you'd like to use with your builds?

If there's a specific version you want, then please set it via live package overrides so that you can be intentional about bumping up the version as and when you need. Otherwise, your builds will always default to the default version in the build image - Whenever we release a new version of the build image, the default CLI version will potentially get updated

With regards to why this build is failing, you'd have to create an issue with the Amplify CLI. Here's their repo: https://github.com/aws-amplify/amplify-cli/issues

@kachkaev
Copy link
Author

kachkaev commented Nov 19, 2023

When I set 12.5.2 as a version override for Amplify CLI, backend build works. When I bump it further (12.6.0 ... 12.8.2) in build image settings, backend build fails every time with the same logs. When I roll Amplify CLI back to 12.5.2, the build works again.

I am trying to understand why a minor version bump of Amplify CLI makes any difference. This does not look right, especially because unsuccessful build logs don't contain any useful information (see logs in the initial comment). I am opening this issue because I think that other developers may find themselves in the same trap.

@swaminator swaminator transferred this issue from aws-amplify/amplify-hosting Nov 21, 2023
@kachkaev
Copy link
Author

kachkaev commented Nov 21, 2023

A bit more info: our project uses pnpm instead of npm/yarn. We call corepack enable; pnpm config set node-linker hoisted; pnpm install in the breBuild step of amplify.yaml. If there were any changes to package management between Amplify CLI 12.5.2 and 12.6.0, that could explain it.

As you can see in my original post, our failing build logs are not very helpful. All we see is 🛑 Failed to pull the backend but there are no other warnings or errors to inspect. It is possible that pnpm is not relevant here, I’m sharing this detail just in case. Amplify CLI 12.5.2 does the job for now, but a breaking change is unexpected in a semver-minor release.

@ykethan
Copy link
Member

ykethan commented Nov 21, 2023

Hey @kachkaev, could you try adding AMPLIFY_ENABLE_DEBUG_OUTPUT to true as a environment variable in the AWS Amplify console this should verbose logging and then run the build.
Could you provide us the output and any identifiers printed here?

@ykethan ykethan added hosting Issues tied to hosting category platform Issues tied to the general CLI platform pending-response Issue is pending response from the issue author labels Nov 21, 2023
@kachkaev
Copy link
Author

kachkaev commented Nov 24, 2023

Just tried AMPLIFY_ENABLE_DEBUG_OUTPUT=true amplify pull --envName dev locally (Amplify CLI 12.8.2). Here is the output:

Details

🛑 Failed to pull the backend.

Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

PullBackendFault: Failed to pull the backend.
at attachBackend (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/attach-backend.js:55:15)
at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/commands/pull.js:64:9)
at async Object.executeAmplifyCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:194:9)
at async executePluginModuleCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:139:5)
at async executeCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:37:9)
at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:121:5)

EEXIST: file already exists, symlink '../uuid/dist/bin/uuid' -> '/path/to/project/amplify/backend/function/arahitoolspocAWSSDK/lib/nodejs/node_modules/.bin/uuid'
Error: EEXIST: file already exists, symlink '../uuid/dist/bin/uuid' -> '/path/to/project/amplify/backend/function/arahitoolspocAWSSDK/lib/nodejs/node_modules/.bin/uuid'
at Object.symlinkSync (node:fs:1741:3)
at onLink (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:139:70)
at getStats (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:49:45)
at startCopy (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:38:10)
at copyDirItem (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:120:10)
at /snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:39
at Array.forEach ()
at copyDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:23)
at onDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:103:10)
at getStats (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:45:37)
at startCopy (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:38:10)
at copyDirItem (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:120:10)
at /snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:39
at Array.forEach ()
at copyDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:23)
at onDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:103:10)
at getStats (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:45:37)
at startCopy (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:38:10)
at copyDirItem (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:120:10)
at /snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:39
at Array.forEach ()
at copyDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:23)
at onDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:103:10)
at getStats (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:45:37)
at startCopy (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:38:10)
at copyDirItem (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:120:10)
at /snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:39
at Array.forEach ()
at copyDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:23)
at onDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:103:10)
at getStats (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:45:37)
at startCopy (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:38:10)
at copyDirItem (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:120:10)
at /snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:39
at Array.forEach ()
at copyDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:23)
at onDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:103:10)
at getStats (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:45:37)
at startCopy (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:38:10)
at copyDirItem (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:120:10)
at /snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:39
at Array.forEach ()
at copyDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:23)
at onDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:103:10)
at getStats (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:45:37)
at startCopy (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:38:10)
at copyDirItem (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:120:10)
at /snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:39
at Array.forEach ()
at copyDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:113:23)
at onDir (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:103:10)
at getStats (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:45:37)
at startCopy (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:38:10)
at handleFilterAndCopy (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:33:10)
at Object.copySync (/snapshot/amplify-cli/build/node_modules/fs-extra/lib/copy-sync/copy-sync.js:26:10)
at downloadBackend (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/attach-backend.js:347:24)
at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/attach-backend.js:117:3)
at async Object.attachBackend (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/index.js:196:3)
at async queryProvider (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/attach-backend-steps/a10-queryProvider.js:36:5)
at async attachBackend (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/attach-backend.js:43:9)
at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/commands/pull.js:64:9)
at async Object.executeAmplifyCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:194:9)
at async executePluginModuleCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:139:5)
at async executeCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:37:9)
at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:121:5)

Session Identifier: 927e3a42-23ea-4332-87f0-fbbf7fc4b3ef

Note that the project uses pnpm, which could play a role here.

@github-actions github-actions bot removed the pending-response Issue is pending response from the issue author label Nov 24, 2023
@ykethan
Copy link
Member

ykethan commented Nov 27, 2023

Hey @kachkaev, thank you for the information.
The error message: EEXIST: file already exists, symlink '../uuid/dist/bin/uuid' -> '/path/to/project/amplify/backend/function/arahitoolspocAWSSDK/lib/nodejs/node_modules/.bin/uuid' indicates the build failed due to a symlink link error in arahitoolspocAWSSDK resource.

The issue appears to be similar to #2948.
To mitigate the issue the workaround would be to remove the node_modules and .temp then running push/re-build the application, the detailed steps are documented here.

@ykethan ykethan added the pending-response Issue is pending response from the issue author label Nov 27, 2023
@kachkaev
Copy link
Author

I’m afraid I won’t be able to try this hack with removing node_modules in early January only. In the meantime, if anyone has the same issue or has successfully applied a workaround, please share your experience.

@github-actions github-actions bot removed the pending-response Issue is pending response from the issue author label Nov 29, 2023
@ykethan
Copy link
Member

ykethan commented Dec 6, 2023

@kachkaev I understand, i am not aware on other workarounds at this moment. But the amplify push should recreate the node modules in the current-cloud-backed.zip. Do reach out if you experience any issues or get a chance to test the workaround.

@ykethan ykethan added the pending-response Issue is pending response from the issue author label Dec 6, 2023
@josefaidt
Copy link
Contributor

Closing due to inactivity

@josefaidt josefaidt closed this as not planned Won't fix, can't repro, duplicate, stale Feb 2, 2024
Copy link

github-actions bot commented Feb 2, 2024

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hosting Issues tied to hosting category pending-response Issue is pending response from the issue author pending-triage Issue is pending triage platform Issues tied to the general CLI platform
Projects
None yet
Development

No branches or pull requests

4 participants