Amplify Push/mock with added function fails: "Packaging lambda function failed with the error Command failed with exit code 243: npm install --production"

[eettaa]

Before opening, please confirm:

• I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
• I have searched for duplicate or closed issues.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v14.18.1

Amplify CLI Version

6.3.1

What operating system are you using?

Ubuntu 18.04.5 LTS

Amplify Categories

function, api

Amplify Commands

push

Describe the bug

When I run amplify push or amplify mock function from my projects TLD (i.e. the dir that contains amplify/)I get the seemingly related errors:
Push:

Packaging lambda function failed with the error 
Command failed with exit code 1: yarn --production
An error occurred during the push operation: Packaging lambda function failed with the error 
Command failed with exit code 1: yarn --production

Mock:

Packaging lambda function failed with the error 
Command failed with exit code 1: yarn

This happens for a clean project where I add a new amplify backend to an (amplify-hosted) frontend using default backend options:

amplify add api
amplify add function

The api schema does not connect to the lambda function, fwiw.

If I cd into amplify/backend/function/<function_name>/src and run npm install [--production] the command succeeds.
Upon rerunning push/mock I get similar npm-flavored errors:
Push:

Packaging lambda function failed with the error 
Command failed with exit code 243: npm install --production
An error occurred during the push operation: Packaging lambda function failed with the error 
Command failed with exit code 243: npm install --production

Mock:

Packaging lambda function failed with the error 
Command failed with exit code 243: npm install

I am on Ubuntu v14.18.1 / yarn 1.22.15 / npm 6.14.15 / amplify 6.3.1
I have verified that this works as expected on macOS big sur 11.2.3 / yarn 1.22.11 / npm 7.18.1 / amplify 6.3.1

I have found related #7883 but believe it is irrelevant as I am still on yarn 1.x
I have found related #5483 but believe it is irrelevant as it is related to older amplify cli versions and has since been corrected.

Thank you for your help!!!

Expected behavior

Push pushes changes (including newly added lambda) to the cloud

Mock spins up local instances of the backend.

Reproduction steps

Using my OS/cli versions:
amplify pull --appId --envName staging // Pull an amplify project that has no backend configured
amplify add api // choose fully default settings
amplify add function // choose fully default settings, hello world template
amplify push

GraphQL schema(s)

# Put schemas below this line

type Todo @model {
  id: ID!
  name: String!
  description: String
}

Log output

Nothing interesting in logs AFAICT, just the same repeated message

# Put your logs below this line
Error: Packaging lambda function failed with the error
Command failed with exit code 243: npm install --production

Additional information

No response

[eettaa]

OP:
I am not sure whether this is relevant, but it might be worth noting that I am in a monorepo and have multiple nested packages. All other packages are using npm

• mono dir
  • .git
  • other dir
  • app dir ( I run the amplify commands from here)
    • frontend code.file
    • package.json
    • package-lock.json
    • amplify
      • backend
        • function
          • functionname
            • src
              • package.json
              • [yarn.lock|package-lock.json]
              • index.js

I am not sure that this is relevant, though, as I notice amplify intelligently detecting npm usage (after I manually run npm install in the functionname/src directory)

[codybreene]

Running into this same issue with yarn, but I never used yarn...

[codybreene]

Packaging lambda function failed with the error Command failed with exit code 255: yarn --production [DEPRECATION] The trollop gem has been renamed to optimist and will no longer be supported. Please switch to optimist as soon as possible. Error: unknown argument '--production'.

[eettaa]

@codybreene what is your os/npm/amplify/yarn versions?

what do you get if you cd into amplify/backend/api/function/functionname/src, delete a yarn.lock file if it exists, run npm install (thus creating a package-lock.json) file, cd back to the project tld and re-run amplify push/amplify mock function?

[codybreene]

I found that I had installed `yarn` in an old Ruby environment, and for some reason that was being called. Still confused, though, because I was never using yarn within this project. Only npm.

…

On Thu, Nov 4, 2021 at 9:34 PM Eric Anderson ***@***.***> wrote: @codybreene <https://github.com/codybreene> what is your os/npm/amplify/yarn versions? what do you get if you cd into amplify/backend/api/function/functionname/src, delete a yarn.lock file if it exists, run npm install (thus creating a package-lock.json) file, cd back to the project tld and re-run amplify push/amplify mock function? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8653 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLVA5SJDUUTNUJVDH74RO3UKNNDZANCNFSM5HJYS5EQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[eettaa]

I'm still struggling with this problem, may try to dig into source code later this week. Are there any amplify experts out there who could recommend good diagnostic steps?

[josefaidt]

Hey @eettaa 👋 thanks for raising this! Was the Amplify CLI installed with npm using sudo? Curious if this issue is related to permissions as it works if you cd into the function's src directory and run npm install

If I cd into amplify/backend/function/<function_name>/src and run npm install [--production] the command succeeds.

[eettaa]

Hi @josefaidt , thanks so much for your reply. I'm still very interested in diagnosing this and will help any way I can.
Yes, I wondered the same thing about permissions. However, nothing obvious jumps out at me.

• Yes, Amplify CLI was installed using sudo. Just to be sure, I uninstalled and reinstalled the cli (globally, using sudo) and can still reproduce the behavior.
• Running sudo amplify mock [function|api] also results in the same error (and I don't think I should have to run the mock command using sudo, anyways...)
• I cannot seem to find other interesting error messages. After running the command, ~/.amplify/logs/ has the same message mentioned above:

Error: Packaging lambda function failed with the error
Command failed with exit code 1: yarn --production

• I do not believe this has to do with AWS/Amplify CLI permissions as e.g. amplify mock api works just fine when there is no lambda in the project. Just to be sure, I re-ran amplify-configure, added a new iam user, followed instructions at https://docs.amplify.aws/cli/start/install/#install-the-amplify-cli, and still reproduced the problem.

Do you have recommendations for other diagnostic steps we could take? I have yet to dig through the cli source code but am wondering if I can identify the failing line there/if it would offer anything of value.

[josefaidt]

Thank you for clarifying @eettaa and providing those details! Do you still experience this issue when the CLI is installed without sudo? What is the motivation behind using sudo to install global dependencies? I can say this is indeed caused by some issue locally and not with the AWS account's permissions. For reference, here is the code powering how we determine which Node package manager to use https://github.com/aws-amplify/amplify-cli/blob/master/packages/amplify-cli-core/src/utils/packageManager.ts#L30-L40, and when looking at this logic we should see the CLI leveraging the locally installed binary.

For what it's worth there is also an open feature request to support a "Bring your own Node package manager" #6382, and also there are command hooks that we could potentially use to bootstrap function dependencies.

[eettaa]

Hi @josefaidt , thanks for your response.
On Ubuntu it appears the default location for global npm installations (/usr/local/lib/node_modules/) is only accessible with sudo privileges. Non-sudo install attempts fail with:

 npm ERR! path /usr/local/lib/node_modules/@aws-amplify/cli/node_modules/esm

... among other errors.
However, I wonder if this is really the problem? For one thing, running sudo amplify [push|mock] doesn't solve the problem. For another, the generated symlink after sudo installation in /usr/local/bin (discovered via which amplify) is:

lrwxrwxrwx 1 root root 48 Nov  8 16:21 amplify -> ../lib/node_modules/@aws-amplify/cli/bin/amplify

indicating anyone can run

and the pointed-to executable is

-rwxr-xr-x 1 root root 238 Oct 26  1985 ../lib/node_modules/@aws-amplify/cli/bin/amplify

Which seems to have the necessary read/execute permissions regardless of user?

When running amplify push, the error message is simply "Command failed with exit code 1: yarn --production" ... Is there any way to understand why this command (or its npm flavor) is failing? Maybe there are extra verbosity flags?
Another thought I had was that the exit code for npm is 243, which is unusual. Documentation says you should subtract 128 from >128 exit codes to get a fatal signal code -> signal code 115 -> EINPROGRESS 115 (according to link) ... does that help at all?

Thank you for your help

[josefaidt]

Hey @eettaa please see this npm docs page for resolving the issue with installing packages globally. From here we can check a few things:

• we can check the permissions on the function's src directory ll amplify/backend/function/<function-name>/src to ensure owner has write access (group and public should show only "read" access, which should remain)
• resolve and attempt installation without sudo
• try upgrading npm, though considering yarn is also failing I do not suspect this will alleviate the issue

[eettaa]

RESOLVED
Hi @josefaidt, thanks for the suggestions:
I resolved this problem by taking your suggestion to install npm/node/etc without sudo. To do this, I used nvm. I also had to fully uninstall all system node packages (found via npm list -g --depth=0 before starting nvm installation) reinstalling said packages globally but without sudo after nvm installation (impossible before nvm/other messier workarounds). I still am not quite sure what inside amplify caused the permissions issue in the first place, but I am very happy to be unblocked. Thank you!

For Posterity:

Before installing nvm, it's important to note the versions of the relevant software, as you will need to reinstall them (first via nvm use <version> then npm install -g <package>)

node -v
npm -v
npm list -g --depth=0

Amplify Directory Permissions

It appears that the ll amplify/backend/function/<function-name>/src permissions were as expected:
Permissions of the function-name dir

-rw-r--r-- 1 <name>  <name>  156 Nov  9 09:40 amplify.state
-rw-r--r-- 1  <name>  <name>   48 Nov  9 09:40 custom-policies.json
-rw-r--r-- 1  <name>  <name>   24 Nov  9 09:40 function-parameters.json
-rw-r--r-- 1  <name>  <name> 4958 Nov  9 09:40 helloworld-cloudformation-template.json
drwxr-xr-x 2  <name> <name> 4096 Nov  9 09:40 src/

and here are the permissions of the src/ dir:

drwxr-xr-x 2 <name> <name> 4096 Nov  9 09:40 ./
drwxr-xr-x 3 <name> <name> 4096 Nov  9 09:40 ../
-rw-r--r-- 1 <name> <name>   63 Nov  9 09:40 event.json
-rw-r--r-- 1 <name> <name>  367 Nov  9 09:40 index.js
-rw-r--r-- 1 <name> <name>  155 Nov  9 09:40 package.json

where is my user/group as expected. It appears the owner always has write access, so it looks okay to me?

[josefaidt]

Ah awesome @eettaa I'm glad to hear you're back up and running! Installing Node and npm packages with sudo can cause some headaches I've found, but those permissions look correct!

Closing issue as the issue has been resolved 🙂

[github-actions]

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels for those types of questions.