amplify push timeout: An error occured during the push operation: Your socket connection to the server was not read from or written to within the timeout period

[andreialecu]

Describe the bug

For the past few days, as the schema started growing it has been almost impossible to reliably push any changes via amplify push --yes

The "Updating resources in the cloud. This may take a few minutes..." message only appears for one in several push attempts.

Some more logs/status updates relating to what it does at every step between compiling the schema and updating the resources in the cloud would be helpful.

I'm not sure if it's some sort of intermittent outage related to S3 or other resources amplify push might be using. This is not an internet problem on my end.

Running amplify api push results in a stack trace that mentions S3 as the reson for the timeout.

I have attached a screenshot below.

Error is:
Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. An error occured during the push operation: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

Amplify CLI Version

4.18.0

Expected behavior

Pushing should not timeout.

Screenshots

Desktop (please complete the following information):

node -v: 13.7.0
Win10 with WSL2 (Ubuntu)

Additional context

This started happening yesterday, presumably related to schema changes but I'm not sure. The processing step before it starts "Updating resources in the cloud. This may take a few minutes..." has been taking longer and longer.

Has been reported several times but the issues are closed:

#462
#501
#2121

[andreialecu]

Here's a screenshot of the error during amplify function push --yes:

[andreialecu]

Here's another screenshot of having to run the command 3 times:

[ammarkarachi]

@andreialecu This usually happens when Lambda sizes are too large, what's the size of your Lambda package that you are trying to deploy?

[andreialecu]

The size of the zip in the dist directory of the function is 1.5 megabytes.

It's this big because of node_modules dependencies. The unpacked size is around 6-7 MB.

[ammarkarachi]

@andreialecu I have been unable to reproduce it but as far as I know there could be a problem with the network. When a packet loss occurs S3 waits on the packet to be resent and waits, if the packets aren't sent in 20 seconds the connection is closed. I would recommend checking the network connection on the machine you are using to publish.

[KayIlory]

This issue is present in Amplify version 4.22 going forward.

[bernardnongpoh]

amplify cli 4.24.3 does not work for me, error same as this issue. I downgraded to @aws-amplify/[email protected], it works as for now.

[ilfoxo]

Same for me, with @aws-amplify/[email protected]. In my network connection there are some retransmissions due to signal quality. Uploading files to S3 through web console works, but if I try to use amplify push always get a socket timeout.

[KayIlory]

@ammarkarachi - how are you.. is it safe to say this has now been resolved. Or maybe being tracked by another issue reference

[ammarkarachi]

@KayIlory This should be available in the latest build let me know if it works for you.

[vidhyasagarj]

Still happening to me in version 4.35.0. Command stuck at \ Uploading Files...8% and fails with socket timeout error

Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
An error occurred during the push operation: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

while pushing a lambda trigger function. It uses npm package node-html-to-image and hence it becomes a little bulky.

[amcdnl]

Still happening - why is this closed?

Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
An error occurred during the push operation: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

Version: @aws-amplify/[email protected]

[TheDutchCoder]

This is indeed still happening, I can no longer push with 4.52.

This is not an internet problem.

[amcdnl]

So I was able to figure out what the issue was ....

• Amplify Push
• Connections Opened ( S3/etc )
• Amplify Build + Your Build
• Build > S3 Socket Timeout time = Failure

[hisham]

+1. Also running into this. My feel is perhaps I have too many lambdas and too big of packages in each lambda for upload. I deploy through AWS Amplify as workaround. Perhaps the ongoing lambda layers rework will finally allow me to use lambda layers in the way I want.

[stevenbdf]

Downgraded to @aws-amplify/[email protected] and it works! as @bernardnongpoh said

[stevenbdf]

It seems that's an error related to the speed of your internet connection. If you are uploading a large set of artifacts and your connection is slow, it will fail.

You can try to increase the AWS_CLIENT_TIMEOUT=900000 so it gives more time to your CLI to upload everything.

Also you can try to create a Lambda Layer to store all your dependencies/packages and leave business logic/resolvers on a regular Lambda Function that import that layer. So when you make a change on your resolver, you won't have to push the large artifact every time, as the layer is already in the cloud.

[stevenbdf]

@josefaidt @ammarkarachi Amplify maintainers, please try to push a large set of functions with a slow internet connection (3G for example) to reproduce the error.

[amcdnl]

Its actually not just on slow connections, I have 1000 down and 50 up and with a very large project and I can see my bandwidth isn't even used 100% during the process and still times out.

[danshev]

Having this issue -- pushing just a few Lambdas with basic dependencies (total Amplify folder is 320 MB).

This issue is dramatically slowing down the development cycle. This issue seems to be persistent.

[linuslagerhjelm]

We also ran into this problem with a project that was ~1GB in total. After a lot of fiddling back and forth we got it to work, here's what we did:

• Deleted all the node_modules - did not solve the issue (foder size ~70MB)
• Removing functions one by one using amplify remove function until there were only two left - did not solve the issue (folder size ~25MB)
• Removed the same functions from the folder #current-cloud-backend- now it worked 🎉 (folder size ~24MB)

I would like to point out that the functions that were listed in the #current-cloud-backend directory were not actually present in the remote cloud environment, which I think might be the key here.

[InnovateWithEric]

Still not repro here. We need to explore complex gql schemas & DDB tables that take >15s to build.
If you still hit this in the future, please respond to this issue with details of your project.

[flesner]

This is completely connection speed related for me. I have a large change which changes the schema and all the lambda functions (20) in my project. Over a 100/5 connection it fails every time. Over a 1000/50 connections it works every time. Setting the AWS_CLIENT_TIMEOUT has no effect at all for me. My workaround was to move the amplify env to an EC2 in the same region as the project.

🛑 An error occurred during the push operation: write EPIPE

amplify -v
8.0.1

[nick-0101]

This occurs for me when installing aws-sdk or pushing a large amount of functions at once.

[lomaky]

Any updates on this issue? it continues to happen, at least 4 out of 5 times I try to deploy

[ryanhollander]

We see this more frequently, recently using CI/CD through Amplify Console. Typically, a successive redeploy operation will succeed just fine. It's not consistent, and we run almost everything out of US-EAST-2.

Amplify team, this has been outstanding for more than a year, is there any more insight or an ETA for mitigation, workaround, or a fix? I'm happy to give you all of the data from our project you'd need to reproduce or share full logs with you, please get in touch. @sachscode @Straubulous

[lomaky]

We see this more frequently, recently using CI/CD through Amplify Console. Typically, a successive redeploy operation will succeed just fine. It's not consistent, and we run almost everything out of US-EAST-2.

Amplify team, this has been outstanding for more than a year, is there any more insight or an ETA for mitigation, workaround, or a fix? I'm happy to give you all of the data from our project you'd need to reproduce or share full logs with you, please get in touch. @sachscode @Straubulous

@ryanhollander We are moving to CDK, Codebuild and Codedeploy for our deployments. Amplify is great to kick off a new project, but once it starts to become bigger it is better to move out of it. We have also been waiting for amplify to support .net6, which is supported by lambda but no luck so far.

[ryanhollander]

@lomaky thanks for the feedback. We are evaluating our options as well.

[sachscode]

We see this more frequently, recently using CI/CD through Amplify Console. Typically, a successive redeploy operation will succeed just fine. It's not consistent, and we run almost everything out of US-EAST-2.

Amplify team, this has been outstanding for more than a year, is there any more insight or an ETA for mitigation, workaround, or a fix? I'm happy to give you all of the data from our project you'd need to reproduce or share full logs with you, please get in touch. @sachscode @Straubulous

@lomaky
I am no longer in the Amplify team. @edwardfoyle , @Straubulous should be able to help.

[ryanhollander]

@sachscode no worries! Thanks for the redirect.

[biller-aivy]

@lomaky
Did you do that with the export function from the cli?

[josefaidt]

related #11835

[s3638844]

happens for me as well. I was able to get around it eventually kind of

• made many changes in code, push, get the above error.
• tried recreating the changes in the UI and applied and deploy from studio failed as well - some abstract message about deployment not complete
• back to CLI: created a new env from, changed to new env, deleted old env, pushed new env - the error this issue calls out occurs
• deleted the build dir in amplify/backend/api//build
• push again -> push doesn't fail up front, it''s still deploying now and looks like it just might work

honestly concerned if this persists I may have to redo my backend on another platform as I can't be destroying and recreating environments for every change

[MusaLawrence]

I found the solution to the issue. I don't usually give the solutions that take me a long time to figure out but this one's on the house. Besides, its 2023 and even with the latest version of amplify this still occurs, and there's no found solution, so let me explain. For those getting this error while trying to push an update via amplify, and who DO NOT have lambda functions within their app, because yes the node modules created by each lambda function will definitely cause this error being the size of each module. In fact I removed lambda from my direct project for that reason. Any lambda function I need to create, I do it from the console. But from the CLI the above error occurs when creating tables in the schema while already having many other tables created, especially complex tables, each referencing, and depending on other tables like the user table to a social media app. It seems to occur if you create too many properties on a new table or existing table at the same time. Especially properties with arrays/AWSJSON types. Amplify seems to rewrite the entire backend during each update. Try adding no more than a few properties at a time with String or Boolean types and only one property at a time with array types. This will solve the issue, but make sure after the fail happens that cloud formation is not in a suspended or still updating state. Easy fix after experiencing it many many many times. But you're welcome, hope this helps

[josefaidt]

This now resolved with #13493, closing

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.