slow start-up when not at AWS Seattle offices (I am in Europe)

[mikn]

Hi! As mentioned during the community meeting, it takes me somewhere around 15-30 seconds before I get to the build stage. This is at a high level what happens:

Combined Application and Network Timeline

Time	Type	Event/Message
21:32:35	App	Command started: strace -t -f -e trace=network twoliter build variant metal-dev
21:32:35	Net	Socket: Initial connection to ECR (44.216.203.94:443)
21:32:35	Log	INFO twoliter::project::lock - Resolving project references to check against lock file
21:32:35	Log	INFO twoliter::project::lock::image - Resolving dependency image dependency '[email protected]/bottlerocket/bottlerocket-core-kit:v5.0.0'
21:32:35	Net	Socket: Secondary ECR connection (44.216.203.94:443)
21:32:36	Net	Socket: AWS service connection (75.2.101.78:443)
21:32:36	Net	Socket: AWS service connection (99.83.145.10:443)
21:32:37	Net	Socket: Additional ECR connection (44.216.203.94:443)
21:32:37	Net	DNS: Failed NSCD lookup
21:32:41	Net	IPv6: Multiple connection attempts failed (ENETUNREACH)
21:32:41	Log	INFO twoliter::project::lock::image - Resolving dependency image dependency '[email protected]/bottlerocket/bottlerocket-kernel-kit:v1.0.0'
21:32:42	Net	Socket: Additional AWS service connection (99.83.145.10:443)
21:32:43	Net	Socket: AWS service connection (75.2.101.78:443)
21:32:44	Net	Socket: Additional AWS service connection (99.83.145.10:443)
21:32:45	Net	Socket: Additional ECR connection (44.216.203.94:443)
21:32:46	Net	Socket: AWS service connection (75.2.101.78:443)
21:32:47	Net	Socket: AWS service connection (99.83.145.10:443)
21:32:48	Net	Socket: AWS service connection (3.164.226.167:443)
21:32:48	Log	INFO twoliter::project::lock::image - Resolving dependency image dependency '[email protected]/bottlerocket/bottlerocket-sdk:v0.50.0'
21:32:48	Net	IPv6: Multiple failed connection attempts
21:32:49	Net	Socket: Final ECR connection (44.216.203.94:443)
21:32:50	Net	Socket: Connection termination begins
21:32:52	Log	INFO cargo-make 0.37.23
21:32:52	Log	WARN Unable to parse Cargo.toml via cargo-metadata, fallbacking
21:32:53	Log	INFO Build File: /home/mikn/devel/molnett/src/platform/base/bottlerocket/build/tools/Makefile.toml
21:32:53	Log	INFO Task: build
21:32:53	Log	INFO Profile: development
21:32:53	Log	INFO Running Task: setup
21:32:53	Log	INFO Running Task: setup-build
21:32:53	Log	INFO Running Task: fetch-sdk

I had this strace output analysed by an AI and produced this log - but this was of a broken run as you can see by the fact that Cargo.toml was not parse:able.

Perhaps we could at least do all of the images in parallel?

[cbgbt]

We originally did not do parallel image pulls because docker was the engine underlying the pulls, and we did not trust it to do them correctly.

But we don't have that constraint now. I'm not sure if it's the only thing to blame here, but we can look into that as a start.