feat: keep downloaded packages (releases only)

[a-gave]

This increase the speed of subsequent builds for the same target (excluding snapshots),
by saving a local copy of downloaded packages in a path defined as version/target/subtarget
that is then mounted on the imagebuilder as a bind-volume,
for example /path/to/asu/dl/23.05.5/ath79/generic:/builder/dl when building for ath79/generic.

Since this increase a few the storage needed, it is added as opt-in.
In addition, to enable this also for snapshots, a mechanism to auto delete old/lower version_codes
should be added, presumably linked to the environment.update part. [0]

This is the result of some tests made via the api at http://0.0.0.0:8000/docs

• with pre-pulled podman images
• building twice for the same device, adding or removing packages among the default_packages, eg. opkg, vim, dnsmasq

Tested running the dev server, building for ath79-generic-8dev_carambola2

• the first build completes in 0:01:58.630602s
• the second build takes 0:01:26.964493s

Tested with the podman-compose setup, building for ath79-nand-glinet_gl-xe300 [1]

• the first build completes in 0:01:55.503813s
• the second completes in 0:01:29.265000s

[0] Note: An alternative approach, would be to instruct apk (opkg)
to use the local instance of squid if enabled
for example adding this before the validate_manifest cmd make manifest...:

export http_proxy="http://10.89.0.1:3128"
sed -i 's|https|http|g' repositories

however in snapshots this apparently could not benefit of the squid cache without becoming error prone.

[1] Note: to complete the tests with podman I had to change these settings in asu/config.py,
because it seems that Path.cwd() when is read inside the container erroneously points to
/app/dl/23.05.5/ath79/generic instead of /home/openwrt/asu/dl/23.05.5/ath79/generic.
Maybe I missed a better option to do the same changes:

# public_path: Path = Path.cwd() / "public"
public_path: Path = Path('/home/openwrt/asu/public/')
# dl_path: Path = Path.cwd() / "dl"
dl_path: Path = Path('/home/openwrt/asu/dl/')

[aparcar]

Thanks this is a fine ideas, however some issues with the current approach:

• An evil ImageBilder or package could poison the cache (not again please)
• Two parallel workers sharing the same dir cause errors
• No automatic cleanup

I'd prefer to downgrade the IB repositories to HTTP and use squid, which should in turn automatically upgrade the connection to https.

This should solve all the above problems and is used anyway for snapshot Downloads.

Would you be willing to modify your PR to do that?

[aparcar]

I tested it and it worked as expected, would you mind adopting your PR?

diff --git a/asu/build.py b/asu/build.py
index bc29a90..d9e7e78 100644
--- a/asu/build.py
+++ b/asu/build.py
@@ -75,14 +75,15 @@ def build(build_request: BuildRequest, job=None):
                 .replace("{version}", build_request.version),
             }
         )
-        if settings.squid_cache:
-            environment.update(
-                {
-                    "UPSTREAM_URL": settings.upstream_url.replace("https", "http"),
-                    "use_proxy": "on",
-                    "http_proxy": "http://127.0.0.1:3128",
-                }
-            )
+
+    if settings.squid_cache:
+        environment.update(
+            {
+                "UPSTREAM_URL": settings.upstream_url.replace("https", "http"),
+                "use_proxy": "on",
+                "http_proxy": "http://127.0.0.1:3128",
+            }
+        )
 
     job.meta["imagebuilder_status"] = "container_setup"
     job.save_meta()
@@ -178,6 +179,7 @@ def build(build_request: BuildRequest, job=None):
         container, ["make", "info"]
     )
 
+
     job.meta["imagebuilder_status"] = "validate_revision"
     job.save_meta()
 
@@ -218,6 +220,15 @@ def build(build_request: BuildRequest, job=None):
     job.meta["imagebuilder_status"] = "validate_manifest"
     job.save_meta()
 
+    returncode, job.meta["stdout"], job.meta["stderr"] = run_cmd(
+        container, ["sed", "-i", "s|https|http|g", "repositories"]
+    )
+    returncode, job.meta["stdout"], job.meta["stderr"] = run_cmd(
+        container, ["sed", "-i", "s|https|http|g", "repositories.conf"]
+    )
+
     returncode, job.meta["stdout"], job.meta["stderr"] = run_cmd(
         container,
         [

[a-gave]

Hi sorry i did some tests and written almost the same stuff [0]

but I can't figure out quickly how to upgrade automatically to https with squid without 'trying' dockerizing an alpine with squid ssl-bump etc

but my setup currently fails also running ./setup.sh with http for snapshots, both with podman-compose and with dev server (giving me connection refused on 127.0.0.1:3128, probably squid is not configured properly)

let's close this and continue with your setup that is working!

[0]

diff --git a/asu/build.py b/asu/build.py
index bc29a90..8a675ac 100644
--- a/asu/build.py
+++ b/asu/build.py
@@ -75,14 +75,23 @@ def build(build_request: BuildRequest, job=None):
                 .replace("{version}", build_request.version),
             }
         )
-        if settings.squid_cache:
-            environment.update(
-                {
-                    "UPSTREAM_URL": settings.upstream_url.replace("https", "http"),
-                    "use_proxy": "on",
-                    "http_proxy": "http://127.0.0.1:3128",
-                }
-            )
+
+    if settings.squid_cache:
+        environment.update(
+            {
+                "UPSTREAM_URL": settings.upstream_url.replace("https", "http"),
+                "use_proxy": "on",
+                "http_proxy": "http://127.0.0.1:3128",
+                "repositories_file": (
+                    "repositories.conf"
+                    if (
+                        not build_request.version.lower().startswith("snapshot")
+                        and int(build_request.version[:2]) < 25
+                    )
+                    else "repositories"
+                ),
+            }
+        )
 
     job.meta["imagebuilder_status"] = "container_setup"
     job.save_meta()
@@ -218,6 +227,11 @@ def build(build_request: BuildRequest, job=None):
     job.meta["imagebuilder_status"] = "validate_manifest"
     job.save_meta()
 
+    if settings.squid_cache:
+        returncode, job.meta["stdout"], job.meta["stderr"] = run_cmd(
+            container, ["bash", "-c", "sed -i 's|https|http|g' ${repositories_file}"]
+        )
+
     returncode, job.meta["stdout"], job.meta["stderr"] = run_cmd(
         container,
         [