Merge pull request #2 from staticfloat/sf/precommand

Add more documentation, and `persist` mode defaults to `on`

README.md

@@ -7,7 +7,77 @@ Requires `julia` to be installed, see [the `julia` buildkite plugin](https://git
 We recommend running all buildkite agents within some kind of isolation sandbox, however if running inside of `Docker`, for this plugin to work, you must [set the container as `privileged`](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities).
 We recommend running the buildkite agent inside of a non-docker sandbox for ease of sandbox nesting; see this repository for more information.
 
-## Basic Usage
+## Usage
 
-First, create a rootfs and host it as a tarball somewhere.  The [`Sandbox.jl` repository](https://github.com/staticfloat/Sandbox.jl) contains a few, we will use the Debian-based `julia` and `python3` rootfs as an example.
+The purpose of this package is to allow users to pre-create a rootfs that contains their needed packages, then launch test suites, CI jobs, etc.. within that environment so that they do not have to continually `apt intsall` all of their dependencies within every step of their build pipelines.
 
+First, create a rootfs and host it as a tarball somewhere.  The [`Sandbox.jl` repository](https://github.com/staticfloat/Sandbox.jl) contains a few, we will use a minimal debian rootfs as an example.
+
+```yaml
+steps:
+  - label: "run code within a sandbox"
+    plugins:
+      # Install Julia v1.6+ to run sandbox
+      - JuliaCI/julia#v1:
+          version: 1.6
+      - staticfloat/sandbox:
+          rootfs_url: "https://github.com/staticfloat/Sandbox.jl/releases/download/debian-minimal-927c9e7f/debian_minimal.tar.gz"
+          rootfs_treehash: "b0fbf90920c3eca7d9e452d8651346172a835635"
+    commands: |
+      echo "this is running in a sandbox!"
+```
+
+The sandbox automatically gets access to a few host directories, such as the current repository checkout, the buildkite plugins directory, etc...
+Most Julia users will want to do something like launch their julia tests within an environment that has, e.g. `pdflatex` available.
+To do so, they would first create a rootfs, perhaps by building a `Dockerfile` with the following contents (these snippets adapted from [the `docker_build_example` in `Sandbox.jl`](https://github.com/staticfloat/Sandbox.jl/tree/main/contrib/docker_build_example)):
+
+```Dockerfile
+# Our base image will be debian
+FROM debian
+
+# Install some useful tools
+RUN apt update && apt install -y curl python3 texlive-full
+```
+
+They would then build it via `docker build -t texlive_rootfs .`.
+Once it is finished building, it can be flattened into an artifact using `Sandbox.export_docker_image()`:
+
+```julia
+using Pkg.Artifacts, Sandbox
+artifact_hash = create_artifact() do dir
+    Sandbox.export_docker_image("texlive_rootfs", dir; verbose=true)
+end
+@info("docker export complete, artifact hash: $(bytes2hex(artifact_hash))")
+```
+
+This artifact can then be packaged into a tarball and uploaded somewhere:
+```julia
+archive_artifact(artifact_hash, "/tmp/texlive_rootfs.tar.gz")
+```
+
+Note that `texlive-full` is quite large, and will generate a large artifact that will take quite a while to export and upload.
+Once the archived artifact is uploaded somewhere (such as an S3 bucket, or a github release), it is usable as a rootfs by listing its URL and treehash.
+
+For users that want to test on a variety of Julia versions (even those that are incompatible with `Sandbox.jl` itself; e.g. Julia v1.5 and earlier) pipelines will need to install Julia v1.6 first, and then within the sandbox, install the desired version of Julia.
+Note that all plugins after `sandbox` will magically be running inside of the sandbox itself, so Julia installation/testing of Julia packages etc.. can all happen from within the sandbox.
+We give here an example, using the `texlive-full` rootfs built in the previous steps, and running a different version of Julia from within the rootfs than is used on the outside to install `sandbox`:
+
+```yaml
+steps:
+  - label: "run tests within a sandbox"
+    plugins:
+      # Install Julia v1.6 to run Sandbox.jl
+      - JuliaCI/julia#v1:
+          version: 1.6
+      - staticfloat/sandbox:
+          rootfs_url: "https://julialang-buildkite.s3.amazonaws.com/rootfs_images/texlive_rootfs.tar.gz"
+          rootfs_treehash: "4748ee25bcb689727fb642eea56fd10c840d5094"
+          workspaces:
+            # Include `/cache` so that `julia` install can properly cache its Julia downloads
+            - "/cache:/cache"
+      # Once inside the sandbox, install a different version of Julia to run our tests
+      - JuliaCI/julia#v1:
+          version: 1.7
+          update_registry: false
+      - JuliaCI/julia-test#v1: ~
+```

docker-compose.yml

@@ -6,5 +6,9 @@ services:
       - ".:/plugin"
       # This just speeds up testing
       - "./tests/depot_cache:/root/.julia"
+      # Persistence testing requires a non-overlayfs home
+      - "./tests/sandbox_persistence:/sandbox_persistence"
+    environment:
+      - SANDBOX_PERSISTENCE_DIR=/sandbox_persistence
     # We need to run `Sandbox.jl` within this docker container
     privileged: true

hooks/post-command

@@ -1,5 +1,12 @@
 #!/bin/truebash
 
-# Un-do our craziness
+# Determine the persistence directory used and clear it out if it exists
+PERSIST_DIR=$(sed -n -E 's/.*--persist ([^ ]+).*/\1/p' </bin/bash 2>/dev/null)
+if [[ -d "${PERSIST_DIR}" ]]; then
+    echo "Clearing out temporary persistence directory ${PERSIST_DIR}"
+    rm -rf "${PERSIST_DIR}"
+fi
+
+# Un-do our crazy `bash` override
 echo "Removing sandbox /bin/bash overrride"
 mv /bin/truebash /bin/bash

hooks/pre-command

@@ -14,6 +14,10 @@ function die() {
     exit 1
 }
 
+if [[ -z "$(which julia 2>/dev/null)" ]]; then
+    die "Sandbox requires a Julia v1.6+ installation to function"
+fi
+
 
 if [[ ! -v "BUILDKITE_PLUGIN_SANDBOX_ROOTFS_URL" ]]; then
     die "Sandbox requires a rootfs tarball URL"
@@ -34,6 +38,3 @@ julia --project="${SANDBOX_REPO}/lib" -e "using Pkg; Pkg.instantiate()"
 echo "--- Installing bash override"
 cp -va /bin/bash /bin/truebash
 julia --project="${SANDBOX_REPO}/lib" "${SANDBOX_REPO}/lib/generate_sandboxed_bash.jl" "${BUILDKITE_PLUGIN_SANDBOX_SHELL:-/bin/bash}"
-
-echo "--- cat ${BUILDKITE_PLUGIN_SANDBOX_SHELL:-/bin/bash}"
-cat "${BUILDKITE_PLUGIN_SANDBOX_SHELL:-/bin/bash}"

lib/Manifest.toml

@@ -93,10 +93,10 @@ uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
 
 [[Sandbox]]
-deps = ["JLLWrappers", "LazyArtifacts", "Libdl", "Preferences", "Random", "SHA", "Scratch", "TOML", "Tar", "UserNSSandbox_jll"]
-git-tree-sha1 = "c8ff0fbea5cab2b89642221afe1b03fda1df0621"
+deps = ["LazyArtifacts", "Libdl", "Preferences", "Random", "SHA", "Scratch", "TOML", "Tar", "UserNSSandbox_jll"]
+git-tree-sha1 = "36cd9e7d56858dabcd6fd65a9414901443ceaea9"
 uuid = "9307e30f-c43e-9ca7-d17c-c2dc59df670d"
-version = "1.0.0"
+version = "1.0.1"
 
 [[Scratch]]
 deps = ["Dates"]

lib/generate_sandboxed_bash.jl

@@ -45,7 +45,7 @@ add_env_dir!(workspace_mappings, "BUILDKITE_BUILD_PATH")
 # Add in the plugins path, since we may want to run plugins within the sandbox
 add_env_dir!(workspace_mappings, "BUILDKITE_PLUGINS_PATH")
 
-# `/tmp` always gets mounted in, as some plugins want access to that
+# `/tmp` always gets mounted in, since buildkite writes hook wrappers out there!
 add_env_dir!(workspace_mappings, "TMPDIR"; default="/tmp")
 
 # Add user-specified workspaces (note they must all be absolute paths)
@@ -83,25 +83,26 @@ config = SandboxConfig(
     verbose,
     uid,
     gid,
+    persist=true,
 )
-with_executor() do exe
-    c = Sandbox.build_executor_command(exe, config, ``)
-    # Write out `/bin/bash` wrapper script that just invokes our `sandbox` executable.
-    open(ARGS[1], write=true) do io
-        println(io, """
-        #!/bin/truebash
-
-        # Don't sandbox `sandbox-buildkite-plugin` itself
-        if [[ "\${BUILDKITE_PLUGIN_NAME}" == "SANDBOX" ]]; then
-            exec /bin/truebash "\$@"
-        fi
-
-        # Ensure that PATH contains the bare minimum that any sane rootfs might need
-        export PATH="\${PATH}:/usr/local/bin:/usr/bin:/bin"
-
-        # Sandbox invocation
-        $(join(c.exec, " ")) /bin/bash "\$@"
-        """)
-    end
-    chmod(ARGS[1], 0o755)
+
+exe = UnprivilegedUserNamespacesExecutor()
+c = Sandbox.build_executor_command(exe, config, ``)
+# Write out `/bin/bash` wrapper script that just invokes our `sandbox` executable.
+open(ARGS[1], write=true) do io
+    println(io, """
+    #!/bin/truebash
+
+    # Don't sandbox `sandbox-buildkite-plugin` itself
+    if [[ "\${BUILDKITE_PLUGIN_NAME}" == "SANDBOX" ]]; then
+        exec /bin/truebash "\$@"
+    fi
+
+    # Ensure that PATH contains the bare minimum that any sane rootfs might need
+    export PATH="\${PATH}:/usr/local/bin:/usr/bin:/bin"
+
+    # Sandbox invocation
+    $(join(c.exec, " ")) /bin/bash "\$@"
+    """)
 end
+chmod(ARGS[1], 0o755)

plugin.yml

@@ -10,6 +10,12 @@ configuration:
       type: string
 
     # Allow mounting arbitrary read-write directories; specified as `host:sandbox`
+    # Note that the sandbox automatically gets access to:
+    # - `${BUILDKITE_BUILD_PATH}`: the repository checkout path
+    # - `${BUILDKITE_PLUGINS_PATH}`: the plugins directory
+    # - `${TMPDIR}`: the temporary directory, since many plugins require communicating through it
+    # - `buildkite-agent`: wherever `buildkite-agent` is located, it gets mapped in as `/usr/bin/buildkite-agent`.
+    # - `/etc/resolv.conf`: We assume you want to be able to resolve DNS queries.
     workspaces:
       type: array
 

tests/.gitignore

@@ -1 +1,2 @@
 depot_cache
+sandbox_persistence

tests/tests.bats

@@ -31,15 +31,19 @@ export BUILDKITE_PLUGIN_SANDBOX_VERBOSE=true
     export BUILDKITE_PLUGIN_SANDBOX_SHELL="${dir}/bash"
     export BUILDKITE_BUILD_PATH="${dir}"
 
-    # First, test out just plain printing
+    # Generate the bash wrapper
     run /plugin/hooks/pre-command
     assert_output --partial "--- Instantiating sandbox environment"
     assert_success
 
+    # Use the bash wrapper to run something
     [[ -f "${dir}/bash" ]]
     run "${dir}/bash" -c "echo hello world; echo foo > ${dir}/foo"
     assert_output --partial "hello world"
     assert_success
     [[ -f "${dir}/foo" ]]
     [[ "$(cat ${dir}/foo)" == "foo" ]]
+
+    # Test that we can extract the `--persist` directive
+    [[ -n "$(sed -n -E 's/.*--persist ([^ ]+).*/\1/p' <"${dir}/bash")" ]]
 }