tests: new approach to run tests in ubuntu and debian using snapd deb…

… from the repo (#14496) * tests: new approach to run tests in ubuntu and debian using snapd deb from the repo The idea of this change is to change de default behaviour for the deb pacage used in our spread tests. To simulate a more real scenario, and having snapd snap build from local, this change is adding as default a new scenario to have snapd installed from the deb repository. Then the deb package is saved in the same place as it was built in during the test, making easy to have both scenarios working together. It is a requirement also so be able to run the tests using a built snapd deb as we are currently doing. * Squashed 'tests/lib/external/snapd-testing-tools/' changes from 1c8efb77e1..33c1d672c6 33c1d672c6 update os.query test 5a428a71ea fix shellcheck 8b423d177a support opensuse 15.6 ab1fa59719 support tests.pkgs download 1e39b307b6 udpate image names for openstack 2a85365d51 remove support for fedora-38 and ubuntu mantic 112bc0a315 append in log-filter 1d4350a818 Make sure the tools are at the begining of the path 45c2f45004 address comments 8743ceaf8c tools: fix support for multi-arch packages on apt-based systems (#54) 33aa8d1e53 fix details in remote.retry test e43b9e314d rename test libraries to avoid static check errors in snapd 4ab8b00afb New spread-filter tool (#53) cdf5cfd47b Remove centos-7 support f3996cc3fa change the spread label 1e309f41c6 change how legacy parameter is determine in remote.pull c43c35f7e3 run remote refresh and wait-for for xenial (skip bionic) 5262d30da7 make sure the test jobs are executed in runners with the spread label cb74259b7a add openstack systems 0b41fd40d3 fix tests.pkgs on arch-linux 558e109793 run fedora-40 spread tests in openstack 6f6187416d fix list implementation b4a5439c9b added more type annotatios to log_helper 58da1e36c3 mypy cleaned 1ff651e680 update wording of remote.pull 18615b1667 just usc scp -O when ssh version is -ge 9 cc68c9868b Added type annotations for log-filter 66f90d10cd Adding -O to scp command to make it compatible in uc24 tests 496cb7b5b3 removing support for centos-8 f2eef30db4 Updated the log helper and log parser 5a375ebf73 Formatting for python utils d3eed3faa5 fix codespell in CODE_OF_CONDUCT.md 18bcca6b14 new log helper d60381fcd9 add run number to filtered filename 5dde2d67b8 consider the tests execution in main 6b9a3aabcc change filtered log name b2756aa579 default file is .filtered.log 500b9dace4 Fix tests workflow 45db26a3d2 fix shellcheck error in log-filter fe45c27b7d create a var to store filter params 5a9b66d7dc filter spread results 51f9b055af New tool used to filter the spread.log output b8d20c1d5b fix snaps.name test with correct siffix spelling f640ac72e3 Add missing test details f0754df304 Filter the error y debug output in log-parser fc10196efd Add suggestions to details 94ac5ffe58 Add details on tests 501578c719 add more checks in os.query to check is-core_xx e8929207ff fix os-query for ubuntu comparing with core 226114641f os.query won't check SPREAD_SYSTEM anymore to compare core systems b89ec98b23 use local variables in os.query tool dacfd81de9 fix is_core functions 1db5214d5f Improve the remote docs (#36) 2e4a3153a2 1 more comment 3a0fc57e1e add explanation about why we check for ( Do | Doing ) 4cf8e635bf fix os.query test after merge b89b4f8647 fix artifacts name d30cee6da0 Merge remote-tracking branch 'upstream/main' 5ef5dcbe8f Tests use artifacts in spread tests (#51) 555c43d2ab Support auto-refresh with Do instead of Doing 96c2b0c19c remove tests support for ubuntu 23.04 (EoL) 74082c0c34 Tests improve remote wait (#49) 5121bfb659 remove support for opensuse leap 15.4 (#48) 30df700d08 Add new systems support (#47) 1f08938925 Support check amazon linux version (#46) 43533bdd97 Change the exit value checking for test formats (#45) 3c88244c04 Update check-test-format to support a dir and a list of files (#44) 510d95f429 add extra check for error in auto-refresh detection function 3289d4031b Try open the log with latin-1 encoding when utf-8 is not working 9db785499f improved how the tools are waiting for system reboot 2a5c4414a3 fix shellcheck errors 5e7b63883d Fixes for osquery and tests pkgs (#43) 4c9145e2ac support reboot waiting for auto-refresh 45768f5188 show changes in unknown status after refresh 8013c30c2a Remove support for ubuntu 22.10 b32b80bf54 Fix remote.rait-for test in bionic 5675c625e9 Enable fedora 38 55f4471957 Support for new oss f2e88b357c New tool used to query spread json reports cacd35ede0 utils/spread-shellcheck: explain disabled warnings (#42) c82afb2dee Support --no-install-recommends parameter when installing dependencies with tests.pkgs b84eea92e2 spread-shellcheck: fix quotes in environment variables (#41) ab1e51c29f New comparison in os-query for core systems (#40) e5ae22a5d4 systemd units can be overwritten 63540b845a Fix error messages in remote pull and push 75e8a426a5 make sure the unit is removed in tests.systemd test 9089ff5c02 Update tests to use the new tests.systemd stop-unit 44ecd5e56a Move tests.systemd stop-units to stop-unit 01a2a83b4b Update tests.systemd to have stop units as systemd.sh 162e93bd35 update tests.systemd CLI options to be the same than retry command 14aa43a405 new feature to re-run failed spread tests (#39) 604cb782db Fix shellcheck in systemd tool bfc71082c8 Update the tests.systemd to allow parameters waiting for service status 8a2d0a99df Adding quiet tool and removing set +-x from tests.pkgs d90935d2a4 A comment explaining about the default values for wait-for 3232c5dba7 Add support for ubuntu 23.04 a7164fba07 remove fedora 35 support, add fedora 37 support 89b9eb5301 Update systems supported 92bb6a0664 Include snap-sufix in the snaps.name tool git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: 33c1d672c6512b23892481eeb013bd01e5eb389a * download snapd snap-confine and ubuntu-core-launcher * skip tests that trust on deb pkg testkeys * fix shellcheck * revert kill timeout * fix tests using snapd.tool and store-state test * new tests.info tool this tool is used to check for re-exec and to know the source of the snapd package * skip tests which are testing with re-exec:0 and snap is comming from the repo * update the systems where snapd comes from the repo * skip another test when re-exec:0 * Squashed 'tests/lib/external/snapd-testing-tools/' changes from 33c1d672c6..912131f2a6 912131f2a6 fix another shellcheck d27cc72346 update tests.pkgs to fix shellchecks git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: 912131f2a6a7d3c807c890d94fe56367aca67039 * Adjust failing test to new re-exec scenario * fix other 2 tests using correct paths * Squashed 'tests/lib/external/snapd-testing-tools/' changes from 912131f2a6..bae7faf8cb bae7faf8cb Update the quoting used to download packages git-subtree-dir: tests/lib/external/snapd-testing-tools git-subtree-split: bae7faf8cb3f04296bac9552ac0d1aa89d4e9139 * some tests fixes Changes considering the changes done in the re-exec workflow * New set of test fixes Also we are inclugin some minor changes to address review comments * update tests info command to check re-execution * fix uc16 which preparing * check SPREAD_SYSTEM is not core in tests.info This is needed because the check in prepare-restore is done before we have an ubuntu-core image * use .skip in tests to determine if a phase has to be skipped Use snap-mount-dir to determine where snaps are mounted * skip updating core snap when the snap pkg is not built from local * save installed core snap when snap deb is used from the repo * copy also .orig core snap * change how we copy the core snap * change the snapname used for core snap * skip saving the core snap * Fix error using snapcraft for testing in arm * make sure fips tests dont run with reexec=0 and snapd.deb from the repo * simplify how fips tests are skipped * fix indentation on test * Make sure we save core snap in TESTSTMP/core_snap * use tests.exec tool to skip tests * updated failing tests ude to new re-execution mechanism * fix shellcheck on test preseed-lxd * Fix tests Fix tests that needed to be updated after snapd snap is build from current * fix how the current snapd snap is found * updated the name of the command is-snapd-pkg-repo
canonical · Oct 10, 2024 · f63a4a8 · f63a4a8
1 parent 2e0eba6
commit f63a4a8
Show file tree

Hide file tree

Showing 47 changed files with 467 additions and 157 deletions.
diff --git a/spread.yaml b/spread.yaml
@@ -82,8 +82,12 @@ environment:
     SNAPD_NO_MEMORY_LIMIT: '$(HOST: echo "${SPREAD_SNAPD_NO_MEMORY_LIMIT:-}")'
 
     SNAPD_PUBLISHED_VERSION: '$(HOST: echo "$SPREAD_SNAPD_PUBLISHED_VERSION")'
+    # Use the snapd package from the repository when possible
+    SNAPD_DEB_FROM_REPO: '$(HOST: echo "${SPREAD_SNAPD_DEB_FROM_REPO:-true}")'
     # Build and use snapd from current branch
     BUILD_SNAPD_FROM_CURRENT: '$(HOST: echo "${SPREAD_BUILD_SNAPD_FROM_CURRENT:-true}")'
+    # Directory where the built snapd snaps and other assets are stored
+    SNAPD_WORK_DIR: '$(HOST: echo "${SPREAD_SNAPD_WORK_DIR:-/tmp/work-dir}")'
 
     # Directory where the nested images and test assets are stored
     NESTED_WORK_DIR: '$(HOST: echo "${NESTED_WORK_DIR:-/tmp/work-dir}")'
@@ -1430,6 +1434,10 @@ suites:
             "$TESTSLIB"/prepare-restore.sh --prepare-suite
         prepare-each: |
             "$TESTSLIB"/prepare-restore.sh --prepare-suite-each
+            # When snapd.deb is from the repo, we just need to test when re-exec is enabled
+            if [ "$SNAP_REEXEC" = "0" ] && tests.info is-snapd-from-archive; then
+                tests.exec skip-test "When snapd.deb is from the repo, we just need to test when re-exec is enabled" && exit 0
+            fi
         restore-each: |
             "$TESTSLIB"/prepare-restore.sh --restore-suite-each
         restore: |

diff --git a/tests/bin/tests.exec b/tests/bin/tests.exec
@@ -0,0 +1 @@
+../lib/tools/tests.exec
diff --git a/tests/bin/tests.info b/tests/bin/tests.info
@@ -0,0 +1 @@
+../lib/tools/tests.info
diff --git a/tests/lib/prepare-restore.sh b/tests/lib/prepare-restore.sh
@@ -587,7 +587,12 @@ prepare_project() {
     # go mod runs as root and will leave strange permissions
     chown test:test -R "$SPREAD_PATH"
 
-    if [ "$BUILD_SNAPD_FROM_CURRENT" = true ]; then
+    # We are testing snapd snap on top of snapd from the archive
+    # of the tested distribution. Download snapd and snap-confine
+    # as they exist in the archive for further use.
+    if tests.info is-snapd-from-archive; then
+        ( cd "${GOHOME}" && tests.pkgs download snapd snap-confine)
+    elif [ "$BUILD_SNAPD_FROM_CURRENT" = true ]; then
         case "$SPREAD_SYSTEM" in
             ubuntu-*|debian-*)
                 build_deb

diff --git a/tests/lib/prepare.sh b/tests/lib/prepare.sh
@@ -9,8 +9,6 @@ set -eux
 # shellcheck source=tests/lib/state.sh
 . "$TESTSLIB/state.sh"
 
-: "${WORK_DIR:=/tmp/work-dir}"
-
 disable_kernel_rate_limiting() {
     # kernel rate limiting hinders debugging security policy so turn it off
     echo "Turning off kernel rate-limiting"
@@ -154,6 +152,24 @@ setup_experimental_features() {
     fi
 }
 
+save_installed_core_snap() {
+    local target_dir="${1-}"
+
+    SNAP_MOUNT_DIR="$(os.paths snap-mount-dir)"
+    core="$(readlink -f "$SNAP_MOUNT_DIR"/core/current)"
+    snap="$(mount | awk -v core="$core" '{ if ($3 == core) print $1 }' | head -n1)"
+    snap_name="$(basename "$snap")"
+
+    # make a copy for later use
+    if [ -n "$target_dir" ]; then
+        mkdir -p "$target_dir"
+
+        cp -av "$snap" "${target_dir}/${snap_name}"
+        cp "$snap" "${target_dir}/${snap_name}.orig"
+    fi
+}
+
+
 # update_core_snap_for_classic_reexec modifies the core snap for snapd re-exec
 # by injecting binaries from the installed snapd deb built from our modified code.
 # $1: directory where updated core snap should be copied (optional)
@@ -175,8 +191,8 @@ update_core_snap_for_classic_reexec() {
     LIBEXEC_DIR="$(os.paths libexec-dir)"
 
     # First of all, unmount the core
-    core="$(readlink -f "$SNAP_MOUNT_DIR/core/current" || readlink -f "$SNAP_MOUNT_DIR/ubuntu-core/current")"
-    snap="$(mount | grep " $core" | head -n 1 | awk '{print $1}')"
+    core="$(readlink -f "$SNAP_MOUNT_DIR"/core/current)"
+    snap="$(mount | awk -v core="$core" '{ if ($3 == core) print $1 }' | head -n1)"
     umount --verbose "$core"
 
     # Now unpack the core, inject the new snap-exec/snapctl into it
@@ -401,15 +417,16 @@ prepare_classic() {
     # This also prevents snapd from automatically installing snapd snap as
     # prerequisite for installing any non-base snap introduced in PR#14173.
     if snap list snapd ; then
-	    snap info snapd
-	    echo "Error: not expecting snapd snap to be installed"
-	    exit 1
+        snap info snapd
+        echo "Error: not expecting snapd snap to be installed"
+        exit 1
     else
-	    build_dir="$WORK_DIR/snapd_snap_for_classic"
-	    rm -rf "$build_dir"
-	    mkdir -p "$build_dir"
-	    build_snapd_snap "$build_dir"
-	    snap install --dangerous "$build_dir/"snapd_*.snap
+        build_dir="$SNAPD_WORK_DIR/snapd_snap_for_classic"
+        rm -rf "$build_dir"
+        mkdir -p "$build_dir"
+        build_snapd_snap "$build_dir"
+        snap install --dangerous "$build_dir/"snapd_*.snap
+        snap wait system seed.loaded
     fi
     snap list snapd
 
@@ -450,10 +467,17 @@ prepare_classic() {
 
         snap list | grep core
 
-        systemctl stop snapd.{service,socket}
-        # repack and also make a side copy of the core snap
-        update_core_snap_for_classic_reexec "$TESTSTMP/core_snap"
-        systemctl start snapd.{service,socket}
+        # With reexec, and on classic, the snapd snap is preferred over the core snap for reexecution target,
+        # so to be as close as possible to the actual real life scenarios, we only update the snapd snap.
+        # The tests alreday ensure that snapd snap is installed.
+        if tests.info is-snapd-from-archive; then
+            save_installed_core_snap "$TESTSTMP/core_snap"
+        else
+            systemctl stop snapd.{service,socket}
+            # repack and also make a side copy of the core snap
+            update_core_snap_for_classic_reexec "$TESTSTMP/core_snap"
+            systemctl start snapd.{service,socket}
+        fi
 
         prepare_reexec_override
         prepare_memory_limit_override
@@ -523,7 +547,7 @@ build_snapd_snap() {
     local snapd_snap_cache
     TARGET="${1}"
 
-    snapd_snap_cache="$WORK_DIR/snapd_snap"
+    snapd_snap_cache="$SNAPD_WORK_DIR/snapd_snap"
     mkdir -p "${snapd_snap_cache}"
     for snap in "${snapd_snap_cache}"/snapd_*.snap; do
         if ! [ -f "${snap}" ]; then
@@ -547,6 +571,7 @@ build_snapd_snap() {
                         ;;
                 esac
                 [ -d "${TARGET}" ] || mkdir -p "${TARGET}"
+                touch "${PROJECT_PATH}"/test-build
                 chmod -R go+r "${PROJECT_PATH}/tests"
                 # TODO: run_snapcraft does not currently guarantee or check the required version for building snapd
                 run_snapcraft --use-lxd --verbosity quiet --output="snapd_from_snapcraft.snap"
@@ -565,7 +590,7 @@ build_snapd_snap_with_run_mode_firstboot_tweaks() {
 
     TARGET="${1}"
 
-    snapd_snap_cache="$WORK_DIR/snapd_snap_with_tweaks"
+    snapd_snap_cache="$SNAPD_WORK_DIR/snapd_snap_with_tweaks"
     mkdir -p "${snapd_snap_cache}"
     for snap in "${snapd_snap_cache}"/snapd_*.snap; do
         if [ -f "${snap}" ]; then
@@ -1192,7 +1217,7 @@ setup_reflash_magic() {
         # FIXME: install would be better but we don't have dpkg on
         #        the image
         # unpack our freshly build snapd into the new snapd snap
-        dpkg-deb -x "$SPREAD_PATH"/../snapd_*.deb "$UNPACK_DIR"
+        dpkg-deb -x "$GOHOME"/snapd_*.deb "$UNPACK_DIR"
         # Debian packages don't carry permissions correctly and we use
         # post-inst hooks to fix that on classic systems. Here, as a special
         # case, fix the void directory we just unpacked.

diff --git a/tests/lib/tools/snapd.tool b/tests/lib/tools/snapd.tool
@@ -47,8 +47,15 @@ main() {
 		esac
 	done
 
-	LIBEXEC_DIR="$(os.paths libexec-dir)"
-	exec "$LIBEXEC_DIR/snapd/$tool" "$@"
+	# Use the tool from the snapd snap when re-exec is enabled
+	# Otherwise use the tool from the distro installation
+	if tests.info is-reexec-in-use; then
+		MOUNT_DIR="$(os.paths snap-mount-dir)"
+		exec "$MOUNT_DIR"/snapd/current/usr/lib/snapd/"$tool" "$@"
+	else
+		LIBEXEC_DIR="$(os.paths libexec-dir)"
+		exec "$LIBEXEC_DIR/snapd/$tool" "$@"
+	fi
 }
 
 main "$@"
diff --git a/tests/lib/tools/tests.exec b/tests/lib/tools/tests.exec
@@ -0,0 +1,58 @@
+#!/bin/bash
+
+set -eu
+
+show_help() {
+    echo "usage: tests.exec skip-test [MSG]"
+    echo "usage: tests.exec is-skipped"
+    echo
+    echo "Supported commands:"
+    echo "    skip-test:  indicates the test has to be skipped and saves the raeson"
+    echo "    is-skipped: check if the test has to be skipped and prints the raeson"
+}
+
+skip_test() {
+    local raeson="${1:-}"
+    echo "$raeson" > tests.exec
+}
+
+is_skipped() {
+    if [ -f tests.exec ]; then
+        echo "skip raeson: $(cat tests.exec)"
+        return 0
+    fi
+    return 1
+}
+
+main() {
+    if [ $# -eq 0 ]; then
+        show_help
+        exit 0
+    fi
+
+    local subcommand="$1"
+    local action=
+    while [ $# -gt 0 ]; do
+        case "$subcommand" in
+            -h|--help)
+                show_help
+                exit 0
+                ;;
+            *)
+                action=$(echo "$subcommand" | tr '-' '_')
+                shift
+                break
+                ;;
+        esac
+    done
+
+    if [ -z "$(declare -f "$action")" ]; then
+        echo "tests.exec: no such command: $subcommand"
+        show_help
+        exit 1
+    fi
+
+    "$action" "$@"
+}
+
+main "$@"
diff --git a/tests/lib/tools/tests.info b/tests/lib/tools/tests.info
@@ -0,0 +1,62 @@
+#!/bin/bash
+
+set -eu
+
+show_help() {
+    echo "usage: tests.info <CMD>"
+    echo
+    echo "Supported commands:"
+    echo "    is-snapd-from-archive:  indicates when the snapd package is from the repository"
+    echo "    is-reexec-enabled:      indicates re-execution has been explicitly enabled through the env"
+    echo "    is-reexec-in-use:       indicates re-execution to the snapd snap is being effectively performed"
+}
+
+# ubuntu 14.04: does not support apt download which is used to get the packages from the archive.
+# ubuntu-core: it is built based on the snapd.deb file, so it requires the deb is created from current.
+is_snapd_from_archive() {
+    [ "$SNAPD_DEB_FROM_REPO" = true ] && \
+    [ -n "$(command -v apt)" ] && \
+    [[ "$SPREAD_SYSTEM" != ubuntu-core-* ]] && \
+    not os.query is-trusty
+}
+
+is_reexec_enabled() {
+    snap debug execution snap | grep -q "is-reexec-enabled: true"
+}
+
+is_reexec_in_use() {
+    snap debug execution snap | grep -q "is-reexecd: true"
+}
+
+main() {
+    if [ $# -eq 0 ]; then
+        show_help
+        exit 0
+    fi
+
+    local subcommand="$1"
+    local action=
+    while [ $# -gt 0 ]; do
+        case "$subcommand" in
+            -h|--help)
+                show_help
+                exit 0
+                ;;
+            *)
+                action=$(echo "$subcommand" | tr '-' '_')
+                shift
+                break
+                ;;
+        esac
+    done
+
+    if [ -z "$(declare -f "$action")" ]; then
+        echo "tests.info: no such command: $subcommand"
+        show_help
+        exit 1
+    fi
+
+    "$action" "$@"
+}
+
+main "$@"
diff --git a/tests/main/classic-custom-device-reg/task.yaml b/tests/main/classic-custom-device-reg/task.yaml
@@ -15,9 +15,9 @@ environment:
     SEED_DIR: /var/lib/snapd/seed
 
 prepare: |
-    if [ "$TRUST_TEST_KEYS" = "false" ]; then
-        echo "This test needs test keys to be trusted"
-        exit
+    # In this scenario, the keys from the snapd pkg are used
+    if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-from-archive; then
+        tests.exec skip-test "This test needs test keys to be trusted" && exit 0
     fi
 
     snap pack "$TESTSLIB/snaps/classic-gadget"
@@ -52,20 +52,15 @@ prepare: |
     systemd-run --unit fakedevicesvc fakedevicesvc localhost:11029
 
 restore: |
-    if [ "$TRUST_TEST_KEYS" = "false" ]; then
-        echo "This test needs test keys to be trusted"
-        exit
-    fi
+    tests.exec is-skipped && exit 0
+
     systemctl stop snapd.service snapd.socket fakedevicesvc
 
     rm -rf "$SEED_DIR"
     systemctl start snapd.socket snapd.service
 
 execute: |
-    if [ "$TRUST_TEST_KEYS" = "false" ]; then
-        echo "This test needs test keys to be trusted"
-        exit
-    fi
+    tests.exec is-skipped && exit 0
 
     #shellcheck source=tests/lib/core-config.sh
     . "$TESTSLIB"/core-config.sh

diff --git a/tests/main/classic-firstboot/task.yaml b/tests/main/classic-firstboot/task.yaml
@@ -14,9 +14,9 @@ environment:
     SEED_DIR: /var/lib/snapd/seed
 
 prepare: |
-    if [ "$TRUST_TEST_KEYS" = "false" ]; then
-        echo "This test needs test keys to be trusted"
-        exit
+    # In this scenario, the keys from the snapd pkg are used
+    if [ "$TRUST_TEST_KEYS" = "false" ] || tests.info is-snapd-from-archive; then
+        tests.exec skip-test "This test needs test keys to be trusted" && exit 0
     fi
 
     snap pack "$TESTSLIB/snaps/basic"
@@ -51,19 +51,13 @@ prepare: |
     cp ./test-snapd-service_1.0_all.snap "$SEED_DIR/snaps/test-snapd-service.snap"
 
 restore: |
-    if [ "$TRUST_TEST_KEYS" = "false" ]; then
-        echo "This test needs test keys to be trusted"
-        exit
-    fi
+    tests.exec is-skipped && exit 0
 
     rm -rf "$SEED_DIR"
     systemctl start snapd.socket snapd.service
 
 execute: |
-    if [ "$TRUST_TEST_KEYS" = "false" ]; then
-        echo "This test needs test keys to be trusted"
-        exit
-    fi
+    tests.exec is-skipped && exit 0
 
     echo "Start the daemon with an empty state, this will make it import "
     echo "assertions from the $SEED_DIR/assertions subdirectory and "