Refactor download role (kubernetes-sigs#5697)

* download file

* download containers

* fix push image to nodes

* pull if none image on host

* fix

* improve docker image tag checks.
do not pull already cached images

* rebase fix merge conflict

* add support download_run_once when upgrade and scale cluster
add some test with download_run_once

* set default values to temp flag for every download cycle

* add save,load abilty for containerd and crio when download_run_once=true

* return redefine image save/load command to  set_docker_image_facts.yml

* move set command to set_container_facts

* ctr in containerd_bin_dir

* fix order of ctr image export arguments

* temporary disable download_run_once for containerd and crio
due containerd/containerd#4075

* remove unused files

* fix strict yaml linter warning and errors

* refactor logical conditions to pull and cache container images

* remove comment due lint check

* document role

* remove image_load_on_localhost, because cached images are always loaded to docker on remote sites

* remove XXX from debug output

.gitlab-ci/packet.yml

@@ -68,6 +68,11 @@ packet_ubuntu18-flannel-containerd:
   extends: .packet
   when: manual
 
+packet_ubuntu18-flannel-containerd-once:
+  stage: deploy-part2
+  extends: .packet
+  when: manual
+
 packet_debian9-macvlan-sep:
   stage: deploy-part2
   extends: .packet
@@ -80,6 +85,13 @@ packet_debian9-calico-upgrade:
   variables:
     UPGRADE_TEST: graceful
 
+packet_debian9-calico-upgrade-once:
+  stage: deploy-part2
+  extends: .packet
+  when: on_success
+  variables:
+    UPGRADE_TEST: graceful
+
 packet_debian10-containerd:
   stage: deploy-part2
   extends: .packet
@@ -90,6 +102,11 @@ packet_centos7-calico-ha:
   extends: .packet
   when: manual
 
+packet_centos7-calico-ha-once-localhost:
+  stage: deploy-part2
+  extends: .packet
+  when: manual
+
 packet_centos7-kube-ovn:
   stage: deploy-part2
   extends: .packet

docs/downloads.md

@@ -13,11 +13,13 @@ There is also a "pull once, push many" mode as well:
 
 NOTE: When `download_run_once` is true and `download_localhost` is false, all downloads will be done on the delegate node, including downloads for container images that are not required on that node. As a consequence, the storage required on that node will probably be more than if download_run_once was false, because all images will be loaded into the docker instance on that node, instead of just the images required for that node.
 
+:warning: [`download_run_once: true` support only for `container_manager: docker`](https://github.com/containerd/containerd/issues/4075) :warning:
+
 On caching:
 
 * When `download_run_once` is `True`, all downloaded files will be cached locally in `download_cache_dir`, which defaults to `/tmp/kubespray_cache`. On subsequent provisioning runs, this local cache will be used to provision the nodes, minimizing bandwidth usage and improving provisioning time. Expect about 800MB of disk space to be used on the ansible node for the cache. Disk space required for the image cache on the kubernetes nodes is a much as is needed for the largest image, which is currently slightly less than 150MB.
-* By default, if `download_run_once` is false, kubespray will not retrieve the downloaded images and files from the remote node to the local cache, or use that cache to pre-provision those nodes. To force the use of the cache, set `download_force_cache` to `True`.
-* By default, cached images that are used to pre-provision the remote nodes will be deleted from the remote nodes after use, to save disk space. Setting download_keep_remote_cache will prevent the files from being deleted. This can be useful while developing kubespray, as it can decrease provisioning times. As a consequence, the required storage for images on the remote nodes will increase from 150MB to about 550MB, which is currently the combined size of all required container images.
+* By default, if `download_run_once` is false, kubespray will not retrieve the downloaded images and files from the download delegate node to the local cache, or use that cache to pre-provision those nodes. If you have a full cache with container images and files and you don’t need to download anything, but want to use a cache - set `download_force_cache` to `True`.
+* By default, cached images that are used to pre-provision the remote nodes will be deleted from the remote nodes after use, to save disk space. Setting `download_keep_remote_cache` will prevent the files from being deleted. This can be useful while developing kubespray, as it can decrease provisioning times. As a consequence, the required storage for images on the remote nodes will increase from 150MB to about 550MB, which is currently the combined size of all required container images.
 
 Container images and binary files are described by the vars like ``foo_version``,
 ``foo_download_url``, ``foo_checksum`` for binaries and ``foo_image_repo``,

roles/download/defaults/main.yml

@@ -476,7 +476,13 @@ dashboard_image_repo: "{{ gcr_image_repo }}/google_containers/kubernetes-dashboa
 dashboard_image_tag: "v1.10.1"
 
 image_pull_command: "{{ docker_bin_dir }}/docker pull"
-image_info_command: "{{ docker_bin_dir }}/docker images -q | xargs {{ docker_bin_dir }}/docker inspect -f \"{{ '{{' }} if .RepoTags {{ '}}' }}{{ '{{' }} (index .RepoTags 0) {{ '}}' }}{{ '{{' }} end {{ '}}' }}{{ '{{' }} if .RepoDigests {{ '}}' }},{{ '{{' }} (index .RepoDigests 0) {{ '}}' }}{{ '{{' }} end {{ '}}' }}\" | tr '\n' ','"
+image_save_command: "{{ docker_bin_dir }}/docker save {{ image_reponame }} | gzip -{{ download_compress }} > {{ image_path_final }}"
+image_load_command: "{{ docker_bin_dir }}/docker load < {{ image_path_final }}"
+image_info_command: "{{ docker_bin_dir }}/docker images -q | xargs {{ docker_bin_dir }}/docker inspect -f \"{{ '{{' }} if .RepoTags {{ '}}' }}{{ '{{' }} (join .RepoTags \\\",\\\") {{ '}}' }}{{ '{{' }} end {{ '}}' }}{{ '{{' }} if .RepoDigests {{ '}}' }},{{ '{{' }} (join .RepoDigests \\\",\\\") {{ '}}' }}{{ '{{' }} end {{ '}}' }}\" | tr '\n' ','"
+
+image_pull_command_on_localhost: "{{ docker_bin_dir }}/docker pull"
+image_save_command_on_localhost: "{{ docker_bin_dir }}/docker save {{ image_reponame }} | gzip -{{ download_compress }} > {{ image_path_cached }}"
+image_info_command_on_localhost: "{{ docker_bin_dir }}/docker images"
 
 downloads:
   netcheck_server:

roles/download/tasks/check_pull_required.yml

@@ -6,19 +6,17 @@
 # nginx:1.15,gcr.io/google-containers/kube-proxy:v1.14.1,gcr.io/google-containers/kube-proxy@sha256:44af2833c6cbd9a7fc2e9d2f5244a39dfd2e31ad91bf9d4b7d810678db738ee9,gcr.io/google-containers/kube-apiserver:v1.14.1,etc...
 - name: check_pull_required |  Generate a list of information about the images on a node
   shell: "{{ image_info_command }}"
-  delegate_to: "{{ download_delegate if download_run_once else inventory_hostname }}"
   no_log: true
   register: docker_images
   failed_when: false
   changed_when: false
   check_mode: no
-  become: "{{ not download_localhost }}"
   when: not download_always_pull
 
 - name: check_pull_required | Set pull_required if the desired image is not yet loaded
   set_fact:
     pull_required: >-
-      {%- if image_reponame in docker_images.stdout.split(',') %}false{%- else -%}true{%- endif -%}
+      {%- if image_reponame | regex_replace('^docker\.io/(library/)?','') in docker_images.stdout.split(',') %}false{%- else -%}true{%- endif -%}
   when: not download_always_pull
 
 - name: check_pull_required | Check that the local digest sha256 corresponds to the given image tag

roles/download/tasks/download_container.yml

@@ -1,19 +1,26 @@
 ---
-- name: container_download | Make download decision if pull is required by tag or sha256
-  include_tasks: set_docker_image_facts.yml
-  when:
-    - download.enabled
-    - download.container
-  tags:
-    - facts
-
 - block:
+    - name: set default values for flag variables
+      set_fact:
+        image_is_cached: false
+        image_changed: false
+        pull_required: "{{ download_always_pull }}"
+      tags:
+        - facts
+
     - name: download_container | Set a few facts
       import_tasks: set_container_facts.yml
-      run_once: "{{ download_run_once }}"
       tags:
         - facts
 
+    - name: download_container | Prepare container download
+      include_tasks: check_pull_required.yml
+      when:
+        - not download_always_pull
+
+    - debug:
+        msg: "Pull {{ image_reponame }} required is: {{ pull_required }}"
+
     - name: download_container | Determine if image is in cache
       stat:
         path: "{{ image_path_cached }}"
@@ -27,56 +34,22 @@
 
     - name: download_container | Set fact indicating if image is in cache
       set_fact:
-        image_is_cached: "{{ cache_image.stat.exists | default(false) }}"
+        image_is_cached: "{{ cache_image.stat.exists }}"
       tags:
         - facts
       when:
         - download_force_cache
 
-    - name: download_container | Upload image to node if it is cached
-      synchronize:
-        src: "{{ image_path_cached }}"
-        dest: "{{ image_path_final }}"
-        use_ssh_args: "{{ has_bastion | default(false) }}"
-        mode: push
-      delegate_facts: no
-      register: upload_image
-      failed_when: not upload_image
-      run_once: "{{ download_run_once }}"
-      until: upload_image is succeeded
-      retries: 4
-      delay: "{{ retry_stagger | random + 3 }}"
+    - name: Stop if image not in cache on ansible host when download_force_cache=true
+      assert:
+        that: not image_is_cached
+        msg: "Image cache file {{ image_path_cached }} not found for {{ image_reponame }} on localhost"
       when:
         - download_force_cache
-        - image_is_cached
-        - not download_localhost
-        - ansible_os_family not in ["CoreOS", "Coreos", "Container Linux by CoreOS", "Flatcar", "Flatcar Container Linux by Kinvolk"]
+        - not download_run_once
 
-    - name: download_container | Load image into docker
-      shell: "{{ docker_bin_dir }}/docker load < {{ image_path_cached if download_localhost else image_path_final }}"
-      delegate_to: "{{ download_delegate if download_run_once else inventory_hostname }}"
-      run_once: "{{ download_run_once }}"
-      register: container_load_status
-      failed_when: container_load_status is failed
-      become: "{{ user_can_become_root | default(false) or not (download_run_once and download_localhost) }}"
-      when:
-        - download_force_cache
-        - image_is_cached
-        - ansible_os_family not in ["CoreOS", "Coreos", "Container Linux by CoreOS", "Flatcar", "Flatcar Container Linux by Kinvolk"]
-
-    - name: download_container | Prepare container download
-      include_tasks: check_pull_required.yml
-      run_once: "{{ download_run_once }}"
-      when:
-        - not download_always_pull
-
-    - debug:
-        msg: "XXX Pull required is: {{ pull_required }}"
-
-    # NOTE: Pre-loading docker images will not prevent 'docker pull' from re-downloading the layers in that image
-    # if a pull is forced. This is a known issue with docker. See https://github.com/moby/moby/issues/23684
     - name: download_container | Download image if required
-      command: "{{ image_pull_command }} {{ image_reponame }}"
+      command: "{{ image_pull_command_on_localhost if download_localhost else image_pull_command }} {{ image_reponame }}"
       delegate_to: "{{ download_delegate if download_run_once else inventory_hostname }}"
       delegate_facts: yes
       run_once: "{{ download_run_once }}"
@@ -86,52 +59,62 @@
       retries: 4
       become: "{{ user_can_become_root | default(false) or not download_localhost }}"
       when:
-        - pull_required | default(download_always_pull)
-
-    # NOTE: image_changed is only valid if a pull is was needed or forced.
-    - name: download_container | Check if image changed
-      set_fact:
-        image_changed: "{{ true if pull_task_result.stdout is defined and not 'up to date' in pull_task_result.stdout else false }}"
-      run_once: true
-      when:
-        - download_force_cache
-      tags:
-        - facts
+        - pull_required
+        - not image_is_cached
 
     - name: download_container | Save and compress image
-      shell: "{{ docker_bin_dir }}/docker save {{ image_reponame }} | gzip -{{ download_compress }} > {{ image_path_cached if download_localhost else image_path_final }}"
-      delegate_to: "{{ download_delegate if download_run_once else inventory_hostname }}"
+      shell: "{{ image_save_command_on_localhost if download_localhost else image_save_command }}"
+      delegate_to: "{{ download_delegate }}"
       delegate_facts: no
       register: container_save_status
       failed_when: container_save_status.stderr
       run_once: true
       become: "{{ user_can_become_root | default(false) or not download_localhost }}"
       when:
-        - download_force_cache
-        - not image_is_cached or (image_changed | default(true))
-        - ansible_os_family not in ["CoreOS", "Coreos", "Container Linux by CoreOS", "Flatcar", "Flatcar Container Linux by Kinvolk"]
+        - not image_is_cached
+        - download_run_once
 
     - name: download_container | Copy image to ansible host cache
       synchronize:
         src: "{{ image_path_final }}"
         dest: "{{ image_path_cached }}"
         use_ssh_args: "{{ has_bastion | default(false) }}"
         mode: pull
-      delegate_facts: no
       when:
-        - download_force_cache
+        - not image_is_cached
+        - download_run_once
         - not download_localhost
         - download_delegate == inventory_hostname
-        - not image_is_cached or (image_changed | default(true))
-        - ansible_os_family not in ["CoreOS", "Coreos", "Container Linux by CoreOS", "Flatcar", "Flatcar Container Linux by Kinvolk"]
+
+    - name: download_container | Upload image to node if it is cached
+      synchronize:
+        src: "{{ image_path_cached }}"
+        dest: "{{ image_path_final }}"
+        use_ssh_args: "{{ has_bastion | default(false) }}"
+        mode: push
+      delegate_facts: no
+      register: upload_image
+      failed_when: not upload_image
+      until: upload_image is succeeded
+      retries: 4
+      delay: "{{ retry_stagger | random + 3 }}"
+      when:
+        - pull_required
+        - download_force_cache
+
+    - name: download_container | Load image into docker
+      shell: "{{ image_load_command }}"
+      register: container_load_status
+      failed_when: container_load_status is failed
+      when:
+        - pull_required
+        - download_force_cache
 
     - name: download_container | Remove container image from cache
       file:
         state: absent
         path: "{{ image_path_final }}"
       when:
         - not download_keep_remote_cache
-        - ansible_os_family not in ["CoreOS", "Coreos", "Container Linux by CoreOS", "Flatcar", "Flatcar Container Linux by Kinvolk"]
-
   tags:
     - download