Mrc-4908 k8s setup

.gitignore

@@ -1 +1,2 @@
 apache/ssl
+k8s/ssl

README.md

@@ -2,60 +2,22 @@
 
 This is a temporary repo designed to give an easy-to-understand deployment of the apache / haproxy / shiny stack; it does not include configurable applications or anything like that. See it online at https://shiny-dev.dide.ic.ac.uk (DIDE network only).
 
-## The components
 
-Some of these are docker images, and they are not set to pull, so you will want to rebuild. All build very quickly.
+## Running in Kubernetes
 
-### apache
+The following is guide to run the shiny server in kubernetes. They are based on running a Kind cluster. The configuration may
+nee to be adjusted for other k8s clusters.
 
-Running an unmodified httpd container (previously was 2.4, we'll update once we know this works). The configuration ([`apache/httpd.conf`](httpd/httpd.conf)) and certificates (`apache/ssl`) will be read-only mounted into the container. You need to fetch the ssl key and certificate, run `./apache/configure_ssl` to do this (only needs to be done if they change or if the `ssl` directory is deleted)
+### Prerequisites
 
-### haproxy
+A production kubernetes cluster using k3s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistant volume. 
 
-Build the image with `./haproxy/build` which builds `mrcide/haproxy` with a configuration that can be seen in [`haproxy/haproxy.cfg`](haproxy/haproxy.cfg) and some utilities which enable some degree of dynamic scaling of shiny servers.
+Run `start-k8s-shiny <env>` to run the shint server in k8s.
 
-### shiny
 
-A lightly modified version of the official shiny container; the original version was more extensively modified
+#### Teardown
 
-### apps
+Run the following: 
 
-Some applications (copied over from the original deployment are in `apps`). These will want to be in a volume; run `./apps/create_volume` to copy them into the volume
-
-### Summary
-
-```
-./apache/configure_ssl
-./haproxy/build
-./shiny/build
-./apps/create_volume
-```
-
-## Bringing the bits up
-
-```
-docker network create twinkle 2> /dev/null || /bin/true
-docker volume create shiny_logs
-docker run -d --name haproxy --network twinkle mrcide/haproxy:dev
-docker run -d --name apache --network twinkle \
-  -p 80:80 \
-  -p 443:443 \
-  -p 9000:9000 \
-  -v "${PWD}/apache/httpd.conf:/usr/local/apache2/conf/httpd.conf:ro" \
-  -v "${PWD}/apache/auth:/usr/local/apache2/conf/auth:ro" \
-  -v "${PWD}/apache/ssl:/usr/local/apache2/conf/ssl:ro" \
-  httpd:2.4
-docker run -d --name shiny-1 --network=twinkle \
-  -v twinkle_apps:/shiny/apps \
-  -v twinkle_logs:/shiny/logs \
-  -p 3838:3838 \
-  mrcide/shiny-server:dev
-docker exec haproxy update_shiny_servers shiny 1
-```
-
-Teardown
-
-```
-docker rm -f haproxy apache shiny-1
-docker network rm twinkle
-```
+1. `kubectl delete -k k8s/overlays/<env>`. Replace <env> with testing or production. 
+2. `kubectl  delete ns twinkle` to remove namespace. 

haproxy/haproxy.cfg

@@ -25,6 +25,7 @@ frontend http-in
     option forwardfor
     default_backend servers
 
+# point this to K8s master nodes (control planes)
 backend servers
     # I think that leastconn is probably the best bet here.
     balance leastconn

k8s/base/deployment.yaml

@@ -0,0 +1,38 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: shiny-deploy
+  labels:
+    app: shiny
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: shiny
+  template:
+    metadata:
+      labels:
+        app: shiny
+    spec:
+      initContainers:
+        - name: init-shiny
+          image: busybox:1.28
+          command: ["sh", "-c", "mkdir -p /shiny/logs /shiny/apps"]
+          volumeMounts:
+            - name: shiny-data
+              mountPath: /shiny
+      containers:
+        - name: shiny
+          image: absternator/shiny-server:dev # todo: change to actual one!!! mrcide-
+          volumeMounts:
+            - name: shiny-data
+              mountPath: /shiny
+          # todo: create appropriate resource requests 
+          # resources:
+          #   requests:
+          #     memory: "128Mi"
+          #     cpu: "250m"
+      volumes:
+        - name: shiny-data
+          persistentVolumeClaim:
+            claimName: shiny-pvc

k8s/base/ingress.yaml

@@ -0,0 +1,26 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: ingress-shiny
+  annotations:
+    nginx.ingress.kubernetes.io/affinity: "cookie"
+    nginx.ingress.kubernetes.io/session-cookie-name: "shinycookie"
+    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
+    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
+spec:
+  ingressClassName: nginx
+  tls:
+    - hosts:
+        - shiny-dev.dide.ic.ac.uk
+      secretName: tls-secret
+  rules:
+    - host: shiny-dev.dide.ic.ac.uk
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: shiny-svc
+                port:
+                  number: 3838

k8s/base/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+metadata:
+  name: shiny
+
+namespace: twinkle # assume this namespace exists
+
+resources:
+  - persistance.yaml
+  - deployment.yaml
+  - service.yaml
+  - ingress.yaml

k8s/base/persistance.yaml

@@ -0,0 +1,13 @@
+# todo: storage capacity as per requirements
+# can use local-path if single node or dont need distributed & replicated
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: shiny-pvc
+spec:
+  storageClassName: longhorn
+  accessModes:
+    - ReadWriteMany
+  resources:
+    requests:
+      storage: 15Gi

k8s/base/service.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: shiny-svc
+spec:
+  type: ClusterIP 
+  selector:
+    app: shiny
+  ports:
+    - protocol: TCP
+      port: 3838
+      targetPort: 3838

k8s/configure_ssl

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+# TODO: update to use vault k8s auth
+set -eu
+
+export VAULT_ADDR=https://vault.dide.ic.ac.uk:8200
+export VAULT_TOKEN=$(vault login -method=github -token-only)
+
+DEST=${PWD}/k8s/ssl
+mkdir -p $DEST 
+
+vault read -field=value /secret/shiny.dide/dev/ssl/cert > \
+      ${DEST}/certificate.pem
+vault read -field=value /secret/shiny.dide/dev/ssl/key > \
+      ${DEST}/key.pem
+
+kubectl -n twinkle create secret tls tls-secret --cert  ${DEST}/certificate.pem --key ${DEST}/key.pem

k8s/krsync

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+set -eu
+# https://serverfault.com/questions/741670/rsync-files-to-a-kubernetes-pod
+#  example usage:
+# ./krsync -av --progress --stats <hostdir>  podName@namespace:<poddir>
+
+if [ -z "$KRSYNC_STARTED" ]; then
+    export KRSYNC_STARTED=true
+    exec rsync -a --delete --blocking-io --rsh "$0" $@
+fi
+
+# Running as --rsh
+namespace=''
+pod=$1
+shift
+
+# If user uses pod@namespace, rsync passes args as: {us} -l pod namespace ...
+if [ "X$pod" = "X-l" ]; then
+    pod=$1
+    shift
+    namespace="-n $1"
+    shift
+fi
+
+
+exec kubectl $namespace exec -i $pod -- "$@"

k8s/overlays/production/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - ../../base
+
+commonLabels:
+  env: production
+
+replicas:
+  - name: shiny-deploy
+    count: 9

k8s/overlays/testing/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - ../../base
+
+commonLabels:
+  env: testing
+
+replicas:
+  - name: shiny-deploy
+    count: 3

k8s/shiny-sync

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+set -eu
+# uses krysnc to sync files to a kubernetes pod and thus into k8s cluster (via persistant volume)
+# RUN this from repository root to ensure correct paths to app files
+NAMESPACE=twinkle
+SYNC_DIR=/shiny/apps
+
+
+FIRST_SHINY_POD=$(kubectl get pods -l app=shiny -n $NAMESPACE -o jsonpath='{.items[0].metadata.name}')
+echo "pod name:  $FIRST_SHINY_POD"
+$PWD/k8s/krsync -av --progress --stats $PWD/apps/* $FIRST_SHINY_POD@$NAMESPACE:$SYNC_DIR

shiny/Dockerfile

@@ -3,5 +3,6 @@ FROM rocker/shiny:latest
 # Override the rocker defaults with our defaults
 COPY shiny-server.conf /etc/shiny-server/shiny-server.conf
 
-VOLUME /shiny/apps
-VOLUME /shiny/logs
+# Install rsync
+RUN apt-get update && apt-get install -y rsync
+

shiny/build

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 set -e
 HERE=$(realpath $(dirname $0))
-IMAGE_NAME="mrcide/shiny-server"
+IMAGE_NAME="absternator/shiny-server"
 TAG_VERSION=dev
 docker pull rocker/shiny:latest
 docker build --rm \

start-k8s-shiny

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+set -eu
+# example use case: ./start-k8s-shiny production
+
+# Set the environment variable to the first command-line argument or default to 'testing'
+ENV=${1:-testing}
+if [[ "$ENV" != "testing" && "$ENV" != "production" ]]; then
+    echo "Error: env must be either 'testing' or 'production'"
+    exit 1
+fi
+# Make the script itself executable
+chmod +x "$0"
+
+# Make the k8s directory and its contents executable
+chmod +x k8s/
+
+# Create a Kubernetes namespace named 'twinkle'
+kubectl create ns twinkle
+
+# Run the script to configure SSL (assuming it's in the k8s directory)
+k8s/configure_ssl ||  { echo "Error: Failed to configure SSL"; exit 1; }
+
+# Apply Kubernetes manifests from overlays based on the specified or default environment
+kubectl apply -k "k8s/overlays/$ENV"
+
+# Wait for the 'shiny-deploy' deployment to be in the 'available' condition within the 'twinkle' namespace
+echo 'Waiting for pods to be ready...'
+kubectl wait -n twinkle --for=condition=available --timeout=300s deployment/shiny-deploy
+
+# Run the script for 'shiny-sync' (assuming it's in the k8s directory)
+k8s/shiny-sync