Mrc-4908 k8s setup #1

absternator · 2024-01-17T13:41:12Z

This PR is the k8s version of setting up and running the shiny server. Follow the readme kubernetes section to set up if wanted. The current shiny server can be reached on https://shiny-dev.dide.ic.ac.uk/ (must be in dide network)

The following is done in this PR:

Kubernetes resources to get the shiny server running
Ingress used for routing, sticky cookies load balancing, and certs
bash scripts to get the full app up and running
shiny-sync scrip to get initial app data into Kubernetes volume

richfitz · 2024-01-19T09:24:49Z

README.md

+
+### Prerequisites
+
+A production kubernetes cluster using k3s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistant volume. 


What does the recommended setup look like for developing this locally? or on a staging machine (separate from the shiny staging, which is really for the users)? Do we need a second cluster somewhere that we use?

the link shows the dev and prod setup and they both can be used locally... for staging, we can use the same cluster and just do another deployment and rename the deployment to something else... But if you wanted a seperate machine for just staging spinning up another cluster might be easiest

richfitz · 2024-01-19T09:25:45Z

k8s/base/deployment.yaml

+              mountPath: /shiny
+      containers:
+        - name: shiny
+          image: absternator/shiny-server:dev # todo: change to actual one!!! mrcide-


What changes did you make here? Can we get the image build script/Dockerfile here and building on BK?

I just changed the image so it has an image to pull from the docker hub... i can add the docker build & push step to the startup script? is that you want? ps.. sorry whats BK?

BK=BuildKite, where we have some non-gha CI builds running,

richfitz · 2024-01-19T09:26:34Z

k8s/base/deployment.yaml

+          volumeMounts:
+            - name: shiny-data
+              mountPath: /shiny
+          # todo: create appropriate resource requests and limits


What is the consequence of these limits? Are they speculative (affecting scheduling) or caps (meaning the scheduler kills processes that exceed them)?

Have updated comments but i think usually requests are set or nothing is fine most of the time ... the most common is to set up requests so that each pod has allocated specific resources before being scheduled...

richfitz · 2024-01-19T09:27:18Z

k8s/base/ingress.yaml

+  annotations:
+    nginx.ingress.kubernetes.io/affinity: "cookie"
+    nginx.ingress.kubernetes.io/session-cookie-name: "shinycookie"
+    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"


Is this in seconds? 48 hours? Seems reasonable

yup in seconds... 48 hours

richfitz · 2024-01-19T09:27:49Z

README.md

@@ -33,7 +33,7 @@ Some applications (copied over from the original deployment are in `apps`). Thes

 ## Bringing the bits up

-```
+```bash


Suggest you remove the docker instructions and the haproxy/httpd bits entirely

richfitz · 2024-01-19T09:30:11Z

k8s/krsync

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+


I suggest some hardening of your bash scripts (basically any time you write one). In general we end up with:

set -eu

which will cause the script to fail if any line errors or if you reference an undefined variable.

richfitz · 2024-01-19T09:30:28Z

k8s/krsync

+    shift
+fi
+
+# customise this to cluster needs!!


What sort of customisation is expected here?

loose comment when i first did have now removed

richfitz · 2024-01-19T09:30:37Z

k8s/overlays/production/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization


richfitz · 2024-01-19T09:31:23Z

k8s/overlays/staging/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1


We probably will need a name for what the users think of staging that won't confuse us - preview or testing perhaps?

have changed naming to testing... because you are right we dont have a specific staging k8s cluster

richfitz · 2024-01-19T09:32:22Z

k8s/shiny-sync

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash


So this is just a temporary thing until we have the management bit working?

yup temporary until we sort the management side

EmmaLRussell

Haven't tried running this up, will do so tomorrow, just a couple of initial questions, and typos.

Can we make a gha to test this cluster? Something like here?

What's the implication for haproxy of Ingress doing the load balancing and sticky cookies? Do we even still need haproxy? Or do we keep it as the entry point to the system from httpd?

EmmaLRussell · 2024-01-23T17:10:43Z

README.md


-### apache
+The following is guide to run the shiny server in kubernetes. They are based on running a Kind cluster. The configuration may
+nee to be adjusted for other k8s clusters.


Suggested change

nee to be adjusted for other k8s clusters.

need to be adjusted for other k8s clusters.

EmmaLRussell · 2024-01-23T17:11:02Z

README.md


-### haproxy
+A production kubernetes cluster using k3s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistant volume. 


Suggested change

A production kubernetes cluster using k3s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistant volume.

A production kubernetes cluster using k8s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistent volume.

EmmaLRussell · 2024-01-23T17:11:41Z

README.md


-Build the image with `./haproxy/build` which builds `mrcide/haproxy` with a configuration that can be seen in [`haproxy/haproxy.cfg`](haproxy/haproxy.cfg) and some utilities which enable some degree of dynamic scaling of shiny servers.
+Run `start-k8s-shiny <env>` to run the shint server in k8s.


Suggested change

Run `start-k8s-shiny <env>` to run the shint server in k8s.

Run `./start-k8s-shiny <env>` to run the shiny server in k8s.

EmmaLRussell · 2024-01-23T17:30:36Z

k8s/base/ingress.yaml

+  ingressClassName: nginx
+  tls:
+    - hosts:
+        - shiny-dev.dide.ic.ac.uk


So we'll need to update this setting for prod, and for running locally?

EmmaLRussell · 2024-01-23T17:36:16Z

k8s/base/kustomization.yaml

+namespace: twinkle # assume this namespace exists
+
+resources:
+  - persistance.yaml


Suggested change

- persistance.yaml

- persistence.yaml

EmmaLRussell · 2024-01-23T17:37:37Z

README.md


-### haproxy
+A production kubernetes cluster using k3s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistant volume. 


storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistant volume.

Sorry, don't really understand this. Why aren't these available? You talk about using local-path below, in persistance.yaml.

have updated to avoid confusion

EmmaLRussell · 2024-01-23T17:38:51Z

k8s/base/deployment.yaml

+              mountPath: /shiny
+      containers:
+        - name: shiny
+          image: absternator/shiny-server:dev # todo: change to actual one!!! mrcide-


BK=BuildKite, where we have some non-gha CI builds running,

absternator · 2024-01-25T15:56:29Z

@EmmaLRussell have added GHA to add shiny server image to build and push... also removed haproxy and apache server and fixed typos.. will seee the testing stuff as linked for testing cluster in another PR...as this needs more thought around what really needs to be tested

absternator added 9 commits January 12, 2024 07:25

feat: kubernetes setup

eb5618b

feat: ssl ingress + persistance local

69d5f98

feat: add start script

4fc22d8

docs: update readme

6e4e6c7

chore: comments for error + persistance

7badd77

fix: longhorn and update scripts

1078b51

update readme

55e8329

chore: remove dead code

37c5cfd

update readme extras

472a689

absternator requested review from richfitz and EmmaLRussell January 17, 2024 13:41

richfitz reviewed Jan 19, 2024

View reviewed changes

refactor: PR comments

31b18a4

absternator requested a review from richfitz January 19, 2024 10:40

Update configure_ssl script to use vault k8s auth

a6f2fa5

EmmaLRussell reviewed Jan 23, 2024

View reviewed changes

absternator added 4 commits January 25, 2024 12:13

cleanups + add gha

16ad36d

gha update

8b19073

remove PR into main docker hub push

d5fd1c4

fix krsync issue

bc066f3

richfitz approved these changes Jan 25, 2024

View reviewed changes

absternator added 2 commits January 25, 2024 15:39

feat: cleanup kustomize for testing and prod

9a9c8f0

readme updae for local

526208e

absternator requested a review from EmmaLRussell January 25, 2024 15:56

absternator merged commit 05db8b5 into main Jan 31, 2024

absternator deleted the mrc-4908-k8s-setup branch January 31, 2024 09:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mrc-4908 k8s setup #1

Mrc-4908 k8s setup #1

absternator commented Jan 17, 2024

richfitz Jan 19, 2024

absternator Jan 19, 2024

richfitz Jan 19, 2024

absternator Jan 19, 2024

EmmaLRussell Jan 23, 2024

richfitz Jan 19, 2024

absternator Jan 19, 2024

richfitz Jan 19, 2024

absternator Jan 19, 2024

richfitz Jan 19, 2024

richfitz Jan 19, 2024

richfitz Jan 19, 2024

absternator Jan 19, 2024

richfitz Jan 19, 2024

absternator Jan 19, 2024

richfitz Jan 19, 2024

absternator Jan 19, 2024

richfitz Jan 19, 2024

absternator Jan 19, 2024

EmmaLRussell left a comment

EmmaLRussell Jan 23, 2024

EmmaLRussell Jan 23, 2024

EmmaLRussell Jan 23, 2024

EmmaLRussell Jan 23, 2024

EmmaLRussell Jan 23, 2024

EmmaLRussell Jan 23, 2024

absternator Jan 25, 2024

EmmaLRussell Jan 23, 2024

absternator commented Jan 25, 2024 •

edited

Loading


		### Prerequisites

		A production kubernetes cluster using k3s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistant volume.

		@@ -0,0 +1,12 @@
		apiVersion: kustomize.config.k8s.io/v1beta1
		kind: Kustomization

	nee to be adjusted for other k8s clusters.
	need to be adjusted for other k8s clusters.


		### haproxy
		A production kubernetes cluster using k3s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistant volume.

	A production kubernetes cluster using k3s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistant volume.
	A production kubernetes cluster using k8s is needed to be setup first. To setup a k8s cluster follow the guide [here](https://mrc-ide.myjetbrains.com/youtrack/articles/RESIDE-A-31/Setting-up-Kubernetes-k8s-Cluster). Note: If using dev cluster (KIND), the storageClassName: local-path & longhorn is unavailable and you will need to provision own storage class and persistent volume.


		Build the image with `./haproxy/build` which builds `mrcide/haproxy` with a configuration that can be seen in [`haproxy/haproxy.cfg`](haproxy/haproxy.cfg) and some utilities which enable some degree of dynamic scaling of shiny servers.
		Run `start-k8s-shiny <env>` to run the shint server in k8s.

	Run `start-k8s-shiny <env>` to run the shint server in k8s.
	Run `./start-k8s-shiny <env>` to run the shiny server in k8s.

Mrc-4908 k8s setup #1

Mrc-4908 k8s setup #1

Conversation

absternator commented Jan 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EmmaLRussell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

absternator commented Jan 25, 2024 • edited Loading

absternator commented Jan 25, 2024 •

edited

Loading