Deploy and test SLURM setup

[noamross]

No description provided.

[noamross]

@n8layman, when @espirado has you test this, look into:
https://books.ropensci.org/targets/crew.html

https://github.com/wlandau/crew.cluster/

[espirado]

@n8layman @collinschwantes I have updated the README for testing purposes. Could you test and create issues on your findings/suggestions.

[n8layman]

From my conversation with @espirado today I won't be able to fork the repo and test the container on my local machine due to incompatibilities with ARM architecture. We're working on setting up a VM so I can remotely test the SLURM workflow.

[espirado]

Since the code change to deploy slurm based reservior is huge will be creating a separate repository for the new code from the eha-server.

[espirado]

Deployed the first attempt on Aegypti got the containers to run but due to hardware incompatibility for GPU and most drivers had to remove GPU. Hopefully will have the usable access tomorrow

[espirado]

@n8layman can we schedule a test session for slurm I have it set up on aegypti. Can you first try to access via
ssh -p 22022 [email protected] ( your normal username and password). This will give you access to the controller. The web interface will be up when the dns resolves to the domain name after a few days. For now we can test via CLI access.

[n8layman]

Sounds great. I can access the controller using ssh as above.

[espirado]

We successfully did a run test on the base slurm environment and worked as expected and will proceed in creating a repo with adequate code examples to integrate various R/Python workflows with slurm for different types of workloads and efficient cluster usage. @n8layman will also assist in coming up with examples that we can use for M3 .

[noamross]

Excellent!

[espirado]

Notes from review

1. Are the workers exclusive - will they reserve that amount of hardware? Can you exceed the number of CPUs on the machine?
2. Can you dynamically set the resources used? Yes
3. How does prioritization change the run? SLURM admin can configure a partition to have a priority value - submitting a job to a partition with a higher priority wins — user groups can be prioritized
4. Handling Authentication?

• Standard partition - bigger than a laptop 8 cores and 16 gbs —> run one R script to the standard partition (standard resources and priority)
• Can you get errors out to a specific place?
• Pull example from slurmR?
• Are defaults for each parameters set by the controller? Yes -
• Likely smaller 64 gb of ram and 32 cores
• Want to make sure this will work with targets
• Using the targets slurm backend to >>> targets are sent to jobs
• Have 100 single CPU targets
• These targets need to go the nodes

All issues Raised will be addressed on next merge request and documentation both for workflow examples and Infrastucture deployment.

[espirado]

Added all nodes(Prospero,sycorax.aegypti). Tests for targets working.