Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for IBM Cloud #434

Merged
merged 20 commits into from
Sep 9, 2024
Merged

Add support for IBM Cloud #434

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
1ebab05
Initial version of IBM Code Engine
macarronesc Jun 11, 2024
bd3f353
Added IBM backend to tests and setup
macarronesc Jun 11, 2024
d3ff567
Initial steps to create a Cluster and VMs on IBM
macarronesc Jun 12, 2024
a7148d8
Working on IBM Code Engine support
macarronesc Jun 13, 2024
94792f3
Trying new way to connect Dask client with workers
macarronesc Jun 17, 2024
3f7098d
Added a lot of changes
macarronesc Jul 3, 2024
3bff4ff
Last tests
macarronesc Jul 12, 2024
19284e9
Fixed Scheduler connection
macarronesc Jul 15, 2024
920f1a8
Implemented worker execution
macarronesc Jul 17, 2024
f8ce77f
Cleaning and added logs
macarronesc Jul 17, 2024
f5ca242
Deleted unused imports
macarronesc Jul 18, 2024
028830d
Improved command variable
macarronesc Jul 18, 2024
06e1075
General cleaning
macarronesc Jul 18, 2024
13d10ca
Added worker threads flag
macarronesc Jul 19, 2024
b0a94a4
Fixed issue with IBM delete vm
macarronesc Jul 22, 2024
722d86e
Added IBM Cloud documentation
macarronesc Jul 22, 2024
74b2273
Add parallel job run creation using worker threads
macarronesc Jul 22, 2024
dc35c0b
Merge remote-tracking branch 'upstream/main'
macarronesc Sep 4, 2024
df705ac
Updated IBM default region
macarronesc Sep 5, 2024
e5b103a
Required changes applied
macarronesc Sep 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ci/scripts/test_imports.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,5 @@ test_import "aws" "import dask_cloudprovider.aws"
test_import "azure" "import dask_cloudprovider.azure"
test_import "digitalocean" "import dask_cloudprovider.digitalocean"
test_import "gcp" "import dask_cloudprovider.gcp"
test_import "openstack" "import dask_cloudprovider.openstack"
test_import "ibm" "import dask_cloudprovider.ibm"
test_import "openstack" "import dask_cloudprovider.openstack"
14 changes: 13 additions & 1 deletion dask_cloudprovider/cloudprovider.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,18 @@ cloudprovider:
docker_image: "daskdev/dask:latest" # docker image to use
bootstrap: true # It is assumed that the OS image does not have Docker and needs bootstrapping. Set this to false if using a custom image with Docker already installed.

ibm:
api_key: null
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there not a global location where IBM stores credentials? For example AWS stores things in ~/.aws/credentials and we read it from there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, according to the documentation it is not stored anywhere visible to the user. That is why it is necessary to add the API key manually.

image: "ghcr.io/dask/dask:latest"
region: us-east
project_id: null
scheduler_cpu: "1.0"
scheduler_mem: 4G
scheduler_timeout: 600 # seconds
worker_cpu: "2.0"
worker_mem: 8G
worker_threads: 1

openstack:
region: "RegionOne" # The name of the region where resources will be allocated in OpenStack. List available regions using: `openstack region list`.
size: null # Openstack flavors define the compute, memory, and storage capacity of computing instances. List available flavors using: `openstack flavor list`
Expand All @@ -132,4 +144,4 @@ cloudprovider:
security_group: null # The security group name that defines firewall rules for instances. List available security groups using: `openstack security group list`
external_network_id: null # The ID of the external network used for assigning floating IPs. List available external networks using: `openstack network list --external`
create_floating_ip: false # Specifies whether to assign a floating IP to each instance, enabling external access. Set to `True` if external connectivity is needed.
docker_image: "daskdev/dask:latest" # docker image to use
docker_image: "daskdev/dask:latest" # docker image to use
1 change: 1 addition & 0 deletions dask_cloudprovider/ibm/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from .code_engine import IBMCodeEngineCluster
Loading
Loading