[QST]: DASK AND CUGRAPH

[williamcolegithub]

What is your question?

Hello! For the life of me I cannot get slurmcluster dask and cugraph to cooperate. I can get many configurations of slurmcluster dask and cudf to work. Cugraph has various errors for me such as a generic cufile error, or different modules dont exist, or code runs indefinitely... etc. All existing documentatino appears to be with localcudacluster which does not work for my setup and is this even truley multi-node + multi-gpu or just multigpu to use local cuda cluster?

I know my environments are consistent and up to date.

Looking for any better examples or hop on a quick call.
Thank you!!!!!

Code of Conduct

• I agree to follow cuGraph's Code of Conduct
• I have searched the open issues and have found no duplicates for this question

[jnke2016]

@williamcolegithub thank you for reaching out. Can you provide more information on how you setup your cluster please?

All existing documentatino appears to be with localcudacluster

LocalCudaCluster only supports single node multi-gpu hence if you want to run multi nodes, you will need to start each worker with a CLI command like dask-cuda-worker along with the scheduler on one of your nodes with dask-scheduler.

[williamcolegithub]

@jnke2016 Heres a reply please see: I see! OK, I will reach out to my slurm team. Yes, I have tried dask-cuda-worker and it resulted in a failure to connect with nanny. So I have been using dask-worker.
From my perspective the documentation was not clear that dask-cuda-worker was essential, it appeared optional. Thank you for clarifying and the fast reply. I will reach out if issues persist.

--- By any chance, is there a distributed notebook you all recommend? I only find examples using local cuda cluster, even notebooks that claim to be multi-node.

[quasiben]

We have examples of how to deploy with SLURM on the HPC Deployment page:
https://docs.rapids.ai/deployment/stable/hpc/

If you run into trouble please ping here

cc @jacobtomlinson

As for a cuGraph notebooks, that is a great question. @acostadon / @jnke2016 / @rlratzel is that something you know ?

[rlratzel]

is there a distributed notebook you all recommend? I only find examples using local cuda cluster

I'm not aware of any notebook examples that are multi-node (and we should fix that in our notebooks that claim to be multi-node). I did however recently commit some scripts which can be used for multi-node cugraph workflows that use dask:

https://github.com/rapidsai/cugraph/tree/branch-25.02/scripts/dask

Those scripts will launch the dask scheduler and worker processes. Once you have those running, you should then be able to use these helper functions here and here with the SCHEDULER_FILE env var used by the scheduler and workers to create and teardown a dask client in your code.

Here's an example:

import cugraph
import dask_cudf
import cugraph.dask as dask_cugraph

from cugraph.testing.mg_utils import start_dask_client, stop_dask_client


if __name__ == "__main__":
    # Must have SCHEDULER_FILE env var set to path of generated scheduler file.
    (client, cluster) = start_dask_client()

    input_data_path="/data/22/graph500-22.e"
    blocksize = dask_cugraph.get_chunksize(input_data_path)

    e_list = dask_cudf.read_csv(input_data_path,
                                blocksize=blocksize,
                                delimiter=" ",
                                names=["src", "dst"],
                                dtype=["int32", "int32"],
                                )

    G = cugraph.Graph()
    G.from_dask_cudf_edgelist(e_list,
                              source="src",
                              destination="dst",
                              )

    results = dask_cugraph.pagerank(G)

    stop_dask_client(client, cluster)

Hope this helps.