[Q&A] cifar10-real-world example not working

[thibaultferaux]

Python version (python3 -V)

3.10.12

NVFlare version (python3 -m pip list | grep "nvflare")

2.5.2

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.5

Operating system

Ubuntu 22.04.3 LTS

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I am new to NVIDIA FLARE and ran into the following problem and don't know what the problem is or how to fix it:

When I try to run the cifar10-real-world example from the repo with the cifar10_fedavg_he job the server get's the following warnings:

2025-01-18 20:33:07,048 - HandleDeadJobCommand - WARNING - received dead job notification: reason='missing job on client'
2025-01-18 20:33:07,052 - WFCommServer - WARNING - [identity=secure_project, run=412e8880-fa38-401b-9d8f-ef9cde72b81c, wf=scatter_gather_ctl]: received dead job report for client site-1
2025-01-18 20:33:07,067 - WFCommServer - WARNING - [identity=secure_project, run=412e8880-fa38-401b-9d8f-ef9cde72b81c, wf=scatter_gather_ctl]: client site-1 is placed on dead client watch list
2025-01-18 20:34:07,221 - WFCommServer - ERROR - Client site-1 is deemed disconnected!
2025-01-18 20:34:07,282 - WFCommServer - ERROR - [identity=secure_project, run=412e8880-fa38-401b-9d8f-ef9cde72b81c, wf=scatter_gather_ctl]: Alive clients 7 < required min 8
2025-01-18 20:34:07,308 - ServerRunner - ERROR - [identity=secure_project, run=412e8880-fa38-401b-9d8f-ef9cde72b81c, wf=scatter_gather_ctl]: Aborting current RUN due to FATAL_SYSTEM_ERROR received: Aborting job due to deployment policy violation
2025-01-18 20:34:07,314 - ServerRunner - INFO - [identity=secure_project, run=412e8880-fa38-401b-9d8f-ef9cde72b81c, wf=scatter_gather_ctl]: asked to abort - triggered abort_signal to stop the RUN

And when I go and check the client logs they and like this:

2025-01-18 20:32:57,463 - CIFAR10ModelLearner - INFO - [identity=site-1, run=412e8880-fa38-401b-9d8f-ef9cde72b81c, peer=secure_project, peer_run=412e8880-fa38-401b-9d8f-ef9cde72b81c, task_name=train, task_id=f7a95a77-71c9-41b4-ad9d-0d48f5e9ad68]: IndexList Path: /tmp/cifar10_splits/cifar10_fedavg_he_alpha1.0_4332841a-afdd-40b3-be00-2f47305661a0/site-1.npy
2025-01-18 20:32:57,464 - CIFAR10ModelLearner - INFO - [identity=site-1, run=412e8880-fa38-401b-9d8f-ef9cde72b81c, peer=secure_project, peer_run=412e8880-fa38-401b-9d8f-ef9cde72b81c, task_name=train, task_id=f7a95a77-71c9-41b4-ad9d-0d48f5e9ad68]: Loading subset index
2025-01-18 20:32:57,469 - CIFAR10ModelLearner - INFO - [identity=site-1, run=412e8880-fa38-401b-9d8f-ef9cde72b81c, peer=secure_project, peer_run=412e8880-fa38-401b-9d8f-ef9cde72b81c, task_name=train, task_id=f7a95a77-71c9-41b4-ad9d-0d48f5e9ad68]: Client subset size: 7410

Can anyone tell me what I did wrong?

[holgerroth]

Your number of active clients (7) is smaller than the required 8 clients. site-1 seems to be dead. Can you share the log.txt file in the workdir of that client to see what the error is?

[thibaultferaux]

This is my log.txt file of my client:

2025-01-18 20:31:15,195 - FederatedClient - INFO - Successfully registered client:site-1 for project secure_project. Token:dd39404c-740c-4fe3-a694-e532391715bf SSID:ebc6125d-0a56-4688-9b08-355fe9e4d61a
2025-01-18 20:31:15,198 - FederatedClient - INFO - Got engine after 0.09662961959838867 seconds
2025-01-18 20:31:15,199 - FederatedClient - INFO - Got the new primary SP: grpc://localhost:8102
2025-01-18 20:32:41,579 - ClientEngine - INFO - Starting client app. rank: 0
2025-01-18 20:32:41,586 - ProcessExecutor - INFO - Worker child process ID: 2081
2025-01-18 20:32:41,588 - ProcessExecutor - INFO - run (412e8880-fa38-401b-9d8f-ef9cde72b81c): waiting for child worker process to finish.
2025-01-18 20:32:51,172 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 127.0.0.1:29711 <= 127.0.0.1:38862] is created: PID: 977
2025-01-18 20:33:02,488 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 977
2025-01-18 20:33:02,498 - ProcessExecutor - INFO - run (412e8880-fa38-401b-9d8f-ef9cde72b81c): child worker process finished with RC -9

Could it be because the client crashed because of OOM?

[holgerroth]

Could be. You can try running it with less clients adjusting this command.

Need to change the min_clients requirements in the meta.json and server config before submitting the job.

[thibaultferaux]

This was the problem. The client crashed because it had not enough compute.

[IsaacYangSLA]

Also, the last log from site-1 is at 20:32:57 and the dead job notification on server is at 20:33:07. The 10-second difference may be an indicator that site-1 didn't respond by timeout.