How to retrieve information about ScatterAndGather rounds within PTLearner?

[elchiodo]

Python version (python3 -V)

3.8

NVFlare version (python3 -m pip list | grep "nvflare")

2.3.10

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.3

Operating system

Ubuntu 20.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I'm following the pneumonia-federated experiment https://github.com/Azure/medical-imaging/blob/main/federated-learning/pneumonia-federated and through some code tweaks I've been able to build a Federated Learning project using Azure Infrastructures. My goal is to retrieve information about ScatterAndGather workflow rounds within PTLearner, so I can then iterate on each round to get metrics validation of my global model and being able to stream it to Tensorboard. Right now I can only obtain a single data point on tensorboard representing the validation of the global model on the last round of ScatterAndGather. Is there any way to obtain round number? I tried to gather it from fl_context through calling different properties but without success. Thank you in advance.

[SYangster]

Hi thanks for reaching out, yes there is a way to obtain the round number from within the PTLearner. It was a good idea to try looking at the FLContext, however if you see ScatterAndGather:221 the AppConstants.CURRENT_ROUND is set as a private, meaning that this property is only visible to local components. Since the ScatterAndGather workflow is server-controlled, the PTLearner on the client will not be able to access this.

Instead we can look at ScatterAndGather:226 where the current round is set as a header on the shareable. This can be accessed from within PTLearner: current_round = shareable.get_header(AppConstants.CURRENT_ROUND)

Also do you mind elaborating on which validation metrics you are interested in capturing?
Currently this job has a ScatterAndGather workflow followed by a CrossSiteModelEval workflow. In ScatterAndGather in the local train of PTLearner:186, local validation is performed at the end of each epoch inside local_train() and the metric is streamed. Then after all rounds of SAG, in CrossSiteModelEval all the models (including the global model) are distributed to each client for local validation.

Let me know if you have any more questions and I hope this helps!

[elchiodo]

Hello, first of all, thank you for the clear and comprehensive response. Regarding the validation metrics, in addition to accuracy and log loss, we have added precision, recall, f1 score, and calculated a confusion matrix. The idea of iterating over the number of rounds in ScatterAndGather was specifically to visualize on TensorBoard how the metrics improve with each round of the workflow. We have just tested and were able to correctly retrieve the number of rounds. Thank you for your assistance.