federated data cleaning

[ramiro350]

Hello i've been curious if there's an already implemented way of perfoming federated data cleaning in the latest version ? i know there's federated statistics, but besides it don't know if there is or is planned to be.

[chesterxgchen]

can you elaborate this ? what do you mean by Federated data cleaning ?

[ramiro350]

like when using an ETL to remove outliers from a dataset, before performing a centralized training, but instead of the data cleansing be made using only each client dataset individually, it's done globally be founding the outliers of the whole clients, like i said, i think federated-statistics maybe be turned in something like that, but don't know if it's already implemented

[chesterxgchen]

Ok, understand, basically federated computing or federated ETL, feature engineering before Federated training.
FLARE can certainly do that. Flare at its core, its a federated computing framework. Its agnostic to the task, model or domain.
you can easily write a client task to do ETL work directly or delegate to Apache Spark to do the ETL.

We haven't add any example on this, but is fairly easy to do. In this case, the data exchange between client and server are just task command, maybe some metadata, the ETL job performed locally on each site's dataset without data exchange.

We don't have customer ask for this so far, even its easy to do. Can you describe your use case and project if you can share ?

[ramiro350]

it's going to be a web app targeted for image classification between hospitals, basically regular federated learning but with the federated ETL, we are still in research phase so we don't have the project to share.