Efficient One-Off Clustering for Personalized Federated Learning

Abstract

In traditional federated learning such as FedAvg, the associations among clients are often ignored when executing on non-independent or heterogeneously distributed datasets, resulting in unsatisfactory accuracy. Although some previous works on clustered federated learning are proposed to address such problems, most of them has a polarized problem. When the number of clusters is small, the model performs poorly and fails to accurately capture the distinction between clients. While the clusters of large number tends to lead to the higher communication costs. Therefore, a critical need is to design an efficient clustered federated solution that can both better capture the diversity between local clients and minimize the communication and computation costs. To this end, we propose an efficient one-off clustered federated learning framework called FedEOC. FedEOC exploits the ``learning-to-learn'' characteristic of meta-learning to enhance the generalization of the model across different clients, so that only a small number of iterations are needed for each client to quickly obtain locally adapted weights. Based on the well initially trained weights on all clients, we can cluster the clients only once to achieve the effect of one-off clustering and multiple-round applying. Additionally, to alleviate the issue of cluster imbalance, FedEOC is equipped with a Decomposition and Consolidation (Dec-Con) mechanism to decompose the clients from the extreme clusters and consolidate them into the most similar ones. The comprehensive experiments conducted on two real-world datasets demonstrate the superior capability of FedEOC from both aspects of accuracy and efficiency.

Keywords: Federated learning, Meta learning, One-off clustering, Decomposition and consolidation mechanism.