Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Design Overview
Experimental Setup
Related Work
Conclusion
References
All Topics
Computer Science
Distributed Systems

Decentralized Learning Made Easy with DecentralizePy

Profile image of Miloš VujasinovićMiloš Vujasinović

2023

https://doi.org/10.1145/3578356.3592587
visibility

…

description

8 pages

link

1 file

Abstract

Decentralized learning (DL) has gained prominence for its potential benefits in terms of scalability, privacy, and fault tolerance. It consists of many nodes that coordinate without a central server and exchange millions of parameters in the inherently iterative process of machine learning (ML) training. In addition, these nodes are connected in complex and potentially dynamic topologies. Assessing the intricate dynamics of such networks is clearly not an easy task. Often in literature, researchers resort to simulated environments that do not scale and fail to capture practical and crucial behaviors, including the ones associated to parallelism, data transfer, network delays, and wall-clock time. In this paper, we propose decentralizepy, a distributed framework for decentralized ML, which allows for the emulation of large-scale learning networks in arbitrary topologies. We demonstrate the capabilities of decentralizepy by deploying techniques such as sparsification and secure aggregation on top of several topologies, including dynamic networks with more than one thousand nodes. CCS Concepts: • Networks → Programming interfaces; • Computing methodologies → Distributed algorithms; Machine learning algorithms; • Computer systems organization → Peer-to-peer architectures.

Related papers

Unifying Intelligence: Federated Learning in Cloud Environments for Decentralized Machine Learning
Srinivasa Rao Angajala

", International Journal of Science and Research (IJSR), , 2023

The rapid growth of data generation and the increasing demand for machine learning models have given rise to novel approaches in the realm of distributed computing. Federated Learning, as a paradigm, allows machine learning models to be trained across decentralized data sources, paving the way for enhanced privacy, efficiency, and scalability. This rsearch-oriented descriptive article explores the implementation of Federated Learning techniques in cloud environments, unraveling the intricacies of decentralized model training, addressing challenges, and examining real-world applications. Keywords, relevant studies, and references are provided to offer a comprehensive resource for researchers and practitioners in the field.

View PDFchevron_right
BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning
shayan siddiqui

ArXiv, 2019

Access to sufficient annotated data is a common challenge in training deep neural networks on medical images. As annotating data is expensive and time-consuming, it is difficult for an individual medical center to reach large enough sample sizes to build their own, personalized models. As an alternative, data from all centers could be pooled to train a centralized model that everyone can use. However, such a strategy is often infeasible due to the privacy-sensitive nature of medical data. Recently, federated learning (FL) has been introduced to collaboratively learn a shared prediction model across centers without the need for sharing data. In FL, clients are locally training models on site-specific datasets for a few epochs and then sharing their model weights with a central server, which orchestrates the overall training process. Importantly, the sharing of models does not compromise patient privacy. A disadvantage of FL is the dependence on a central server, which requires all cl...

View PDFchevron_right
Get More for Less in Decentralized Learning Systems
Miloš Vujasinović

arXiv (Cornell University), 2023

Decentralized learning (DL) systems have been gaining popularity because they avoid raw data sharing by communicating only model parameters, hence preserving data confidentiality. However, the large size of deep neural networks poses a significant challenge for decentralized training, since each node needs to exchange gigabytes of data, overloading the network. In this paper, we address this challenge with JWINS, a communication-efficient and fully decentralized learning system that shares only a subset of parameters through sparsification. JWINS uses wavelet transform to limit the information loss due to sparsification and a randomized communication cutoff that reduces communication usage without damaging the performance of trained models. We demonstrate empirically with 96 DL nodes on non-IID datasets that JWINS can achieve similar accuracies to full-sharing DL while sending up to 64% fewer bytes. Additionally, on low communication budgets, JWINS outperforms the state-of-the-art communication-efficient DL algorithm CHOCO-SGD by up to 4× in terms of network savings and time.

View PDFchevron_right
Overmind: A Collaborative Decentralized Machine Learning Framework
Puttakul SAKUL-UNG

Advances in Science, Technology and Engineering Systems Journal

This paper is an extension of work originally presented in PM2.5 Prediction-based Weather Forecast Information and Missingness Challenges: A Case Study Industrial and Metropolis Areas, which focused on imputation algorithm to solve missingness challenge and demonstrated a basic prediction system to prove the proposed algorithm, II-MPA. Distributed and decentralized systems, recently, have been proven for their effectiveness in multiple perspectives. This paper introduces "Overmind", the solution that governs and builds the network of decentralized machine learning as a prediction framework named after its functionality: it aims to discover a set of data and associated attributes for assigning machine learnings in the collaborative decentralized manner. Overmind also empowers feature transfer learning with data preservation. It demonstrates how discovered features are transferred and shared among synergic agents in the network. This model is tested and evaluated the accuracy against the traditional single machine learning prediction model in the original work. The results are satisfactory in both prediction performance and transfer learning.

View PDFchevron_right
Train Me If You Can: Decentralized Learning on the Deep Edge
MIGUEL BORGES COSTA

Applied Sciences

The end of Moore’s Law aligned with data privacy concerns is forcing machine learning (ML) to shift from the cloud to the deep edge. In the next-generation ML systems, the inference and part of the training process will perform at the edge, while the cloud stays responsible for major updates. This new computing paradigm, called federated learning (FL), alleviates the cloud and network infrastructure while increasing data privacy. Recent advances empowered the inference pass of quantized artificial neural networks (ANNs) on Arm Cortex-M and RISC-V microcontroller units (MCUs). Nevertheless, the training remains confined to the cloud, imposing the transaction of high volumes of private data over a network and leading to unpredictable delays when ML applications attempt to adapt to adversarial environments. To fill this gap, we make the first attempt to evaluate the feasibility of ANN training in Arm Cortex-M MCUs. From the available optimization algorithms, stochastic gradient descent...

View PDFchevron_right
Biscotti: A Ledger for Private and Secure Peer-to-Peer Machine Learning
Muhammad Shayan

2019

Federated Learning is the current state of the art in supporting secure multi-party machine learning (ML): data is maintained on the owner's device and the updates to the model are aggregated through a secure protocol. However, this process assumes a trusted centralized infrastructure for coordination, and clients must trust that the central service does not use the byproducts of client data. In addition to this, a group of malicious clients could also harm the performance of the model by carrying out a poisoning attack. As a response, we propose Biscotti: a fully decentralized peer to peer (P2P) approach to multi-party ML, which uses blockchain and cryptographic primitives to coordinate a privacy-preserving ML process between peering clients. Our evaluation demonstrates that Biscotti is scalable, fault tolerant, and defends against known attacks. For example, Biscotti is able to protect the privacy of an individual client's update and the performance of the global model at ...

View PDFchevron_right
Low Precision Decentralized Distributed Training over IID and non-IID Data
SANGAMESH KODGE

2021

Decentralized distributed learning is the key to enabling large-scale machine learning (training) on the edge devices utilizing private user-generated local data, without relying on the cloud. However, practical realization of such on-device training is limited by the communication and compute bottleneck. In this paper, we propose and show the convergence of low precision decentralized training that aims to reduce the computational complexity and communication cost of decentralized training. Many feedback-based compression techniques have been proposed in the literature to reduce the communication cost. To the best of our knowledge, there is no work that applies and shows compute efficient training techniques such quantization, pruning etc., for peer-to-peer decentralized learning setups. Since real-world applications have significant skew in the data distribution, we design "RangeEvoNorm" as the normalization activation layer which is better suited for of low precision tr...

View PDFchevron_right
Decentralized Machine Learning Models with Cryptographic Techniques
IJERA Journal

Everything from medical screening to disease outbreak detection could benefit from machine learning models trained on sensitive real-world data. And, thanks to the widespread use of mobile devices, even more detailedand sensitive-information is becoming available. Traditional machine learning, on the other hand, involves a data pipeline that uses a central server (on-premises or in the cloud) to host the trained model and make predictions. Distributed Machine Learning (FL), on the other hand, is a method of downloading the current model and computing an updated model using local data at the device itself (a.k.a. edge computing).These locally trained models are then sent back to the central server, where they are aggregated (i.e. weights are averaged), and a single consolidated and improved global model is then sent back to the devices. The interaction of parameters and the resulting model, however, may still reveal information about the training data used. Two approaches have been used in this report to address these privacy concerns, which are based on Homographic Encryption and Secret Sharing techniques, among others. The report summarises previous research in these areas and makes recommendations for future research.

View PDFchevron_right
Federated Learning: Collaborative Machine Learning without Centralized Training Data
Vishal Thomas

international journal of engineering technology and management sciences

Federated learning (also known as collaborative learning) is a machine learning technique that trains an algorithm without transferring data samples across numerous decentralized edge devices or servers. This strategy differs from standard centralized machine learning techniques in which all local datasets are uploaded to a single server, as well as more traditional decentralized alternatives, which frequently presume that local data samples are uniformly distributed. Federated learning allows several actors to collaborate on the development of a single, robust machine learning model without sharing data, allowing crucial issues such as data privacy, data security, data access rights, and access to heterogeneous data to be addressed. Defence, telecommunications, internet of things, and pharmaceutical industries are just a few of the sectors where it has applications.

View PDFchevron_right
The Effect of Training Parameters and Mechanisms on Decentralized Federated Learning based on MNIST Dataset
Chaouki Abdallah

ArXiv, 2021

Federated Learning is an algorithm suited for training models on decentralized data, but the requirement of a central "server" node is a bottleneck. In this document, we first introduce the notion of Decentralized Federated Learning (DFL). We then perform various experiments on different setups, such as changing model aggregation frequency, switching from independent and identically distributed (IID) dataset partitioning to non-IID partitioning with partial global sharing, using different optimization methods across clients, and breaking models into segments with partial sharing. All experiments are run on the MNIST handwritten digits dataset. We observe that those altered training procedures are generally robust, albeit non-optimal. We also observe failures in training when the variance between model weights is too large. The open-source experiment code is accessible through GitHub1.

View PDFchevron_right

References (39)

  1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irv- ing, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: a system for Large-Scale machine learn- ing (OSDI'16). https://www.usenix.org/conference/osdi16/technical- sessions/presentation/abadi
  2. Dan Alistarh, Demjan Grubic, Jerry Z. Li, Ryota Tomioka, and Milan Vojnovic. 2017. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NIPS'17). https://proceedings.neurips.cc/paper_files/paper/2017/file/ 6c340f25839e6acdc73414517203f5f0-Paper.pdf
  3. Dan Alistarh, Torsten Hoefler, Mikael Johansson, Sarit Khirirat, Nikola Konstantinov, and Cédric Renggli. 2018. The Convergence of Sparsified Gradient Methods (NIPS'18). https://proceedings.neurips.cc/paper_ files/paper/2018/file/314450613369e0ee72d0da7f6fee773c-Paper.pdf
  4. Batiste Le Bars, Aurélien Bellet, Marc Tommasi, Erick Lavoie, and Anne-Marie Kermarrec. 2023. Refined Convergence and Topology Learning for Decentralized Optimization with Heterogeneous Data (AISTATS'23). arXiv:2204.04452
  5. Aurélien Bellet, Anne-Marie Kermarrec, and Erick Lavoie. 2022. D- Cliques: Compensating for Data Heterogeneity with Topology in De- centralized Federated Learning (SRDS'22). https://doi.org/10.1109/ SRDS55811.2022.00011
  6. Juan Benet. 2014. IPFS -Content Addressed, Versioned, P2P File System. CoRR (2014). arXiv:1407.3561
  7. Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcol- let, and Nicholas D Lane. 2020. Flower: A Friendly Federated Learning Research Framework. (2020). arXiv:2007.14390
  8. Luca Boccassi et al. 2023. ZeroMQ: An open-source universal messag- ing library. https://zeromq.org
  9. Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, et al. 2019. Towards Federated Learning at Scale: System Design (ML- Sys'19). https://proceedings.mlsys.org/paper_files/paper/2019/file/ bd686fd640be98efaae0091fa301e613-Paper.pdf
  10. Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. 2017. Practical Secure Aggregation for Privacy-Preserving Machine Learning (CCS '17). https://doi.org/10.1145/3133956.3133982
  11. Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. 2019. Leaf: A benchmark for federated settings. In 2nd Intl. Workshop on Federated Learning for Data Privacy and Confidentiality (FL-NeurIPS'19). arXiv:1812.01097
  12. Akash Dhasade, Nevena Dresevic, Anne-Marie Kermarrec, and Rafael Pires. 2022. TEE-based decentralized recommender systems: The raw data sharing redemption (IPDPS'22). https://doi.org/10.1109/ IPDPS53621.2022.00050
  13. Paulo Gouveia, João Neves, Carlos Segarra, Luca Liechti, Shady Issa, Valerio Schiavoni, and Miguel Matos. 2020. Kollaps: Decentralized and Dynamic Topology Emulation (EuroSys '20). Article 23. https: //doi.org/10.1145/3342195.3387540
  14. Chaoyang He, Songze Li, Jinhyun So, Mi Zhang, Hongyi Wang, Xi- aoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, and Salman Avestimehr. 2020. FedML: A re- search library and benchmark for federated machine learning. (2020). arXiv:2007.13518
  15. Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B. Gibbons. 2020. The Non-IID Data Quagmire of Decentralized Machine Learning. In Proceedings of the 37th International Conference on Machine Learning (ICML'20). Article 408. http://proceedings.mlr.press/v119/hsieh20a/ hsieh20a.pdf
  16. Márk Jelasity, Spyros Voulgaris, Rachid Guerraoui, Anne-Marie Ker- marrec, and Maarten Van Steen. 2007. Gossip-based peer sampling. ACM Transactions on Computer Systems (TOCS) 25, 3 (2007), 8-es.
  17. Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Gar- rett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchin- son, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Kho- dak, Jakub Konecný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebas- tian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. 2020. Advances and open problems in federated learning. Foundations and Trends in Machine Learning 14, 1-2 (2020). https://doi.org/10.1561/2200000083
  18. Anastasia Koloskova, Tao Lin, Sebastian U Stich, and Martin Jaggi. 2020. Decentralized Deep Learning with Arbitrary Communication Com- pression (ICLR'20). https://openreview.net/forum?id=SkgGCkrKvH
  19. Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. 2020. A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (ICML'20). https://proceedings. mlr.press/v119/koloskova20a.html
  20. Anastasia Koloskova, Sebastian Stich, and Martin Jaggi. 2019. De- centralized stochastic optimization and gossip algorithms with com- pressed communication (ICML'19). https://proceedings.mlr.press/v97/ koloskova19a.html
  21. Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2014. The CIFAR-10 dataset. 55, 5 (2014). https://www.cs.toronto.edu/~kriz/cifar.html
  22. Fan Lai, Yinwei Dai, Sanjay Singapuram, et al. 2022. FedScale: Bench- marking Model and System Performance of Federated Learning at Scale (ICML'22). https://proceedings.mlr.press/v162/lai22a.html
  23. Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. 2017. Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gra- dient Descent (NIPS'17). https://proceedings.neurips.cc/paper_files/ paper/2017/file/f75526659f31040afeb61cb7133e4e6d-Paper.pdf
  24. Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. 2018. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (ICLR'18). https://openreview.net/forum?id= SkhQHMW0W
  25. Yang Liu, Tao Fan, Tianjian Chen, Qian Xu, and Qiang Yang. 2021. FATE: An Industrial Grade Platform for Collaborative Learning With Data Protection. J. Mach. Learn. Res. 22, 226 (2021). http://jmlr.org/ papers/v22/20-815.html
  26. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data (AISTATS'17). https://proceedings. mlr.press/v54/mcmahan17a/mcmahan17a.pdf
  27. Message Passing Interface Forum. 2021. MPI: A Message-Passing In- terface Standard Version 4.0. https://www.mpi-forum.org/docs/mpi- 4.0/mpi40-report.pdf
  28. Christodoulos Pappas, Dimitris Chatzopoulos, Spyros Lalis, and Manolis Vavalis. 2021. IPLS: A Framework for Decentralized Fed- erated Learning (IFIP Networking'21). https://doi.org/10.23919/ IFIPNetworking52078.2021.9472790
  29. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Brad- bury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS'19. https://proceedings.neurips.cc/paper_files/paper/2019/ file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf
  30. Holger R Roth, Yan Cheng, Yuhong Wen, Isaac Yang, Ziyue Xu, YuanT- ing Hsieh, Kristopher Kersten, Ahmed Harouni, Can Zhao, Kevin Lu, Zhihong Zhang, Wenqi Li, Andriy Myronenko, Dong Yang, Sean Yang, Nicola Rieke, Abood Quraini, Chester Chen, Daguang Xu, Nic Ma, Prerna Dogra, Mona G Flores, and Andrew Feng. 2022. NVIDIA FLARE: Federated Learning from Simulation to Real-World. In Work- shop on Federated Learning: Recent Advances and New Challenges. https://openreview.net/forum?id=hD9QaIQTL_f
  31. Rishi Sharma et al. 2022. decentralizepy: An open-source decen- tralized learning research framework. https://github.com/sacs- epfl/decentralizepy
  32. Nikko Strom. 2015. Scalable distributed DNN training using commodity GPU cloud computing. In 16th Annual Conference of the International Speech Communication Association (INTER- SPEECH'15). https://www.isca-speech.org/archive_v0/interspeech_ 2015/papers/i15_1488.pdf
  33. Thijs Vogels, Hadrien Hendrikx, and Martin Jaggi. 2022. Beyond spectral gap: the role of the topology in decentralized learning (NeurIPS'22). https://proceedings.neurips.cc/paper_files/paper/2022/ file/61162d94822d468ee6e92803340f2040-Paper-Conference.pdf
  34. Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. 2020. Prac- tical Low-Rank Communication Compression in Decentralized Deep Learning (NeurIPS'20). https://proceedings.neurips.cc/paper_files/ paper/2020/file/a376802c0811f1b9088828288eb0d3f0-Paper.pdf
  35. Milos Vujasinovic. 2023. Secure Aggregation on Sparse Mod- els in Decentralized Learning Systems. Master's thesis. EPFL. https://www.epfl.ch/labs/sacs/wp-content/uploads/2023/02/Secure_ Aggregation_on_Sparse_Models_in_Decentralized_Learning_ Systems___Milos_Vujasinovic.pdf
  36. Lin Xiao, Stephen Boyd, and Seung-Jean Kim. 2007. Distributed average consensus with least-mean-square deviation. J. Parallel and Distrib. Comput. 67, 1 (2007). https://doi.org/10.1016/j.jpdc.2006.08.010
  37. Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. 2018. Applied federated learning: Improving google keyboard query suggestions. (2018). arXiv:1812.02903
  38. Tongtian Zhu, Fengxiang He, Lan Zhang, Zhengyang Niu, Mingli Song, and Dacheng Tao. 2022. Topology-aware generalization of decentralized SGD (ICML'22). https://proceedings.mlr.press/v162/ zhu22d.html
  39. Alexander Ziller, Andrew Trask, Antonio Lopardo, et al. 2021. PySyft: A library for easy federated learning. In Federated Learning Systems. https://doi.org/10.1007/978-3-030-70604-3_5

Related papers

Federated Learning using Peer-to-peer Network for Decentralized Orchestration of Model Weights
Suresh Shetty

2021

In recent times, Machine learning and Artificial intelligence have become one of the key emerging fields of computer science. Many researchers and businesses are benefited by machine learning models that are trained by data processing at scale. However, machine learning, and particularly Deep Learning requires large amounts of data, that in several instances are proprietary and confidential to many businesses. In order to respect individual organization’s privacy in collaborative machine learning, federated learning could play a crucial role. Such implementations of privacy preserving federated learning find applicability in various ecosystems like finance, health care, legal, research and other fields that require preservation of privacy. However, many such implementations are driven by a centralized architecture in the network, where the aggregator node becomes the single point of failure, and is also expected with lots of computing resources at its disposal. In this paper, we pro...

View PDFchevron_right
Communication Efficient Framework for Decentralized Machine Learning
Anis Elgabli

2020 54th Annual Conference on Information Sciences and Systems (CISS)

In this paper, we propose a fast, privacy-aware, and communication-efficient decentralized framework to solve the distributed machine learning (DML) problem. The proposed algorithm is based on the Alternating Direction Method of Multipliers (ADMM) algorithm. The key novelty in the proposed algorithm is that it solves the problem in a decentralized topology where at most half of the workers are competing the limited communication resources at any given time. Moreover, each worker exchanges the locally trained model only with two neighboring workers, thereby training a global model with a lower amount of communication overhead in each exchange. We prove that GADMM converges faster than the centralized batch gradient descent for convex loss functions, and numerically show that it converges faster and more communication-efficient than the state-of-the-art communication-efficient algorithms such as the Lazily Aggregated Gradient (LAG) and dual averaging, in linear and logistic regression tasks on synthetic and real datasets.

View PDFchevron_right
Establishing the Parameters of a Decentralized Neural Machine Learning Model
Aline Ioste

Anais do XIX Encontro Nacional de Inteligência Artificial e Computacional (ENIAC 2022)

The decentralized machine learning models face a bottleneck of high-cost communication. Trade-offs between communication and accuracy in decentralized learning have been addressed by theoretical approaches. Here we propose a new practical model that performs several local training operations before a communication round, choosing among several options. We show how to determine a configuration that dramatically reduces the communication burden between participant hosts, with a reduction in communication practice showing robust and accurate results both to IID and NON-IID data distributions.

View PDFchevron_right
FedGrad: Optimisation in Decentralised Machine Learning
Mann Patel

Cornell University - arXiv, 2022

Federated Learning is a machine learning paradigm where we aim to train machine learning models in a distributed fashion. Large number of clients/edge devices collaborate with each other to train a single model on the central. Clients do not share their own dataset with each other, decoupling computation and data on the same device. In this paper, we propose yet another adaptive federated optimization method, and some other ideas in the field of federated learning. We also perform experiments using these methods and showcase the improvement in overall performance of federated learning.

View PDFchevron_right
BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning
Bicheng Ying

ArXiv, 2021

Decentralized algorithm is a form of computation that achieves a global goal through local dynamics that relies on low-cost communication between directly-connected agents. On large-scale optimization tasks involving distributed datasets, decentralized algorithms have shown strong, sometimes superior, performance over distributed algorithms with a central node. Recently, developing decentralized algorithms for deep learning has attracted great attention. They are considered as low-communication-overhead alternatives to those using a parameter server or the Ring-Allreduce protocol. However, the lack of an easy-to-use and efficient software package has kept most decentralized algorithms merely on paper. To fill the gap, we introduce BlueFog, a python library for straightforward, high-performance implementations of diverse decentralized algorithms. Based on a unified abstraction of various communication operations, BlueFog offers intuitive interfaces to implement a spectrum of decentra...

View PDFchevron_right

Related topics

  • Computer Science
  • Distributed Computing
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved