BlueFog Tutorials

Welcome to the BlueFog tutorials!

In this repository, we've put together a collection of awesome Jupyter notebooks. These notebooks serve two purposes:

• Help readers understand the basic concepts and theories of the decentralized optimization.
• Help readers understand how to implement decentralized algorithms with the BlueFog library.

Contents

1 Preliminary

Learn how to write your first "hello world" program over the real multi-CPU system with BlueFog.

2 Average Consensus Algorithm

Learn how to achieve the globally averaged consensus among nodes in a decentralized manner.

3 Decentralized Gradient Descent

Learn how to solve a general distributed (possibly stochastic) optimization problem in a decentralized manner.

4 Decentralized Gradient Descent with Bias-Correction

Learn how to accelerate your decentralized (possibly stochastic) optimization algorithms with various bias-correction techniques.

5 Decentralized Optimization over directed and time-varying networks

Learn how to solve distributed optimization in a decentralized manner if the connected topology is directed or time-varying.

6 Asynchronous Decentralized Optimization

Learn how to solve a general distributed optimization problem with asynchronous decentralized algorithms.

7 Decentralized Deep Learning

Learn how to train a deep neural network with decentralized optimization algorithms.

Call for Contributions

This tutorial only contains the very basic concepts, algorithms, theories, and implementations for decentralized optimization. It misses many important recent progress in the algorithm development and theory in the decentralized optimization community. We hope you will consider using BlueFog in the experiment of your new decentralized algorithm and summarize your ideas into a Jupyter notebook tutorial.

About BlueFog Team

The BlueFog Team involves several researchers and engineers that target to make decentralized algorithms practical for large-scale optimization and deep learning. We hope to bridge the gap between the theoretical progress of decentralized algorithms in the academia and the real implementation in the industry. We hope more researchers and engineers can join us to contribute to the community of decentralized optimization.

Other Resources:

Faster Learning over Networks and BlueFog, BlueFog Team, invited talk at MLA, 2020 [slides]

Parallel, Distributed, and Decentralized optimization methods, Wotao Yin, Tutorial in ECOM2021, 2021 [Materials]

Citation

Feel free to share the BlueFog repo and this tutorial to anyone that has an interest. If you use BlueFog, please cite it as follows:

@software{bluefog2021_4616052,
  author       = {BlueFog Team},
  title        = {BlueFog: Make Decentralized Algorithms Practical For Optimization and Deep Learning},
  month        = Mar.,
  year         = 2021,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.4616052},
  url          = {https://doi.org/10.5281/zenodo.4616052}
}