CrayLabs and user contibuted examples of using SmartSim for various simulation and machine learning applications.

Overview

SmartSim Example Zoo

This repository contains CrayLabs and user contibuted examples of using SmartSim for various simulation and machine learning applications.

The CrayLabs team will attempt to keep examples updated with current releases but all user contibuted examples should specify the release they were created with.

Contibuting Examples

We welcome any and all contibutions to this repository. The CrayLabs team will do their best to review in a timely manner. We ask that, if you contribute examples, please include a description and all references to code and relavent previous implemenations or open source code that the work is based off of for the benefit of anyone who would like to try out your example.

Examples by Paper

The following examples are implemented based on existing research papers. Each example lists the paper, previous works, and links to the implementation (possibly stored within this repository or a seperate repository)

1. DeepDriveMD

  • Contibuting User: CrayLabs
  • Tags: OpenMM, CVAE, online inference, unsupervised online learning, PyTorch, ensemble

This use case highlights many features of SmartSim and SmartRedis and together they can be used to orchestrate complex workflows with coupled applications without using the filesystem for exchanging information.

More specifically, this use case is based on the original DeepDriveMD work. DeepDriveMD was furthered with an asynchronous streaming version. SmartSim extends the streaming implementation through the use of the SmartSim architecture. The main difference between the SmartSim implementation and the previous implementations, is that neither ML models, nor Molecular Dynamics (MD) intermediate results are stored on the file system. Additionally, the inference portion of the workflow takes place inside the database instead of a seperate task launched on the system.

2. TensorFlowFoam

  • Contributing User: CrayLabs
  • Tags: Online Inference, TensorFlow, OpenFOAM, supervised learning

This example shows how to use TensorFlow inside of OpenFOAM simulations using SmartSim.

More specifically, this SmartSim use case adapts the TensorFlowFoam work which utilized a deep neural network to predict steady-state turbulent viscosities of the Spalart-Allmaras (SA) model. This use case highlights that a machine learning model can be evaluated using SmartSim from within a simulation with minimal external library code. For the OpenFOAM use case herein, only four SmartRedis client API calls are needed to initialize a client connection, send tensor data for evaluation, execute the TensorFlow model, and retrieve the model inference result.

In general, this example provides a useful driver script for those looking to run OpenFOAM with SmartSim.

3. ML-EKE

  • Contributing User: CrayLabs
  • Tags: Online inference, MOM6, climate modeling, ensemble, parameterization replacement

This example was a collaboration between CrayLabs (HPE), NCAR, and the university of Victoria. Using SmartSim, this example shows how to run an ensemble of simulations all using the SmartSim architecture to replace a parameterization (MEKE) within each global ocean simulation (MOM6).

Paper Abstract:

We demonstrate the first climate-scale, numerical ocean simulations improved through distributed, online inference of Deep Neural Networks (DNN) using SmartSim. SmartSim is a library dedicated to enabling online analysis and Machine Learning (ML) for traditional HPC simulations. In this paper, we detail the SmartSim architecture and provide benchmarks including online inference with a shared ML model on heterogeneous HPC systems. We demonstrate the capability of SmartSim by using it to run a 12-member ensemble of global-scale, high-resolution ocean simulations, each spanning 19 compute nodes, all communicating with the same ML architecture at each simulation timestep. In total, 970 billion inferences are collectively served by running the ensemble for a total of 120 simulated years. Finally, we show our solution is stable over the full duration of the model integrations, and that the inclusion of machine learning has minimal impact on the simulation runtimes.

Since this is original research done by CrayLabs, there is no previous implementation.

Examples by Simulation Model

LAMMPS

SmartSim examples with LAMMPS which is a Molecular Dynamics simulation model.

1. Online Analysis of Atom Position

  • Contibuting User: CrayLabs
  • Tags: Molecular Dynamics, online analysis, visualizations.

LAMMPS has dump styles which are custom I/O methods that can be implmentated by users. CrayLabs implemented a SMARTSIM dump style which uses the SmartRedis clients to stream data to an Orchestrator database created by SmartSim.

Once the data is in the database, any application with a SmartRedis client can consume that data. For this example, we have a simple Python script that uses iPyVolume to plot the data every 100 iterations.

Examples by System

High Performance Computing Systems are a bit like snowflakes, they are all different. Since each one has their own quirks, some examples for specific and popular systems can be of benefit to new users.

National Center for Atmospheric Research (NCAR)

1. Cheyenne

  • Contibuting User: CrayLabs
  • implementation (this repo)
  • WLM: PBSPro
  • System: SGI 8600
  • CPU: intel
  • GPU: None

2. Casper

  • Contibuting user: @jedwards4b
  • Implementation (this repo)
  • WLM: PBSPro
  • GPU: Nvidia
  • CPU: Intel
  • SmartSim Version: 0.3.2
  • SmartRedis Version: 0.2.0

Oak Ridge National Lab

1. Summit

  • Contributing user: CrayLabs
  • implementation (this repo)
  • System:
  • OS: Red Hat Enterprise Linux (RHEL)
  • CPU: Power9
  • GPU: Nvidia V100
Owner
Cray Labs
Cray Labs
Book Item Based Collaborative Filtering

Book-Item-Based-Collaborative-Filtering Collaborative filtering methods are used

Şebnem 3 Jan 06, 2022
As we all know the BGMI Loot Crate comes with so many resources for the gamers, this ML Crate will be the hub of various ML projects which will be the resources for the ML enthusiasts! Open Source Program: SWOC 2021 and JWOC 2022.

Machine Learning Loot Crate 💻 🧰 🔴 Welcome contributors! As we all know the BGMI Loot Crate comes with so many resources for the gamers, this ML Cra

Abhishek Sharma 89 Dec 28, 2022
A Streamlit demo to interactively visualize Uber pickups in New York City

Streamlit Demo: Uber Pickups in New York City A Streamlit demo written in pure Python to interactively visualize Uber pickups in New York City. View t

Streamlit 230 Dec 28, 2022
List of Data Science Cheatsheets to rule the world

Data Science Cheatsheets List of Data Science Cheatsheets to rule the world. Table of Contents Business Science Business Science Problem Framework Dat

Favio André Vázquez 11.7k Dec 30, 2022
Machine Learning Study 혼자 해보기

Machine Learning Study 혼자 해보기 기여자 (Contributors) ✨ Teddy Lee 🏠 HongJaeKwon 🏠 Seungwoo Han 🏠 Tae Heon Kim 🏠 Steve Kwon 🏠 SW Song 🏠 K1A2 🏠 Wooil

Teddy Lee 1.7k Jan 01, 2023
A complete guide to start and improve in machine learning (ML)

A complete guide to start and improve in machine learning (ML), artificial intelligence (AI) in 2021 without ANY background in the field and stay up-to-date with the latest news and state-of-the-art

Louis-François Bouchard 3.3k Jan 04, 2023
Azure Cloud Advocates at Microsoft are pleased to offer a 12-week, 24-lesson curriculum all about Machine Learning

Azure Cloud Advocates at Microsoft are pleased to offer a 12-week, 24-lesson curriculum all about Machine Learning

Microsoft 43.4k Jan 04, 2023
Iris-Heroku - Putting a Machine Learning Model into Production with Flask and Heroku

Puesta en Producción de un modelo de aprendizaje automático con Flask y Heroku L

Jesùs Guillen 1 Jun 03, 2022
Implementation of the Object Relation Transformer for Image Captioning

Object Relation Transformer This is a PyTorch implementation of the Object Relation Transformer published in NeurIPS 2019. You can find the paper here

Yahoo 158 Dec 24, 2022
SIMD-accelerated bitwise hamming distance Python module for hexidecimal strings

hexhamming What does it do? This module performs a fast bitwise hamming distance of two hexadecimal strings. This looks like: DEADBEEF = 1101111010101

Michael Recachinas 12 Oct 14, 2022
A GitHub action that suggests type annotations for Python using machine learning.

Typilus: Suggest Python Type Annotations A GitHub action that suggests type annotations for Python using machine learning. This action makes suggestio

40 Sep 18, 2022
The project's goal is to show a real world application of image segmentation using k means algorithm

The project's goal is to show a real world application of image segmentation using k means algorithm

2 Jan 22, 2022
Code Repository for Machine Learning with PyTorch and Scikit-Learn

Code Repository for Machine Learning with PyTorch and Scikit-Learn

Sebastian Raschka 1.4k Jan 03, 2023
This is my implementation on the K-nearest neighbors algorithm from scratch using Python

K Nearest Neighbors (KNN) algorithm In this Machine Learning world, there are various algorithms designed for classification problems such as Logistic

sonny1902 1 Jan 08, 2022
An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models

Seldon Core: Blazing Fast, Industry-Ready ML An open source platform to deploy your machine learning models on Kubernetes at massive scale. Overview S

Seldon 3.5k Jan 01, 2023
Multiple Linear Regression using the LinearRegression class from sklearn.linear_model library

Multiple-Linear-Regression-master - A python program to implement Multiple Linear Regression using the LinearRegression class from sklearn.linear model library

Kushal Shingote 1 Feb 06, 2022
Iterative stochastic gradient descent (SGD) linear regressor with regularization

SGD-Linear-Regressor Iterative stochastic gradient descent (SGD) linear regressor with regularization Dataset: Kaggle “Graduate Admission 2” https://w

Zechen Ma 1 Oct 29, 2021
Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

Horovod Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. The goal of Horovod is to make dis

Horovod 12.9k Jan 07, 2023
Fourier-Bayesian estimation of stochastic volatility models

fourier-bayesian-sv-estimation Fourier-Bayesian estimation of stochastic volatility models Code used to run the numerical examples of "Bayesian Approa

15 Jun 20, 2022
Meerkat provides fast and flexible data structures for working with complex machine learning datasets.

Meerkat makes it easier for ML practitioners to interact with high-dimensional, multi-modal data. It provides simple abstractions for data inspection, model evaluation and model training supported by

Robustness Gym 115 Dec 12, 2022