A highly modular PyTorch framework with a focus on Neural Architecture Search (NAS).

Related tags

Deep Learninguninas
Overview

UniNAS

A highly modular PyTorch framework with a focus on Neural Architecture Search (NAS).

under development

(which happens mostly on our internal GitLab, we push only every once in a while to Github)

  • APIs may change
  • argparse arguments may be moved to more fitting classes
  • there may be incomplete or not-yet-working pieces of code
  • ...

Features

  • modular and therefore reusable
    • data set loading,
    • network building code and topologies,
    • methods to train architecture weights,
    • sets of operations (primitives),
    • weight initializers,
    • metrics,
    • ... and more
  • everything is configurable from the command line and/or config files
    • improved reproducibility, since detailed run configurations are saved and logged
    • powerful search network descriptions enable e.g. highly customizable weight sharing settings
    • the underlying argparse mechanism enables using a GUI for configurations
  • compare results of different methods in the same environment
  • import and export detailed network descriptions
  • integrate new methods and more with fairly little effort
  • NAS-Benchmark integration
    • NAS-Bench 201
  • ... and more

Where is this code from?

Except for a few pieces, the code is entirely self-written. However, sometimes the (official) code is useful to learn from or clear up some details, and other frameworks can be used for their nice features.

Other meta-NAS frameworks

  • Deep Architect
    • highly customizable search spaces, hyperparameters, ...
    • the searchers (SMBO, MCTS, ...) focus on fully training (many) models and are not differentiable
  • D-X-Y NAS-Projects
  • Auto-PyTorch
    • stronger focus on model selection than optimizing one architecture
  • Vega
  • NNI

Repository notes

Dynamic argparse tree

Everything is an argument. Learning rate? Argument. Scheduler? Argument. The exact topology of a Network, including how many of each cell and whether they share their architecture weights? Also arguments.

This is enabled by the idea that each used class (method, network, cells, regularizers, ...) can add arguments to argparse, including which further classes are required (e.g. a method needs a network, which needs a stem).

It starts with the Main class adding a Task (cls_task), which itself adds all required components (cls_*).

To see all available (meta) arguments, run Main.list_all_arguments() in uninas/main.py

Graphical user interface

Since putting together the arguments correctly is not trivial (and requires some familiarity with the code base), an easier approach is using a GUI.

Have a look at uninas/gui/tk_gui/main.py, a tkinter GUI frontend.

The GUI can automatically filter usable classes, display available arguments, and display tooltips; based only on the implemented argparse (meta) arguments in the respective classes.

Some meta arguments take a single class name:

e.g: cls_task, cls_trainer, cls_data, cls_criterion, cls_method

The chosen classes define their own arguments, e.g.:

  • cls_trainer="SimpleTrainer"
  • SimpleTrainer.max_epochs=100
  • SimpleTrainer.test_last=10

Their names are also available as wildcards, automatically using their respectively set class name:

  • cls_trainer="SimpleTrainer"
  • {cls_trainer}.max_epochs --> SimpleTrainer.max_epochs
  • {cls_trainer}.test_last --> SimpleTrainer.test_last

Some meta arguments take a comma-separated list of class names:

e.g. cls_metrics, cls_initializers, cls_regularizers, cls_optimizers, cls_schedulers

The chosen classes also define their own arguments, but always include an index, e.g.:

  • cls_regularizers="DropOutRegularizer, DropPathRegularizer"
  • DropOutRegularizer#0.prob=0.5
  • DropPathRegularizer#1.max_prob=0.3
  • DropPathRegularizer#1.drop_id_paths=false

And they are also available as indexed wildcards:

  • cls_regularizers="DropOutRegularizer, DropPathRegularizer"
  • {cls_regularizers#0}.prob --> DropOutRegularizer#0.prob
  • {cls_regularizers#1}.max_prob --> DropPathRegularizer#1.max_prob
  • {cls_regularizers#1}.drop_id_paths --> DropPathRegularizer#1.drop_id_paths

Register

UniNAS makes heavy use of a registering mechanism (via decorators in uninas/register.py). Classes of the same type (e.g. optimizers, networks, ...) will register in one RegisterDict.

Registered classes can be accessed via their name in the Register, no matter of their actual location in the code. This enables e.g. saving network topologies as nested dictionaries, no matter how complicated they are, since the class names are enough to find the classes in the code. (It also grants a certain amount of refactoring-freedom.)

Exporting networks

(Trained) Networks can easily be used by other PyTorch frameworks/scripts, see verify.py for an easy example.

Citation

The framework

we will possibly create a whitepaper at some point

@misc{kl2020uninas,
  author = {Kevin Alexander Laube},
  title = {UniNAS},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/cogsys-tuebingen/uninas}}
}

Inter-choice dependent super-network weights

  1. Train super-networks, e.g. via experiments/demo/inter_choice_weights/icw1_train_supernet_nats.py
    • you will need Cifar10, but can also easily use fake data or download it
    • to generate SubImageNet see uninas/utils/generate/data/subImageNet
  2. Evaluate the super-network, e.g. via experiments/demo/inter_choice_weights/icw2_eval_supernet.py
  3. View the evaluation results in the save dir, in TensorBoard or plotted directly
@article{laube2021interchoice,
  title={Inter-choice dependent super-network weights},
  author={Kevin Alexander Laube, Andreas Zell},
  journal={arXiv preprint arXiv:2104.11522},
  year={2021}
}
Owner
Cognitive Systems Research Group
Autonomous Mobile Robots; Bioinformatics; Chemo- and Geoinformatics; Evolutionary Algorithms; Machine Learning
Cognitive Systems Research Group
Estimation of human density in a closed space using deep learning.

Siemens HOLLZOF challenge - Human Density Estimation Add project description here. Installing Dependencies: Install Python3 either system-wide, user-w

3 Aug 08, 2021
A paper using optimal transport to solve the graph matching problem.

GOAT A paper using optimal transport to solve the graph matching problem. https://arxiv.org/abs/2111.05366 Repo structure .github: Files specifying ho

neurodata 8 Jan 04, 2023
[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models Codes for this paper The Lottery Tickets Hypo

VITA 59 Dec 28, 2022
Bayesian regularization for functional graphical models.

BayesFGM Paper: Jiajing Niu, Andrew Brown. Bayesian regularization for functional graphical models. Requirements R version 3.6.3 and up Python 3.6 and

0 Oct 07, 2021
Fully convolutional networks for semantic segmentation

FCN-semantic-segmentation Simple end-to-end semantic segmentation using fully convolutional networks [1]. Takes a pretrained 34-layer ResNet [2], remo

Kai Arulkumaran 186 Dec 25, 2022
Lightweight mmm - Lightweight (Bayesian) Media Mix Model

Lightweight (Bayesian) Media Mix Model This is not an official Google product. L

Google 342 Jan 03, 2023
Linear algebra python - Number of operations and problems in Linear Algebra and Numerical Linear Algebra

Linear algebra in python Number of operations and problems in Linear Algebra and

Alireza 5 Oct 09, 2022
Source Code and data for my paper titled Linguistic Knowledge in Data Augmentation for Natural Language Processing: An Example on Chinese Question Matching

Description The source code and data for my paper titled Linguistic Knowledge in Data Augmentation for Natural Language Processing: An Example on Chin

Zhengxiang Wang 3 Jun 28, 2022
DSTC10 Track 2 - Knowledge-grounded Task-oriented Dialogue Modeling on Spoken Conversations

DSTC10 Track 2 - Knowledge-grounded Task-oriented Dialogue Modeling on Spoken Conversations This repository contains the data, scripts and baseline co

Alexa 51 Dec 17, 2022
CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

CoTr: Efficient 3D Medical Image Segmentation by bridging CNN and Transformer This is the official pytorch implementation of the CoTr: Paper: CoTr: Ef

218 Dec 25, 2022
BARTScore: Evaluating Generated Text as Text Generation

This is the Repo for the paper: BARTScore: Evaluating Generated Text as Text Generation Updates 2021.06.28 Release online evaluation Demo 2021.06.25 R

NeuLab 196 Dec 17, 2022
Julia package for contraction of tensor networks, based on the sweep line algorithm outlined in the paper General tensor network decoding of 2D Pauli codes

Julia package for contraction of tensor networks, based on the sweep line algorithm outlined in the paper General tensor network decoding of 2D Pauli codes

Christopher T. Chubb 35 Dec 21, 2022
i-RevNet Pytorch Code

i-RevNet: Deep Invertible Networks Pytorch implementation of i-RevNets. i-RevNets define a family of fully invertible deep networks, built from a succ

Jörn Jacobsen 378 Dec 06, 2022
Tandem Mass Spectrum Prediction with Graph Transformers

MassFormer This is the original implementation of MassFormer, a graph transformer for small molecule MS/MS prediction. Check out the preprint on arxiv

Röst Lab 13 Oct 27, 2022
Everything you want about DP-Based Federated Learning, including Papers and Code. (Mechanism: Laplace or Gaussian, Dataset: femnist, shakespeare, mnist, cifar-10 and fashion-mnist. )

Differential Privacy (DP) Based Federated Learning (FL) Everything about DP-based FL you need is here. (所有你需要的DP-based FL的信息都在这里) Code Tip: the code o

wenzhu 83 Dec 24, 2022
An official source code for paper Deep Graph Clustering via Dual Correlation Reduction, accepted by AAAI 2022

Dual Correlation Reduction Network An official source code for paper Deep Graph Clustering via Dual Correlation Reduction, accepted by AAAI 2022. Any

yueliu1999 109 Dec 23, 2022
Large-scale language modeling tutorials with PyTorch

Large-scale language modeling tutorials with PyTorch 안녕하세요. 저는 TUNiB에서 머신러닝 엔지니어로 근무 중인 고현웅입니다. 이 자료는 대규모 언어모델 개발에 필요한 여러가지 기술들을 소개드리기 위해 마련하였으며 기본적으로

TUNiB 172 Dec 29, 2022
An educational AI robot based on NVIDIA Jetson Nano.

JetBot Looking for a quick way to get started with JetBot? Many third party kits are now available! JetBot is an open-source robot based on NVIDIA Jet

NVIDIA AI IOT 2.6k Dec 29, 2022
Finetune the base 64 px GLIDE-text2im model from OpenAI on your own image-text dataset

Finetune the base 64 px GLIDE-text2im model from OpenAI on your own image-text dataset

Clay Mullis 82 Oct 13, 2022
Hand-distance-measurement-game - Hand Distance Measurement Game

Hand Distance Measurement Game This is program is made to calculate the distance

Priyansh 2 Jan 12, 2022