Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Last update: Sep 08, 2022

Overview

Cross-framework Python Package for Evaluation of Latent-based Generative Models

Latte

Latte (for LATent Tensor Evaluation) is a cross-framework Python package for evaluation of latent-based generative models. Latte supports calculation of disentanglement and controllability metrics in both PyTorch (via TorchMetrics) and TensorFlow.

Installation

For developers working on local clone, cd to the repo and replace latte with .. For example, pip install .[tests]

pip install latte-metrics           # core (numpy only)
pip install latte-metrics[pytorch]  # with torchmetrics wrapper
pip install latte-metrics[keras]    # with tensorflow wrapper
pip install latte-metrics[tests]    # for testing

Running tests locally

pip install .[tests]
pytest tests/ --cov=latte

Example

Functional API

import latte
from latte.functional.disentanglement.mutual_info import mig
import numpy as np

latte.seed(42)

z = np.random.randn(16, 8)
a = np.random.randn(16, 2)

mutual_info_gap = mig(z, a, discrete=False, reg_dim=[4, 3])

Modular API

import latte
from latte.metrics.core.disentanglement import MutualInformationGap
import numpy as np

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.compute()

TorchMetrics API

import latte
from latte.metrics.torch.disentanglement import MutualInformationGap
import torch

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update(z, attributes)

mig_val = mig.compute()

Keras Metric API

import latte
from latte.metrics.keras.disentanglement import MutualInformationGap
from tensorflow import keras as tfk

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.result()

Documentation

https://latte.readthedocs.io/en/latest

Supported metrics

Metric	Latte Functional	Latte Modular	TorchMetrics	Keras Metric
Disentanglement Metrics
📝 Mutual Information Gap (MIG)	🧪	🧪	🧪	??
📝 Dependency-blind Mutual Information Gap (DMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Mutual Information Gap (XMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Latent Information Gap (DLIG)	🧪	🧪	🧪	🧪
📝 Separate Attribute Predictability (SAP)	🧪	🧪	🧪	🧪
📝 Modularity	🧪	🧪	🧪	🧪
📝 β-VAE Score	👀	👀	👀	👀
📝 FactorVAE Score	👀	👀	👀	👀
📝 DCI Score	👀	👀	👀	👀
📝 Interventional Robustness Score (IRS)	👀	👀	👀	👀
📝 Consistency	👀	👀	👀	👀
📝 Restrictiveness	👀	👀	👀	👀
Interpolatability Metrics
📝 Smoothness	🧪	🧪	🧪	🧪
📝 Monotonicity	🧪	🧪	🧪	🧪
📝 Latent Density Ratio	🕣	🕣	🕣	🕣
📝 Linearity	👀	👀	👀	👀

Bundled metric modules

🧪 Experimental (subject to changes) | ✔️ Stable | 🔨 In Progress | 🕣 In Queue

Metric Bundle	Latte Functional	Latte Modular	TorchMetrics	Keras Metric	Included
Dependency-aware Disentanglement	🧪	🧪	🧪	🧪	MIG, DMIG, XMIG, DLIG
LIAD-based Interpolatability	🧪	🧪	🧪	🧪	Smoothness, Monotonicity

Cite

For individual metrics, please cite the paper according to the link in the 📝 icon in front of each metric.

If you find our package useful please cite our repository and arXiv preprint as

@article{
  watcharasupat2021latte,
  author = {Watcharasupat, Karn N. and Lee, Junyoung and Lerch, Alexander},
  title = {{Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models}},
  eprint={2112.10638},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url = {https://github.com/karnwatcharasupat/latte}
  doi = {10.5281/zenodo.5786402}
}

Comments

Documentation: Metric Descriptions

Might be nice to provide a short description for each metric in addition to the paper links. The readme might get too long with it, but either some doc in the repo or maybe on a github.io page?
type: documentation priority: high

opened by alexanderlerch 2
Add Smoothness and Monotonicity support
Smoothness

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

Monotonicity

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add Modularity support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add SAP support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add DMIG, DLIG, XMIG support
DMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

XMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

DLIG

[ x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add MIG support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Support issue for on-the-fly computation in TF2 graph mode
The current delegate-to-NumPy technique used in TF is only compatible with TF2 eager mode since Tensor.numpy() would not work in graph mode. As a result, graph-mode users will only be able to use Latte in the evaluation stage when the model weights are no longer changing but not on-the-fly during the training stage.

However, certain computation steps required for some metrics (especially MI-based ones) necessarily require scikit-learn ops and there is no (maintainable) way to create consistent TF mirrors of those functions.

One potential solution is to wrap the core functions in tf.numpy_function or tf.py_function but we will have to figure out a way to make the wrapper less painful to implement/maintain since the variable args/kwargs option currently used by the dtype converter is not allowed in these functions. A naive workaround would be to make a tf.numpy_function wrapper for every highest-possible level function with fixed args but this would be considered a last-resort solution.

Links:

https://www.tensorflow.org/api_docs/python/tf/numpy_function

https://www.tensorflow.org/api_docs/python/tf/py_function

type: enhancement priority: medium !! needs more brains !!
opened by karnwatcharasupat 3

Releases(v0.0.1-alpha5)

v0.0.1-alpha5(Jan 20, 2022)
What's Changed

Add contributing guide by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/16

[ADD] add example notebooks by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/18

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha3...v0.0.1-alpha5
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha3(Dec 16, 2021)
Added metric bundle support

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha2...v0.0.1-alpha3
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha2(Dec 9, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha1...v0.0.1-alpha2
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha1(Dec 1, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-dev5...v0.0.1-alpha1
Source code(tar.gz)
Source code(zip)

Owner

Karn Watcharasupat

Lab Cat 🐱🌈 | Audio Signal Processing Research Student. NTU EEE Class of 2022. Georgia Tech Music Tech Visiting Researcher.

GitHub Repository https://latte.readthedocs.io/en/latest

implementation for paper "ShelfNet for fast semantic segmentation"

ShelfNet-lightweight for paper (ShelfNet for fast semantic segmentation) This repo contains implementation of ShelfNet-lightweight models for real-tim

252 Sep 16, 2022

Very simple NCHW and NHWC conversion tool for ONNX. Change to the specified input order for each and every input OP. Also, change the channel order of RGB and BGR. Simple Channel Converter for ONNX.

scc4onnx Very simple NCHW and NHWC conversion tool for ONNX. Change to the specified input order for each and every input OP. Also, change the channel

16 Dec 22, 2022

A novel framework to automatically learn high-quality scanning of non-planar, complex anisotropic appearance.

appearance-scanner About This repository is an implementation of the neural network proposed in Free-form Scanning of Non-planar Appearance with Neura

14 Oct 18, 2022

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

The Neural Process Family This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CN

892 Dec 28, 2022

Multi-Stage Spatial-Temporal Convolutional Neural Network (MS-GCN)

Multi-Stage Spatial-Temporal Convolutional Neural Network (MS-GCN) This code implements the skeleton-based action segmentation MS-GCN model from Autom

8 Nov 29, 2022

Code for NeurIPS 2021 paper 'Spatio-Temporal Variational Gaussian Processes'

Spatio-Temporal Variational GPs This repository is the official implementation of the methods in the publication: O. Hamelijnck, W.J. Wilkinson, N.A.

26 Sep 16, 2022

DeepStruc is a Conditional Variational Autoencoder which can predict the mono-metallic nanoparticle from a Pair Distribution Function.

ChemRxiv | [Paper] XXX DeepStruc Welcome to DeepStruc, a Deep Generative Model (DGM) that learns the relation between PDF and atomic structure and the

13 Aug 01, 2022

The open-source and free to use Python package miseval was developed to establish a standardized medical image segmentation evaluation procedure

miseval: a metric library for Medical Image Segmentation EVALuation The open-source and free to use Python package miseval was developed to establish

59 Dec 10, 2022

Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Related tags

Overview

Latte

Installation

Running tests locally

Example

Functional API

Modular API

TorchMetrics API

Keras Metric API

Documentation

Supported metrics

Bundled metric modules

Cite

Comments

Documentation: Metric Descriptions

Add Smoothness and Monotonicity support

Smoothness

Monotonicity

Add Modularity support

Add SAP support

Add DMIG, DLIG, XMIG support

DMIG

XMIG

DLIG

Add MIG support

Support issue for on-the-fly computation in TF2 graph mode

Releases(v0.0.1-alpha5)

v0.0.1-alpha5(Jan 20, 2022)

What's Changed

v0.0.1-alpha3(Dec 16, 2021)

v0.0.1-alpha2(Dec 9, 2021)

v0.0.1-alpha1(Dec 1, 2021)

Owner

Karn Watcharasupat

implementation for paper "ShelfNet for fast semantic segmentation"

Very simple NCHW and NHWC conversion tool for ONNX. Change to the specified input order for each and every input OP. Also, change the channel order of RGB and BGR. Simple Channel Converter for ONNX.

A novel framework to automatically learn high-quality scanning of non-planar, complex anisotropic appearance.

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

Multi-Stage Spatial-Temporal Convolutional Neural Network (MS-GCN)

Code for NeurIPS 2021 paper 'Spatio-Temporal Variational Gaussian Processes'

DeepStruc is a Conditional Variational Autoencoder which can predict the mono-metallic nanoparticle from a Pair Distribution Function.

The open-source and free to use Python package miseval was developed to establish a standardized medical image segmentation evaluation procedure

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

A modern pure-Python library for reading PDF files

Nested Graph Neural Network (NGNN) is a general framework to improve a base GNN's expressive power and performance

GUPNet - Geometry Uncertainty Projection Network for Monocular 3D Object Detection

U-Net implementation in PyTorch for FLAIR abnormality segmentation in brain MRI

Contrastive Language-Image Pretraining

Python parser for DTED data.

2021 credit card consuming recommendation

Regularizing Nighttime Weirdness: Efficient Self-supervised Monocular Depth Estimation in the Dark (ICCV 2021)

Replication of Pix2Seq with Pretrained Model

PyTorch implementation of the cross-modality generative model that synthesizes dance from music.

A method to perform unsupervised cross-region adaptation of crop classifiers trained with satellite image time series.