An end-to-end machine learning library to directly optimize AUC loss

Overview

LibAUC

An end-to-end machine learning library for AUC optimization.

Why LibAUC?

Deep AUC Maximization (DAM) is a paradigm for learning a deep neural network by maximizing the AUC score of the model on a dataset. There are several benefits of maximizing AUC score over minimizing the standard losses, e.g., cross-entropy.

  • In many domains, AUC score is the default metric for evaluating and comparing different methods. Directly maximizing AUC score can potentially lead to the largest improvement in the model’s performance.
  • Many real-world datasets are usually imbalanced . AUC is more suitable for handling imbalanced data distribution since maximizing AUC aims to rank the predication score of any positive data higher than any negative data

Links

Installation

$ pip install libauc

Usage

Official Tutorials:

  • 01.Creating Imbalanced Benchmark Datasets [Notebook][Script]
  • 02.Training ResNet20 with Imbalanced CIFAR10 [Notebook][Script]
  • 03.Training with Pytorch Learning Rate Scheduling [Notebook][Script]
  • 04.Training with Imbalanced Datasets on Distributed Setting [Coming soon]

Quickstart for beginner:

>>> #import library
>>> from libauc.losses import AUCMLoss
>>> from libauc.optimizers import PESG
...
>>> #define loss
>>> Loss = AUCMLoss(imratio=0.1)
>>> optimizer = PESG(imratio=0.1)
...
>>> #training
>>> model.train()    
>>> for data, targets in trainloader:
>>>	data, targets  = data.cuda(), targets.cuda()
        preds = model(data)
        loss = Loss(preds, targets) 
        optimizer.zero_grad()
        loss.backward(retain_graph=True)
        optimizer.step()
...	
>>> #restart stage
>>> optimizer.update_regularizer()		
...   
>>> #evaluation
>>> model.eval()    
>>> for data, targets in testloader:
	data, targets  = data.cuda(), targets.cuda()
        preds = model(data)

Please visit our website or github for more examples.

Citation

If you find LibAUC useful in your work, please cite the following paper:

@article{yuan2020robust,
title={Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification},
author={Yuan, Zhuoning and Yan, Yan and Sonka, Milan and Yang, Tianbao},
journal={arXiv preprint arXiv:2012.03173},
year={2020}
}

Contact

If you have any questions, please contact us @ Zhuoning Yuan [[email protected]] and Tianbao Yang [[email protected]] or please open a new issue in the Github.

Comments
  • Only compatible with Nvidia GPU

    Only compatible with Nvidia GPU

    I tried running the example tutorial but I got the following error. ''' AssertionError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx '''

    opened by Beckham45 2
  • Extend to Multi-class Classification Task and Be compatible with PyTorch scheduler

    Extend to Multi-class Classification Task and Be compatible with PyTorch scheduler

    Hi Zhuoning,

    This is an interesting work! I am wondering if the DAM method can be extended to a multi-class classification task with long-tailed imbalanced data. Intuitively, this should be possible as the famous sklearn tool provides auc score for multi-class setting by using one-versus-rest or one-versus-one technique.

    Besides, it seems that optimizer.update_regularizer() is called only when the learning rate is reduced, thus it would be more elegant to incorporate this functional call into a pytorch lr scheduler. E.g.,

    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer)
    scheduler.step()    # override the step to fulfill: optimizer.update_regularizer()
    
    

    For current libauc version, the PESG optimizer is not compatible with schedulers in torch.optim.lr_scheduler . It would be great if this feature can be supported in the future.

    Thanks for your work!

    opened by Cogito2012 2
  • When to use retain_graph=True?

    When to use retain_graph=True?

    Hi,

    When to use retain_graph=True in the loss backward function?

    In 2 examples (2 and 4), it is True. But not in the other examples.

    I appreciate your time.

    opened by dfrahman 1
  • Using AUCMLoss with imratio>1

    Using AUCMLoss with imratio>1

    I'm not very familiar with the maths in the paper so please forgive me if i'm asking something obvious.

    The AUCMLoss uses the "imbalance ratio" between positive and negative samples. The ratio is defined as

    the ratio of # of positive examples to the # of negative examples

    Or imratio=#pos/#neg

    When #pos<#neg, imratio is some value between 0 and 1. But when #pos>#neg, imratio>1

    Will this break the loss calculations? I have a feeling it would invalidate the many 1-self.p calculations in the LibAUC implementation, but as i'm not familiar with the maths I can't say for sure.

    Also, is there a problem (mathematically speaking) with calculating imratio=#pos/#total_samples, to avoid the problem above? When #pos<<#neg, #neg approximates #total_samples.

    opened by ayhyap 1
  • AUCMLoss does not use margin argument

    AUCMLoss does not use margin argument

    I noticed in the AUCMLoss class that the margin argument is not used. Following the formulation in the paper, the forward function should be changed in line 20 from 2*self.alpha*(self.p*(1-self.p) + \ to 2*self.alpha*(self.p*(1-self.p)*self.margin + \

    opened by ayhyap 1
  • How to train multi-label classification tasks? (like chexpert)

    How to train multi-label classification tasks? (like chexpert)

    I have started using this library and I've read your paper Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification, and I'm still not sure how to train a multi-label classification (MLC) model.

    Specifically, how did you fine-tune for the Chexpert multi-label classification task? (i.e. classify 5 diseases, where each image may have presence of 0, 1 or more diseases)

    • The first step pre-training with Cross-entropy loss seems clear to me
    • You mention: "In the second step of AUC maximization, we replace the last classifier layer trained in the first step by random weights and use our DAM method to optimize the last classifier layer and all previous layers.". The new classifier layer is a single or multi-label classifier?
    • In the Appendix I, figure 7 shows only one score as output for Deep AUC maximization (i.e. only one disease)
    • In the code, both AUCMLoss() and APLoss_SH() receive single-label outputs, not multi-label outputs, apparently

    How do you train for the 5 diseases? Train sequentially Cardiomegaly, then Edema, and so on? or with 5 losses added up? or something else?

    opened by pdpino 4
  • Example for tensorflow

    Example for tensorflow

    Thank you for the great library. Does it currently support tensorflow? If so, could you provide an example of how it can be used with tensorflow? Thank you very much

    opened by Kokkini 1
Releases(1.1.4)
  • 1.1.4(Jul 26, 2021)

    What's New

    • Added PyTorch dataloader for CheXpert dataset. Tutorial for training CheXpert is available here.
    • Added support for training AUC loss on CPU machines. Note that please remove lines with .cuda() from the code.
    • Fixed some bugs and improved the training stability
    Source code(tar.gz)
    Source code(zip)
  • 1.1.3(Jun 16, 2021)

  • 1.1.2(Jun 14, 2021)

    What's New

    1. Add SOAP optimizer contributed by @qiqi-helloworld @yzhuoning for optimizing AUPRC. Please check the tutorial here.
    2. Update ResNet18, ResNet34 with pretrained models on ImageNet1K
    3. Add new strategy for AUCM Loss: imratio is calculated over a mini-batch if initial value is not given
    4. Fixed some bugs and improved the training stability
    Source code(tar.gz)
    Source code(zip)
  • V1.1.0(May 10, 2021)

    What's New:

    • Fixed some bugs and improved the training stability
    • Changed default settings in loss function for binary labels to be 0 and 1
    • Added Pytorch dataloaders for CIFAR10, CIFAR100, CAT_vs_Dog, STL10
    • Enabled training DAM with Pytorch leanring scheduler, e.g., ReduceLROnPlateau, CosineAnnealingLR
    Source code(tar.gz)
    Source code(zip)
Official implementation of Densely connected normalizing flows

Densely connected normalizing flows This repository is the official implementation of NeurIPS 2021 paper Densely connected normalizing flows. Poster a

Matej Grcić 31 Dec 12, 2022
LaneAF: Robust Multi-Lane Detection with Affinity Fields

LaneAF: Robust Multi-Lane Detection with Affinity Fields This repository contains Pytorch code for training and testing LaneAF lane detection models i

155 Dec 17, 2022
Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction

Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction. arxiv This repository contains python scripts for tr

12 Dec 12, 2022
Visual Adversarial Imitation Learning using Variational Models (VMAIL)

Visual Adversarial Imitation Learning using Variational Models (VMAIL) This is the official implementation of the NeurIPS 2021 paper. Project website

14 Nov 18, 2022
A hyperparameter optimization framework

Optuna: A hyperparameter optimization framework Website | Docs | Install Guide | Tutorial Optuna is an automatic hyperparameter optimization software

7.4k Jan 04, 2023
A high-performance anchor-free YOLO. Exceeding yolov3~v5 with ONNX, TensorRT, NCNN, and Openvino supported.

YOLOX is an anchor-free version of YOLO, with a simpler design but better performance! It aims to bridge the gap between research and industrial communities. For more details, please refer to our rep

7.7k Jan 06, 2023
CondNet: Conditional Classifier for Scene Segmentation

CondNet: Conditional Classifier for Scene Segmentation Introduction The fully convolutional network (FCN) has achieved tremendous success in dense vis

ycszen 31 Jul 22, 2022
Codebase to experiment with a hybrid Transformer that combines conditional sequence generation with regression

Regression Transformer Codebase to experiment with a hybrid Transformer that combines conditional sequence generation with regression . Development se

International Business Machines 27 Jan 05, 2023
A stable algorithm for GAN training

DRAGAN (Deep Regret Analytic Generative Adversarial Networks) Link to our paper - https://arxiv.org/abs/1705.07215 Pytorch implementation (thanks!) -

195 Oct 10, 2022
PyTorch implementations of Top-N recommendation, collaborative filtering recommenders.

PyTorch implementations of Top-N recommendation, collaborative filtering recommenders.

Yoonki Jeong 129 Dec 22, 2022
Deep functional residue identification

DeepFRI Deep functional residue identification Citing @article {Gligorijevic2019, author = {Gligorijevic, Vladimir and Renfrew, P. Douglas and Koscio

Flatiron Institute 156 Dec 25, 2022
Pytorch implementation of FlowNet by Dosovitskiy et al.

FlowNetPytorch Pytorch implementation of FlowNet by Dosovitskiy et al. This repository is a torch implementation of FlowNet, by Alexey Dosovitskiy et

Clément Pinard 762 Jan 02, 2023
MusicYOLO framework uses the object detection model, YOLOx, to locate notes in the spectrogram.

MusicYOLO MusicYOLO framework uses the object detection model, YOLOX, to locate notes in the spectrogram. Its performance on the ISMIR2014 dataset, MI

Xianke Wang 2 Aug 02, 2022
A module for solving and visualizing Schrödinger equation.

qmsolve This is an attempt at making a solid, easy to use solver, capable of solving and visualize the Schrödinger equation for multiple particles, an

506 Dec 28, 2022
This repository contains code demonstrating the methods outlined in Path Signature Area-Based Causal Discovery in Coupled Time Series presented at Causal Analysis Workshop 2021.

signed-area-causal-inference This repository contains code demonstrating the methods outlined in Path Signature Area-Based Causal Discovery in Coupled

Will Glad 1 Mar 11, 2022
Weakly Supervised Learning of Rigid 3D Scene Flow

Weakly Supervised Learning of Rigid 3D Scene Flow This repository provides code and data to train and evaluate a weakly supervised method for rigid 3D

Zan Gojcic 124 Dec 27, 2022
GMFlow: Learning Optical Flow via Global Matching

GMFlow GMFlow: Learning Optical Flow via Global Matching Authors: Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Dacheng Tao We streamline the

Haofei Xu 298 Jan 04, 2023
Human segmentation models, training/inference code, and trained weights, implemented in PyTorch

Human-Segmentation-PyTorch Human segmentation models, training/inference code, and trained weights, implemented in PyTorch. Supported networks UNet: b

Thuy Ng 474 Dec 19, 2022
Deep Learning for Natural Language Processing SS 2021 (TU Darmstadt)

Deep Learning for Natural Language Processing SS 2021 (TU Darmstadt) Task Training huge unsupervised deep neural networks yields to strong progress in

Oliver Hahn 1 Jan 26, 2022
Cereal box identification in store shelves using computer vision and a single train image per model.

Product Recognition on Store Shelves Description You can read the task description here. Report You can read and download our report here. Step A - Mu

Nicholas Baraghini 1 Jan 21, 2022