The best solution of the Weather Prediction track in the Yandex Shifts challenge

Overview

License Apache 2.0 Python 3.7

yandex-shifts-weather

The repository contains information about my solution for the Weather Prediction track in the Yandex Shifts challenge https://research.yandex.com/shifts/weather

This information includes two my Jupyter-notebooks (deep_ensemble_with_uncertainty_and_spec_fe_eval.ipynb for evaluating only, and deep_ensemble_with_uncertainty_and_spec_fe.ipynb for all stages of my experiment), several auxiliary Python modules (from https://github.com/yandex-research/shifts/tree/main/weather), and my pre-trained models (see the models/yandex-shifts/weather subdirectory).

The proposed solution is the best on this track (see SNN Ens U MT Np SpecFE method in the corresponding leaderboard).

You can read a detailed description of all the algorithmic techniques implemented in my solution and their motivation in this paper: More layers! End-to-end regression and uncertainty on tabular data with deep learning.

If you use my solution in your work, please cite my paper using the following Bibtex:

@article{morelayers2021,
  author    = {Bondarenko, Ivan},
  title     = {More layers! End-to-end regression and uncertainty on tabular data with deep learning},
  journal   = {arXiv preprint arXiv:2112.03566},
  year      = {2021},
}

Reproducibility

For reproducibility you need to Python 3.7 or later (I checked all with Python 3.7). You have to clone this repository and install all dependencies (I recommend do it in a special Python environment):

git clone https://github.com/bond005/yandex-shifts-weather.git
pip install -r requirements.txt

My solution is based on deep learning and uses Tensorflow. So, you need a CUDA-compatible GPU for more efficient reproducibility of my code (I used two variants: nVidia V100 and nVidia GeForce 2080Ti).

After installing the dependencies, you need to do the following steps:

  1. Download all data of the Weather Prediction track in the Yandex Shifts challenge and unarchive it into the data/yandex-shifts/weather subdirectory. The training and development data can be downloaded here, and the data for final evaluation stage is available below this link. As a result, there will be four CSV-files in the abovementioned subdirectory:
  • train.csv
  • dev_in.csv
  • dev_out.csv
  • eval.csv
  1. Download the baseline models developed by the challenge organizers at the link, and unarchive their into the models/yandex-shifts/weather-baseline-catboost subdirectory. This step is optional, and it is only needed to compare my solution with the baseline on the development set (my pre-trained models are part of this repository, and they are available in the models/yandex-shifts/weather subdirectory).

  2. Run Jupyter notebook deep_ensemble_with_uncertainty_and_spec_fe_eval.ipynb for inference process reproducing. As a result, you will see the quality evaluation of my pre-trained models on the development set in comparison with the baseline. You will also get the predictions on the development and evaluation sets in the corresponding CSV files models/yandex-shifts/weather/df_submission_dev.csv and models/yandex-shifts/weather/df_submission.csv.

  3. If you want to reproduce the united process of training and inference (and not just inference, as in step 3), then you have to run another Jupyter notebook deep_ensemble_with_uncertainty_and_spec_fe.ipynb. All pre-trained models in the models/yandex-shifts/weather subdirectory will be rewritten. After that predictions will be generated (similar to step 3, but using new models). This step is optional.

Key ideas of the proposed method

A special deep ensemble (i.e. ensemble of deep neural networks) with uncertainty, hierarchicalmultitask learning and some other features is proposed. It is built on the basis of the following key techniques:

  1. A simple preprocessing is applied to the input data:
  • imputing: the missing values are replaced in all input columns following a simple constant strategy (fill value is −1);
  • quantization: each input column is discretized into quantile bins, and the number of these bins is detected automatically; after that the bin identifier can be considered as a quantized numerical value of the original feature;
  • standardization: each quantized column is standardized by removing the mean and scaling to unit variance;
  • decorrelation: all possible linear correlations are removed from the feature vectors discretized by above-mentioned way; the decorrelation is implemented using principal component analysis (PCA).
  1. An even simpler preprocessing is applied to targets: it is based only on removing the mean and scaling to unit variance.

  2. A deep neural network is built for regression with uncertainty:

  • self-normalization: this is a self-normalized neural network, also known as SNN [Klambaueret al. 2017];
  • inductive bias: neural network weights are tuned using a hierarchical multitask learning [Caruana 1997; Søgaardet al. 2016] with the temperature prediction as a high-level regression task and the coarsened temperature class recognition as a low-level classification task;
  • uncertainty: a special loss function similar to RMSEWithUncertainty is applied to training;
  • robustness: a supervised contrastive learning based on N-pairs loss [Sohn2016] is applied instead of the crossentropy-based classification as a low-level task in the hierarchical multitask learning; it provides more robustness of the trainable neural network [Khosla et al. 2020]
  1. A special technique of deep ensemble creation is implemented: it uses a model average approach like bagging, but new training and validation sub-sets for corresponding ensemble items are generated using stratification based on coarsened temperature classes.

The deep ensemble size is 20. Hyper-parameters of each neural network in the ensemble (hidden layer size, number of hidden layers and alpha-dropout as special version of dropout in SNN are the same. They are selected using a hybrid approach: first automatically, and then manually. The automatic approach is based on a Bayesian optimization with Gaussian Process regression, and it discovered the following hyper-parameter values for training the neural network in a single-task mode:

  • hidden layer size is 512;
  • number of hidden layers is 12;
  • alpha-dropout is 0.0003.

After the implementation of the hierarchical multitask learning, the depth was manually increased up to 18 hidden layers: the low-level task (classification or supervised constrastive learning) was added after the 12th layer, and the high-level task (regression with uncertainty) was added after the 18th layer. General architecture of single neural network is shown on the following figure. All "dense" components are feed-forward layers with self-normalization. Alpha-dropout is not shown to oversimplify the figure. The weather_snn_1_output layer estimates the mean and the standard deviation of normal distribution, which is implemented using theweather_snn_1_distribution layer. Another output layer named as weather_snn_1_projection calculates low-dimensional projections for the supervised contrastive learning.

Rectified ADAM [L. Liu et al. 2020] with Lookahead [M. R. Zhang et al. 2019] is used for training with the following parameters: learning rate is 0.0003, synchronization period is 6, and slow weights step size is 0.5. You can see the more detailed description of other training hyper-parameters in the Jupyter notebook deep_ensemble_with_uncertainty_and_spec_fe.ipynb.

Experiments

Experiments were conducted according to data partitioning in [Malinin et al. 2021, section 3.1]. Development and evaluation sets were not used for training and for hyper-parameter search. The quality of each modification of the method was first estimated on the development set, because all participants had access to the development set targets. After the preliminary estimation, the selected modification of the method was submitted to estimate R-AUC MSE on the evaluation set with targets concealed from participants.

In comparison with the baseline, the quality of the deep learning method is better (i.e. R-AUC MSE is less) on both datasets for testing:

  • the development set (for preliminary testing):
  1. proposed deep ensemble = 1.015;
  2. baseline (CatBoost ensemble) = 1.227;
  • the evaluation set (for final testing):
  1. proposed deep ensemble = 1.141;
  2. baseline (CatBoost ensemble) = 1.335.

Also, the results of the deep learning method are better with all possible values of the uncertainty threshold for retention.

The total number of all submitted methods at the evaluation phase is 73. Six selected results (top-5 results of all participants and the baseline result) are presented in the following table. The first three places are occupied by the following modifications of the proposed deep learning method:

  • SNN Ens U MT Np SpecFE is the final solution with "all-inclusive";
  • SNN Ens U MT Np v2 excludes the feature quantization;
  • SNN Ens U MT excludes the feature quantization too, and classification is used instead of supervised contrastive learning as the low-level task in the hierarchical multitask learning.
Rank Team Method R-AUC MSE
1 bond005 SNN Ens U MT Np SpecFE 1.1406288012
2 bond005 SNN Ens U MT Np v2 1.1415403291
3 bond005 SNN Ens U MT 1.1533415892
4 CabbeanWeather Steel box v2 1.1575201873
5 KDDI Research more seed ens 1.1593224114
55 Shifts Team Shifts Challenge 1.3353865316

The time characteristics of the solution with GPU using (Nvidia V100 or GeForce 2080 Ti) are:

  • the average inference time per sample is 0.063 milliseconds;
  • the total training time is about a day.
Owner
Ivan Yu. Bondarenko
Ivan Yu. Bondarenko
Tooling for the Common Objects In 3D dataset.

CO3D: Common Objects In 3D This repository contains a set of tools for working with the Common Objects in 3D (CO3D) dataset. Download the dataset The

Facebook Research 724 Jan 06, 2023
[IJCAI-2021] A benchmark of data-free knowledge distillation from paper "Contrastive Model Inversion for Data-Free Knowledge Distillation"

DataFree A benchmark of data-free knowledge distillation from paper "Contrastive Model Inversion for Data-Free Knowledge Distillation" Authors: Gongfa

ZJU-VIPA 47 Jan 09, 2023
PyTorch implementation for paper "Full-Body Visual Self-Modeling of Robot Morphologies".

Full-Body Visual Self-Modeling of Robot Morphologies Boyuan Chen, Robert Kwiatkowskig, Carl Vondrick, Hod Lipson Columbia University Project Website |

Boyuan Chen 32 Jan 02, 2023
Official repo for QHack—the quantum machine learning hackathon

Note: This repository has been frozen while we consider the submissions for the QHack Open Hackathon. We hope you enjoyed the event! Welcome to QHack,

Xanadu 118 Jan 05, 2023
Feed forward VQGAN-CLIP model, where the goal is to eliminate the need for optimizing the latent space of VQGAN for each input prompt

Feed forward VQGAN-CLIP model, where the goal is to eliminate the need for optimizing the latent space of VQGAN for each input prompt. This is done by

Mehdi Cherti 135 Dec 30, 2022
pytorch implementation of GPV-Pose

GPV-Pose Pytorch implementation of GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Voting. (link) UPDATE A new version

40 Dec 01, 2022
Volumetric Correspondence Networks for Optical Flow, NeurIPS 2019.

VCN: Volumetric correspondence networks for optical flow [project website] Requirements python 3.6 pytorch 1.1.0-1.3.0 pytorch correlation module (opt

Gengshan Yang 144 Dec 06, 2022
Generative Flow Networks

Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation Implementation for our paper, submitted to NeurIPS 2021 (also chec

Emmanuel Bengio 381 Jan 04, 2023
Official PyTorch implementation of "Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition" in AAAI2022.

AimCLR This is an official PyTorch implementation of "Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Reco

Gty 44 Dec 17, 2022
Fine-Tune EleutherAI GPT-Neo to Generate Netflix Movie Descriptions in Only 47 Lines of Code Using Hugginface And DeepSpeed

GPT-Neo-2.7B Fine-Tuning Example Using HuggingFace & DeepSpeed Installation cd venv/bin ./pip install -r ../../requirements.txt ./pip install deepspe

Nikita 180 Jan 05, 2023
DANA paper supplementary materials

DANA Supplements This repository stores the data, results, and R scripts to generate these reuslts and figures for the corresponding paper Depth Norma

0 Dec 17, 2021
Implementation of Ag-Grid component for Streamlit

streamlit-aggrid AgGrid is an awsome grid for web frontend. More information in https://www.ag-grid.com/. Consider purchasing a license from Ag-Grid i

Pablo Fonseca 556 Dec 31, 2022
Architecture Patterns with Python (TDD, DDD, EDM)

architecture-traning Architecture Patterns with Python (TDD, DDD, EDM) Chapter 5. 높은 기어비와 낮은 기어비의 TDD 5.2 도메인 계층 테스트를 서비스 계층으로 옮겨야 하는가? 도메인 계층 테스트 def

minsung sim 2 Mar 04, 2022
Dynamica causal Bayesian optimisation

Dynamic Causal Bayesian Optimization This is a Python implementation of Dynamic Causal Bayesian Optimization as presented at NeurIPS 2021. Abstract Th

nd308 18 Nov 22, 2022
Camview - A CLI-tool used to stream CCTV online footage based on URL params

CamView A CLI-tool used to stream CCTV online footage based on URL params Get St

Finn Lancaster 54 Dec 09, 2022
A simplified framework and utilities for PyTorch

Here is Poutyne. Poutyne is a simplified framework for PyTorch and handles much of the boilerplating code needed to train neural networks. Use Poutyne

GRAAL/GRAIL 534 Dec 17, 2022
Pytorch ImageNet1k Loader with Bounding Boxes.

ImageNet 1K Bounding Boxes For some experiments, you might wanna pass only the background of imagenet images vs passing only the foreground. Here, I'v

Amin Ghiasi 11 Oct 15, 2022
In real-world applications of machine learning, reliable and safe systems must consider measures of performance beyond standard test set accuracy

PixMix Introduction In real-world applications of machine learning, reliable and safe systems must consider measures of performance beyond standard te

Andy Zou 79 Dec 30, 2022
Implementation for Curriculum DeepSDF

Curriculum-DeepSDF This repository is an implementation for Curriculum DeepSDF. Full paper is available here. Preparation Please follow original setti

Haidong Zhu 69 Dec 29, 2022
U-2-Net: U Square Net - Modified for paired image training of style transfer

U2-Net: U Square Net Modified for paired image training of style transfer This is an unofficial repo making use of the code which was made available b

Doron Adler 43 Oct 03, 2022