Orthogonalizing Convolutional Layers with the Cayley Transform

This repository contains implementations and source code to reproduce experiments for the ICLR 2021 spotlight paper Orthogonalizing Convolutional Layers with the Cayley Transform by Asher Trockman and Zico Kolter.

Check out our tutorial on FFT-based convolutions and how to orthogonalize them in this Jupyter notebook.

(more information and code coming soon)

Getting Started

You can clone this repo using:

git clone https://github.com/locuslab/orthogonal-convolutions --recursive

where the --recursive is necessary for the submodules.

The most important dependency is PyTorch >= 1.8. If you like, you can set up a new conda environment:

conda create --name orthoconv python=3.6
conda activate orthoconv
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
conda install --file requirements.txt

The Orthogonal Convolutional Layer

Our orthogonal convolutional layer can be found in layers.py. The actual layer is the module CayleyConv. It depends on the function cayley, our implementation of the Cayley transform for (semi-)orthogonalization. Additionally, CayleyConv is a subclass of StridedConv, which emulates striding functionality by reshaping the input tensor.

Running Experiments

The script train.py can be used to run most of the experiments from our paper. To try the "flagship" experiment demonstrating better clean accuracy and -norm-bounded deterministic certifiable robustness, run:

python train.py --epochs=200 --conv=CayleyConv --linear=CayleyLinear

To compare with BCOP as in our paper, run:

python train.py --epochs=200 --conv=BCOP --linear=BjorckLinear