Pytorch implementation of "Forward Thinking: Building and Training Neural Networks One Layer at a Time"

Last update: Oct 06, 2022

Related tags

Deep Learning forward-thinking-pytorch

Overview

forward-thinking-pytorch

Pytorch implementation of Forward Thinking: Building and Training Neural Networks One Layer at a Time

Requirements

Python 2.7
numpy
pytorch
Matplotlib

Usage

  $ ./run.sh

to run forward-thinking & forward-thinking (deep) & backpropagation & backpropagation (deep).

For forward-thinking experiment (5 layers),

$ python forward_thinking.py

If you want to do this with very deep model (20 layers),

$ python forward_thinking.py --epoch 200 --deep

As a comparison, learn by backpropagation:

$ python backpropagate.py

Backpropagate with very deep model (20 layers),

$ python backpropagate.py --epoch 200 --deep

Result

For 5 layers, forward-thinking learns slightly faster than backpropagation. Dips are observed when new layers are added.

When the model becomes very deep (20 layers), backpropagation cannot train the model. On the other hand, forward-thinking can train the model as usual.

Owner

Kim Heecheol

University of Tokyo, Intelligent systems & Informatics Lab.

GitHub Repository https://arxiv.org/pdf/1706.02480.pdf

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

STCN Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang [a

456 Dec 12, 2022

A developer interface for creating Chat AIs for the Chai app.

ChaiPy A developer interface for creating Chat AIs for the Chai app. Usage Local development A quick start guide is available here, with a minimal exa

28 Dec 28, 2022

Code implementation from my Medium blog post: [Transformers from Scratch in PyTorch]

transformer-from-scratch Code for my Medium blog post: Transformers from Scratch in PyTorch Note: This Transformer code does not include masked attent

27 Dec 21, 2022

Code for "Primitive Representation Learning for Scene Text Recognition" (CVPR 2021)

Primitive Representation Learning Network (PREN) This repository contains the code for our paper accepted by CVPR 2021 Primitive Representation Learni

76 Jan 02, 2023

Code for our NeurIPS 2021 paper 'Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation'

Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation (NeurIPS 2021) Code for our NeurIPS 2021 paper 'Exploiting the Intri

53 Dec 25, 2022

A Semantic Segmentation Network for Urban-Scale Building Footprint Extraction Using RGB Satellite Imagery

A Semantic Segmentation Network for Urban-Scale Building Footprint Extraction Using RGB Satellite Imagery This repository is the official implementati

42 Dec 08, 2022

Dense Prediction Transformers

Vision Transformers for Dense Prediction This repository contains code and models for our paper: Vision Transformers for Dense Prediction René Ranftl,

1.3k Dec 28, 2022

Train Yolov4 using NBX-Jobs

yolov4-trainer-nbox Train Yolov4 using NBX-Jobs. Use the powerfull functionality available in nbox-SDK repo to train a tiny-Yolo v4 model on Pascal VO

1 Jan 12, 2022

This repository contains the code to replicate the analysis from the paper "Moving On - Investigating Inventors' Ethnic Origins Using Supervised Learning"

Replication Code for 'Moving On' - Investigating Inventors' Ethnic Origins Using Supervised Learning This repository contains the code to replicate th

0 Jan 04, 2022

A Research-oriented Federated Learning Library and Benchmark Platform for Graph Neural Networks. Accepted to ICLR'2021 - DPML and MLSys'21 - GNNSys workshops.

FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks A Research-oriented Federated Learning Library and Benchmark Platform

175 Dec 01, 2022

Package towards building Explainable Forecasting and Nowcasting Models with State-of-the-art Deep Neural Networks and Dynamic Factor Model on Time Series data sets with single line of code. Also, provides utilify facility for time-series signal similarities matching, and removing noise from timeseries signals.

DeepXF: Explainable Forecasting and Nowcasting with State-of-the-art Deep Neural Networks and Dynamic Factor Model Also, verify TS signal similarities

88 Dec 27, 2022

Pytorch implementation of "Forward Thinking: Building and Training Neural Networks One Layer at a Time"

Related tags

Overview

forward-thinking-pytorch

Requirements

Usage

Result

Owner

Kim Heecheol

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

A developer interface for creating Chat AIs for the Chai app.

Code implementation from my Medium blog post: [Transformers from Scratch in PyTorch]

Code for "Primitive Representation Learning for Scene Text Recognition" (CVPR 2021)

Code for our NeurIPS 2021 paper 'Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation'

A Semantic Segmentation Network for Urban-Scale Building Footprint Extraction Using RGB Satellite Imagery

Dense Prediction Transformers

Train Yolov4 using NBX-Jobs

This repository contains the code to replicate the analysis from the paper "Moving On - Investigating Inventors' Ethnic Origins Using Supervised Learning"

A Research-oriented Federated Learning Library and Benchmark Platform for Graph Neural Networks. Accepted to ICLR'2021 - DPML and MLSys'21 - GNNSys workshops.

Type4Py: Deep Similarity Learning-Based Type Inference for Python

A simple implementation of Kalman filter in Multi Object Tracking

Dense matching library based on PyTorch

Implementation for "Domain-Specific Bias Filtering for Single Labeled Domain Generalization"

An efficient framework for reinforcement learning.

🎁 3,000,000+ Unsplash images made available for research and machine learning

Simulation code and tutorial for BBHnet training data

Source codes of CenterTrack++ in 2021 ICME Workshop on Big Surveillance Data Processing and Analysis

Unsupervised Attributed Multiplex Network Embedding (AAAI 2020)