ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Last update: Nov 21, 2022

Overview

ZSL-KG

ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Reference paper: Zero-shot Learning with Common Sense Knowledge graphs.

Performance

Performance ZSL-KG compared to other existing graph-based zero-shot learning frameworks.

Method	Ontonotes (Strict)	BBN (Strict)	SNIPS-NLU (Acc.)	AWA2 (H)	aPY (H)	ImageNet (All T-1)	Avg.
GCNZ	41.5	21.5	82.5	73.3	58.1	1.0	46.3
SGCN	42.6	24.9	50.3	73.7	56.8	1.5	41.6
DGP	41.1	24.0	64.4	75.1	55.7	1.4	43.6
ZSL-KG	45.2	26.7	89.0	74.6	61.6	1.7	49.8

ZSL-KG outperforms existing graph-based frameworks on five out of six benchmark datasets.

For more details on the experiments, refer to nayak-arxiv20-code.

Installation

The package requires python >= 3.7. To install the package, type the following command:

pip install .

Example Usage

In our framework, we use AutoGNN to easily create graph neural networks for zero-shot learning.

from zsl_kg.class_encoder.auto_gnn import AutoGNN
from zsl_kg.common.graph import NeighSampler

trgcn = {
    "input_dim": 300,
    "output_dim": 2049,
    "type": "trgcn",
    "gnn": [
        {
            "input_dim": 300,
            "output_dim": 2048,
            "activation": nn.LeakyReLU(0.2),
            "normalize": True,
            "sampler": NeighSampler(100, mode="topk"),
            "fh": 100,
        },
        {
            "input_dim": 2048,
            "output_dim": 2049,
            "activation": None,
            "normalize": True,
            "sampler": NeighSampler(50, mode="topk"),
        },
    ],
}

class_encoder = AutoGNN(trgcn)

Our framework supports the following graph neural networks: gcn, gat, rgcn, lstm, trgcn. You can change the type to any of the available to graph neural networks to instantly create a new graph neural network.

For more examples, refer to nayak-arxiv20-code.

Run Tests

To run the tests, please type the following command:

pytest

Citation

Please cite the following paper if you are using our framework.

@article{nayak:arxiv20,
  Author = {Nayak, Nihal V. and Bach, Stephen H.},
  Title = {Zero-Shot Learning with Common Sense Knowledge Graphs},
  Volume = {arXiv:2006.10713 [cs.LG]},
  Year = {2020}}

Comments

Graph usage query in object classification AWA2

Hi, I'm not entirely sure if this is where I should be asking this but I was wondering if you could help me clarify a question.

1 When performing zero shot object classification in AWA2, what is the graph that is used? (I assume a knowledge graph such as ConceptNet?)

2 If this is a knowledge graph, than what is the "input" to the trgcn? As in where are these "concepts" coming from, is it from the class attributes?

opened by shadowbourne 2
Quick note

Dear Team behind @zsl-kg,

This framework is really cool, however, I was quite disappointed to see you are re-writing everything from scratch when you could have used an awesome framework like PyTorch Geometric.

Best regards, Thomas Chaton

opened by tchaton 1

Releases(v0.0.1)

v0.0.1(Aug 27, 2021)

This is the initial release. Contains all the necessary code to run experiments in nayak-arxiv20-code.
Source code(tar.gz)
Source code(zip)

Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction".

GNN_PPI Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction". Lear

2 Dec 14, 2022

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Episodic Transformers (E.T.) Episodic Transformer for Vision-and-Language Navigation Alexander Pashevich, Cordelia Schmid, Chen Sun Episodic Transform

62 Dec 24, 2022

Neighborhood Contrastive Learning for Novel Class Discovery

Neighborhood Contrastive Learning for Novel Class Discovery This repository contains the official implementation of our paper: Neighborhood Contrastiv

56 Dec 9, 2022

Awesome Deep Graph Clustering is a collection of SOTA, novel deep graph clustering methods

ADGC: Awesome Deep Graph Clustering ADGC is a collection of state-of-the-art (SOTA), novel deep graph clustering methods (papers, codes and datasets).

297 Dec 27, 2022

The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Neural Deformation Graphs Project Page | Paper | Video Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction Aljaž Božič, Pablo P

134 Dec 16, 2022

Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

multilingual-mrc-isdg Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph". This r

5 Dec 7, 2022

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

PyTorch Implementation of Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers 1 Using Colab Please notic

489 Jan 7, 2023

sense-py-AnishaBaishya created by GitHub Classroom

Compute Statistics Here we compute statistics for a bunch of numbers. This project uses the unittest framework to test functionality. Pass the tests T

1 Oct 21, 2021

This repo is the code release of EMNLP 2021 conference paper "Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories".

Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories This repo is the code release of EMNLP 2021 con

12 Nov 22, 2022

ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Related tags

Overview

ZSL-KG

Performance

Installation

Example Usage

Run Tests

Citation

You might also like...

Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction".

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Neighborhood Contrastive Learning for Novel Class Discovery

Awesome Deep Graph Clustering is a collection of SOTA, novel deep graph clustering methods

The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

sense-py-AnishaBaishya created by GitHub Classroom

This repo is the code release of EMNLP 2021 conference paper "Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories".

Comments

Graph usage query in object classification AWA2

Quick note

Releases(v0.0.1)

v0.0.1(Aug 27, 2021)

Owner

Bats Research

Music Source Separation; Train & Eval & Inference piplines and pretrained models we used for 2021 ISMIR MDX Challenge.

Code for "My(o) Armband Leaks Passwords: An EMG and IMU Based Keylogging Side-Channel Attack" paper

Official implementation of the RAVE model: a Realtime Audio Variational autoEncoder

A font family with a great monospaced variant for programmers.

Original Pytorch Implementation of FLAME: Facial Landmark Heatmap Activated Multimodal Gaze Estimation

Crowd-sourced Annotation of Human Motion.

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations

Brain tumor detection using Convolution-Neural Network (CNN)

Lbl2Vec learns jointly embedded label, document and word vectors to retrieve documents with predefined topics from an unlabeled document corpus.

Tools for manipulating UVs in the Blender viewport.

Infrastructure as Code (IaC) for a self-hosted version of Gnosis Safe on AWS

An open-source outlier detection package by Getcontact Data Team

SiamMOT is a region-based Siamese Multi-Object Tracking network that detects and associates object instances simultaneously.

Vision Deep-Learning using Tensorflow, Keras.

A Lightweight Face Recognition and Facial Attribute Analysis (Age, Gender, Emotion and Race) Library for Python

Implementation of gaze tracking and demo

SoK: Vehicle Orientation Representations for Deep Rotation Estimation

PatchMatch-RL: Deep MVS with Pixelwise Depth, Normal, and Visibility

Gauge equivariant mesh cnn

Code for the KDD 2021 paper 'Filtration Curves for Graph Representation'