Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Last update: Sep 20, 2022

Overview

Behavioral Testing of Clinical NLP Models

This repository contains code for testing the behavior of clinical prediction models based on patient letters. For a detailed description of the testing framework see our paper What Do You See in this Patient? Behavioral Testing of Clinical NLP Models.

Usage

Install requirements: pip install -r requirements.txt

Run main.py, e.g. for diagnosis prediction test on gender, age and ethnicity:

python main.py 
    --test_set_path ./path_to_test_set
    --model_path bvanaken/CORe-clinical-diagnosis-prediction
    --task diagnosis
    --shift_keys gender,age,ethnicity
    --save_dir ./results
    --gpu False

Parameter	Description
test_set_path	Path to original test set file
model_path	Path to model or Huggingface model hub checkpoint
task	Current options: diagnosis, mortality
shift_keys	Which patient characteristics to test. Current options: age, gender, ethnicity, weight, intersectional (gender + ethnicity)
save_dir	Directory to save results, default: "./results"
gpu	Whether to use a gpu during inference or not, default: False

Using Non-Transformer models

The framework currently focuses on testing Transformer-based models. However, it is easy to extend it to any other prediction model. To do so, simply create a new class implementing the Predictor interface and add it to the TASK_MAP in main.py.

Cite

@inproceedings{vanAken2021,
  author    = {Betty van Aken and
               Sebastian Herrmann and
               Alexander Löser},
  title     = {What Do You See in this Patient? Behavioral Testing of Clinical NLP Models},
  booktitle = {Bridging the Gap: From Machine Learning Research to Clinical Practice, 
               Research2Clinics Workshop @ NeurIPS 2021},
  year      = {2021}
}

Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Related tags

Overview

Behavioral Testing of Clinical NLP Models

Usage

Using Non-Transformer models

Cite

Owner

Betty van Aken

object recognition with machine learning on Respberry pi

Multi-Joint dynamics with Contact. A general purpose physics simulator.

Using machine learning to predict and analyze high and low reader engagement for New York Times articles posted to Facebook.

General purpose Slater-Koster tight-binding code for electronic structure calculations

Pre-Training 3D Point Cloud Transformers with Masked Point Modeling

Towards End-to-end Video-based Eye Tracking

Self-supervised Label Augmentation via Input Transformations (ICML 2020)

Free Book about Deep-Learning approaches for Chess (like AlphaZero, Leela Chess Zero and Stockfish NNUE)

The-Secret-Sharing-Schemes - This interactive script demonstrates the Secret Sharing Schemes algorithm

SCALoss: Side and Corner Aligned Loss for Bounding Box Regression (AAAI2022).

Speeding-Up Back-Propagation in DNN: Approximate Outer Product with Memory

Vector AI — A platform for building vector based applications. Encode, query and analyse data using vectors.

A Python implementation of the Locality Preserving Matching (LPM) method for pruning outliers in image matching.

Multi-layer convolutional LSTM with Pytorch

PyTorch implementations of the NeRF model described in "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis"

Python implementation of a live deep learning based age/gender/expression recognizer

Unofficial implementation of the paper: PonderNet: Learning to Ponder in TensorFlow

A Deep Reinforcement Learning Framework for Stock Market Trading

Codes for NeurIPS 2021 paper "On the Equivalence between Neural Network and Support Vector Machine".

Real-Time Seizure Detection using EEG: A Comprehensive Comparison of Recent Approaches under a Realistic Setting