Tutorial in Python targeted at Epidemiologists. Will discuss the basics of analysis in Python 3

Overview

Python-for-Epidemiologists

Join the chat at https://gitter.im/zEpid/community DOI

This repository is an introduction to epidemiology analyses in Python. Additionally, the tutorials for my library zEpid are hosted here. For more information on zEpid, see GitHub or ReadTheDocs.

The directory of this guide is

  1. Python Basics
  2. Basics of pandas (data management library)
  3. Epidemiology analyses in Python
    1. Basics
    2. Missing data
    3. Causal inference
      1. Time-fixed treatments
      2. Time-varying treatments
    4. Sensitivity analyses

Required packages for tutorial

To complete the tutorial, user must have the following packages installed: numpy, pandas, zepid, matplotlib, statsmodels, lifelines, and sklearn

IDE (Integrated Development Environment)

No IDE is required to complete the tutorial. All files are available in ipynb also known as jupyter notebooks. Code can be either downloaded or copied from the notebooks.

Here are some IDEs I have used in the past (and what I believe to be their advantages and disadvantages

Rodeo

This is the IDE I used for a long time. It is set up like RStudio

Advantages:

Basically RStudio but for Python, decent interface, easy to run line-by-line, easy to visualize plots (although it encourage bad habits)

Disadvantages:

Does not have all the features of RStudio (will delete changes if closed without saving), sucks up a lot of memory, sometimes the auto-complete would stop working if I hit more than 300+ lines of code, the environment tab is not great (don't expect it to open anything like RStudio)

Aside: their website has great tutorials how to run some basic stuff in Python if you are new to analysis in Python https://rodeo.yhat.com/

jupyter notebooks

Designed to be like a lab notebook, or like R markdown. Supports a pseudo-line-by-line concept Good for writing, since it allows for MarkDown. While I know a lot of people like jupyter, I only really use it for examples of code, not my personal programming. I never liked how it had to open via a Web Browser. I would rather have it be separate program. However, all guides were made using this IDE

PyCharm

This is the IDE I currently use

Advantages:

Easily set up virtual environments, interacts natively with Git, supports different file formats with plug-ins (e.g. .md), enforces certain coding conventions, better debug code features, organization of files under the project tab are convenient

Disadvantages:

Not great for running line-by-line code (it can do it, just not as elegantly), little more hardcore (I wouldn't really consider it a beginner's IDE. It requires some knowledge of set-up of Python)

IDLE

Ships with the basic Python 3.x installation. It is very basic and does not support line-by-line. Wouldn't recommend unless you are just starting with Python and don't want to commit to an IDE yet

Spyder

Ships with conda. Not bad but I didn't use it that much (I couldn't get the hang of it). Similarly it is an RStudio copy. Can't say too much since I haven't used it extensively

Basic Introduction to Python

If you have never used Python before, I have created some introductory materials to Python and the data management library I use, pandas. These are basic guides, but they also point to other resources. Please READ ALL OF THE BELOW BEFORE PROCEEDING.

Installing Python

To install, Python 3.x, we can download it directly from: https://www.python.org/downloads/

The installer provides an option to add Python3 to PATH, it is highly recommended you do this, since it allows you to avoid having to do it manually

Open Command Prompt / Terminal. When opened, type python and this should open Python in the same window. From here, you can quit by typing 'quit()' or closing the window. If this does NOT work, make sure your environmental variable was created properly

Installing Python Packages

Packages are what stores Python functions that we will use. These packages are contributed by various members of the community (including me)) and there is a wide array. To be able to download packages, we need to make sure we have an environmental variable created for python. We will discuss how to install packages

Python 3.x conveniently comes with a package manager. Basically it stores all the packages and we can use it to download new ones or update already downloaded ones.

To download a new package: Open Command Prompt/Terminal and use the following code (we will be installing pandas)

pip install pandas

To update a Python package, type the following command into Command Prompt. For example, we will update our pandas package

pip install pandas --upgrade

That concludes the basics. Please review parts 1 and 2 of the tutorials next

Comments
  • Cochran-Mantel-Haenszel

    Cochran-Mantel-Haenszel

    Thank you @pzivich for this amazing resource. Having the Hernan/Robbins causal model code in python is super helpful... g-estimation!

    I have a request... do you have a Cochran-Mantel-Haenszel script? If you get the chance, please, it would be useful to us to have in this repo. Thank you in advance!

    opened by opioiddatalab 2
  • Slight changes in Incidence Rate Ratio

    Slight changes in Incidence Rate Ratio

    Incidence Ratio Rate Paragraph

    • Fixed repetition
    • T1 & T0 are defined the same way. I believe that T0 is the person-time contributed by people NOT treated with ART
    opened by jaimiles23 0
  • Updates for v0.8.0

    Updates for v0.8.0

    Checklist for various notebooks to update with v0.8.0 release (hasn't released yet)

    • [x] IPTW update. Lots of major changes, so notebook needs to be completely overhauled

    • [x] Demonstrate new diagnostic functions for IPTW, g-formula, AIPW, TMLE

    • [x] Demonstrate g-bound argument

    • [x] Remove TMLE machine learning custom models. This is being removed in favor of cross-fitting. Can leave how to apply for now, but add the warning and mention will be cut in v0.9.0

    opened by pzivich 0
  • Notebooks not rendering in GitHub

    Notebooks not rendering in GitHub

    Sometimes GitHub has trouble rendering the notebooks. AFAIK the rendering system is behind the scenes at GitHub. Others have this same problem across repos and it sometimes occurs to me as well.

    If the notebook won't render in GitHub, you can copy the URL to the notebook you want to view and use the following site to view the notebook: https://nbviewer.jupyter.org/

    opened by pzivich 0
  • Replicate

    Replicate "Causal Inference"

    Issue to track progress on implementation of Hernan and Robins "Causal Inference" chapters

    • [x] Chapter 12: Inverse probability weights

    • [x] Chapter 13: Parametric g-formula

    • [x] Chapter 14: G-estimation of structural nested models

    • [x] ~Chapter 16: G-estimation for IV analysis~

    • [ ] Chapter 17: Causal survival analysis

    • [ ] Part III: Time-varying treatments

    ~G-estimation is not currently implemented. I will need to implement these before chapter 14 can be done.~

    Currently there are no plans to replicate Chapter 15 (propensity scores and regression) or Chapter 16 (instrumental variables) since the first method does not require zEpid and I am unfamiliar with the second. Maybe instrumental variables will be added in the future?

    For Chapter 16, I am considering demonstrating the usage of g-estimation instead of two-stage least-squares. Specifically, using the same data as done in Chapter 16 but following Technical Point 16.3

    enhancement 
    opened by pzivich 0
  • Tutorials

    Tutorials

    On the website, create quick tutorials demonstrating each of the implemented estimators, descriptions of how they work, and why you might want to use them. Might be more digestible than the current docs (also better justify why to choose one over the other)

    Reference to base on https://lifelines.readthedocs.io/en/latest/jupyter_notebooks/Proportional%20hazard%20assumption.html https://github.com/CamDavidsonPilon/lifelines/blob/master/docs/jupyter_notebooks/Proportional%20hazard%20assumption.ipynb

    TODO

    • [x] Basic measures

    • [x] splines

    • [x] IPTW: time-fixed treatment

    • [ ] IPTW: stochastic treatment

    • [ ] IPTW: time-varying treatment

    • [x] IPCW

    • [x] IPMW: single variable

    • [ ] IPMW: monotone

    • [ ] IPMW: nonmonotone (to add after implemented)

    • [x] G-formula: time-fixed binary treatment, binary outcome

    • [x] G-formula: time-fixed categorical treatment, binary outcome

    • [ ] G-formula: time-fixed continuous treatment, binary outcome (to add after implemented)

    • [x] G-formula: time-fixed binary treatment, continuous outcome

    • [x] G-formula: Monte Carlo

    • [x] G-formula: Iterative Conditional

    • [x] G-estimation of SNM

    • [x] AIPTW

    • [ ] AIPMW

    • [x] TMLE

    • [x] TMLE: stochastic treatment

    • [ ] LTMLE (to add after implemented)

    • [x] Quantitative bias analysis

    • [x] Functional form assessment

    • [x] Generalizability

    • [ ] Transportability (IPSW, g-transport, AIPSW)

    • [x] Monte Carlo g-formula by-hand (helps to explain underlying process)

    opened by pzivich 1
Releases(v0.8.0)
Owner
Paul Zivich
Epidemiology post-doc working in epidemiologic methods and infectious diseases.
Paul Zivich
Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution

FAU Implementation of the paper: Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution. Yingruo

Evelyn 78 Nov 29, 2022
3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks

3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks Introduction This repository contains the code and models for the follo

124 Jan 06, 2023
[ICCV 2021] Learning A Single Network for Scale-Arbitrary Super-Resolution

ArbSR Pytorch implementation of "Learning A Single Network for Scale-Arbitrary Super-Resolution", ICCV 2021 [Project] [arXiv] Highlights A plug-in mod

Longguang Wang 229 Dec 30, 2022
Using this codebase as a tool for my own research. Making some modifications to the original repo for my own purposes.

For SwapNet Create a list.txt file containing all the images to process. This can be done with the GNU find command: find path/to/input/folder -name '

Andrew Jong 2 Nov 10, 2021
A visualization tool to show a TensorFlow's graph like TensorBoard

tfgraphviz tfgraphviz is a module to visualize a TensorFlow's data flow graph like TensorBoard using Graphviz. tfgraphviz enables to provide a visuali

44 Nov 09, 2022
Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

Python Streaming Anomaly Detection (PySAD) PySAD is an open-source python framework for anomaly detection on streaming multivariate data. Documentatio

Selim Firat Yilmaz 181 Dec 18, 2022
A GPU-optional modular synthesizer in pytorch, 16200x faster than realtime, for audio ML researchers.

torchsynth The fastest synth in the universe. Introduction torchsynth is based upon traditional modular synthesis written in pytorch. It is GPU-option

torchsynth 229 Jan 02, 2023
Emotion classification of online comments based on RNN

emotion_classification Emotion classification of online comments based on RNN, the accuracy of the model in the test set reaches 99% data: Large Movie

1 Nov 23, 2021
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 01, 2023
[CVPR'22] Official PyTorch Implementation of Collaborative Transformers for Grounded Situation Recognition

[CVPR'22] Collaborative Transformers for Grounded Situation Recognition Paper | Model Checkpoint This is the official PyTorch implementation of Collab

Junhyeong Cho 29 Dec 10, 2022
This is an official implementation of "Polarized Self-Attention: Towards High-quality Pixel-wise Regression"

Polarized Self-Attention: Towards High-quality Pixel-wise Regression This is an official implementation of: Huajun Liu, Fuqiang Liu, Xinyi Fan and Don

DeLightCMU 212 Jan 08, 2023
Accepted at ICCV-2021: Workshop on Computer Vision for Automated Medical Diagnosis (CVAMD)

Is it Time to Replace CNNs with Transformers for Medical Images? Accepted at ICCV-2021: Workshop on Computer Vision for Automated Medical Diagnosis (C

Christos Matsoukas 80 Dec 27, 2022
BOVText: A Large-Scale, Multidimensional Multilingual Dataset for Video Text Spotting

BOVText: A Large-Scale, Bilingual Open World Dataset for Video Text Spotting Updated on December 10, 2021 (Release all dataset(2021 videos)) Updated o

weijiawu 47 Dec 26, 2022
Jittor implementation of Recursive-NeRF: An Efficient and Dynamically Growing NeRF

Recursive-NeRF: An Efficient and Dynamically Growing NeRF This is a Jittor implementation of Recursive-NeRF: An Efficient and Dynamically Growing NeRF

33 Nov 30, 2022
🚀 An end-to-end ML applications using PyTorch, W&B, FastAPI, Docker, Streamlit and Heroku

🚀 An end-to-end ML applications using PyTorch, W&B, FastAPI, Docker, Streamlit and Heroku

Made With ML 82 Jun 26, 2022
Temporal-Relational CrossTransformers

Temporal-Relational Cross-Transformers (TRX) This repo contains code for the method introduced in the paper: Temporal-Relational CrossTransformers for

83 Dec 12, 2022
Unified file system operation experience for different backend

megfile - Megvii FILE library Docs: http://megvii-research.github.io/megfile megfile provides a silky operation experience with different backends (cu

MEGVII Research 76 Dec 14, 2022
GLNet for Memory-Efficient Segmentation of Ultra-High Resolution Images

GLNet for Memory-Efficient Segmentation of Ultra-High Resolution Images Collaborative Global-Local Networks for Memory-Efficient Segmentation of Ultra-

VITA 298 Dec 12, 2022
Files for a tutorial to train SegNet for road scenes using the CamVid dataset

SegNet and Bayesian SegNet Tutorial This repository contains all the files for you to complete the 'Getting Started with SegNet' and the 'Bayesian Seg

Alex Kendall 800 Dec 31, 2022
Readings for "A Unified View of Relational Deep Learning for Polypharmacy Side Effect, Combination Therapy, and Drug-Drug Interaction Prediction."

Polypharmacy - DDI - Synergy Survey The Survey Paper This repository accompanies our survey paper A Unified View of Relational Deep Learning for Polyp

AstraZeneca 79 Jan 05, 2023