Pyspark sam - Analyze Big Sequence Alignments with PySpark in AWS EMR

Related tags

Third-party APIs Wrapperspyspark_sam
Overview

pyspark_sam

This repo hosts my code for the article "Analyze Big Sequence Alignments with PySpark in AWS EMR".

Prerequisite

  1. Spark

  2. AWS CLI

  3. AWS Account

Run

Follow the instruction in the article. Once you have uploaded the files into your S3 bucket, run

aws emr create-cluster --name "Spark_step_pip" \
    --release-label emr-6.5.0 \
    --applications Name=Spark \
    --log-uri s3://[your_S3_bucket]/logs/ \
    --instance-type m5.xlarge \
    --instance-count 3 \
    --bootstrap-actions Path=s3://[your_S3_bucket]/emr_bootstrap.sh \
    --use-default-roles --auto-terminate \
    --steps "Type=Spark,Name=SparkProgram,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--py-files,s3://[your_S3_bucket]/helper_function.py,s3://[your_S3_bucket]/spark_3mer.py,s3://[your_S3_bucket]/test.sam,[your_S3_bucket],sankey.json]"

When the job finishes, download the sankey.json. And run this command to visualize:

python sankey.py sankey.json

Authors

  • Sixing Huang - Concept and Coding

License

This project is licensed under the MIT License - see the LICENSE file for details

Owner
Sixing Huang
A triple Neo4j certified data scientist. I am currently working at BGI in Shenzhen.
Sixing Huang
GitHub Repository
Simple web browser to visualize HiC tracks

HiCBrowser : A simple web browser to visualize Hi-C and other genomic tracks Fidel Ramirez, José Villaveces, Vivek Bhardwaj Installation You can insta

The deepTools ecosystem 14 Jun 21, 2022
A Python library for PagerDuty.

Pygerduty Python Library for PagerDuty's REST API and Events API. This library was originally written to support v1 and is currently being updated to

Dropbox 164 Dec 20, 2022
DB-Drive-CSV - This is app is can be used to access CSV file as JSON from Google Drive.

DB Drive CSV This is app is can be used to access CSV file as JSON from Google Drive. How To Use Create file/ upload file to Google Drive There's 2 fi

Hartawan Bahari M. 5 Oct 20, 2022
Muzan-Discord-Nuker - A simple discord server nuker in python

Muzan-Discord-Nuker This is Just a simple discord server nuker in python. ✨ Feat

Afnan 3 May 14, 2022
A Python library for the Buildkite API

PyBuildkite A Python library and client for the Buildkite API. Usage To get the package, execute: pip install pybuildkite Then set up an instance of

Peter Yasi 29 Nov 30, 2022
Bot Realm of the Mad God Exalt (ROTMG). (Auto_nexus, Auto_HP, Auto_Loot)

Bot_ROTMG Bot Realm of the Mad God Exalt (ROTMG). (Auto_nexus, Auto_HP, Auto_Loot) *Este projeto visa apenas o aprendizado, quem faz mal uso é o único

Guilherme Silva Uchoa 1 Oct 30, 2021
The community bot for the Python Discord community

Python Utility Bot This project is a Discord bot specifically for use with the Python Discord server. It provides numerous utilities and other tools t

Python Discord 998 Jan 03, 2023
VC-Music , Playing music without bot.

VC-Userbot A Telegram Userbot to play or streaming Audio and Video songs / files in Telegram Voice Chats. It's made with PyTgCalls and Pyrogram Requir

RioProjectX 8 Aug 04, 2022
Confirm that files have been uploaded to Backblaze Cloud Backup successfully

Backblaze Backup Checker This Python script compares metadata captured from files within source folders against data parsed from Backblaze Cloud Backu

18 Jul 29, 2022
❤️ Hi There Im EzilaX ❤️ A next gen powerful telegram group manager bot 😱 for manage your groups and have fun with other cool modules Made By Sadew Jayasekara 🔥

❤️ EzilaX v1 ❤️ Unmaintained. The new repo of @EzilaXBot is Public. (It is no longer based on this source code. The completely rewritten bot available

Sadew Jayasekara 18 Nov 24, 2021
Python SDK for Facebook's Graph API

Facebook Python SDK This client library is designed to support the Facebook Graph API and the official Facebook JavaScript SDK, which is the canonical

Mobolic 2.7k Jan 07, 2023
A simple python discord bot with commands for moderation and utility.

Discord Bot A simple python discord bot with commands for moderation, utility and fun. Moderation $kick user reason - Kick a user from the server

3 Jan 07, 2022
This Python script will automate the process of uploading your project to GitHub.

ProjectToGithub This Python script will help you to upload your project to Github without having to type in any commands !!! Quick Start guide First C

Imira Randeniya 1 Sep 11, 2022
Turns any script into a telegram bot

pytobot Turns any script into a telegram bot Install pip install --upgrade pytobot Usage Script: while True: message = input() if message == "

Dmitry Kotov 17 Jan 06, 2023
Send OpenWeatherMap alerts (One Call API) to telegram users.

OpenWeatherMap Telegram Alert Send OpenWeatherMap alerts (One Call API) to telegram users. Installation Requirements: $ apt install python3-yaml pytho

Michael Hacker 1 Jun 04, 2022
Telegram bot to provide links of different types of files you send

File To Link Bot - IDN-C-X Telegram bot to provide links of different types of files you send. WHAT CAN THIS BOT DO Is it a nuisance to send huge file

IDNCoderX 3 Oct 26, 2021
A bot to get Statistics like the Playercount from your Minecraft-Server on your Discord-Server

Hey Thanks for reading me. Warning: My English is not the best I have programmed this bot to show me statistics about the player numbers and ping of m

spaffel 12 Sep 24, 2022
Temperature Monitoring and Prediction Using a Modified Lambda Architecture

Temperature Monitoring and Prediction Using a Modified Lambda Architecture A more detailed write up can be seen in this paper. Original Lambda Archite

Parsa Yousefi 2 Jun 27, 2022
A full-fledged discord bot with moderation and a lot more.

HOT-BOT-POL-POT ⭐ Star me on GitHub m'lady.... hot-bot-pol-pot is a moderation discord bot written using enhanced-dpy library with many functionalitie

Pure Cheekbones 4 Oct 08, 2022
HTTP API for TON (The Open Network)

HTTP API for The Open Network Since TON nodes uses its own ADNL binary transport protocol, a intermediate service is needed for an HTTP connection. TO

66 Dec 28, 2022