ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete

Overview

ShadowClone

ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete.

You can make full use of the Free Tiers provided by cloud providers and supercharge your mundane cli tools with shadow clone jutsu (Naruto style)!

Installation

Please visit the wiki for installation and intial configuration instructions

Usage

⚡ python shadowclone.py -h
usage: shadowclone.py [-h] -i INPUT [-s SPLITNUM] [-o OUTPUT] -c COMMAND

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
  -s SPLITNUM, --split SPLITNUM
                        Number of lines per chunk of file
  -o OUTPUT, --output OUTPUT
  -c COMMAND, --command COMMAND
                        command to execute

How it works

We create a container image during the initial setup and register it as a runtime for our function in AWS/GCP/Azure whatever. When you execute ShadowClone on your computer, instances of that container are activated automatically and are only active for the duration of its execution. How many instances to activate is dynamically decided at runtime depending on the size of the input file provided and the split factor. The input is then split into chunks and equally distributed between all the instances to execute in parallel. For example, if your input file has 10,000 lines and you set the split factor to 100 lines, then it will be split into 100 chunks of 100 lines each and 100 instances will be run in parallel!

Features

  • Extremely fast
  • No need to maintain a VPS (or a fleet of it :))
  • Costs almost nothing per month
    • Compatible with free tiers of most cloud services
  • Cloud agnostic
    • Same script works with AWS, GCP, Azure etc.
  • Supports upto 1000 parallel invocations
  • Dynamically decide the number of invocations
  • Run any tool in parallel on the cloud
  • Pipe output to other tools

Comparison

This tool was inspired by the awesome Axiom and Fleex projects and goes beyond the concept of VPS for running the tools by using serverless functions and containers.

Features Axiom/Fleex ShadowClone
Instances 10-100s* 1000s
Cost Per instance/per minute Mostly Free**
Startup Time 4-5 minutes 2-3 seconds
Idle Cost $++ Free
On Demand Scalability No

*Most cloud providers do not allow spinning up too many instances by default, so you are limited to around 10-15 instances at max. You have to make a request to the support to increase this number.

** AWS & Azure allow 1 million invocations per month for free. Google allows 2 million invocations per month for free. You will be charged only if you go above these limits

Demo

DNS Bruteforcing using a 43mb file - 34 seconds

asciicast

Running httpx on 94K subdomains in 1 min

asciicast

References

Lithops documentation

Free Tiers

Cloud Provider Free Allowance Link
Google Functions 2 Million invocations, 400,000 GB-seconds per month Google Cloud Free Program
AWS Lambda 1 Million invocations, Up to 3.2 million seconds of compute time per month Free Cloud Computing Services - AWS Free Tier
Azure Functions 1 Million invocations Microsoft Azure Free Services

Obviously, you can make any number of function invocations per month. The table above only shows how many invocations are free.

Similar Tools

Disclaimer

This tool is designed as a proof-of-concept implementation and it's usage is intended for educational purpose only. Usage for attacking targets without prior mutual consent is illegal. It's the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program.

YouTube playlist Files downloaded by FDM are not organized according to the original order on YouTube

Youtube-Playlist-File-Organizer YouTube playlist Files downloaded by Free Download Manager are not organized according to the original order on YouTub

David Mainoo 3 Dec 27, 2021
To send an Instagram message using Python

To send an Instagram message using Python, you must have an Instagram account and install the Instabot library in your Python virtual environment.

Coding Taggers 1 Dec 18, 2021
Tweet stream in OBS browser source

Tweetron TweetronはOBSブラウザーソースを使用してツイートを画面上に表示するツールソフトです Windowsのみ対応 (Windows10動作確認済) ダウンロード こちらから最新版をダウンロードしてください (現在ベータテスト版を配布しています) Download ver0.0.

Cube 0 Apr 05, 2022
A Wrapper for ScarletAPI

ScarletAPI A Wrapper for ScarletAPI still a work in progress Docs these are the

Amashi 0 Mar 24, 2022
A simple tool which automate commands of discord economy bots

A simple tool which automate commands of discord economy bots. Fully configurable using an intuitive configuration made in YAML

SkydenFly 5 Sep 18, 2022
A tool for creating credentials for accessing S3 buckets

s3-credentials A tool for creating credentials for accessing S3 buckets For project background, see s3-credentials: a tool for creating credentials fo

Simon Willison 138 Jan 06, 2023
A simple language translator with python and google translate api

Language translator with python A simple language translator with python and google translate api Install pip and python 3.9. All the required depende

0 Nov 11, 2021
A Python interface to AFL, allowing for easy injection of testcases and other functionality.

Fuzzer This module provides a Python wrapper for interacting with AFL (American Fuzzy Lop: http://lcamtuf.coredump.cx/afl/). It supports starting an A

Shellphish 614 Dec 26, 2022
A Python script to backup all repos (public or private) of a user.

GithubBackupAllRepos A Python script to backup all repos (public or private) of a user. Features Clone public and private repos Load specified SSH key

Podalirius 15 Jan 03, 2023
Spotify Web API client for Python 3

Welcome to the GitHub repository of Tekore! We provide a client for the Spotify Web API for Python, complete with all available endpoints and authenti

Felix Hildén 186 Dec 22, 2022
A simple MTProto-based bot that can download various types of media (>10MB) on a local storage

TG Media Downloader Bot 🤖 A telegram bot based on Pyrogram that downloads on a local storage the following media files: animation, audio, document, p

Alessio Tudisco 11 Nov 01, 2022
Sielzz Music adalah proyek bot musik telegram, memungkinkan Anda memutar musik di telegram grup obrolan suara.

Hi, I am: Requirements 📝 FFmpeg NodeJS nodesource.com Python 3.8 or higher PyTgCalls MongoDB Get STRING_SESSION from below: 🎖 History Features 🔮 Th

1 Nov 29, 2021
一个基于Python3的Bot。目前支持以Docker的方式部署在vps上。支持Aria2、本子下载、网易云音乐下载、Pixiv榜单下载、Youtue-dl支持、搜图。

介绍 一个基于Python3的Bot。目前支持以Docker的方式部署在vps上。 主要功能: 文件管理 修改主界面为 filebrowser,账号为admin,密码为admin,主界面路径:http://ip:port,请自行修改密码 FolderMagic自带的webdav:路径:http://

Ben 650 Jan 08, 2023
A telegram bot writen in python for mirroring files on the internet to Google Drive

owner of this repo :- AYUSH contact me :- AYUSH Slam Mirror Bot This is a telegram bot writen in python for mirroring files on the internet to our bel

Thanusara Pasindu 1 Nov 21, 2021
Python library for interacting with the Wunderlist 2 REST API

Overview Wunderpy2 is a thin Python library for accessing the official Wunderlist 2 API. What does a thin library mean here? Only the bare minimum of

mieubrisse 24 Dec 29, 2020
Asynchronous Guilded API wrapper for Python

Welcome to guilded.py, a discord.py-esque asynchronous Python wrapper for the Guilded API. If you know discord.py, you know guilded.py. Documentation

shay 115 Dec 30, 2022
“Hey there 👋 I'm szrosebot .A Powerful, Smart And Simple Group Manager with some extra features..

A Powerful, Smart And Simple Group Manager szrose bot This is the clone of DewmiBotit is a Powerful, Smart And Simple Group Manager bot made by hiruna

slgeekshow 36 Oct 30, 2022
Pdisk Link Converter Telegram Bot, Convert link in a single click

Pdisk Converter Bot Make short link by using Pdisk API key Installation The Easy Way Required Variables BOT_TOKEN: Create a bot using @BotFather, and

Ayush Kumar Jaiswal 6 Jul 28, 2022
This repository provides a set functions to extract paragraphs from AWS Textract responses.

extract-paragraphs-with-aws-textract Since AWS Textract (the AWS OCR service) does not have a native function to extract paragraphs, this repository p

Juan Anzola 3 Jan 26, 2022
Automated JSON API based communication with Fronius Symo

PyFronius - a very basic Fronius python bridge A package that connects to a Fronius device in the local network and provides data that is provided via

Niels Mündler 10 Dec 30, 2022