ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete

Overview

ShadowClone

ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete.

You can make full use of the Free Tiers provided by cloud providers and supercharge your mundane cli tools with shadow clone jutsu (Naruto style)!

Installation

Please visit the wiki for installation and intial configuration instructions

Usage

⚡ python shadowclone.py -h
usage: shadowclone.py [-h] -i INPUT [-s SPLITNUM] [-o OUTPUT] -c COMMAND

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
  -s SPLITNUM, --split SPLITNUM
                        Number of lines per chunk of file
  -o OUTPUT, --output OUTPUT
  -c COMMAND, --command COMMAND
                        command to execute

How it works

We create a container image during the initial setup and register it as a runtime for our function in AWS/GCP/Azure whatever. When you execute ShadowClone on your computer, instances of that container are activated automatically and are only active for the duration of its execution. How many instances to activate is dynamically decided at runtime depending on the size of the input file provided and the split factor. The input is then split into chunks and equally distributed between all the instances to execute in parallel. For example, if your input file has 10,000 lines and you set the split factor to 100 lines, then it will be split into 100 chunks of 100 lines each and 100 instances will be run in parallel!

Features

  • Extremely fast
  • No need to maintain a VPS (or a fleet of it :))
  • Costs almost nothing per month
    • Compatible with free tiers of most cloud services
  • Cloud agnostic
    • Same script works with AWS, GCP, Azure etc.
  • Supports upto 1000 parallel invocations
  • Dynamically decide the number of invocations
  • Run any tool in parallel on the cloud
  • Pipe output to other tools

Comparison

This tool was inspired by the awesome Axiom and Fleex projects and goes beyond the concept of VPS for running the tools by using serverless functions and containers.

Features Axiom/Fleex ShadowClone
Instances 10-100s* 1000s
Cost Per instance/per minute Mostly Free**
Startup Time 4-5 minutes 2-3 seconds
Idle Cost $++ Free
On Demand Scalability No

*Most cloud providers do not allow spinning up too many instances by default, so you are limited to around 10-15 instances at max. You have to make a request to the support to increase this number.

** AWS & Azure allow 1 million invocations per month for free. Google allows 2 million invocations per month for free. You will be charged only if you go above these limits

Demo

DNS Bruteforcing using a 43mb file - 34 seconds

asciicast

Running httpx on 94K subdomains in 1 min

asciicast

References

Lithops documentation

Free Tiers

Cloud Provider Free Allowance Link
Google Functions 2 Million invocations, 400,000 GB-seconds per month Google Cloud Free Program
AWS Lambda 1 Million invocations, Up to 3.2 million seconds of compute time per month Free Cloud Computing Services - AWS Free Tier
Azure Functions 1 Million invocations Microsoft Azure Free Services

Obviously, you can make any number of function invocations per month. The table above only shows how many invocations are free.

Similar Tools

Disclaimer

This tool is designed as a proof-of-concept implementation and it's usage is intended for educational purpose only. Usage for attacking targets without prior mutual consent is illegal. It's the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program.

Python client and API for monitoring and controling energy diversion devices from MyEnergi

Python client and API for monitoring and controling energy diversion devices from MyEnergi A set of library functions and objects for interfacing with

1 Dec 17, 2021
Free and Open Source Machine Translation API. 100% self-hosted, no limits, no ties to proprietary services. Built on top of Argos Translate.

LibreTranslate Try it online! | API Docs Free and Open Source Machine Translation API, entirely self-hosted. Unlike other APIs, it doesn't rely on pro

UAV4GEO 3.5k Jan 03, 2023
An Python SDK for QQ based on mirai-api-http v2.

Argon 一个基于 graia-broadcast 和 mirai-api-http v2 的 Python SDK。 本项目适用于 mirai-api-http 2.0 以上版本。 目前仍处于开发阶段,内部接口可能会有较大的变化。 The Stasis / 停滞 为维持 GraiaProject

BlueGlassBlock 1 Oct 29, 2021
Typed interactions with the GitHub API v3

PyGitHub PyGitHub is a Python library to access the GitHub API v3 and Github Enterprise API v3. This library enables you to manage GitHub resources su

5.7k Jan 06, 2023
Python wrapper for WhatsApp web-based on selenium

alright Python wrapper for WhatsApp web made with selenium inspired by PyWhatsApp Why alright ? I was looking for a way to control and automate WhatsA

Jordan Kalebu 193 Jan 06, 2023
Its The Basic Commands Of Termux

Its The Basic Commands Of Termux

ANKIT KUMAR 1 Dec 27, 2021
Takes upcoming items from a Google Calendar and posts them to Slack.

Google Calendar to Slack by Jason Snell - [email protected] This Python s

6 Aug 21, 2022
An Anime Theme Telegram group management bot. With lot of features.

Emilia Project Emilia-Prjkt is a modular bot running on python3 with anime theme and have a lot features. Easiest Way To Deploy On Heroku This Bot is

ZenitsuID #M•R•T™ 3 Feb 03, 2022
A Powerful, Smart And Simple Userbot In Telethon.

Owner: Masterolic 🇮🇳 BLACK LIGHTNING A Powerful, Smart And Simple Userbot In Telethon. Credits This is A Remix Bot Of Many UserBot. DARKCOBRA Friday

Masterolic 1 Nov 28, 2021
Pluggable Telethon - Telegram UserBot

A stable pluggable Telegram userbot, based on Telethon.

Team Ultroid 2.3k Dec 30, 2022
Code for "Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals," CoRL 2021.

Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals This repository contains code for "Multimodal trajectory prediction conditioned

Nachiket Deo 113 Dec 28, 2022
The system to host your files on the Discord application

Distorage The system to host your files on the Discord application Documentation Documentation Distorage How to use the package You can install it wit

6 Jun 27, 2022
Wrapper for shh/rsync for use with OpenFOAM and blue bear

bbsync wrapper for shh/rsync for use with OpenFOAM and blue bear About The Project bbsync is a wrapper for shh/rsync for use with OpenFOAM and blue be

1 Dec 10, 2021
A Pancakeswap and Uniswap trading client (and bot) with limit orders, marker orders, stop-loss, custom gas strategies, a GUI and much more.

Pancakeswap and Uniswap trading client Adam A A Pancakeswap and Uniswap trading client (and bot) with market orders, limit orders, stop-loss, custom g

570 Mar 09, 2022
🔏 Discordちゃんねる ◆wGFzKUzY7E

使い方 discord.pyをインストール. python -m pip install -r requirements.txtを実行. bot.pyと同じ階層に.tokenを用意. bot.pyを実行. ※現状、使用しているライブラリの関係でWindowsOSは未対応です。 コマンド ニックネーム

Gattxxa 3 Feb 02, 2022
A tool to build scripts to toggle between minimal & default services in Windows based on user defined lists.

A tool to build scripts to toggle between minimal & default services in Windows based on user defined lists.

AMIT 29 Jan 01, 2023
3X Fast Telethon Based Bot

📺 YouTube Song Downloader Bot For Telegram 🔮 3X Fast Telethon Based Bot ⚜ Easy To Deploy 🤗

@Dk_king_offcial 1 Dec 09, 2021
Cancel all your follow requests on Instagram.

Unrequester This python code unrequests all your follow requests on Instagram, using selenium. Everything's step-by-step and understanding it is like

ChamRun 3 Apr 09, 2022