Hook Slinger acts as a simple service that lets you send, retry, and manage event-triggered POST requests, aka webhooks

Overview

logo

>> A generic service to send, retry, and manage webhooks. <<

forthebadge forthebadge forthebadge

Table of Contents

Description

What?

Hook Slinger acts as a simple service that lets you send, retry, and manage event-triggered POST requests, aka webhooks. It provides a fully self-contained docker image that is easy to orchestrate, manage, and scale.

Why?

Technically, a webhook is a mere POST request—triggered by a system—when a particular event occurs. The following diagram shows how a simple POST request takes the webhook nomenclature when invoked by an event trigger.

Webhook Concept

However, there are a few factors that make it tricky to manage the life cycle of a webhook, such as:

  • Dealing with server failures on both the sending and the receiving end.
  • Managing HTTP timeouts.
  • Retrying the requests gracefully without overloading the recipients.
  • Avoiding retry loop on the sending side.
  • Monitoring and providing scope for manual interventions.
  • Scaling them quickly; either vertically or horizontally.
  • Decoupling webhook management logic from your primary application logic.

Properly dealing with these concerns can be cumbersome; especially when sending webhooks is just another small part of your application and you just want it to work without you having to deal with all the hairy details every time. Hook Slinger aims to alleviate this pain point.

How?

Hook Slinger exposes a single endpoint where you can post your webhook payload, destination URL, auth details, and it'll make the POST request for you asynchronously in the background. Under the hood, the service uses:

  • FastAPI to provide a Uvicorn driven ASGI server.

  • Redis and RQ for implementing message queues that provide the asynchrony and robust failure handling mechanism.

  • Rqmonitor to provide a dashboard for monitoring the status of the webhooks and manually retrying the failed jobs.

  • Rich to make the container logs colorful and more human friendly.

The simplified app architecture looks something this:

Topology

In the above image, the webhook payload is first sent to the app and the app leverages the worker instance to make the POST request. Redis DB is used for fast bookkeeping and async message queue implementation. The monitor instance provides a GUI to monitor and manage the webhooks. Multiple worker instances can be spawned to achieve linear horizontal scale-up.

Installation

  • Make sure you've got Docker and Docker Compose installed in your system.

  • Clone the repository and head over to the root directory.

  • To start the orchestra, run:

    make start_servers
    

    This will:

    • Start an app server that can be accessed from port 5000.

    • Start an Alpine-based Redis server that exposes port 6380.

    • Start a single worker that will carry out the actual tasks.

    • Start a rqmonitor instance that opens port 8899.

  • To shut down everything, run:

    make stop_servers
    

TODO: Generalize it more before making it installable with a docker pull command.

Usage

Exploring the Interactive API Docs

To try out the entire workflow interactively, head over to the following URL on your browser:

http://localhost:5000/docs

You should see a panel like this:

API Docs

This app implements a rudimentary token-based authentication system where you're expected to send an API token by adding Authorization: Token field to your request header. To do that here, click the POST /hook_slinger/ ribbon and that will reveal the API description like this:

API Description

Copy the default token value from the description corpus, then click the green button on the top right that says Authorize, and paste the value in the prompt box. Click the Authorize button again and that'll conclude the login step. In your production application, you should implement a robust authentication system or at least change this default token.

To send a webhook, you'll need a URL where you'll be able to make the POST request. For this demonstration, let's pick this webhook site service to monitor the received webhooks. It gives you a unique URL against which you'll be able to make the post requests and monitor them in a dashboard like this:

Webhook Site

On the API docs page, click the Try it out button near the request body section:

API Request

This should reveal a panel like the following one where you can make your request:

API Request

Notice that the section is prefilled with an example request payload. You can use this exact payload to make a request. Go ahead and click the execute button. If you scroll down a little, you'll notice the HTTP response:

API Response

Now, if you head over to the webhook site URL, you should be able to see your API payload:

API Response

To monitor the webhook tasks, head over to the following URL:

http://localhost:8899/

You should be presented with a GUI like this:

RQ Monitor

If you click Workers on the left panel, you'll be presented with a panel where you can monitor all the workers:

RQ Monitor

The Jobs panel lists all the tasks, and from there you'll be able to requeue a failed job. By default, Hook Slinger retries a failed job 3 times with 5 seconds linear backoff. However, this can be configured using environment variables in the .env file.

RQ Monitor

Sending A Webhook Via cURL

Run the following command on your terminal; this assumes that you haven't changed the auth token (you should):

curl -X 'POST' \
  'http://localhost:5000/hook_slinger/' \
  -H 'accept: application/json' \
  -H 'Authorization: Token $5$1O/inyTZhNvFt.GW$Zfckz9OL.lm2wh3IewTm8YJ914wjz5txFnXG5XW.wb4' \
  -H 'Content-Type: application/json' \
  -d '{
  "to_url": "https://webhook.site/37ad9530-59c3-430d-9db6-e68317321a9f",
  "to_auth": "",
  "tag": "Dhaka",
  "group": "Bangladesh",
  "payload": {
    "greetings": "Hello, world!"
  }
}' | python -m json.tool

You should expect the following output:

{
    "status": "queued",
    "ok": true,
    "message": "Webhook registration successful.",
    "job_id": "Bangladesh_Dhaka_a07ca786-0b7a-4029-bac0-9a7c6eb68a98",
    "queued_at": "2021-07-23T20:15:04.389690"
}

Sending A Webhook Via Python

For this purpose, you can use an HTTP library like httpx.

Make the request with the following script:

import asyncio
from http import HTTPStatus
from pprint import pprint

import httpx


async def send_webhook() -> None:

    wh_payload = {
        "to_url": "https://webhook.site/aa7e2e7e-a62d-4505-8879-13bd806da6d5",
        "to_auth": "",
        "tag": "Dhaka",
        "group": "Bangladesh",
        "payload": {"greetings": "Hello, world!"},
    }

    async with httpx.AsyncClient(http2=True) as session:
        headers = {
            "Content-Type": "application/json",
            "Authorization": (
                "Token $5$1O/inyTZhNvFt.GW$Zfckz9OL.lm2wh3IewTm8YJ914wjz5txFnXG5XW.wb4"
            ),
        }

        response = await session.post(
            "http://localhost:5000/hook_slinger",
            headers=headers,
            json=wh_payload,
        )

        # Hook Slinger returns http code 202, accepted, for a successful request.
        if response.status_code == HTTPStatus.ACCEPTED:
            result = response.json()
            pprint(result)


asyncio.run(send_webhook())

This should return a similar response as before:

{
    'job_id': 'Bangladesh_Dhaka_139fc35a-d2a5-4d01-a6af-e980c52f55bc',
    'message': 'Webhook registration successful.',
    'ok': True,
    'queued_at': '2021-07-23T20:15:04.389690',
    'status': 'queued'
}

Exploring the Container Logs

Hook Slinger overloads the Python root logger to give you a colorized and user-friendly logging experience. To explore the logging messages of the application server, run:

make app_logs

Notice the colorful logs cascading down from the app server:

App Logs

Now, to explore the worker instance logs, in a separate terminal, run:

make worker_logs

You should see something like this:

Worker Logs

Scaling Up the Service

Hook Slinger offers easy horizontal scale-up, powered by the docker-compose --scale command. In this case, scaling up means, spawning new workers in separate containers. Let's spawn 3 worker containers this time. To do so, first shut down the orchestra by running:

make stop_servers

Now, run:

make worker_scale n=3

This will start the App server, Redis DB, RQmonitor, and 3 Worker instances. Spawning multiple worker instances are a great way to achieve job concurrency with the least amount of hassle.

Philosophy & Limitations

Hooks Slinger is designed to be simple, transparent, upgradable, and easily extensible to cater to your specific needs. It's not built around AMQP compliant message queues with all the niceties and complexities that come with them—this is intentional.

Also, if you scrutinize the end-to-end workflow, you'll notice that it requires making HTTP requests from the sending service to the Hook Slinger. This inevitably adds another point of failure. However, from the sending service's POV, it's sending the HTTP requests to a single service, and the target service is responsible for fanning out the webhooks to the destinations. The developers are expected to have control over both services, which theoretically should mitigate the failures. The goal is to transfer some of the code complexity around managing webhooks from the sending service over to the Hook Slinger. Also, I'm playing around with some of the alternatives to using HTTP POST requests to send the payloads from the sending end to the Hook Slinger. Suggestions are always appreciated.

🍰
Owner
Redowan Delowar
Hacking healthcare @DendiSoftware. Writing & talking about—Statistics, Machine Learning, System Arch, APIs, Redis, Docker, Python, Go, etc.
Redowan Delowar
A utility that allows you to use DI in fastapi without Depends()

fastapi-better-di What is this ? fastapi-better-di is a utility that allows you to use DI in fastapi without Depends() Installation pip install fastap

Maxim 9 May 24, 2022
This is an API developed in python with the FastApi framework and putting into practice the recommendations of the book Clean Architecture in Python by Leonardo Giordani,

This is an API developed in python with the FastApi framework and putting into practice the recommendations of the book Clean Architecture in Python by Leonardo Giordani,

0 Sep 24, 2022
Signalling for FastAPI.

fastapi-signals Signalling for FastAPI.

Henshal B 7 May 04, 2022
OpenAPI for Todolist RESTful API

swagger-client OpenAPI for Todolist RESTful API This Python package is automatically generated by the Swagger Codegen project: API version: 1 Package

Iko Afianando 1 Dec 19, 2021
Qwerkey is a social media platform for connecting and learning more about mechanical keyboards built on React and Redux in the frontend and Flask in the backend on top of a PostgreSQL database.

Flask React Project This is the backend for the Flask React project. Getting started Clone this repository (only this branch) git clone https://github

Peter Mai 22 Dec 20, 2022
A kedro-plugin to serve Kedro Pipelines as API

General informations Software repository Latest release Total downloads Pypi Code health Branch Tests Coverage Links Documentation Deployment Activity

Yolan Honoré-Rougé 12 Jul 15, 2022
Adds simple SQLAlchemy support to FastAPI

FastAPI-SQLAlchemy FastAPI-SQLAlchemy provides a simple integration between FastAPI and SQLAlchemy in your application. It gives access to useful help

Michael Freeborn 465 Jan 07, 2023
fastapi-mqtt is extension for MQTT protocol

fastapi-mqtt MQTT is a lightweight publish/subscribe messaging protocol designed for M2M (machine to machine) telemetry in low bandwidth environments.

Sabuhi 144 Dec 28, 2022
Starlette middleware for Prerender

Prerender Python Starlette Starlette middleware for Prerender Documentation: https://BeeMyDesk.github.io/prerender-python-starlette/ Source Code: http

BeeMyDesk 14 May 02, 2021
Redis-based rate-limiting for FastAPI

Redis-based rate-limiting for FastAPI

Glib 6 Nov 14, 2022
Beyonic API Python official client library simplified examples using Flask, Django and Fast API.

Beyonic API Python Examples. The beyonic APIs Doc Reference: https://apidocs.beyonic.com/ To start using the Beyonic API Python API, you need to start

Harun Mbaabu Mwenda 46 Sep 01, 2022
Minecraft biome tile server writing on Python using FastAPI

Blocktile Minecraft biome tile server writing on Python using FastAPI Usage https://blocktile.herokuapp.com/overworld/{seed}/{zoom}/{col}/{row}.png s

Vladimir 2 Aug 31, 2022
fastapi-admin2 is an upgraded fastapi-admin, that supports ORM dialects, true Dependency Injection and extendability

FastAPI2 Admin Introduction fastapi-admin2 is an upgraded fastapi-admin, that supports ORM dialects, true Dependency Injection and extendability. Now

Glib 14 Dec 05, 2022
Opinionated authorization package for FastAPI

FastAPI Authorization Installation pip install fastapi-authorization Usage Currently, there are two models available: RBAC: Role-based Access Control

Marcelo Trylesinski 18 Jul 04, 2022
A minimum reproducible repository for embedding panel in FastAPI

FastAPI-Panel A minimum reproducible repository for embedding panel in FastAPI Follow either This Tutorial or These steps below ↓↓↓ Clone the reposito

Tyler Houssian 15 Sep 22, 2022
Pagination support for flask

flask-paginate Pagination support for flask framework (study from will_paginate). It supports several css frameworks. It requires Python2.6+ as string

Lix Xu 264 Nov 07, 2022
Local Telegram Bot With FastAPI & Ngrok

An easy local telegram bot server with python, fastapi and ngrok.

Ömer Faruk Özdemir 7 Dec 25, 2022
FastAPI Socket.io with first-class documentation using AsyncAPI

fastapi-sio Socket.io FastAPI integration library with first-class documentation using AsyncAPI The usage of the library is very familiar to the exper

Marián Hlaváč 9 Jan 02, 2023
FastAPI with Docker and Traefik

Dockerizing FastAPI with Postgres, Uvicorn, and Traefik Want to learn how to build this? Check out the post. Want to use this project? Development Bui

51 Jan 06, 2023
京东图片点击验证码识别

京东图片验证码识别 本项目是@yqchilde 大佬的 JDMemberCloseAccount 识别图形验证码(#45)思路验证,若你也有思路可以提交Issue和PR也可以在 @yqchilde 的 TG群 找到我 声明 本脚本只是为了学习研究使用 本脚本除了采集处理验证码图片没有其他任何功能,也

AntonVanke 37 Dec 22, 2022