Microservice example with Python, Faust-Streaming and Kafka (Redpanda)

Overview

Microservices Orchestration with Python, Faust-Streaming and Kafka (Redpanda)

Example project for PythonBenin meetup. It demonstrates how to use Faust-Streaming to build a microservice using Python and Kafka/Redpanda.

Installation

This project requires poetry. Once poetry is installed, launch the following commands:

git clone [email protected]:lerela/orchestration-python-kafka.git
cd orchestration-python-kafka
poetry install

We will use Redpanda as a replacement for Kafka, but Kafka can also be used. Start a development instance of Redpanda with (do not use this in production!):

docker run -d --pull=always --name=redpanda-1 --rm \
  -p 9092:9092 \
  -p 9644:9644 \
  docker.vectorized.io/vectorized/redpanda:latest \
  redpanda start \
  --overprovisioned \
  --smp 1  \
  --memory 1G \
  --reserve-memory 0M \
  --node-id 0 \
  --check=false

Usage

Start a worker

poetry run faust --datadir=worker-1/ -A orchestration worker --without-web -l info

The parameter --datadir is useful to start multiple workers on the same host (the folder name must be unique for each worker). --without-web disables the Faust web interface since we do not need it.

Send a message on the topic

Using Faust-Streaming: poetry run faust -A orchestration send @topic '{"origin": "Haie Vive", "destination": "Adogleta", "duration": "940"}'

Directly using Redpanda (would work with any Kafka client):

  1. Open a producer with: docker exec -it redpanda-1 rpk topic produce zem-ride --brokers=localhost:9092
  2. Directly send the messages as JSON (press enter to send a message): {"origin": "Calavi", "destination": "Troisième Pont", "duration": 2560}

Troubleshooting

PartitionsMismatch

If you see the following error:

faust.exceptions.PartitionsMismatch: The source topic 'zem-ride' for table 'durations'
has 1 partitions, but the changelog
topic 'zem-app-durations-changelog' has 8 partitions.

Please make sure the topics have the same number of partitions
by configuring Kafka correctly.

Then Faust has wrongly created 8 partitions for the durations changelog (the topic that synchronizes the table "durations"). Just delete the topic while the worker is running with docker exec -it redpanda-1 rpk topic delete zem-app-durations-changelog, and it will be properly re-created. Alternatively, you can delete it when the worker is stopped and manually re-create the topic with rpk topic create zem-app-durations-changelog -p 1 -r 1. Of course, increase the partition number if you wish to use multiple workers.

License

This micro-project is an example of a talk I gave at PythonBenin and is licensed under the MIT License.

Klein - A micro-framework for developing production-ready web services with Python

Klein, a Web Micro-Framework Klein is a micro-framework for developing production-ready web services with Python. It is 'micro' in that it has an incr

Twisted Matrix Labs 814 Jan 08, 2023
NO LONGER MAINTAINED - A Flask extension for creating simple ReSTful JSON APIs from SQLAlchemy models.

NO LONGER MAINTAINED This repository is no longer maintained due to lack of time. You might check out the fork https://github.com/mrevutskyi/flask-res

1k Jan 04, 2023
Pyramid - A Python web framework

Pyramid Pyramid is a small, fast, down-to-earth, open source Python web framework. It makes real-world web application development and deployment more

Pylons Project 3.7k Dec 30, 2022
Swagger/OpenAPI First framework for Python on top of Flask with automatic endpoint validation & OAuth2 support

Connexion Connexion is a framework that automagically handles HTTP requests based on OpenAPI Specification (formerly known as Swagger Spec) of your AP

Zalando SE 4.2k Jan 07, 2023
WAZO REST API for the call management of the C4 infrastructure

wazo-router-calld wazo-router-calld provides REST API for the C4 infrastructure. Installing wazo-router-calld The server is already provided as a part

Wazo Platform 4 Dec 21, 2022
Pyrin is an application framework built on top of Flask micro-framework to make life easier for developers who want to develop an enterprise application using Flask

Pyrin A rich, fast, performant and easy to use application framework to build apps using Flask on top of it. Pyrin is an application framework built o

Mohamad Nobakht 10 Jan 25, 2022
An alternative serializer implementation for REST framework written in cython built for speed.

drf-turbo An alternative serializer implementation for REST framework written in cython built for speed. Free software: MIT license Documentation: htt

Mng 74 Dec 30, 2022
A simple Tornado based framework designed to accelerate web service development

Toto Toto is a small framework intended to accelerate web service development. It is built on top of Tornado and can currently use MySQL, MongoDB, Pos

Jeremy Olmsted-Thompson 61 Apr 06, 2022
An abstract and extensible framework in python for building client SDKs and CLI tools for a RESTful API.

django-rest-client An abstract and extensible framework in python for building client SDKs and CLI tools for a RESTful API. Suitable for APIs made wit

Certego 4 Aug 25, 2022
A python application to log QSOs directly to QRZ.com from the command line

qrzlogger This script is a QRZ.com command line QSO logger. It does the following: asks the user for a call sign displays available call sign info pul

Michael Clemens 15 Jul 16, 2021
Daniel Vaz Gaspar 4k Jan 08, 2023
A public API written in Python using the Flask web framework to determine the direction of a road sign using AI

python-public-API This repository is a public API for solving the problem of the final of the AIIJC competition. The task is to create an AI for the c

Lev 1 Nov 08, 2021
Official mirror of https://gitlab.com/pgjones/quart

Quart Quart is an async Python web microframework. Using Quart you can, render and serve HTML templates, write (RESTful) JSON APIs, serve WebSockets,

Phil Jones 2 Oct 05, 2022
An easy-to-use high-performance asynchronous web framework.

An easy-to-use high-performance asynchronous web framework.

Aber 264 Dec 31, 2022
Serverless Python

Zappa - Serverless Python About Installation and Configuration Running the Initial Setup / Settings Basic Usage Initial Deployments Updates Rollback S

Rich Jones 11.9k Jan 01, 2023
Light, Flexible and Extensible ASGI API framework

Starlite Starlite is a light, opinionated and flexible ASGI API framework built on top of pydantic and Starlette. Check out the Starlite documentation

Na'aman Hirschfeld 1.6k Jan 09, 2023
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

Sanic Community Organization 16.7k Dec 28, 2022
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

Sanic Community Organization 16.7k Jan 08, 2023
A comprehensive reference for all topics related to building and maintaining microservices

This pandect (πανδέκτης is Ancient Greek for encyclopedia) was created to help you find and understand almost anything related to Microservices that i

Ivan Bilan 64 Dec 09, 2022
Free and open source full-stack enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.

Readme web2py is a free open source full-stack framework for rapid development of fast, scalable, secure and portable database-driven web-based applic

2k Dec 31, 2022