A django integration for huey task queue that supports multi queue management

Related tags

Djangodjango-huey
Overview

Version

django-huey

This package is an extension of huey contrib djhuey package that allows users to manage multiple queues.

Installation

Using pip package manager run:

# pip install Django  if not installed
# pip install huey    if not installed
pip install django-huey

Note: use a virtualenv to isolate your dependencies. Note 2: django and huey must be installed.

Then, in your settings.py file add django_huey to the INSTALLED_APPS:

INSTALLED_APPS = [
	...
    'django_huey',
]

Configuration

In settings.py you must add the HUEYS setting:

HUEYS = {
    'first': {#this name will be used in decorators below
        'huey_class': 'huey.RedisHuey',  
        'name': 'first_tasks',  
        'consumer': {
            'workers': 2,
            'worker_type': 'thread',
        },
    },
    'emails': {#this name will be used in decorators below
        'huey_class': 'huey.RedisHuey',  
        'name': 'emails_tasks',  
        'consumer': {
            'workers': 5,
            'worker_type': 'thread',
        },
    }
}

Note: This setting is incompatible with HUEY setting.

Usage

Now you will be able to run multiple queues using:

python manage.py run_djangohuey --queue first
python manage.py run_djangohuey --queue emails

Each queue must be run in a different terminal.

Configuring tasks

You can use usual huey decorators to register tasks, but they must be imported from django_huey as shown below:

from django_huey import db_task, task

@db_task(queue='first')
	# perform some db task

@task(queue='emails')
	# send some emails

All the args and kwargs defined in huey decorators should work in the same way, if not, let us know.

You might also like...
Bootstrap 3 integration with Django.

django-bootstrap3 Bootstrap 3 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 3. Want to use Bootstr

Bootstrap 4 integration with Django.

django-bootstrap 4 Bootstrap 4 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 4. Requirements Pytho

Plug and play continuous integration with django and jenkins

django-jenkins Plug and play continuous integration with Django and Jenkins Installation From PyPI: $ pip install django-jenkins Or by downloading th

Django admin CKEditor integration.

Django CKEditor NOTICE: django-ckeditor 5 has backward incompatible code moves against 4.5.1. File upload support has been moved to ckeditor_uploader.

TinyMCE integration for Django

django-tinymce django-tinymce is a Django application that contains a widget to render a form field as a TinyMCE editor. Quickstart Install django-tin

Bootstrap 3 integration with Django.

django-bootstrap3 Bootstrap 3 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 3. Want to use Bootstr

Django + Next.js integration

Django Next.js Django + Next.js integration From a comment on StackOverflow: Run 2 ports on the same server. One for django (public facing) and one fo

This is a repository for collecting global custom management extensions for the Django Framework.

Django Extensions Django Extensions is a collection of custom extensions for the Django Framework. Getting Started The easiest way to figure out what

File and Image Management Application for django
File and Image Management Application for django

Django Filer django Filer is a file management application for django that makes handling of files and images a breeze. Contributing This is a an open

Comments
  • HueyException <task> not found in TaskRegistry

    HueyException not found in TaskRegistry

    When using django-huey, the consumer is able to correctly find the registered tasks, however when running the Django server and trying to enqueue a task, I get an "HueyException not found in TaskRegistry" error. From doing wayy too much debugging, I finally tracked this down to the fact that the huey package initializes itself with RegistryA, and then django_huey initializes itself with a separate RegistryB. So when trying to enqueue a task, it attempts to find the task in RegistryA which is empty, because all of the tasks are in RegistryB.

    I use import django_huey as huey everywhere, and then use @huey.task(queue="test"), but I'm still getting this error. I'm not sure what I'm doing wrong 😞

    opened by wgordon17 7
  • django-huey-monitor not working with djangohuey

    django-huey-monitor not working with djangohuey

    I use djangohuey --queue [name of queue] to run consumer, this runs my task, but the django-huey-monitor does not track the tasks, or are not created in the huey_monitor database tables. Which configuration am I missing?

    opened by EvelynArinaitwe 2
  • Unable to use `django-huey` with `SqliteHuey`

    Unable to use `django-huey` with `SqliteHuey`

    Settings:

    DJANGO_HUEY = {
        'default': 'xxx',
        'queues': {
            'xxx': {
                'huey_class': 'huey.SqliteHuey',
                'name': 'xxx_tasks',
                'consumer': {
                    'workers': 4,
                    'worker_type': 'thread',
                },
            },
            'yyy': { 
                'huey_class': 'huey.SqliteHuey',
                'name': 'yyy_tasks',
                'consumer': {
                    'workers': 2,
                    'worker_type': 'thread',
                },
            }
        }
    }
    

    Exception raised:

    huey.exceptions.ConfigurationError: "redis" python module not found, cannot use Redis storage backend. Run "pip install redis" to install.
    

    redis should be optional.

    opened by fabiocaccamo 2
  • Cannot configure djangohuey with different workers.

    Cannot configure djangohuey with different workers.

    Is there a way to configure the djangohuey settings, so that a specific worker is assigned to a specific queue? At the moment, I implement different queues with different number of workers in each consumer, but anytime a worker is free, it is taken on by any of the queue tasks that are waiting. what I want is for example: worker1 listen to only queue_a and worker2&worker3 listen to queue_b, regardless of which worker is free. Is this possible? Here are my settings:

    DJANGO_HUEY = {
        'default': 'first', #this name must match with any of the queues defined below.
        'queues': {
            'queue_a': {
                'huey_class': 'huey.RedisHuey',
                'name': 'tasks_a',
                "immediate": False,
                'results': True,
                'blocking': True,
                'connection': {
                        'host': 'localhost',
                        'port': 6379,
                        'db': 0,
                        'connection_pool': None,
                        'read_timeout': 1,
                        'url': None,  
                    },
                'consumer': {
                    'workers': 1,
                    'worker_type': 'thread',
                },
            },
            'queue_b': {
                'huey_class': 'huey.RedisHuey',
                'name': 'tasks_b',
                "immediate": False,
                'blocking': True,
                
                'consumer': {
                    'workers': 2,
                    'worker_type': 'thread',
                },
            },
        }
    }
    
    opened by EvelynArinaitwe 1
Releases(v1.1.1)
  • v1.1.1(Feb 7, 2022)

  • v1.1.0(Jan 19, 2022)

  • v1.0.1(Jan 14, 2022)

    1.0.1 - 2022-01-14

    Added

    • Close db connections before task body. https://github.com/coleifer/huey/commit/e77acf307bfdade914ab7f91c65dbbc183af5d8f
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(May 19, 2021)

    1.0.0 - 2021-05-19

    Note: This release contains breaking changes, see them below with the migration instructions.

    Added

    • Allow definition of a default queue.

    Changed

    • HUEYS django setting renamed to DJANGO_HUEY.
    • Change command run_djangohuey to djangohuey.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Apr 18, 2021)

    0.2.0 - 2021-04-18

    Added

    Nothing added this release

    Changed

    Nothing changed this release

    Fixed

    • When a huey name was not provided, default django db name was used. Now it's defaulted to queue name.

    Removed

    • Removed incompatibility with HUEY setting used by huey project.
    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Apr 4, 2021)

Owner
GAIA Software
Cooperativa de Desarrollo de Software | Quilmes, Buenos Aires, Argentina
GAIA Software
Bootstrap 4 integration with Django.

django-bootstrap 4 Bootstrap 4 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 4. Requirements Pytho

Zostera B.V. 980 Dec 29, 2022
PEP-484 type hints bindings for the Django web framework

mypy-django Type stubs to use the mypy static type-checker with your Django projects This project includes the PEP-484 compatible "type stubs" for Dja

Machinalis 223 Jun 17, 2022
Awesome Django Blog App

Awesome-Django-Blog-App Made with love django as the backend and Bootstrap as the frontend ! i hope that can help !! Project Title Django provides mul

ANAS NABIL 2 Feb 08, 2022
A Django backed for PostgreSQL using Psycopg 3

A Django backend for PostgreSQL using Psycopg 2 The backend passes the entire Django test suite, but it needs a few modifications to Django and to i

Daniele Varrazzo 42 Dec 16, 2022
A pickled object field for Django

django-picklefield About django-picklefield provides an implementation of a pickled object field. Such fields can contain any picklable objects. The i

Gintautas Miliauskas 167 Oct 18, 2022
A simple app that provides django integration for RQ (Redis Queue)

Django-RQ Django integration with RQ, a Redis based Python queuing library. Django-RQ is a simple app that allows you to configure your queues in djan

RQ 1.6k Jan 06, 2023
Fast / fuzzy PostgreSQL counts for Django

Created by Stephen McDonald Introduction Up until PostgreSQL 9.2, COUNT queries generally required scanning every row in a database table. With millio

stephenmcd 85 Oct 25, 2021
Declarative model lifecycle hooks, an alternative to Signals.

Django Lifecycle Hooks This project provides a @hook decorator as well as a base model and mixin to add lifecycle hooks to your Django models. Django'

Robert Singer 1k Dec 31, 2022
A BitField extension for Django Models

django-bitfield Provides a BitField like class (using a BigIntegerField) for your Django models. (If you're upgrading from a version before 1.2 the AP

DISQUS 361 Dec 22, 2022
Repo for All the Assignments I have to submit for Internship Application !😅

Challenges Repository for All the Assignments I have to submit for Internship Application ! 😅 As You know, When ever We apply for an Internship, They

keshav Sharma 1 Sep 08, 2022
A Minimalistic Modern Django Boilerplate

A Minimalistic Modern Django Boilerplate This boilerplate is mainly for educational purposes. It is meant to be cloned as a starter code for future tu

Jonathan Adly 21 Nov 02, 2022
Django-static-site - A simple content site framework that harnesses the power of Django without the hassle

coltrane A simple content site framework that harnesses the power of Django with

Adam Hill 57 Dec 06, 2022
Dynamic Django settings.

Constance - Dynamic Django settings A Django app for storing dynamic settings in pluggable backends (Redis and Django model backend built in) with an

Jazzband 1.5k Jan 07, 2023
Dashboad Full Stack utilizando o Django.

Dashboard FullStack completa Projeto finalizado | Informações Cadastro de cliente Menu interatico mostrando quantidade de pessoas bloqueadas, liberada

Lucas Silva 1 Dec 15, 2021
I managed to attach the Django Framework to my Telegram Bot and set a webhook

I managed to attach the Django Framework to my Telegram Bot and set a webhook. I've been developing it from 10th of November 2021 and I want to have a basic working prototype.

Valentyn Vovchak 2 Sep 08, 2022
Pipeline is an asset packaging library for Django.

Pipeline Pipeline is an asset packaging library for Django, providing both CSS and JavaScript concatenation and compression, built-in JavaScript templ

Jazzband 1.4k Jan 03, 2023
Automatic caching and invalidation for Django models through the ORM.

Cache Machine Cache Machine provides automatic caching and invalidation for Django models through the ORM. For full docs, see https://cache-machine.re

846 Nov 26, 2022
Streamlining Django forms to provide all the wins of single-page-applications without the pain.

nango Streamlining Django forms to provide all the wins of single-page-applications without the pain. Key features Available to all Django deployments

Nick Farrell 107 Dec 12, 2022
Django-gmailapi-json-backend - Email backend for Django which sends email via the Gmail API through a JSON credential

django-gmailapi-json-backend Email backend for Django which sends email via the

Innove 1 Sep 09, 2022
Reusable workflow library for Django

django-viewflow Viewflow is a lightweight reusable workflow library that helps to organize people collaboration business logic in django applications.

Viewflow 2.3k Jan 08, 2023