Push Prometheus metrics to VictoriaMetrics or other exporters

Overview

test-all codecov

prometheus-push-client

Push metrics from your periodic long-running jobs to existing Prometheus/VictoriaMetrics monitoring system.

Currently supports pushes directly to VictoriaMetrics:

For pure Prometheus setups, several options are supported:

  • to pushgateway or prom-aggregation-gateway in OpenMetrics format via HTTP. Please read corresponding docs about appropriate use cases and limitations;
  • to StatsD or statsd-exporter in StatsD format via UDP. Prometheus and StatsD metric types are not fully compatible, so currenly all metrics become StatsD gauges, but rate, increase, histogram_quantile and other PromQL functions produce same results as if types never changed.

Install it via pip:

pip install prometheus-push-client

Metrics

This library uses prometheus-client metric implementation, but adds some minor tweaks.

Separate registry

New metric constructors use separate PUSH_REGISTRY as a default, not to interfere with other metrics already defined and monitored in existing projects.

Default labelvalues

With regular prometheus_client, defaults may be defined for either none or all the labels (with labelvalues), but that's not enough. Moreover labelvalues sometimes doesn't work as expected.

We probably want to define some defaults, like hostname, or more importantly, if we use VictoriaMetrics cluster, VictoriaMetrics_AccountID= (try 0 as a default) label must always be set, and metrics without it will be ignored.

Following example shows how to use defaults, and how to override them if necessary.

import prometheus_push_client as ppc


counter1 = ppc.Counter(
    name="c1",
    labelnames=["VictoriaMetrics_AccountID", "host", "event_type"],
    default_labelvalues={
        "VictoriaMetrics_AccountID": 0,
        "host": socket.gethostname(),
    }
)


# regular usage
counter1.labels(event_type="login").inc()

# overriding defaults
counter1.labels(host="non-default", event_type="login").inc()
# same effect as above: defaults are applied in `labelvalues`
# order for "missing" labels in the beginning
counter1.labels("non-default", "login").inc()

Metrics with no labels are initialized at creation time. This can have unpleasant side-effect: if we initialize lots of metrics not used in currently running job, batch clients will have to push their non-changing values in every synchronization session.

To avoid that we'll have to properly isolate each task's metrics, which can be impossible or rather tricky, or we can create metrics with default, non-changing labels (like hostname). Such metrics will be initialized on first use (inc), and we'll be pushing only those we actually utilized.

Clients

Batch clients

Batch clients spawn synchronization jobs "in background" (meaning in a thread or asyncio task) to periodically send all metrics from ppc.PUSH_REGISTRY to the destination.

Clients will attempt to stop gracefully, synchronizing registry "one last time" after job exits or crashes. Sometimes this may mess up sampling, but the worst case I could artifically create looks like this:

graceful push effect

Best way to use them is via decorators / context managers. These clients are intended to be used with long running, but finite tasks, which could be spawned anywhere, therefor not easily accessible by the scraper. If that's not the case -- just use "passive mode" w/ the scraper instead.

def influx_udp_async(host, port, period=15.0):
def influx_udp_thread(host, port, period=15.0):
def statsd_udp_async(host, port, period=15.0):
def statsd_udp_thread(host, port, period=15.0):
def influx_http_async(url, verb="POST", period=15.0):
def influx_http_thread(url, verb="POST", period=15.0):
def openmetrics_http_async(url, verb="POST", period=15.0):
def openmetrics_http_thread(url, verb="POST", period=15.0):

Usage example:

import prometheus_push_client as ppc


req_hist = ppc.Histogram(
    name="external_requests",
    namespace="acme"
    subsystem="job123",
    unit="seconds",
    labelnames=["service"]
)


@ppc.influx_udp_async("victoria.acme.inc.net", 9876, period=15)
async def main(urls):
    # the job ...
    req_hist.labels(gethostname(url)).observe(response.elapsed)

# OR

async def main(urls):
    async with ppc.influx_udp_async("victoria.acme.inc.net", 9876, period=15):
        # the job ...
        req_hist.labels(gethostname(url)).observe(response.elapsed)

Please read about mandatory job tag within url while using pushgateway.

Streaming clients

If for some reason every metric change needs to be synced, UDP streaming clients are implemented in this library.

def influx_udp_aiostream(host, port):
def influx_udp_stream(host, port):
def statsd_udp_aiostream(host, port):
def statsd_udp_stream(host, port):

Usage is completely identical to batch clients' decorators / context managers.

⚠️ Histogram and Summary .time() decorator doesn't work in this mode atm, because it can't be monkey-patched easily.

Transports

Main goal is not to interrupt measured jobs with errors from monitoring code. Therefor all transports will attempt to catch all network errors, logging error info and corresponding tracebacks to stdout.

You might also like...
Google Foobar challenge solutions from my experience and other's on the web.
Google Foobar challenge solutions from my experience and other's on the web.

Google Foobar challenge Google Foobar challenge solutions from my experience and other's on the web. Note: Problems indicated with "Mine" are tested a

Custom component to calculate estimated power consumption of lights and other appliances
Custom component to calculate estimated power consumption of lights and other appliances

Custom component to calculate estimated power consumption of lights and other appliances. Provides easy configuration to get virtual power consumption sensors in Home Assistant for all your devices which don't have a build in power meter.

The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss.
The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss.

The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss. Greater accuracy is achieved thanks to the line-by-line comparison of pages, comparison of response code and reflections.

Emulate and Dissect MSF and *other* attacks
Emulate and Dissect MSF and *other* attacks

Need help in analyzing Windows shellcode or attack coming from Metasploit Framework or Cobalt Strike (or may be also other malicious or obfuscated code)? Do you need to automate tasks with simple scripting? Do you want help to decrypt MSF generated traffic by extracting keys from payloads?

External Network Pentest Automation using Shodan API and other tools.

Chopin External Network Pentest Automation using Shodan API and other tools. Workflow Input a file containing CIDR ranges. Converts CIDR ranges to ind

to learn how to do pull request and do contribution to other's repo
to learn how to do pull request and do contribution to other's repo

Hacktoberfest-2021 - open-source-contribution An Open Source repository to Teach people How to contribute to open sources. 💥 🔥 JOIN PVX PROGRAMMING

python scripts and other files to generate induction encoder PCBs in Kicad
python scripts and other files to generate induction encoder PCBs in Kicad

induction_encoder python scripts and other files to generate induction encoder PCBs in Kicad Targeting the Renesas IPS2200 encoder chips.

scap is a tool for putting code in places and for other purposes

Scap is the deployment script used by Wikimedia Foundation to publish code and configuration on production web servers.

Python script for changing the SSH banner content with other content

Banner-changer-py Python script for changing the SSH banner content with other content. The Script will take the content of a specified file range and

Comments
  • Can I use this to push historical data?

    Can I use this to push historical data?

    I came here from https://stackoverflow.com/a/67562080/7424510.

    Suppose, I've got a temperature sensor, and I've got a list of UNIX time stamps and temperatures, like so:

    [
        (1661863713, 22),
        (1661863714, 21),
        (1661863715, 22)
    ]
    

    Can I push this with this client? How?

    opened by joernschellhaas 1
  • http fixes, pushgateway client

    http fixes, pushgateway client

    • http: ensure data ends with "\n\n"
    • openmetrics: data type metric header
    • added openmetrics http clients for pushgateway / prom-aggregation-gateway
    opened by gistart 1
Releases(0.0.8)
Owner
olegm
olegm
JurjenLang, an interpreted programming language

JurjenLang An interpreted programming language Getting started Follow these three steps on your computer to get started git clone https://github.com/J

JVerbruggen 5 May 03, 2022
Multi View Stereo on Internet Images

Evaluating MVS in a CPC Scenario This repository contains the set of artficats used for the ENGN8601/8602 research project. The thesis emphasizes on t

Namas Bhandari 1 Nov 10, 2021
Schemdule is a tiny tool using script as schema to schedule one day and remind you to do something during a day.

Schemdule is a tiny tool using script as schema to schedule one day and remind you to do something during a day. Platform Python Install Use pip: pip

StardustDL 4 Sep 13, 2021
This library is an ongoing effort towards bringing the data exchanging ability between Java/Scala and Python

PyJava This library is an ongoing effort towards bringing the data exchanging ability between Java/Scala and Python

Byzer 6 Oct 17, 2022
Generate a wordlist to fuzz amounts or any other numerical values.

Generate a wordlist to fuzz amounts or any other numerical values. Based on Common Security Issues in Financially-Oriented Web Applications.

Ivan Šincek 3 Oct 14, 2022
The Begin button and menu for the Meadows operating system. The start button for UNIX/Linux.

By: Seanpm2001, Meadows Et; Al. Top README.md Read this article in a different language Sorted by: A-Z Sorting options unavailable ( af Afrikaans Afri

Sean P. Myrick V19.1.7.2 4 Aug 28, 2022
Sublime Text 2/3 style auto completion for ST4

Hippie Autocompletion Sublime Text 2/3 style auto completion for ST4: cycle through words, do not show popup. Simply hit Tab to insert completion, hit

Alexander Schepanovski 20 May 19, 2022
Make your functions return something meaningful, typed, and safe!

Make your functions return something meaningful, typed, and safe! Features Brings functional programming to Python land Provides a bunch of primitives

dry-python 2.5k Jan 03, 2023
The Doodle Master seeks to turn your UI mockups into real code.

Doodle Master The Doodle Master seeks to turn your UI mockups into real code. Currently this repository just serves to demonstrate a Proof Of Concept

Karanbir Chahal 2.4k Dec 09, 2022
A simple and efficient computing package for Genshin Impact gacha analysis

GGanalysisLite计算包 这个版本的计算包追求计算速度,而GGanalysis包有着更多计算功能。 GGanalysisLite包通过卷积计算分布列,通过FFT和快速幂加速卷积计算。 测试玩家得到的排名值rank的数学意义是:与抽了同样数量五星的其他玩家相比,测试玩家花费的抽数大于等于比例

一棵平衡树 34 Nov 26, 2022
IDA Pro plugin that shows the comments in a database

ShowComments A Simple IDA Pro plugin that shows the comments in a database Installation Copy the file showcomments.py to the plugins folder under IDA

Fernando Mercês 32 Dec 10, 2022
A Python module for decorators, wrappers and monkey patching.

wrapt The aim of the wrapt module is to provide a transparent object proxy for Python, which can be used as the basis for the construction of function

Graham Dumpleton 1.8k Jan 06, 2023
A student information management system in Python

Student-information-management-system 本项目是一个学生信息管理系统,这个项目是用Python语言实现的,也实现了图形化界面的显示,同时也实现了管理员端,学生端两个登陆入口,同时底层使用的是Redis做的数据持久化。 This project is a stude

liuyunfei 7 Nov 15, 2022
My Dotfiles of Arco Linux

Arco-DotFiles My Dotfiles of Arco Linux Apps Used Htop LightDM lightdm-webkit2-greeter Alacritty Qtile Cava Spotify nitrogen neofetch Spicetify Thunar

$BlueDev5 6 Dec 11, 2022
Contains a Jupyter Notebook for calculating remaining plants required based on field/lathhouse data.

Davis-Sunflowers-Su21 Project goals: Plants influence their reproduction and mating system in many ways. Various factors such as time of flowering, ab

1 Feb 10, 2022
The most widely used Python to C compiler

Welcome to Cython! Cython is a language that makes writing C extensions for Python as easy as Python itself. Cython is based on Pyrex, but supports mo

7.6k Jan 03, 2023
Monitoring of lake dynamics

slamcore_utils Description This repo contains the slamcore-setup-dataset script. It can be used for installing a sample dataset for offline testing an

10 Jun 23, 2022
mypy plugin for PynamoDB

pynamodb-mypy A plugin for mypy which gives it deeper understanding of PynamoDB (beyond what's possible through type stubs). Usage Add it to the plugi

1 Oct 21, 2022
The RAP community of practice includes all analysts and data scientists who are interested in adopting the working practices included in reproducible analytical pipelines (RAP) at NHS Digital.

The RAP community of practice includes all analysts and data scientists who are interested in adopting the working practices included in reproducible analytical pipelines (RAP) at NHS Digital.

NHS Digital 50 Dec 22, 2022
a bit of my project :) and I use some of them for my school lesson or study for an exam! but some of them just for myself.

Handy Project a bit of my project :) and I use some of them for my school lesson or study for an exam! but some of them just for myself. the handy pro

amirkasra esmaeilian 13 Jul 05, 2021