Python Service for MISP Feed Management

Overview

Python Service for MISP Feed Management

This set of scripts is designed to offer better reliability and more control over the fetching of feeds into MISP. For the moment, the schedule is broken up into multiple components, at the top of each plugin and in config.py:

  • MISP_TIMES: An array of times (24hr format) when enabled MISP feeds will be fetched and cached.
  • TEXT_TIMES: An array of times (24hr format) when enabled plaintext and CSV feeds will be fetched and cached.
  • HOURLY_FEEDS An array of the ID's of enabled feeds that you wish to run at the beginning of every hour.
  • FULL_EXPORT_TIME The time (24hr format) that you want to run a full text export of attributes.

In addition to this are "ENABLE" options for all external services. By default, Abuse.ch is configured to run every hour.

Am still working out the best way of going about granular scheduling.

Variable Notes:

  • MISP_ADMIN_KEY: MISP feeds must be fetched by a Site Admin user.
  • MISP_USER_KEY: This can be the key of an Org Admin, Sync User or your own custom role. They must be able to both manage and publish events, and hold the Tag Editor permission.

Installation:

  • Recommended: Ensure that the fetch_feeds and cache_feeds Scheduled Tasks are not enabled. Also, disable the default Abuse.ch feeds as this project includes a module that loads the data with more context and into a separate event each day.
  • SCP this folder to your MISP server.
  • Alter the paths in misp-feeds.service and start_worker.sh to point to where you've dropped the folder.
  • Correct the user in misp-feeds.service if it is not ubuntu.
  • Complete the variables at the top of the feed_manager.py, misp_export.py, otx_misp.py, twitter_misp.py and xforce_misp.py scripts.
  • Run the following (in the misp-feeds folder):
chmod +x start_worker.sh
apt install nodejs
pip3 install -r requirements.txt
sudo mv misp-feeds.service /etc/systemd/system
sudo chown root:root /etc/systemd/system/misp-feeds.service
sudo systemctl daemon-reload
sudo systemctl start misp-feeds.service
  • nodejs is required for cfscrape (used by the Twitter module to get Ghostbin pastes).
  • Check misp_feeds.log for errors. You can also run both of the Python scripts from the command line for standalone, ad-hoc operation.

Module Notes:

Export:

  • This is a rough script that I use for exporting a plaintext list of attributes for ingestion into external facilities. They're output to a subfolder of the MISP webroot, so ensure the script user has permission to write here and there's adequate access control in place.
  • A full export is run once a day for the number of days defined by EXPORT_DAYS. Incremental updates are made daily.
  • The sample values for EXPORT_TAGS and EXPORT_TYPES should give you an idea of how to configure this. 'domain' and 'hostname' can be output separately or together. Use EXPORT_MERGE_HOSTNAME to configure this.

Plugins:

At the top of each plugin are three variables which determine its operation:

  • PLUGIN_NAME: The friendly name of the Plugin. Only used for logging and ad-hoc operation.

  • PLUGIN_ENABLED: Boolean setting to enable/disable the plugin.

  • PLUGIN_TIMES: The times throughout the day to run the plugin. Also accepts 'hourly', which will run it on the hour every hour.

Default plugins are as follows:

  • Abuse.ch: Pulls URLhaus, Feodo Tracker, MalwareBazaar and ThreatFox into a single event per day. Attributes are tagged according to the feed tags and/or classification.
  • CleanMX: Virus and Phishing feeds are pulled into a single event per day. No tagging yet.
  • OTX: Individual pulses form a separate events in MISP. OTX tags can be spammy so are ignored, but Adversary, Malware and ATT&CK techniques are used. Galaxy tags are attempted, and if no appropriate tag can be found, the feed supplied tag is used.
  • RiskIQ: Individual articles form a separate events in MISP. The same method of tagging is employed as OTX.
  • Twitter: Pulls IOC's found on Twitter into a single event per day. GitHub, PasteBin and GhostBin links are followed and also scraped. Attributes are tagged with the hashtags included in the Tweet and the same method as OTX.
  • X-Force: Individual articles form a separate events in MISP. X-Force articles are not tagged, so the Title of the article is parsed to identify Galaxy tags that match Title keywords.
Owner
Chris
Security Architect / Malware Wrangler
Chris
Workshop OOP - Workshop OOP - Discover object-oriented programming

About: This is an open-source bot, the code is open for anyone to see, fork and

Francis Clairicia-Rose-Claire-Joséphine 5 May 02, 2022
Sigma coding youtube - This is a collection of all the code that can be found on my YouTube channel Sigma Coding.

Sigma Coding Tutorials & Resources YouTube • Facebook Support Sigma Coding Patreon • GitHub Sponsor • Shop Amazon Table of Contents Overview Topics Re

Alex Reed 927 Jan 08, 2023
Moleey Panel with python 3

Painel-Moleey pkg upgrade && pkg update pkg install python3 pip install pyfiglet pip install colored pip install requests pip install phonenumbers pkg

Moleey. 1 Oct 17, 2021
Explores the python bytecode, provides some tools to access it for fun and profit.

Pyasmtools - looking at the python bytecode for fun and profit. The pyasmtools library is made up of two parts A python bytecode disassembler . See Py

Michael Moser 299 Jan 04, 2023
Prophet is a tool to discover resources detailed for cloud migration, cloud backup and disaster recovery

Prophet is a tool to discover resources detailed for cloud migration, cloud backup and disaster recovery

22 May 31, 2022
Pseudometa's dotfiles

pseudometa's dotfiles Table of Contents Why this repository? How this Repository works Special Explanations Got an idea for an improvement? Contact Wh

pseudometa 23 Dec 27, 2022
A Python library for inspecting JVM class files (.class)

lawu Lawu is a human-friendly library for assembling, disassembling, and exploring JVM class files. It's highly suitable for automation tasks. Documen

Tyler Kennedy 45 Oct 23, 2022
Astroquery is an astropy affiliated package that contains a collection of tools to access online Astronomical data.

Astroquery is an astropy affiliated package that contains a collection of tools to access online Astronomical data.

The Astropy Project 631 Jan 05, 2023
Simple web application, which has a single endpoint, dedicated to annotation parsing and convertion.

Simple web application, which has a single endpoint, dedicated to annotation parsing and conversion.

Pavel Paranin 1 Nov 01, 2021
A simple python script where the user inputs the current ingredients they have in their kitchen into ingredients.txt

A simple python script where the user inputs the current ingredients they have in their kitchen into ingredients.txt and then runs the main.py script, and it will output what recipes can be created b

Jordan Leich 3 Nov 02, 2022
Project aims to map out common user behavior on the computer

User-Behavior-Mapping-Tool Project aims to map out common user behavior on the computer. Most of the code is based on the research by kacos2000 found

trustedsec 136 Dec 23, 2022
Sodium is a general purpose programming language which is instruction-oriented (a new programming concept that we are developing and devising) [Still developing...]

Sodium Programming Language Sodium is a general purpose programming language which is instruction-oriented (a new programming concept that we are deve

Instruction Oriented Programming 22 Jan 11, 2022
Скрипт позволяет заводить задачи в Панель мониторинга YouTrack на основе парсинга сайта safe-surf.ru

Скрипт позволяет заводить задачи в Панель мониторинга YouTrack на основе парсинга сайта safe-surf.ru

Bad_karma 3 Feb 12, 2022
🎅🏻 Helping santa understand ✨ python ✨

☃️ Advent of code 2021 ☃️ Helping santa understand ✨ python ✨

Fluffy 2 Dec 25, 2021
Run CodeServer on Google Colab using Inlets in less than 60 secs using your own domain.

Inlets Colab Run CodeServer on Colab using Inlets in less than 60 secs using your own domain. Features Optimized for Inlets/InletsPro Use your own Cus

2 Dec 30, 2021
The LiberaPay archive module for the SeanPM life archive project.

By: Top README.md Read this article in a different language Sorted by: A-Z Sorting options unavailable ( af Afrikaans Afrikaans | sq Shqiptare Albania

Sean P. Myrick V19.1.7.2 1 Aug 26, 2022
An Airdrop alternative for cross-platform users only for desktop with Python

PyDrop An Airdrop alternative for cross-platform users only for desktop with Python, -version 1.0 with less effort, just as a practice. ##############

Bernardo Olisan 6 Mar 25, 2022
A custom advent of code I am completing

advent-of-code-custom A custom advent of code I am doing in python. The link to the problems I am solving is here: https://github.com/seldoncode/Adven

Rocio PV 2 Dec 11, 2021
Sodium is a general purpose programming language which is instruction-oriented

Sodium is a general purpose programming language which is instruction-oriented (a new programming concept that we are developing and devising)

Satin Wuker 22 Jan 11, 2022
Sorter makes file organisation and management easier.

Sorter Sorter makes file organisation easier. It simply helps you organise several files that contain similar characteristics into a single folder. Yo

Aswa Paul 34 Aug 14, 2022