Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service

Last update: Dec 20, 2022

Overview

hashlookup-forensic-analyser

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

Usage

usage: hashlookup-analyser.py [-h] [-v] [-d DIR] [--print-all] [--print-unknown] [--include-stats]

Analyse a forensic target to find and report files found and not found in hashlookup CIRCL public service

optional arguments:
  -h, --help         show this help message and exit
  -v, --verbose      Verbose output
  -d DIR, --dir DIR  Directory to analyse
  --print-all        Print all files result including known and unknown
  --print-unknown    Print all files unknown to hashlookup service
  --include-stats    Include statistics in the CSV export

Example

[email protected] ~/git/hashlookup-forensic-analyser/bin $ python3 hashlookup-analyser.py --print-all -d /usr/local/bin/ --include-stats
unknown,/usr/local/bin/octopress
unknown,/usr/local/bin/safe_yaml
unknown,/usr/local/bin/bayes.rb
unknown,/usr/local/bin/redcarpet
unknown,/usr/local/bin/listen
unknown,/usr/local/bin/f2py
unknown,/usr/local/bin/f2py3.8
unknown,/usr/local/bin/tabulate
unknown,/usr/local/bin/jekyll
unknown,/usr/local/bin/pdf2txt.py
unknown,/usr/local/bin/rougify
unknown,/usr/local/bin/summarize.rb
unknown,/usr/local/bin/camelot
unknown,/usr/local/bin/kramdown
unknown,/usr/local/bin/posix-spawn-benchmark
unknown,/usr/local/bin/f2py3
unknown,/usr/local/bin/__pycache__/dumppdf.cpython-38.pyc
unknown,/usr/local/bin/__pycache__/pdf2txt.cpython-38.pyc
known,/usr/local/bin/scss
known,/usr/local/bin/sass-convert
known,/usr/local/bin/dumppdf.py
known,/usr/local/bin/sass
stats,Analysed directory /usr/local/bin/ on kolmogorov running Linux-5.10.0-1045-oem-x86_64-with-glibc2.29 at 2021-10-03 10:09:18.254424+00:00- Found 4 on hashlookup.circl.lu - Unknown files 18 - Excluded files 0

License

The software is open source software released under the "Simplified BSD License".

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Comments

Added some python tests using github actions
added some tests in github actions such as :

> bandit --recursive --skip B106,B110,B404,B602,B603,B607,B303 > black --check > flake8 . --count --ignore=B,E203,E722,W503,W605 --max-complexity=28 --max-line-length=220 --show-source --statistics > isort --check-only --profile black > mypy --allow-redefinition --ignore-missing-imports --install-types --non-interactive

changed the Module name platform to pl because the variable platform has the same name and a test was failed. bin/hashlookup-analyser.py:20: error: Incompatible types in assignment (expression has type "str", variable has type Module)
opened by signorrayan 4
Stream processing and cached/async lookups?
I wonder if this script might be enhanced for use cases handling some or all of:

large numbers of local objects

high redundancy across execution runs or local folders in a single run

low bandwidth uplinks

@adulau: In case there is interest I am happy to provide a minimally invasive pull request (have to implement before ... of course).

Questions:

Which python version is targeted as bottom? Is it 3.6+ or 3.8 or ...? I do find indicators in the source but not declaration ... I assume it is a version every analyst has on their OS but that I do not know :wink:

Could one use a bulk query endpoint as per "Bulk search of SHA-1 hashes"?

Is this binary linux version compiled from the python source per nuitka, pythran et al. or is this build from some other source language hosted elsewhere?

If it is compiled from python source I would need to know how, so I can provide a compilable version directly and no reviewer needs to hint me at all things breaking in that compilation.
enhancement
opened by sthagen 4
Add support for multiple bloom filter files

This PR adds support for multiple bloom filters via the "--bloomfilters x y z" argument. (cfr https://github.com/hashlookup/hashlookup-forensic-analyser/issues/10)

opened by wllm-rbnt 2

Consistent string interpolation

Why?

Use one and only one approach for string interpolation.

What?

All but one interpolation tasks for strings are solved per f-strings. The requests get call uses the good all format method on strings for construction of the address parameter.

How?

This is just a proposal with single line scope that shall ease maintenance and thus maybe a diff is enough to either implement or kindly ignore the change suggested:

diff --git a/bin/hashlookup-analyser.py b/bin/hashlookup-analyser.py
index f59dfec..e9e1845 100644
--- a/bin/hashlookup-analyser.py
+++ b/bin/hashlookup-analyser.py
@@ -36,7 +36,7 @@ if not args.dir:
 def lookup(value=None):
     if value is None:
         return False
-    r = requests.get('https://hashlookup.circl.lu/lookup/sha1/{}'.format(value), headers=headers)
+    r = requests.get(f'https://hashlookup.circl.lu/lookup/sha1/{value}', headers=headers)
     return r.json()

opened by sthagen 2

Stuck at named pipes

file=57C8EDB95DF3F0AD4EE2DC2B8CFD4157, mode=4607, finfo=os.stat_result(st_mode=4607, st_ino=10480, st_dev=1793, st_nlink=2, st_uid=0, st_gid=0, st_size=0, st_atime=1612118667, st_mtime=1612118667, st_ctime=1612118667)

file 57C8EDB95DF3F0AD4EE2DC2B8CFD4157 57C8EDB95DF3F0AD4EE2DC2B8CFD4157: fifo (named pipe)

opened by michael-hamm 1
Missing dependency declaration of `pytz`
Why?

Without additionally installing pytz from pypi.org per pip the script does fail to start.

What?

The pytzpackage is not part of the standard install of python but instead hosted on the Python Package Index at pypi.org/project/pytz

Example failure:

$ python bin/hashlookup-analyser.py --include-stats -d . -v Traceback (most recent call last): File "bin/hashlookup-analyser.py", line 11, in <module> import pytz ModuleNotFoundError: No module named 'pytz'

How?

Add the pytz dependency to the requirements file and maybe add an install section to the documentation.

The latter might go like this:

Install

$ pip install -r REQUIREMENTS
opened by sthagen 1
Add option to list files that have known hash, but for which the filename doesn't match any of the known filenames for that hash

Some attack techniques replace a 'known' file by another 'known' file, allowing them to exploit some processing flow that triggers the binary at the target location

This kind of scenario could be detected with this proposed new functionality

opened by Wachizungu 0
[Idea] Option to unarchive

This is not an easy suggestion and it may also introduce risks but having an option to get archived files to get extracted recursively & hashed, I thought could also be an interesting add to hashlookup :)

(E.g of lib - https://githubplus.com/enzok/sflock or 7zip etc.)
enhancement

opened by Maijin 5

Releases(v1.1)

v1.1(Jul 9, 2022)

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service or the Bloom filter from CIRCL hashlookup. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

This release includes the ability to load multiple Bloom filters at the same time. Thanks @wllm-rbnt for the contribution.
Source code(tar.gz)
Source code(zip)
v1.0(May 7, 2022)
A new version of the hashlookup-forensic-analyser has been released including a new --live-linux option to verify running process on a Linux machine if these are known or unknown binaries on hashlookup service.

This version 1.0 is considered as stable following the feedback received by the different users. Don't hesitate to provide feedback or ideas by opening issues.

New --live-linux option

python3 hashlookup-analyser.py --cache --print-unknown --live-linux. The option is actually reading the /proc directory to find all running processes and due to the fact the ./exe file in proc directory is the actual file, you can quickly find the known/unkown files from the running processes.

Complete changelog available below:

v1.0 (2022-05-07)

New

[progress] disable progress option --disable-progress as progress is by default nowadays. [Alexandre Dulaunoy]

[live_linux] new --live-linux option to display known and unknown processes running from a running linux instance. [Alexandre Dulaunoy]

The process hashes are read from the /proc/PID/exe which is the actual file running before being loaded in memory.

Changes

[lint] increase code complexity. [Alexandre Dulaunoy]

[code] black -S [Alexandre Dulaunoy]

[hash] skip bandit check on weak hash. [Alexandre Dulaunoy]

[hashlib] the library is not used for security per se but for forensic and use SHA1 as the lookup service gives more potential results in SHA1 (due to old NSRL dataset) [Alexandre Dulaunoy]

[file] black -S [Alexandre Dulaunoy]

[doc] updated with new options. [Alexandre Dulaunoy]

Fix

[action] remove mypy too many FP ref -> https://github.com/python/mypy/issues/12246. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)
v0.9(Feb 18, 2022)
hashlookup-forensic-analyser version 0.9 released with MIME type statistics

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service or the Bloom filter from CIRCL hashlookup.

This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

For the curious, 📖 A sample summary report is available.

New

[report] add MIME type statistics. [Alexandre Dulaunoy]

[feature] stats about mime_type added. [Alexandre Dulaunoy]

Changes

[doc] fix the sample report link. [Alexandre Dulaunoy]

[hashlookup-forensic-analyser] add mermaid pie chart for mime types. [Alexandre Dulaunoy]

[doc] sample report updated. [Alexandre Dulaunoy]

[doc] sample report updated. [Alexandre Dulaunoy]

[requirements] filemagic added. [Alexandre Dulaunoy]

Fix

[hashlookup-analyser] blake -S [Alexandre Dulaunoy]

[hashlookup-analyser] add missing type annotation. [Alexandre Dulaunoy]

Other

Set theme jekyll-theme-dinky. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)
v0.8(Jan 2, 2022)
hashlookup-forensic-analyser version 0.8 released including a report functionality

A new --report option added to generate a report directory including a markdown summary and a JSON export of the results. A sample report in Markdown is available. The JSON includes all the found and unknown files discovered.

New

[hashlookup] --report option added to generate a report directory including a markdown summary and a JSON export of the results. [Alexandre Dulaunoy]

Changes

[doc] cleanup. [Alexandre Dulaunoy]

[doc] add sample gist page of a summary report. [Alexandre Dulaunoy]

Fix

[hashlookup] fix headline report typo. [Alexandre Dulaunoy]

[hashlookup] shadowed variable removed. [Alexandre Dulaunoy]

[hashlookup] shadowed loop variable removed. [Alexandre Dulaunoy]

Other

Update README.md. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)
v0.7(Dec 28, 2021)
hashlookup-forensic-analyser version 0.7 released including various bugs fixed

Changes

[hashlookup] if a file cannot be read (e.g. Permission denied) for hashing. [Alexandre Dulaunoy]

Those are skipped, accounted in the nonanalysed files.

[ci] updated. [Alexandre Dulaunoy]

Fix

[bug] better handling of filename and CSV export (fix issue with comma separated filenames) [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)

v0.6(Dec 24, 2021)

hashlookup-forensic-analyser version 0.6 released including various bugs fixed and improved logging

usage: hashlookup-analyser.py [-h] [-v] [--extended-debug] [--progress] [-d DIR] [--print-all] [--print-unknown] [--include-stats] [--format FORMAT] [--cache] [--bloomfilter BLOOMFILTER]

Analyse a forensic target to find and report files found and not found in hashlookup CIRCL public service.

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Verbose output
  --extended-debug      Debug file processed along with the mode and type.
  --progress            Pring progress of the file lookup on stderr.
  -d DIR, --dir DIR     Directory to analyse
  --print-all           Print all files result including known and unknown
  --print-unknown       Print all files unknown to hashlookup service
  --include-stats       Include statistics in the CSV export
  --format FORMAT       Output format (default is CSV)
  --cache               Enable local cache of known and unknown hashes in /tmp/hashlookup-forensic-analyser
  --bloomfilter BLOOMFILTER
                        Specify filename of a bloomfilter in DCSO bloomfilter format

New

[hashlookup] --progress option to display the number of files analysed, excluded, unknown and found in hashlookup. [Alexandre Dulaunoy]

and a completely useless spinner added too.

This fixes #7

Changes

[hashlookup] enable the spinner by default and log on stderr. [Alexandre Dulaunoy]
[doc] README updated with new options. [Alexandre Dulaunoy]
[bin] requests missing. [Alexandre Dulaunoy]
[import] order is important. [Alexandre Dulaunoy]
[workflow] long lines again. [Alexandre Dulaunoy]
[formatting] black -S --target-version py38 [Alexandre Dulaunoy]
[hashlookup] --extended-debug option added to output the filetype. [Alexandre Dulaunoy]

only useful for debugging
[LICENSE] added. [Alexandre Dulaunoy]

Fix

[bin] typo fixed. [Alexandre Dulaunoy]
[hashlookup] exclude char and block device - catch incorrect fstat on size. [Alexandre Dulaunoy]
[chg] excluded named pipe - FIFO. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)

v0.5(Dec 14, 2021)
hashlookup-forensic-analyser version 0.5

Bloom filter support

If you don't want to share your lookups online and do faster lookup, hashlookup provides a bloom filter to download.

The file is around 700MB and can be stored locally in your home directory. hashlookup-analyser works in the same way, --bloomfilter option allows to specify the filename locatoon of the bloom filter.

python3 bin/hashlookup-analyser.py --bloomfilter <yourdir>/hashlookup-full.bloom --include-stats -d /bin

v0.5 (2021-12-14)

New

[doc] add a quick documentation about the bloom filter feature. [Alexandre Dulaunoy]

[bloomfilter] use a bloomfilter source instead of the live request. [Alexandre Dulaunoy]

[email protected] ~/git/hashlookup-forensic-analyser/bin (main)$ python3 hashlookup-analyser.py --cache -d /home/adulau/sample/ --print-unknown --bloomfilter /tmp/x.bloom --include-stats hashlookup_result,filename,sha-1,size unknown,/home/adulau/sample/xxx,278DFA419DC12FB8CD9785A873956885BBE11779,17 unknown,/home/adulau/sample/vi,DFA8A9B6417CE3C43D91B0E69E68C26A0230DD48,3194152 stats,Analysed directory /home/adulau/sample/ on kolmogorov running Linux-5.10.0-1052-oem-x86_64-with-glibc2.29 at 2021-12-06 06:48:56.892009+00:00- Found 1 on hashlookup.circl.lu (hashlookup-blomfilter)- Unknown files 2 - Excluded files 0

New: [gh] based on the pull-request #6 from @signorrayan and feedback from @sthagen. [Alexandre Dulaunoy]

Changes

[workflow] we have big screens nowadays. [Alexandre Dulaunoy]

[hashlookup] black -S --target-version py38 [Alexandre Dulaunoy]

[doc] --bloomfilter option added. [Alexandre Dulaunoy]

[doc] updated with new --cache option. [Alexandre Dulaunoy]

[bin] CSV now includes SHA-1 of the files. [Alexandre Dulaunoy]

[bin] new --cache feature added. [Alexandre Dulaunoy]

if cache is called, this will create a file per hash in a fixed directory. at each run, if --cache is called, it will check the existence of a file. The directory contains the JSON result saved from the previous hashlookup query of the server.

[README] a status badge added. [Alexandre Dulaunoy]

[bin] make isort happy. [Alexandre Dulaunoy]

[bin] make black happy. [Alexandre Dulaunoy]

[bin] consistency update fix #4. [Alexandre Dulaunoy]

Fix

[doc] bloom filter link fixed. [Alexandre Dulaunoy]

[bloomfilter] bug fix as bloomfilter just returns a True or False. [Alexandre Dulaunoy]

[bin] remove ambiguity on file handle type. [Alexandre Dulaunoy]

[bin] BinaryIO and not TextIO while reading file. [Alexandre Dulaunoy]

[bin] blake formatting (need a post-hook ;-) [Alexandre Dulaunoy]

[bin] skip type test on empty dict. [Alexandre Dulaunoy]

[bin] incorrect version. [Alexandre Dulaunoy]

[bin] as referenced in #6 - platform assignment can lead to some confusion. [Alexandre Dulaunoy]

[REQUIREMENTS] pytz dep was missing fix #5. [Alexandre Dulaunoy]

Source code(tar.gz)
Source code(zip)
v0.2(Oct 16, 2021)
hashlookup-forensic-analyser is a script to analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

What's Changed

Clarification regarding the CSV output format

Size of the file analysed added in the output

Improved performance - reduce lines by @signorrayan in https://github.com/hashlookup/hashlookup-forensic-analyser/pull/1

Pre-built Linux binary of the script added (if Python is not present on the target system to be analysed)

New Contributors

@signorrayan made their first contribution in https://github.com/hashlookup/hashlookup-forensic-analyser/pull/1

Screenshot

Full Changelog: https://github.com/hashlookup/hashlookup-forensic-analyser/compare/v0.1...v0.2
Source code(tar.gz)
Source code(zip)
v0.1(Oct 3, 2021)

hashlookup forensic analyser - version 0.1 release

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation. The CIRCL public service already includes multiple sources such as Linux distribution, NIST NSRL and many others.

The tool is written in Python 3 using a minimal set of dependencies.

Source code(tar.gz)
Source code(zip)

Owner

hashlookup

GitHub Repository

Growtopia Save.dat Stealer

savedat-stealer Growtopia Save.dat Stealer (Auto Send To Webhook) How To Use After Change Webhook URL Compile script to exe Give to target Done Info C

9 May 01, 2022

Dependency Combobulator is an Open-Source, modular and extensible framework to detect and prevent dependency confusion leakage and potential attacks.

Dependency Combobulator Dependency Combobulator is an Open-Source, modular and extensible framework to detect and prevent dependency confusion leakage

84 Dec 23, 2022

A simple linux keylogger project.

The project This project is a simple linux keylogger. When activated, it registers all the actions made with the keyboard. The log files are registere

1 Oct 24, 2021

PrivateRoom - Make your work private by building a system using arduino which instantly kills a program when someone enters your room/cabin

privateRoom Make your work private by building a system using arduino which instantly kills a program when someone enters your room/cabin STEPS: Uploa

3 Nov 08, 2022

the metasploit script(POC/EXP) about CVE-2021-22005 VMware vCenter Server contains an arbitrary file upload vulnerability

CVE-2021-22005-metasploit the metasploit script(POC/EXP) about CVE-2021-22005 VMware vCenter Server contains an arbitrary file upload vulnerability pr

25 Nov 15, 2022

Malware for Discord, designed to steal passwords, tokens, and inject discord folders for long-term use.

Vital What is Vital? Vital is malware primarily used to collect and extract information from the Discord desktop client. While it has other features (

59 Dec 01, 2022

A Python Scanner for log4j

log4j-Scanner scanner for log4j cat web-urls.txt | python3 log4j.py ID.burpcollaborator.net web-urls.txt http://127.0.0.1:8080 https://www.google.c

5 Jun 26, 2022

On the 11/11/21 the apache 2.4.49-2.4.50 remote command execution POC has been published online and this is a loader so that you can mass exploit servers using this.

ApacheRCE ApacheRCE is a small little python script that will allow you to input the apache version 2.4.49-2.4.50 and then input a list of ip addresse

3 Dec 04, 2022

Analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service

Related tags

Overview

hashlookup-forensic-analyser

Usage

Example

License

Comments

Why?

What?

How?

Why?

What?

How?

Install

Releases(v1.1)

v1.1(Jul 9, 2022)

v1.0(May 7, 2022)

New --live-linux option

v1.0 (2022-05-07)

New

Changes

Fix

v0.9(Feb 18, 2022)

hashlookup-forensic-analyser version 0.9 released with MIME type statistics

New

Changes

Fix

Other

v0.8(Jan 2, 2022)

hashlookup-forensic-analyser version 0.8 released including a report functionality

New

Changes

Fix

Other

v0.7(Dec 28, 2021)

hashlookup-forensic-analyser version 0.7 released including various bugs fixed

Changes

Fix

v0.6(Dec 24, 2021)

hashlookup-forensic-analyser version 0.6 released including various bugs fixed and improved logging

New

Changes

Fix

v0.5(Dec 14, 2021)

hashlookup-forensic-analyser version 0.5

Bloom filter support

v0.5 (2021-12-14)

New

Changes

Fix

v0.2(Oct 16, 2021)

What's Changed

New Contributors

Screenshot

v0.1(Oct 3, 2021)

hashlookup forensic analyser - version 0.1 release

Owner

hashlookup

Growtopia Save.dat Stealer

Dependency Combobulator is an Open-Source, modular and extensible framework to detect and prevent dependency confusion leakage and potential attacks.

A simple linux keylogger project.

PrivateRoom - Make your work private by building a system using arduino which instantly kills a program when someone enters your room/cabin

the metasploit script(POC/EXP) about CVE-2021-22005 VMware vCenter Server contains an arbitrary file upload vulnerability

Malware for Discord, designed to steal passwords, tokens, and inject discord folders for long-term use.

A Python Scanner for log4j

On the 11/11/21 the apache 2.4.49-2.4.50 remote command execution POC has been published online and this is a loader so that you can mass exploit servers using this.

Proof of Concept Exploit for vCenter CVE-2021-21972

BloodyAD is an Active Directory Privilege Escalation Framework

MITMSDR for INDIAN ARMY cybersecurity hackthon

Add a Web Server based on Rogue Mysql Server to allow remote user get

Exploiting CVE-2021-44228 in Unifi Network Application for remote code execution and more

ProxyLogon Pre-Auth SSRF To Arbitrary File Write

Python tool for enumerating directories and for fuzzing

DNSSEQ: PowerDNS with FALCON Signature Scheme

How to exploit a double free vulnerability in 2021. 'Use-After-Free for Dummies'

CVE-2021-41773 Path Traversal for Apache 2.4.49

Python script that sends CVE-2021-44228 log4j payload requests to url list

Keystroke logging, often referred to as keylogging or keyboard capturing

New `--live-linux` option