Confirm that files have been uploaded to Backblaze Cloud Backup successfully

Overview

Backblaze Backup Checker

This Python script compares metadata captured from files within source folders against data parsed from Backblaze Cloud Backup's log files (bz_done files, stored locally), reporting any files that do not appear to have been uploaded or that have conflicting cryptographic hash values, as well as files that would not be uploaded due to filter/exclusion rules.

The primary use case is to provide confidence that files have been backed up with expected integrity, or if not, what filter/exclusion rules may be blocking the backup.

Due to limited availability of test data, this script is currently in alpha state. It is likely that false positives will be reported (see Known Issues below).

Prerequisites

Python 3.7 or later is required, with the tqdm progress bar module installed (pip install tqdm).

This script has been tested using Backblaze client version 7.0.2.470 on macOS 11.5, and client version 8.0.0.517 on Windows 10 20H2.

This script is only compatible with 'Version 5' log records (the standard used by Backblaze clients since October 2014). Any log records that are in an older format will not be processed.

Script behaviour

Backblaze configuration and log data, as well as files in source folders, are opened in read-only mode and will not be modified.

For each source folder specified by the user, file path and size metadata will be only captured for those files that 'pass' the filter/exclusion rules and have not been created or modified in the last 24 hours (to allow the Backblaze client time to upload them). Filter/exclusion rules will be read automatically from Backblaze configuration data; both Backblaze-default and user-configured rules will be processed.

Under default script configuration, SHA1 hash metadata will also be generated for files that are <= 100MB in size, in order to confirm file integrity. (Files > 100MB in size are split in the Backblaze logs; the original hash value is not retained, and I am unaware of any technique to construct the true SHA1 hash from the available split data.)

Symlinks will be ignored during processing as these are not backed up by Backblaze. Mac hidden files .DS_Store and .localized will also be ignored.

Progress is reported within the console (and if specified, to log files). At script conclusion, up to four text files will be created in either the current working folder or a user-specified output folder, with filenames and comma-separated contents as follows:

  1. [datetime]_backblaze_uploaded_files.txt: list of files that are present and correct in Backblaze log data, indicating successful upload.
  2. [datetime]_backblaze_mismatch_files.txt: list of files whose path is either not present in Backblaze log data, or the path is present but SHA1/size metadata does not match between the local file and Backblaze log data. Files listed in this output may not have been uploaded to Backblaze successfully - but be aware that there are likely to be false positives (see Known Issues below).
  3. [datetime]_backblaze_excluded_by_filter_files.txt: list of files that have not been uploaded to Backblaze due to filter/exclusion rules, with details of which rules the files are 'failing' on.
  4. [datetime]_backblaze_recent_files_not_processed.txt: list of files created or modified in the last 24 hours are listed in this output and not processed during script execution (as Backblaze may not have had time to upload them yet).

Recommended usage

It is recommended that:

  1. Specific folders containing user data are processed rather than full operating system drives. This is because various permissions errors and temporary files will be encountered if an entire operating system drive is processed, which will complicate the output (although the script should still run).
  2. The script is run only after sufficient time has been allowed for files to have been uploaded to Backblaze. Files created or updated within the last 24 hours will be identified and not processed, but this approach is not robust (further script development to account for data within bz_todo log files would enable accurate filtering of files that are still being uploaded).

Script usage

Syntax:

python3 bbcheck.py source_path [source_path ...] [flags]

Usage example (Windows) with two source folders:

python3 bbcheck.py C:\Users\john\Documents "E:\data folder"

Usage example (Mac) with one source folder:

python3 bbcheck.py /Users/john/Documents

Absolute or relative paths may be provided; all metadata generated will revert to absolute paths.

Flags can be viewed using: python3 bbcheck.py --help, and are as follows:

  • -l or --log: opt to write the status messages displayed in the console to a log file in the log folder.
  • -d or --debug: display debug messages and write these to a dedicated debug log file in the log folder.
  • --logfolder [str]: folder to write logs to (if not specified, the default of bbcheck_logs will be used).
  • -o [str] or --output [str]: folder to write check results to (if not specified, results will be written to the current working folder).
  • -b [str] or --bzdata-folder [str]: by default, the script will attempt to read Backblaze configuration and log data from standard client install locations for Windows and Mac. An alternative path to the bzdata folder may be provided with this flag.
  • -s or --only-check-size: by default, SHA1 hash values will be generated for the integrity check for files <= 100MB in size. Hash data may take a long time to generate for large source folders; this flag sets the script to instead check integrity using file size metadata, which should execute quickly (but is not a true integrity check and is therefore less reliable).
  • --hash-files [str ... str]: instead of generating SHA1 hash data during script execution, hash file(s) created using the hash mode in Vericopy may be used as a lookup. This approach allows for quick successive executions of this script, and is reliable on condition that files within the source folders are not changed between script executions.

Usage example (Windows) incorporating flags:

python3 bbcheck.py C:\Users\john\Documents "E:\data folder" -l -o output_folder -s --hash-files e_drive_hashes.txt -b "C:\ProgramData\Backblaze Alternative Location\bzdata"

Known issues

  1. Hash/size mismatches may be reported for some files, where Backblaze logs reference an incorrect hash/size value for a file - but upon recovery of the file using the Backblaze web interface, the correct original file (with correct hash/size) will be downloaded. In testing this behaviour seems prevalent for certain file types, such as .doc files, but as yet I have not discovered the reason for this behaviour.
  2. Although symlinks are not processed during script execution, Mac alias links will be processed and raised (incorrectly) as missing files.
  3. Depending on upload speed and duration the client has been running since files are created or modified, it is possible that files may be reported as missing that are still uploading (further script development to account for data within bz_todo log files would mitigate this).

Privacy, log data, and uninstallation

This script runs entirely locally; neither Backblaze nor any other third party services are communicated with.

Script output is stored by default in the folder the script is executed in. If -l or -d is used to output logs to a file, these are stored by default in folder bbcheck_logs (created in the folder that the script is executed in). Debug logs may contain sensitive information, such as system details (including Python version and operating system), command line arguments used, and events occurring with data processed during script execution.

Full uninstallation can be achieved by:

  1. Deleting the script and any other downloaded files (e.g. the readme and license).
  2. Deleting script output and the logs folder (bbcheck_logs by default).
  3. If desired, removing the tqdm library and Python runtime.

Contributing

If you would like to contribute, please fork the repository and use a feature branch. Pull requests are warmly welcome.

Licensing

The code in this project is licensed under the MIT License.

You might also like...
💻  A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline!
💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline!

LocalStack - A fully functional local AWS cloud stack LocalStack provides an easy-to-use test/mocking framework for developing Cloud applications. Cur

Cloud-native, data onboarding architecture for the Google Cloud Public Datasets program
Cloud-native, data onboarding architecture for the Google Cloud Public Datasets program

Public Datasets Pipelines Cloud-native, data pipeline architecture for onboarding datasets to the Google Cloud Public Datasets Program. Overview Requi

Prisma Cloud utility scripts, and a Python SDK for Prisma Cloud APIs.

pcs-toolbox Prisma Cloud utility scripts, and a Python SDK for Prisma Cloud APIs. Table of Contents Support Setup Configuration Script Usage CSPM Scri

Python client for using Prefect Cloud with Saturn Cloud

prefect-saturn prefect-saturn is a Python package that makes it easy to run Prefect Cloud flows on a Dask cluster with Saturn Cloud. For a detailed tu

:electric_plug: Generating short urls with python has never been easier
:electric_plug: Generating short urls with python has never been easier

pyshorteners A simple URL shortening API wrapper Python library. Installing pip install pyshorteners Documentation https://pyshorteners.readthedocs.i

API to retrieve the number of grades on the OGE website (Website listing the grades of students) to know if a new grade is available. If a new grade has been entered, the program sends a notification e-mail with the subject.
API to retrieve the number of grades on the OGE website (Website listing the grades of students) to know if a new grade is available. If a new grade has been entered, the program sends a notification e-mail with the subject.

OGE-ESIREM-API Introduction API to retrieve the number of grades on the OGE website (Website listing the grades of students) to know if a new grade is

A Bot to Upload files to Many Cloud services. Powered by Telethon.
A Bot to Upload files to Many Cloud services. Powered by Telethon.

oVo MultiUpload V1.0 👀 A Bot to Upload files to Many Cloud services. Powered by Telethon _ 🎯 Follow me and star this repo for more telegram bots. @H

Bot simply search for the files from provided channel according to given query and gives link to those files as buttons!

Auto Filter Bot ㅤㅤㅤㅤㅤㅤㅤ ㅤㅤㅤㅤㅤㅤㅤ You can call this as an Auto Filter Bot if you like :D Bot simply search for the files from provided channel according

Aggrokatz is an aggressor plugin extension for Cobalt Strike which enables pypykatz to interface with the beacons remotely and allows it to parse LSASS dump files and registry hive files to extract credentials and other secrets stored without downloading the file and without uploading any suspicious code to the beacon.
Comments
  • _csv.Error: line contains NUL

    _csv.Error: line contains NUL

    I tried to run it but ran into an error (see below).

    My Environment:

    • Windows 10 Version 20H2 (OS Build 19042.1237)
    • Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32
    • Backblaze log data location: C:\ProgramData\Backblaze\bzdata

    Output: python3 bbcheck.py -d E:\

    2021-10-02 21:39:15,419 - DEBUG - python_version: 3.9.7
    2021-10-02 21:39:15,444 - DEBUG - system: Windows
    2021-10-02 21:39:15,444 - DEBUG - machine: AMD64
    2021-10-02 21:39:15,457 - DEBUG - platform: Windows-10-10.0.19042-SP0
    2021-10-02 21:39:15,458 - DEBUG - version: 10.0.19042
    2021-10-02 21:39:15,459 - DEBUG - mac_ver: ('', ('', '', ''), '')
    2021-10-02 21:39:15,460 - DEBUG - commandline_args: {'source_paths': ['E:\\'], 'log': False, 'debug': True, 'logfolder': 'bbcheck_logs', 'output': None, 'bzdata_folder': None, 'only_check_size': False, 'hash_files': None}
    2021-10-02 21:39:15,460 - INFO - Logs will be stored in folder 'bbcheck_logs'
    2021-10-02 21:39:15,461 - INFO - Source path(s) 'E:\' will be compared against Backblaze log data in 'C:\ProgramData\Backblaze\bzdata'
    2021-10-02 21:39:15,467 - INFO - Findings will be written to current working folder
    Traceback (most recent call last):
      File "D:\Downloads\bbz\bbcheck.py", line 1073, in <module>
        main()
      File "D:\Downloads\bbz\bbcheck.py", line 1050, in main
        check_backup(
      File "D:\Downloads\bbz\bbcheck.py", line 753, in check_backup
        bzdone_metadata = parse_bz_done_files(bzdone_file_paths, only_check_size_flag)
      File "D:\Downloads\bbz\bbcheck.py", line 483, in parse_bz_done_files
        for row in reader:
    _csv.Error: line contains NUL
    

    *EDIT: Just re-ran script with '-d' option and replaced output with the new output.

    opened by jeffkujath 7
  • XML parsing error

    XML parsing error

    Hi, first of all - thanks for making this nice script!

    I run into XML parsing error:

    INFO - Source path(s) '/Users/msoida/Downloads' will be compared against Backblaze log data in '/Library/Backblaze.bzpkg/bzdata'
    INFO - Findings will be written to current working folder
    
    Traceback (most recent call last):
      File "/Users/msoida/Downloads/backblaze-backup-checker-0.1.1/bbcheck.py", line 1102, in <module>
        main()
      File "/Users/msoida/Downloads/backblaze-backup-checker-0.1.1/bbcheck.py", line 1085, in main
        datetime_string=datetime_string,
      File "/Users/msoida/Downloads/backblaze-backup-checker-0.1.1/bbcheck.py", line 759, in check_backup
        exclude_rules = get_excludes(bzdata_folder_path)
      File "/Users/msoida/Downloads/backblaze-backup-checker-0.1.1/bbcheck.py", line 586, in get_excludes
        root = xml.etree.ElementTree.parse(exclude_file_path).getroot()
      File "/Users/msoida/.pyenv/versions/3.7.3/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
        tree.parse(source, parser)
      File "/Users/msoida/.pyenv/versions/3.7.3/lib/python3.7/xml/etree/ElementTree.py", line 598, in parse
        self._root = parser._parse_whole(source)
    xml.etree.ElementTree.ParseError: XML or text declaration not at start of entity: line 2, column 0
    

    After some debugging, I found the problem to be in /Library/Backblaze.bzpkg/bzdata/bzexcluderules_mandatory.xml - this file had an empty line at the start. After removing this empty line, the script works great.

    Now, I have no idea why this line was there. Since I have BackBlaze running for many years now, it's quite possible this happend due to some automatic file structure update many years ago. Whatever the reason is, it seems the BB client does not mind this extra line, and the Python XML parser does not like it.

    While I expect that such problems are probably not too common, it would be nice to defend the rest of this awesome script against them ;)

    PS: I'm using MacOS 10.15.7/Python 3.7.3, but I doubt it's relevant in this case.

    opened by msoida 2
Releases(v0.1.2)
  • v0.1.2(Oct 4, 2021)

  • v0.1.1(Oct 2, 2021)

  • v0.1(Oct 1, 2021)

    Initial release - due to limited availability of test data, this script is currently in alpha state. It is likely that false positives will be reported (see Known Issues in README).

    Source code(tar.gz)
    Source code(zip)
NMux is the version of the NMscript in termux

NMscript-termux-version Termux-Version NMux is the termux version of NMscript which is NMscript? NMscript is a simple script written in Python that he

cabeson sin z 5 Apr 23, 2022
Wordle-bot: A Discord bot to track you and your friends' Wordle scores.

wordle-bot A Discord bot to track you and your friends' Wordle scores, so you can see who's the best! To submit a score to wordle-bot, just paste the

Spencer Murray 8 Feb 16, 2022
A program that automates the boring parts of completing the Daily accounting spreadsheet at Taos Ski Valley

TSV_Daily_App A program that automates the boring parts of completing the Daily accounting spreadsheet at my old job. To see how it works you will nee

Devin Beck 2 Jan 01, 2022
Program that automates the bump of the Disboard Bot. Done 100% in Python with PyAutoGUI library

Auto-Discord-Bump Program that automates the bump of the Disboard Bot done 100% in python with PyAutoGUI How to configue You will need 3 things before

Mateus 1 Dec 19, 2021
Trellox Tool is written in Python3 and designed to pull and list Trello boards.

TrelloX Trellox Tool is written in Python3 and designed to list and pull Trello boards. It can be used by penetration testers/bug bounty hunters to de

Ali Fathi Ali Sawehli 1 Dec 05, 2021
dex.guru python sdk

dexguru-sdk.py dexguru-sdk.py allows you to access dex.guru public methods from your async python scripts. Installation To install latest version, jus

DexGuru 17 Dec 06, 2022
对hermit 的API进行简单的封装,做成了这个python moudle

hermit-py 对hermit 的API进行简单的封装,做成了这个Python Moudle,推荐通过wheel的方式安装。 目前对点击、滑动、模拟输入、找组件、等支持较好,支持查看页面的实时布局信息,再通过布局信息进行点击滑动等操作。 支持剪贴板相关的操作,支持设置剪贴的任意语言内容。

LookCos 40 Jun 25, 2022
Source code from thenewboston Discord Bot with Python tutorial series.

Project Setup Follow the steps below to set up the project on your environment. Local Development Create a virtual environment with Python 3.7 or high

Bucky Roberts 24 Aug 19, 2022
A melhor maneira de atender seus clientes no Telegram!

Clientes.Chat Sobre o serviço Configuração Banco de Dados Variáveis de Ambiente Docker Python Heroku Contribuição Sobre o serviço A maneira mais organ

Gabriel R F 10 Oct 12, 2022
Discord bot that plays cricket with the user

CricBot Table of content Commands Installation Game rules License Commands S.No Command Use 1. cric Open the home window. This command is not necessa

Raveesh Yadav 1 Nov 19, 2021
Tracks twitter spaces and sends it to a discord webhook.

Tracks twitter spaces and sends it to a discord webhook. Uses the twitter api to find twitter spaces and then the m3u8 url for the space is found using selenium and will have it posted using a discor

Sam Phung 20 Dec 17, 2022
This is the repository for HalpyBOT, the Hull Seals IRC Chatbot Assistant.

HalpyBOT 1.4.2 This is the repository for HalpyBOT, the Hull Seals IRC Chatbot Assistant. Description This repository houses all of the files required

The Hull Seals 3 Nov 03, 2022
An example Music Bot written in Disnake and uses slash commands to operate.

Music Bot An example music bot that is written in Disnake [Maintained discord.py Fork] Disnake Disnake is a maintained and updated fork of discord.py.

6 Jan 08, 2022
A program that generates discord.py code

discord-py-generator A program that generates discord.py code Setup in cmds.txt file add your user id, client id and bot token you can change the bot

3 Dec 15, 2022
Efetuar teste de automação usando linguagem gherkin

🚀 Teste-de-Automação - QA---CI-T 🚀 Descrição • Primeira Parte • Segunda Parte • Terceira Parte Contributors Descrição Efetuamos testes de automação

Eliel martins 6 Dec 07, 2021
Cytotron - A unique discord bot like never before. Add it to your server to keep it active, motiviated, and amazing!!

Cytotron - Take your server to the next level Most of the details are in the website. Go to https://cytotron-bot.gq for more information. If that link

LeviathanProgramming 6 Jun 13, 2021
Bulk convert image types with Python

Bulk Image Converter 🔥 Helper script to convert a folder's worth of images from one filetype to another, and optionally delete originals Use Setup /

1 Nov 13, 2021
This is a Anti Channel Ban Robots

AntiChannelBan This is a Anti Channel Ban Robots delete and ban message sent by channels Heroku Deployment 💜 Heroku is the best way to host ur Projec

BᵣₐyDₑₙ 25 Dec 10, 2021
This is a simple collection of instructions and scripts to accompany the computerphile video about mininet and openflow.

How to get going. This project should work on Linux or MacOS. I used Ubuntu 20.04 and provide some notes here. Note, this is certainly not intended as

Richard G. Clegg 70 Jan 02, 2023
This is a simple bot that can be used to upload images to a third-party cloud (image hosting). Currently, only the imgbb.com website supports the bot. I Will do future updates

TGImageHosting This is a simple bot that can be used to upload images to a third party cloud (image hosting). Currently, only the imgbb.com website su

Abhijith N T 0 Dec 26, 2021