It's a simple python script to take backup of directories (compressing) then the same to move your mentioned S3 bucket with the help of AWS IAM User.

Overview

Directory Backup Moved to S3 (Pyscript)

Build


Description

Here it's a python script that needs to use this script simply create a directory backup and moved that compressed backup file to a configured S3 bucket with the help of python script and AWS IAM User with S3 full access. So, let's roll down.


Feature

  • Easy to configure for anyone
  • It generates the directory compressed format
  • Which one we entered directory converted to a backup/compressed form to S3
  • All the steps I have provided including AWS IAM User and S3 Bucket Creation

Modules used


Pre-Requests

  • Basic Knowledge of python
  • Basic Knowledge of AWS IAM, S3 service
  • Need to change your IAM user creds and bucket name at var.py

IAM User Creation steps (with screenshot)

  1. log into your AWS account as a root user and go to IAM user
  2. goto Access Managment >> Users alt_txt
  3. Click Add User (top right corner) alt_txt
  4. Enter any username as you like and Choose "Programmatic access" >> Click Next Permissions alt_txt
  5. Set Permissions >> Select "Attach existing policies directly" >> Choose "AmazonS3FullAccess" >> Click Next Tags alt_txt
  6. Add Tags(Optional) >> Enter a key and value as you like either you can leave as blank alt_txt
  7. Review your user details and click "Create User" alt_txt
  8. Store your credentials to your local alt_txt

Reference URL:: IAM User creation article


S3 Bucket Creation (with screenshot)

  1. Go to S3 > Click Create Bucket alt_txt
  2. Any bucket name as you wish and then please enable versioning (that you upload same file multiple times or modified versions uploaded that the S3 stored as a version bases like Git) alt_txt
  3. Click create bucket

alt_txt

Reference URL:: Creating S3 bucket please use this doc and you can secure your bucket with IAM user using S3 bucket policy


Pre-Requested (Dependency packages)

yum install -y git
yum install -y python3
yum install -y python3-pip

How to get

git clone https://github.com/yousafkhamza/backup-to-s3-pyscript.git
cd backup-to-s3-pyscript
pip3 install -r requirements.txt

Change your creds and bucket name in at var.py file

Command to run the script::

python3 to-S3.py Python/test
# Pyhon/test               <------------ Directory to take backup and move the backup to S3

Output be like

$ $ python3 to-S3.py Python/httpd
Start to Upload that the httpd.tar.gz your S3 Bucket yousaf-test
Backup successfully uploaded to your S3 bucket yousaf-test

View of S3 bucket

Screenshot alt_txt


Behind the code

vim to-S3.py

import boto3
import tarfile
import os
import sys
import posixpath
import var
from boto3.s3.transfer import S3Transfer

directory = sys.argv[1]
dirname = os.path.split(directory)[-1]
if directory.endswith ('/'):
    print ('Please remove / after the directory path you have entered')
else:
    if posixpath.isdir(directory):
        tarname = '/tmp/{}.tar.gz'.format(dirname)
        tar = tarfile.open(tarname,'w:gz')
        tar.add(directory)
        tar.close()
        print('Start to Upload that the', dirname+'.tar.gz', 'your S3 Bucket', var.BUCKET_NAME)

# S3 uploading started
        client = boto3.client('s3', aws_access_key_id=var.AWS_ACCESS_KEY_ID,aws_secret_access_key=var.AWS_SECRET_ACCESS_KEY)
        transfer = S3Transfer(client)
        transfer.upload_file(tarname, var.BUCKET_NAME, 'backup/{}.tar.gz'.format(dirname))
        print('Backup succesfully uploaded to your S3 bucket', var.BUCKET_NAME)
    
# Remove temporary backup from local
        os.remove(tarname)
    else:
        print('Please enter a valid directory path')

var.py

AWS_ACCESS_KEY_ID = 'AKTBRI2N5IAT3ND'             <--------------   Replace your acess key
AWS_SECRET_ACCESS_KEY = 'asUNmMPrC99HoiiQPjehetFtVsPv'          <--------- Replace your secret key
BUCKET_NAME = 'yousaf-test'                <----------- Replace your bucket name

Conclusion

It's a simple python script to take backup of directories (compressing) then the same to move your mentioned S3 bucket with the help of AWS IAM User. this script may be helpful who had face issues moving backups to S3 so it might be useful for cloud engineers.

⚙️ Connect with Me


Owner
Yousaf K Hamza
Junior DevOps Enginner
Yousaf K Hamza
AirDrive lets you store unlimited files to cloud for free. Upload & download files from your personal drive at any time using its super-fast API.

AirDrive lets you store unlimited files to cloud for free. Upload & download files from your personal drive at any time using its super-fast API.

Sougata 4 Jul 12, 2022
Algofi Python SDK is useful for developers who want to programatically interact with the Algofi lending protocol

algofi-py-sdk Algofi Python SDK Documentation https://algofi-py-sdk.readthedocs.

Algofi 41 Dec 15, 2022
Reddit bot that uses sentiment analysis

Reddit Bot Project 2: Neural Network Boogaloo Reddit bot that uses sentiment analysis from NLTK.VADER WIP_WIP_WIP_WIP_WIP_WIP Link to test subreddit:

TpK 1 Oct 24, 2021
Open Source Discord Account Creator

Alter Token Generator Open Source Discord Account Creator This program abuses the discord api and uses the 2Captcha captcha solving service to make di

24 Dec 13, 2022
This is a python bot that automatically logs in, clicks the new button, and sends heroes to work in the bombcrypto game

This is a python bot that automatically logs in, clicks the new button, and sends heroes to work in the bombcrypto game. It is fully open source and free.

856 Jan 04, 2023
"Nesse projeto criei uma automação para abrir as tarefas no Jira em massa pegando de uma determinada fila do Zendesk."

automacao-Zendesk "Nesse projeto criei uma automação para abrir as tarefas no Jira em massa pegando de uma determinada fila do Zendesk." en-us "In thi

tokoyamy 1 Dec 20, 2021
This is simple maker for level card in discord bot.

mariocard This is simple maker for level card in discord bot in discord.py or pycord. Installing Python 3.8 or higher is required # Linux/macOS pip3 i

3 Jan 29, 2022
Python API for British Geological Survey magnetic field calculator

Magnetic field calculator Python API for British Geological Survey magnetic field calculator. Description This project magnetic field calculator. It u

Filip Š 3 Mar 11, 2022
WebhookHub - A discord WebHook Manager with much more features coming soon

WebhookHub A discord WebHook Manager with much more features coming soon This is

5 Feb 19, 2022
Weather App using openweathermap API

This is my hobby project used to learn how to use public api for project.In this i used the api of openweathermap to featch the weather details of various city across the globe by giving city name as

Subramanya K S 1 Nov 06, 2021
Fix Twitter video embeds in Discord

TwitFix very basic flask server that fixes twitter embeds in discord by using youtube-dl to grab the direct link to the MP4 file and embeds the link t

Robin Universe 682 Dec 28, 2022
Volt is yet another discord api wrapper for Python. It supports python 3.8 +

Volt Volt is yet another discord api wrapper for Python. It supports python 3.8 + How to install [Currently Not Supported.] pip install volt.py Speed

Minjun Kim (Lapis0875) 11 Nov 21, 2022
gnosis safe tx builder

Ape Safe: Gnosis Safe tx builder Ape Safe allows you to iteratively build complex multi-step Gnosis Safe transactions and safely preview their side ef

228 Dec 22, 2022
A telegram bot to download mega.nz links. (made with pyrogram).

Mega Link Downloader Bot This is a telegram bot to download mega.nz links and return them as files/videos - Made by a 100% noob! (When I mean noob I r

171 Dec 27, 2022
Cloud-native, data onboarding architecture for the Google Cloud Public Datasets program

Public Datasets Pipelines Cloud-native, data pipeline architecture for onboarding datasets to the Google Cloud Public Datasets Program. Overview Requi

Google Cloud Platform 109 Dec 30, 2022
The smart farm is an idea that designing Smart Farm by IoT

The smart farm is an idea that designing Smart Farm by IoT. Using Raspberry Pi 4 detect the data from different sensors(Raindrop sensor and DHT22 sensor), and push the data to Azure IoT central.

Jiage 1 Jan 11, 2022
Support for Competitive Coding badges to add in Github readme or portfolio websites.

Support for Competitive Coding badges to add in Github readme or portfolio websites.

Akshat Aggarwal 2 Feb 14, 2022
Script to post multiple status(posts) on twitter

Script to post multiple status on twitter (i.e. TWITTER STORM) This program can post upto maximum limit of twitter(around 300 tweets) within seconds.

Sandeep Kumar 4 Sep 09, 2021
Automatically scrape all of your artifacts in Genshin Impact.

Genshin Artifact Scraper Automatically scrape all of your artifacts in Genshin Impact. Features: Simple recalibration (2 steps). GUI to select OCR reg

21 Dec 17, 2022
对hermit 的API进行简单的封装,做成了这个python moudle

hermit-py 对hermit 的API进行简单的封装,做成了这个Python Moudle,推荐通过wheel的方式安装。 目前对点击、滑动、模拟输入、找组件、等支持较好,支持查看页面的实时布局信息,再通过布局信息进行点击滑动等操作。 支持剪贴板相关的操作,支持设置剪贴的任意语言内容。

LookCos 40 Jun 25, 2022