Upload files or folders (even with subfolders) to Amazon S3 in a totally automatized way taking advantage of:
-
Amazon S3 Multipart Upload: The uploaded files are processed transparently in parts improving the throughput and the quick recovery from any network issues.
-
Resilent Retry System: Intelligent S3 Upload has been built to detect any error during the uploading process and to perform any retries whenever is necessary.
-
User Friendly Interface: Just check the demo to see with your own eyes how the upload process is performed.
Installation
Clone the repository
git clone https://github.com/polius/intelligent-s3-upload.git
Install the dependencies
python3 -m pip install boto3 --user
python3 -m pip install requests --user
Setup
Before executing the Intelligent S3 Upload, modify the credentials.json file.
{
"aws_access_key_id": "",
"aws_secret_access_key": "",
"region_name": "",
"bucket_name": "",
"bucket_prefix": "",
"storage_class": "",
"skip_s3_existing_files": true,
"server_side_encryption": true,
"slack_url": ""
}
- aws_access_key_id | aws_secret_access_key: Credentials generated by Amazon IAM.
- region_name: The AWS Region Code where the bucket is located.
- bucket_name: The bucket name created by Amazon S3.
- bucket_path: (Optional) The bucket path to store the uploaded objects.
- storage_class: The type of storage to use for the uploaded object. These are the possible values:
| storage_class |
|---|
| STANDARD |
| REDUCED_REDUNDANCY |
| STANDARD_IA |
| ONEZONE_IA |
| INTELLIGENT_TIERING |
| GLACIER |
| DEEP_ARCHIVE |
| OUTPOSTS |
- skip_s3_existing_files: Skip uploading objects if these already exists in S3. Possible values: [ true | false ]
- server_side_encryption: Enable Server-side encryption using the Amazon S3 key (SSE-S3). Possible values: [ true | false ]
- slack_url: (Optional) Enter a Webhook URL to send a message to Slack when a upload finishes.
AWS Policy
To be able to run the script check that your policy meets the following requirements:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucketMultipartUploads",
"s3:AbortMultipartUpload",
"s3:ListBucket",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::yourbucket/*",
"arn:aws:s3:::yourbucket"
]
}
]
}
Execution
python3 upload.py --path "{PATH}"
Replace the {PATH} string with the absolute file/folder path.
License
This project is licensed under the MIT license. See the LICENSE file for more info.
1 Dec 13, 2021
428 Jan 8, 2023
71 Jul 19, 2022
1 Nov 6, 2021
0 Dec 18, 2021
4 Dec 10, 2021
240 Jan 6, 2023
219 Oct 3, 2022
59 Nov 27, 2022
55 Dec 12, 2022
16 Nov 01, 2022
45.3k Jan 02, 2023
37 Oct 29, 2022
1 Jan 31, 2022
36 Dec 23, 2022
69 Dec 26, 2022
1 Nov 22, 2021
1 Jan 28, 2022
5 Sep 21, 2022
228 Dec 22, 2022
81 Dec 26, 2022
25 Oct 31, 2022
25 Dec 13, 2022
1 Sep 27, 2022
32 Oct 01, 2022
0 Jan 19, 2022
229 Jan 07, 2023
26 May 01, 2022
1 Nov 01, 2021