Terraform wrapper to manage state across multiple cloud providers(AWS, GCP, and Azure)

Overview

Terraform Remote State Manager(tfremote)

tf is a python package for managing terraform remote state for: Google(Gcloud), AWS, and Azure. It sets a defined structure for all cloud providers by removing the overheard of configuring and managing the path in storage buckets.

It works with:

👉 Google Storage Bucket

👉 AWS S3

👉 Azure Storage

❗️ Note Best practice is to make sure buckets are versioned.

Install package

pip install tfremote --upgrade

Environment setup

  • Install Python 3.6+

  • Using virtualenv is strongly recommended:

python3 -m venv <venv name>

Default log level is WARNING, to change:

export TF_LOG_LEVEL to any of these: 'CRITICAL', 'ERROR', 'WARNING', 'INFO', 'DEBUG'

❗️ Important - Two variables are required for using tf package (used set creat path in remote storage):

  • teamid
  • prjid

Required variables can be defined using:

  • As inline variables e.g.: -var='teamid=demo-team' -var='prjid=demo-project'
  • Inside .tfvars file e.g.: -var-file=<tfvars file location>

Two optional variables:

workspace and state_key can be defined using:

  • -w=<workspace_name>. If no workspace is provided default workspace is used.

  • s=<state_key name>. If no key is provided terraform is used.

Path created in S3 backend: /<teamid>/<prjid>/<workspace>/<state-key>.tfstate

For more information refer to Terraform documentation

Setup environment variables

Workspace list file location TF_WORKSPACE_FILE_LOCATION

export TF_WORKSPACE_FILE_LOCATION=<workspace yml file location>

Reference file: link

AWS

❗️ Important - s3 bucket for remote state should reside in us-west-2

Set these env variables:

export TF_AWS_BUCKET=<your_remote_state_bucket_name>
export TF_AWS_BUCKET_REGION=us-west-2

One of below environment variable is required:

export TF_AWS_PROFILE=<aws profile to use>

or

export AWS_ACCESS_KEY_ID=<aws access key>
export AWS_SECRET_ACCESS_KEY=<aws secret access key>

Azure

To create storage for remote state there is handy script.

Run scripts/remote_state.sh (fill in the required information)

Set below env variables:

export TF_AZURE_STORAGE_ACCOUNT=<remote state storage account name>
export TF_AZURE_CONTAINER=<remote state container>
export ARM_ACCESS_KEY=<storage account access key>

GCP(gcloud)

https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform

Set below env variables:

export TF_GCLOUD_BUCKET=<remote state storage bucket name>
export TF_GCLOUD_CREDENTIALS=json credentials file path>

Usage

For GCP(gcloud):

tf plan -c=gcloud -var=teamid=demo-team -var=prjid=demo-app -w=demo-workspace

The structure in Google Storage Bucket:

alt text

For AWS:

tf plan -c=aws -var=teamid=demo-team -var=prjid=demo-app -w=demo-workspace

The structure in AWS S3:

alt text

If you need to specify state_key in S3, specify -s=tryme-key

For Azure:

tf plan -c=azure -var=teamid=demo-team -var=prjid=demo-app -w=demo-workspace

The structure in Azure Storage:

alt text

For more available options:

tf --help
usage: tf [-h] [-var] [-var-file] [-c] [-w] [-wp] [-s] [-no-color] [-json] [-out] [-f] [-nf] [-v]

Terraform remote state wrapper package
--------------------------------------
Usage: Set below env variables to begin (more information: https://github.com/tomarv2/tfremote):
TF_WORKSPACE_FILE_LOCATION
aws: TF_AWS_BUCKET, TF_AWS_BUCKET_REGION=us-west-2, TF_AWS_PROFILE or AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
azure: TF_AZURE_STORAGE_ACCOUNT, TF_AZURE_CONTAINER, ARM_ACCESS_KEY
gcloud: TF_GCLOUD_BUCKET, TF_GCLOUD_CREDENTIALS

optional arguments:
  -h, --help  show this help message and exit
  -var        Set Terraform configuration variable. This flag can be set multiple times
  -var-file   Set Terraform configuration variables from a file. This flag can be set multiple times
  -c          Specify cloud provider (default: 'aws'). Supported values: gcloud, aws, or azure
  -w          Specify existing workspace name(default: 'default')
  -wp         Overwrite workspace directory path structure
  -s          File name in remote state (default: 'terraform.tfstate')
  -no-color   Disables terminal formatting sequences in the output
  -json       Enables the machine readable JSON UI output
  -out        Writes the generated plan to the given filename in an opaque file format
  -f          Enable FIPS endpoints (default: True)
  -nf         Disable FIPS endpoints
  -v          show program's version number and exit
You might also like...
RichWatch is wrapper around AWS Cloud Watch to display beautiful logs with help of Python library Rich.
RichWatch is wrapper around AWS Cloud Watch to display beautiful logs with help of Python library Rich.

RichWatch is TUI (Textual User Interface) for AWS Cloud Watch. It formats and pretty prints Cloud Watch's logs so they are much more readable. Because

Bot made with Microsoft Azure' cloud service
Bot made with Microsoft Azure' cloud service

IttenWearBot Autori: Antonio Zizzari Simone Giglio IttenWearBot è un bot intelligente dotato di sofisticate tecniche di machile learning che aiuta gli

This repository is used to simplify the process of cloning the SSM documents across the AWS regions.

SSM Cloner Introduction This module is created in order to simplify the process of copying the SSM documents from one region to another regions. As an

tfquery: Run SQL queries on your Terraform infrastructure.  Query resources and analyze its configuration using a SQL-powered framework.
tfquery: Run SQL queries on your Terraform infrastructure. Query resources and analyze its configuration using a SQL-powered framework.

🌩️ tfquery 🌩️ Run SQL queries on your Terraform infrastructure. Ask questions that are hard to answer 🚀 What is tfquery? tfquery is a framework tha

Terraform module to ship CloudTrail logs stored in a S3 bucket into a Kinesis stream for further processing and real-time analysis.
Terraform module to ship CloudTrail logs stored in a S3 bucket into a Kinesis stream for further processing and real-time analysis.

AWS infrastructure to ship CloudTrail logs from S3 to Kinesis This repository contains a Terraform module to ship CloudTrail logs stored in a S3 bucke

Cookies is a project inspired by python cookiecutter but used for terraform generation.

Introduction Cookies is a project inspired by python cookiecutter but used for terraform generation. How to run your terraform After you download your

Python binding for Terraform.

Python libterraform Python binding for Terraform. Installation $ pip install libterraform NOTE Please install version 0.3.1 or above, which solves the

Compares and analyzes GCP IAM roles.

gcp-iam-analyzer I wrote this to help in my day to day working in GCP. A lot of the time I am doing role comparisons to see which role has more permis

Periodically check the manuscript state in the scholar one system and send email when finding a new state.

ScholarOne-manuscript-checker Periodically check the manuscript state in the scholar one system and send email when finding a new state. Parameters ne

Releases(v0.0.4)
Owner
tomarv2
Sr. DevOps/DataOps Engineer
tomarv2
GitHub Actions Poll Mode AutoScaler (GAPMAS)

GitHub Actions Poll Mode AutoScaler, or GAPMAS, is a simple tool that helps you run ephemeral GitHub Actions self-hosted runners on your own infrastructure.

Frode Nordahl 4 Nov 04, 2022
KaydyPurge - Python Purge Script for Discord made by Kaydy Cain#0001

How to Install Open terminal Execute "git clone https://github.com/apolo1337/Kay

apolo 5 Jan 27, 2022
Python Telegram Bot Template

Python Telegram Bot Template Concepts Customizable python-telegram-bot template implementing code refactoring to streamline development process: handl

Jansen A. Simanullang 1 Jan 25, 2022
fair-test is a library to build and deploy FAIR metrics tests APIs supporting the specifications used by the FAIRMetrics working group.

☑️ FAIR test fair-test is a library to build and deploy FAIR metrics tests APIs supporting the specifications used by the FAIRMetrics working group. I

Maastricht University IDS 6 Oct 30, 2022
MAASTA is a wrapper to create an Ansible inventory for MAAS instances that are provisioned by Terraform.

MAASTA is a wrapper to create an Ansible inventory for MAAS instances that are provisioned by Terraform.

Saeid Bostandoust 144 Dec 16, 2022
iCloudPy is a simple iCloud webservices wrapper library written in Python

iCloudPy 🤟 Please star this repository if you end up using the library. It will help me continue supporting this product. 🙏 iCloudPy is a simple iCl

Mandar Patil 49 Dec 26, 2022
The official Discord Python framework for thenewboston blockchain.

Project Setup Follow the steps below to set up the project on your environment. Mac Setup Homebrew requires the Xcode command-line tools from Apple's

Bucky Roberts 18 Jul 15, 2022
A Telegram Music Bot written in Python using Pyrogram and Py-Tgcalls

A Telegram Music Bot written in Python using Pyrogram and Py-Tgcalls. This is Also The Source Code of The UserBot Which is Playing Music in @S1-BOTS Support Group ❤️

SAF ONE 224 Dec 28, 2022
DEPRECATED - Official Python Client for the Discogs API

⚠️ DEPRECATED This repository is no longer maintained. You can still use a REST client like Requests or other third-party Python library to access the

Discogs 483 Dec 31, 2022
Flaga ze Szturmu na AWS.

Witaj Jesteś na GitHub'ie i czytasz właśnie plik README.md który znajduje się wewnątrz repozytorium Flaga z 7 i 8 etapu Szturmu na AWS. W tym etapie w

9 May 16, 2022
Scrape Twitter for Tweets

Backers Thank you to all our backers! 🙏 [Become a backer] Sponsors Support this project by becoming a sponsor. Your logo will show up here with a lin

Ahmet Taspinar 2.2k Jan 02, 2023
gnosis safe tx builder

Ape Safe: Gnosis Safe tx builder Ape Safe allows you to iteratively build complex multi-step Gnosis Safe transactions and safely preview their side ef

228 Dec 22, 2022
VaccineAlarm is a simple python script that allows user to get notified when their desired vaccine doses are available at vaccine centers near them.

Introduction VaccineAlarm is a simple python script that allows user to get notified when their desired vaccine doses are available at vaccine centers

Ankit Tripathi 5 Nov 26, 2021
ANKIT-OS/STYLISH-TEXT is a special repository. Its Is A Telegram Bot Which Can Translate Your Text Into 100+ Language

🔥 ᴳᴼᴼᴳᴸᴱ⁻ᵀᴿᴬᴺᔆᴸᴬᵀᴱᴿ 🔥 The owner would not be responsible for any kind of bans due to the bot. • ⚡ INSTALLING ⚡ • • 🛠️ Lᴀɴɢᴜᴀɢᴇs Aɴᴅ Tᴏᴏʟs 🔰 • If

ANKIT KUMAR 1 Dec 23, 2021
An Auto-Grinding bot made for Pokemeow. Efficient but not many features yet

PokeGrinder 🤖 This is an Auto-Grinding bot made for Pokemeow. Efficient but not many features yet. Supported features This bot can currently handle :

Xombie 9 Feb 01, 2022
Torrent-Igruha SDK Python

Простой пример использования библиотеки: Устанавливаем библиотеку python -m

LORD_CODE 2 Jun 25, 2022
SQS + Lambda를 활용한 문자 메시지 및 이메일, Voice call 호출을 간단하게 구현하는 serverless 템플릿

AWS SQS With Lambda notification 서버 구축을 위한 Poc TODO serverless를 통해 sqs 관련 리소스(람다, sqs) 배포 가능한 템플릿 작성 및 배포 poc차원에서 간단한 rest api 호출을 통한 sqs fifo 큐에 메시지

김세환 4 Aug 08, 2021
Script to get a notification when a product, on Amazon Warehouse, is available within a target price

Amazon_Warehouse_Scraping This script aims to scrape Amazon Warehouse and send an email back if there are products whose price matches with the target

2 Oct 25, 2021
A customizable, multilanguage Telegram shop bot with Telegram Payments support

Greed A customizable, multilanguage Telegram shop bot with Telegram Payments support! Demo Send a message to @greedtestbot on Telegram to view a demo

Stefano Pigozzi 328 Dec 29, 2022
Um painel de consultas completo, com metodos atualizados.

Meu pix para eu comprar um café :D "25ef499b-d184-4aa1-9797-0a294be40d83" Painel-de-Consultas Completo. Feito por JOESTAR-TEAM Painel de consultas Com

Dio brando 10 Nov 19, 2021