metamaker
Simple command line tool to train and deploy your machine learning models with AWS SageMaker
Features
metamaker enables you to:
- Build a docker image for training and inference with poetry and FastAPI
- Train your own machine learning model with SageMaker
- Deploy inference endpoint with SageMaker
Usage
- Create poetry project and install metamaker
❯ poetry new your_module
❯ cd your_module
❯ poetry add metamaker
- Define scripts for traning and inference in
main.py
Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data) ">
from pathlib import Path from typing import Any, Dict from metamaker import MetaMaker # Import your model, and input/output data classs: # # Model ... machine learning model class you want to use # Input ... input data class for inference # Output ... ouput data class for inference # # Note that the Input and Output are used as type hints to # create API endpoint with FastAPI like below: # # @fastapi_app.post("/invocations") # def predict(data: Input) -> Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data)
- Write metamaker configs in
metamaker.yaml
# Specify metamaker handler like: `path.to.module:app_name`
handler: main:app
# dataset_path and artifact_path should be directories and end with '/'
dataset_path: s3://your-bucket/path/to/dataset/
artifact_path: s3://your-bucket/path/to/artifacts/
hyperparameter_path: ./hparams.yaml
image:
name: metamaker
includes:
- your_module/
- main.py
excludes:
- __pycache__/
- '*.py[cod]'
training:
execution_role: arn:aws:iam::xxxxxxxxxxxx:role/SageMakerExecutionRole
instance:
type: ml.m5.large
count: 1
inference:
endpoint_name: your_endpoint
instance:
type: ml.t2.meduim
count: 1
- Build docker image and push to ECR
metamaker build --deploy .
- Train your model with SageMaker and deploy endpoint
metamker sagemaker train --deploy
2 Sep 06, 2022
144 Dec 22, 2022
164 Dec 14, 2022
2 Jul 22, 2022
30 Sep 08, 2022
194 Dec 29, 2022
3 Dec 14, 2022
1 Feb 02, 2022
2.5k Dec 28, 2022
4 Dec 09, 2022
69 Oct 24, 2022
22 Nov 10, 2022
249 Dec 23, 2022
1.7k Jan 02, 2023
572 Dec 28, 2022
5.9k Jan 04, 2023
305 Dec 30, 2022
4 Apr 06, 2022
3 May 28, 2022
4 Nov 14, 2022