metamaker
Simple command line tool to train and deploy your machine learning models with AWS SageMaker
Features
metamaker enables you to:
- Build a docker image for training and inference with poetry and FastAPI
- Train your own machine learning model with SageMaker
- Deploy inference endpoint with SageMaker
Usage
- Create poetry project and install metamaker
❯ poetry new your_module
❯ cd your_module
❯ poetry add metamaker
- Define scripts for traning and inference in
main.py
Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data) ">
from pathlib import Path from typing import Any, Dict from metamaker import MetaMaker # Import your model, and input/output data classs: # # Model ... machine learning model class you want to use # Input ... input data class for inference # Output ... ouput data class for inference # # Note that the Input and Output are used as type hints to # create API endpoint with FastAPI like below: # # @fastapi_app.post("/invocations") # def predict(data: Input) -> Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data)
- Write metamaker configs in
metamaker.yaml
# Specify metamaker handler like: `path.to.module:app_name`
handler: main:app
# dataset_path and artifact_path should be directories and end with '/'
dataset_path: s3://your-bucket/path/to/dataset/
artifact_path: s3://your-bucket/path/to/artifacts/
hyperparameter_path: ./hparams.yaml
image:
name: metamaker
includes:
- your_module/
- main.py
excludes:
- __pycache__/
- '*.py[cod]'
training:
execution_role: arn:aws:iam::xxxxxxxxxxxx:role/SageMakerExecutionRole
instance:
type: ml.m5.large
count: 1
inference:
endpoint_name: your_endpoint
instance:
type: ml.t2.meduim
count: 1
- Build docker image and push to ECR
metamaker build --deploy .
- Train your model with SageMaker and deploy endpoint
metamker sagemaker train --deploy
2 Dec 10, 2021
20 Dec 22, 2022
249 Dec 23, 2022
11.1k Dec 27, 2022
1 Dec 26, 2021
1 Oct 23, 2021
6 Nov 13, 2022
1 Jan 03, 2022
4 Feb 15, 2022
5 Nov 26, 2022
1 Aug 27, 2022
4 Nov 30, 2022
5 Jan 30, 2022
5 Nov 18, 2022
![[Ͼ⁴] Ƶephyr](https://avatars.githubusercontent.com/u/76829032?v=4&s=60)
2 Dec 02, 2021
1k Jan 07, 2023
1 Jan 06, 2022
405 Jan 03, 2023
69 Oct 24, 2022
131 Dec 25, 2022