Architecture

Checkpoint/Artifact Lake on Object Storage

Using S3 as the durable repository for ML model checkpoints, trained model artifacts, training logs, and experiment metadata. A centralized, versioned artifact store on object storage.

3 connections 3 resources

Summary

What it is

Using S3 as the durable repository for ML model checkpoints, trained model artifacts, training logs, and experiment metadata. A centralized, versioned artifact store on object storage.

Where it fits

ML training produces large, versioned artifacts (checkpoints can be tens of GB each). S3 provides the scalable, durable storage that keeps these artifacts accessible across experiments, teams, and clusters — serving as the "source of truth" for model lineage.

Misconceptions / Traps
  • Checkpoint frequency has a direct cost impact. Frequent checkpointing (every N steps) generates significant storage volume. Implement retention policies to garbage-collect old checkpoints.
  • S3 write latency affects training throughput if checkpointing is synchronous. Use asynchronous checkpoint uploads to avoid GPU idle time during saves.
Key Connections
  • scoped_to Object Storage for AI Data Pipelines — ML artifact management
  • depends_on S3 API — artifacts stored in S3
  • constrained_by Egress Cost — downloading checkpoints across regions/clouds is expensive

Definition

What it is

Using S3 as the durable, versioned repository for ML training checkpoints, model weights, pipeline artifacts, and experiment metadata — with lifecycle policies for retention and cost management.

Why it exists

ML training produces frequent checkpoints (every N steps) and final model artifacts. These must be durable, versioned, and shareable across teams. S3 provides cheap, durable, HTTP-accessible storage with versioning, making it the natural checkpoint and artifact repository.

Primary use cases

ML training checkpoint storage, model registry artifact storage, experiment tracking metadata, pipeline artifact management.

Connections 3

Outbound 3
scoped_to2
S3Object Storage for AI Data Pipelines
depends_on1
S3 API

Resources 3

DocsHigh
docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints.ht...

SageMaker documentation on saving model checkpoints to S3 during training for fault tolerance and resume.

DocsHigh
pytorch.org/docs/stable/checkpoint.html

PyTorch checkpointing documentation covering state dict serialization patterns used with S3-backed artifact stores.

DocsHigh
mlflow.org/docs/latest/tracking/artifacts-stores.html

MLflow artifact stores documentation showing how to persist ML artifacts, models, and metadata to S3.