Architecture

Offline Embedding Pipeline

A batch pattern where embeddings are generated from S3-stored data on a schedule, with resulting vectors written back to object storage or a vector index.

4 connections 3 resources

Summary

What it is

A batch pattern where embeddings are generated from S3-stored data on a schedule, with resulting vectors written back to object storage or a vector index.

Where it fits

This pattern is the cost-effective way to add semantic search to S3 data. Instead of real-time embedding on every query, data is vectorized in batch — keeping inference costs predictable and avoiding always-on GPU infrastructure.

Misconceptions / Traps
  • "Offline" means batch, not "never updated." A daily or weekly refresh is typical. Freshness requirements determine the schedule.
  • Embedding pipeline failures can leave the vector index out of sync with S3 data. Idempotent, resumable pipelines are essential.
Key Connections
  • depends_on S3 API — reads source data from and writes embeddings to S3
  • constrained_by High Cloud Inference Cost — the motivating economic constraint
  • scoped_to LLM-Assisted Data Systems, S3

Definition

What it is

A batch pattern where embeddings are generated from S3-stored data on a schedule, and the resulting vectors are written back to object storage or a vector index.

Why it exists

Real-time embedding generation is expensive and unnecessary for many use cases. Processing S3 data in batch keeps inference costs predictable and avoids the need for always-on GPU infrastructure.

Primary use cases

Periodic embedding refresh for document corpora on S3, bulk vectorization of historical data, populating vector indexes for RAG systems.

Recent developments

Latest signals
  • Batch embedding speedup: 3-5× vs one-at-a-time on the same hardware. When ingesting documents, batch embedding calls speed up a 500-chunk corpus by 3-5× compared to processing documents one by one — the canonical optimization for the offline pipeline pattern. Per SGLang — Offline batch processing.
  • SGLang's offline Engine for in-process batch work. SGLang is the high-throughput serving framework for HF-style models — offers OpenAI-compatible HTTP APIs AND an offline Engine for in-process batch embeddings. The batch-engine architecture eliminates the HTTP round-trip overhead per chunk. Per Glukhov — LLM Hosting in 2026.
  • Mixpeek = unified multimodal batch pipeline. Self-hosted solution generating text + image + video + audio embeddings in a unified pipeline with automatic batching — eliminates the need to run + orchestrate separate model servers per modality. Per Mixpeek — Best Self-Hosted Embedding Models 2026.
  • Memory-bandwidth bound, not compute bound — choose GPUs accordingly. Embedding workloads are memory-bandwidth-bound (not compute-bound) — the A100 80GB PCIe with ~1.94TB/s memory bandwidth + 80GB VRAM is an excellent choice for large batch caching. Per Spheron — Self-Host Embeddings and Rerankers: TEI on GPU Cloud 2026.
  • llama.cpp --batch-size 512 increases throughput. For llama.cpp-based offline embedding pipelines, increasing batch size with the --batch-size flag is the simplest path to higher throughput — the default batch is undersized for offline-batch use cases. Per Daily.dev — Running LLMs Locally with Ollama, llama.cpp.
  • LM Studio supports GGUF embedding models via OpenAI-compatible /v1/embeddings. Local-first embedding generation via LM Studio + GGUF format (e.g. nomic-embed-text-v1.5); exposes an OpenAI-compatible endpoint so application code stays portable between local + hosted backends. Per Hakuna Matata Tech — LM Studio Text Embedding.
  • CPU-only deployment is viable for small/quantized models + offline batch. For embedding workloads with smaller models, heavy quantization, or low-traffic internal tools, CPU-only deployment works — but requires careful expectation-setting on throughput. Per Alex Genovese — Self-Hosted LLM Stack Practical Guide.

Connections 4

Outbound 4
scoped_to2
LLM-Assisted Data SystemsS3
depends_on1
S3 API
constrained_by1
High Cloud Inference Cost

Resources 3

BlogHigh
aws.amazon.com/blogs/big-data/generate-vector-embeddings-for...

AWS Big Data Blog showing how to build a batch embedding pipeline that reads from S3, generates vectors via Lambda, and ingests into OpenSearch.

GitHubHigh
github.com/aws-samples/text-embeddings-pipeline-for-rag

Official AWS sample repository providing a complete pipeline to convert documents stored in S3 into text embeddings for RAG applications.

BlogMedium
blog.skypilot.co/large-scale-embedding/

SkyPilot engineering blog demonstrating 9x faster embedding generation at scale across cloud GPUs for 30M+ records, with S3 as the source/sink.