Hybrid S3 + Vector Index
A pattern that stores raw data on S3 and maintains a vector index over embeddings that points back to S3 objects.
Summary
A pattern that stores raw data on S3 and maintains a vector index over embeddings that points back to S3 objects.
This pattern bridges structured storage (S3) with semantic retrieval (vector search). It is the architecture behind RAG systems that ground LLM responses in S3-stored documents.
- The vector index and the raw data can drift. If S3 objects are updated or deleted without updating the index, search results return stale or broken references.
- Hybrid does not mean "query both simultaneously." Typically, vector search retrieves references first, then the application fetches the raw data from S3 in a second step.
depends_onS3 API — raw data stored in S3solvesCold Scan Latency — pre-computed embeddings avoid scanning raw contentconstrained_byHigh Cloud Inference Cost — generating embeddings is expensive- LanceDB
implementsHybrid S3 + Vector Index - Embedding Generation, Semantic Search
enablesHybrid S3 + Vector Index scoped_toVector Indexing on Object Storage, S3
Definition
A pattern that stores raw data (documents, media, logs) on S3 and maintains a vector index (embeddings + similarity search) that points back to the S3 objects.
S3 is excellent for durable, cheap storage of unstructured content, but it has no concept of semantic similarity. A vector index adds a semantic retrieval layer without duplicating the raw data.
Retrieval-augmented generation (RAG) over S3-stored corpora, semantic document search, content recommendation systems backed by S3 data.
Recent developments
- Amazon S3 Vectors reached GA (December 2025) — 40× the scale of preview. Native vector support directly inside S3 — no separate vector database service required for the storage layer. GA expanded availability from 5 to 14 AWS regions. Per AWS — Amazon S3 Vectors generally available and AWS — S3 Vectors GA with increased scale.
- 2 billion vectors per index; 1,000 vectors/sec write throughput per index. GA-tier capacity is sufficient for large enterprise RAG corpora. 50 metadata keys per vector for fine-grained filtering — closes the metadata-filter gap that pushed early users back to Pinecone/Weaviate. Per AWS — Amazon S3 Vectors product page.
- "Storage-First" architecture decouples compute from storage — up to 90% TCO reduction for large RAG. S3 Vectors' core pitch: traditional vector databases keep all indexes in RAM (expensive); S3 Vectors persists indexes in object storage and pages compute only when queried. Per InfoQ — Amazon S3 Vectors Reaches GA with Storage-First Architecture.
- Hybrid tiered pattern: S3 Vectors (cold) + OpenSearch (hot). Standard 2026 pattern: infrequently-queried embeddings live in S3 Vectors; OpenSearch auto-promotes vectors that get hot. OpenSearch ships the auto-management primitive natively. Per Caylent — Architecting GenAI at Scale: AWS S3 Vector Store + Hybrid Vector Storage.
- Pinecone, Weaviate, Milvus all face cost-disruption pressure. S3 Vectors' price point (storage-first vs RAM-first) puts margin pressure on the dedicated vector-DB cohort. 2026 narrative shift: dedicated vector DBs win on latency + throughput; S3 Vectors wins on cost + scale + integration. Per DEV — AWS S3 Vectors: Finally, Cloud Scalable Vector Storage.
- Hybrid metadata-filter retrieval is the dominant query shape. Up to 100 results per query + 50 metadata keys per vector enables filter-then-vector-search patterns (categories, dates, user preferences, tags). The "vector search + metadata filter" shape replaces pure-vector-search as the production default. Per Medium — What Is Amazon S3 Vectors? Use Cases When to Use It Cost.
Connections 12
Outbound 5
scoped_to2depends_on1solves1constrained_by1Resources 3
AWS Architecture Blog describing a production-grade 1B+ vector search solution built on LanceDB with S3 as the storage layer, demonstrating the hybrid pattern at scale.
Official Milvus documentation for configuring S3 as the object storage backend for vector data and index persistence.
LanceDB's official example of running a serverless vector database directly on S3 with AWS Lambda.