Browse the Index

Amazon's Simple Storage Service and the broader ecosystem of S3-compatible object storage. The root concept of this entire index.

The storage paradigm of flat-namespace, HTTP-accessible binary objects with metadata. Data is addressed by bucket and key, not by filesystem path.

The convergence of data lake storage (raw files on object storage) with data warehouse capabilities — ACID transactions, schema enforcement, SQL acces...

The pattern of storing raw, heterogeneous data in object storage for later processing. Data arrives in its original form and is transformed downstream...

The category of specifications (Iceberg, Delta, Hudi) that bring table semantics — schema, partitioning, ACID transactions, time-travel — to collectio...

The practice of building and querying vector indexes over embeddings derived from data stored in S3.

The intersection of large language models and S3-centric data infrastructure. Scoped strictly to cases where LLMs operate on, enhance, or derive value...

The discipline of maintaining catalogs, schemas, statistics, and descriptive information about objects and datasets stored in S3.

Techniques for tracking and managing changes to datasets stored in object storage over time, including snapshots, branching, and rollback.

Amazon's fully managed object storage service — the origin and reference implementation of the S3 API.

An open-source, S3-compatible object storage server designed for high performance and self-hosted deployment.

A distributed storage system providing object, block, and file storage in a unified platform. S3 compatibility via its RADOS Gateway (RGW).

A scalable, distributed object storage system in the Hadoop ecosystem with an S3-compatible interface.

An open table format for large analytic datasets. Manages metadata, snapshots, and schema evolution for collections of data files (typically Parquet) ...

An open table format and storage layer providing ACID transactions, scalable metadata, and schema enforcement on data stored in object storage. Origin...

A table format and data management framework optimized for incremental data processing — upserts, deletes, and change data capture — on object storage...

An in-process analytical database engine (like SQLite for analytics) that reads Parquet, Iceberg, and other formats directly from S3 without requiring...

A distributed SQL query engine for federated analytics across heterogeneous data sources, with deep support for S3-backed data lakes and lakehouses.

A column-oriented DBMS designed for real-time analytical queries, with native support for reading from and writing to S3.

A distributed compute engine for large-scale data processing — batch ETL, streaming, SQL, and machine learning — over S3-stored data.

A vector database that stores data in the Lance columnar format directly on object storage. Designed for serverless vector search without a separate i...

An MPP analytical database with native lakehouse capabilities, able to directly query S3 data in Parquet, ORC, and Iceberg formats.

A distributed stream processing framework that processes data in real-time, with S3 as checkpoint store, state backend, and output sink.

The HTTP-based API for object storage operations — PUT, GET, DELETE, LIST, multipart upload. The de-facto standard for object storage interoperability...

A columnar file format specification designed for efficient analytical queries. Stores data by column, enabling predicate pushdown, projection pruning...

A cross-language in-memory columnar data format specification with libraries for zero-copy reads, IPC, and efficient analytics.

The specification defining how a logical table is represented as metadata files, manifest lists, manifests, and data files on object storage. Provides...

The specification for ACID transaction logs over Parquet files on object storage. Defines how writes, deletes, and schema changes are recorded in a JS...

The specification for managing incremental data processing on object storage — record-level upserts, deletes, change logs, and timeline-based metadata...

Optimized Row Columnar file format specification — a columnar format with built-in indexing, compression, and predicate pushdown support, originally d...

A row-based data serialization format with rich schema definition and built-in schema evolution support. Schemas are stored with the data.

A unified architecture combining data lake storage (files on S3) with warehouse capabilities (ACID, schema enforcement, SQL access) by using a table f...

A layered data quality pattern — Bronze (raw), Silver (cleansed), Gold (business-ready) — with each layer stored on object storage.

The design pattern of keeping data in S3 while running independent, elastically scaled compute engines against it.

A pattern that stores raw data on S3 and maintains a vector index over embeddings that points back to S3 objects.

A batch pattern where embeddings are generated from S3-stored data on a schedule, with resulting vectors written back to object storage or a vector in...

A pattern of running ML/LLM models on local hardware against data stored in or pulled from S3, avoiding cloud-based inference APIs.

A data quality pattern where data lands in a raw S3 zone, undergoes validation, and is promoted to a curated zone only after passing audits.

Moving data between hot, warm, and cold storage tiers based on access frequency. S3 itself offers tiering (Standard, Infrequent Access, Glacier).

Too many small objects in S3 degrade query performance and increase API call costs. Each file requires a separate GET request, and S3 charges per-requ...

Slow first-query performance against S3-stored data, caused by object discovery, metadata fetching, and data transfer over HTTP.

Changing data schemas (adding columns, renaming fields, altering types) in S3-stored datasets without breaking downstream consumers.

Older ETL systems designed for HDFS or traditional databases that cannot efficiently write to modern S3-based lakehouse architectures.

The expense of running LLM/ML inference via cloud APIs (per-token or per-request pricing) against S3 data at scale.

The slowness and cost of listing large numbers of objects in S3's flat namespace using prefix-based scans. Paginated at 1,000 objects per request.

Table format metadata (manifests, snapshots, statistics) grows as S3 datasets grow, eventually slowing planning, compaction, and garbage collection.

The difficulty of efficiently skipping irrelevant S3 objects during queries. Requires careful partitioning strategy, predicate pushdown, and metadata ...

Dependence on a single S3 provider's proprietary features, pricing, or integrations that makes migration difficult.

The cost charged by cloud providers for data transferred out of their S3 service — to the internet, another region, or another cloud.

The differences in consistency guarantees across S3-compatible storage providers. AWS S3 is now strongly consistent; other providers may differ.

The S3 API has no atomic rename operation. Renaming requires copy-then-delete — a two-step, non-atomic process.

A class of model that converts unstructured data (text, images, audio) into fixed-dimensional vector representations suitable for similarity search.

A large language model for broad text tasks. In scope when applied to metadata extraction, summarization, schema inference, or querying of S3-stored c...

An LLM specialized for code understanding and generation. A subtype of General-Purpose LLM with enhanced ability to work with structured and semi-stru...

A compact model (typically under 10B parameters) suitable for local or edge deployment, often distilled from a larger model to retain key capabilities...

Converting unstructured content stored in S3 (documents, images, logs) into vector representations for similarity search.

Querying S3-derived vector embeddings to find content by meaning rather than exact keyword match.

Using LLMs to extract structured metadata (entities, categories, summaries, key-value pairs) from unstructured objects stored in S3.

Using LLMs to infer or suggest schemas from semi-structured data (JSON, CSV, nested formats) stored in S3.

Using LLMs to categorize, tag, or label S3-stored objects based on content analysis — by topic, sensitivity level, or compliance category.

Using LLMs to translate natural language questions into executable queries (SQL, API calls) over S3-backed datasets.