Schema Inference
Using LLMs to infer or suggest schemas from semi-structured data (JSON, CSV, nested formats) stored in S3.
Summary
Using LLMs to infer or suggest schemas from semi-structured data (JSON, CSV, nested formats) stored in S3.
Schema inference automates the tedious process of determining what fields, types, and structures exist in S3 data. It accelerates onboarding new datasets and proposes schema evolution changes for existing tables.
- LLM-inferred schemas are suggestions, not ground truth. Always validate against actual data samples before applying to production tables.
- Sampling matters. Schema inference from a small sample may miss rare fields or variant types that appear in the full dataset.
depends_onGeneral-Purpose LLM — requires language understanding for schema analysissolvesSchema Evolution — automates schema change proposalsaugmentsApache Iceberg — can suggest schema changes for Iceberg tablesscoped_toLLM-Assisted Data Systems, Table Formats
Definition
Using LLMs to infer, suggest, or validate schemas from semi-structured data (JSON, CSV with inconsistent headers, nested formats) stored in S3.
Semi-structured data arriving in S3 often has no declared schema. Manually inspecting files to determine field names, types, and nesting is tedious and error-prone. LLMs can analyze sample data and propose schemas automatically.
Recent developments
- SI-LLM paper formalizes LLM-driven hierarchical schema inference. Uses LLMs to infer hierarchical entity types, attributes, and inter-type relationships across tabular-data repositories — achieving state-of-the-art or comparable results to prior structured-inference methods on standard benchmarks. First credible academic frame for "LLM as schema-inference engine." Per arXiv 2509.04632 — Schema Inference for Tabular Data Using LLMs.
- CSV schema inference is a silent performance killer in production. Inferring an accurate CSV schema requires scanning the full file (since later rows may introduce new types or null patterns); Parquet/Avro self-contain metadata headers and avoid this cost. 2026 production guidance: declare schemas explicitly for CSV ingestion; rely on inference only at dev/exploration time. Per Miles Cole — Should You Infer Schema in Production?.
- Databricks Auto Loader supports schema inference + evolution.
cloudFiles.inferColumnTypes+cloudFiles.schemaEvolutionModeflags handle CSV/JSON arriving in S3 with shifting structure — additive evolution, rescue-data column, and explicit failure modes for breaking changes. Per Databricks — Auto Loader Schema Inference and Evolution. - ClickHouse supports CSV/Parquet schema inference with type-detection toggles. Including the
--input_format_csv_use_best_effort_in_schema_inferenceandtreat all columns as stringsflags for messy production inputs — a common 2026 pattern when LLM-generated CSVs land in ingestion. Per ClickHouse — Automatic Schema Inference from Input Data. - Spark + Synapse + Foundry all expose explicit inference primitives. Spark SQL auto-discovers partitioning + Parquet schemas; Synapse serverless reads Parquet metadata at query time; Palantir Foundry exposes a UI "Infer schema for CSV/JSON" pipeline step. Schema inference is now a first-class data-platform primitive, not an ETL afterthought. Per Spark — Parquet Files docs and Palantir — Infer schema for CSV/JSON.
- LLM-vs-engine inference handoff is the new architecture pattern. Engine inference (Spark/ClickHouse/DuckDB) wins on type accuracy + cost for self-describing formats (Parquet, Avro). LLM inference wins on weakly-structured data (nested JSON, semi-structured logs, semi-tabular spreadsheets) where it can name fields semantically + infer hierarchical structure. Production stacks now route by input format. Per arXiv — SI-LLM + Miles Cole — Schema Inference in Production.
Connections 7
Outbound 5
scoped_to2depends_on1solves1augments1Inbound 2
Resources 3
Databricks Auto Loader documentation for automated schema inference from S3-stored files, sampling up to 50 GB to detect column types and handle schema evolution.
Official AWS Glue documentation for schema inference from S3 data sources, covering automatic type detection, crawler-based discovery, and Glue Studio's Infer Schema feature.
Research paper on using LLMs for schema inference on tabular data repositories, inferring entity types, attributes, and relationships from column headers and cell values.