Synthetic Training Data Generation
Generating fine-tuning corpora with a teacher model over your own documents — the self-instruct/distillation pattern — then filtering hard and training a student on the survivors.
Summary
Generating fine-tuning corpora with a teacher model over your own documents — the self-instruct/distillation pattern — then filtering hard and training a student on the survivors.
It is the data-supply side of fine-tune-own-data: PEFT and the trainer frameworks (TRL, Axolotl, LLaMA-Factory) consume instruction/preference datasets, and synthetic generation is how those datasets get made without annotation budgets or shipping documents to a labeling vendor. With teacher inference on local vLLM/Ollama and corpora written as JSONL/Parquet to your own object storage, the entire loop stays sovereign.
- "Open model = free distillation" is wrong in both directions. DeepSeek-R1 (MIT) and Qwen3 (Apache-2.0) impose no output restrictions; Llama 4 allows it but forces the student's name to begin with "Llama"; Gemma's terms define a model trained on Gemma-generated synthetic data as a "Model Derivative" that inherits Gemma's use restrictions (even though "Google claims no rights in Outputs" themselves). Hosted APIs add contract terms — Gemini's API terms flatly prohibit using the service "to develop models that compete." Read the actual license of your teacher.
- Generation is the easy 20%; curation is the product. Self-Instruct's gains came after filtering invalid/similar examples; Cosmopedia (25B tokens from a Mixtral teacher) leaned on prompt diversity engineering plus 10-gram benchmark decontamination against MMLU/HellaSwag.
- Unfiltered synthetic data happily bakes in the teacher's errors and style collapse — dedup and quality classification (the NeMo Curator stages) are not optional at scale.
depends_onGeneral-Purpose LLM — the teacher; its license propagates constraints into your corpus and student.enablesSmall / Distilled Model — distillation over synthetic corpora is how task-specific small models get made.used_byHugging Face TRL / Axolotl — the trainers that consume the generated SFT/DPO datasets.
Definition
Using a strong "teacher" model to generate a fine-tuning corpus — instructions, answers, reasoning traces, preference pairs — often grounded in your own documents, then training a smaller "student" model on it. The shape traces back to Self-Instruct (Wang et al., Dec 2022): generate candidate examples from the model itself, filter out invalid and near-duplicate ones, fine-tune on what survives. Run with a local teacher (served via vLLM or Ollama), the whole loop — documents in, distilled model out — never leaves your hardware.
Hand-labeling instruction data is the expensive step in fine-tuning on data you control; a teacher model converts raw internal documents into training pairs at near-zero marginal cost. The trade-offs are quality (filtering and deduplication are where pipelines succeed or fail) and licensing: what you may legally do with teacher outputs depends entirely on the teacher's license, and the differences between open-weight licenses are material. The resulting corpora are ordinary JSONL/Parquet artifacts, typically versioned and staged on object storage alongside checkpoints.
Recent developments
- NeMo Curator 26.04 released (April 2026). NVIDIA's Apache-2.0 curation toolkit — GPU-accelerated dedup (~16x over CPU for fuzzy dedup), quality classification, and synthetic data generation stages — is the current heavyweight QC pipeline; it powers the Nemotron dataset builds (8T+ tokens). Per NVIDIA-NeMo/Curator.
- DeepSeek-R1 made distillation-explicit licensing normal (2025). The R1 model card states the series "support commercial use, allow for any modifications and derivative works, including, but not limited to, distillation for training other LLMs" (MIT), and DeepSeek shipped six R1-distilled Qwen/Llama students as proof of pattern. Per HF model card.
- Llama 4's license permits distillation but brands the student (2025). "If you use the Llama Materials or any outputs or results … to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include 'Llama' at the beginning of any such AI model name" — plus "Built with Llama" attribution and a 700M-MAU licensing gate. Per Llama 4 Community License.
- distilabel passed to community maintainers (1.5.3, Jan 2025). Argilla's Apache-2.0 pipeline framework for synthetic generation + AI feedback remains the common open tooling; backends include Ollama, vLLM, and llama.cpp for fully local teachers. Original authors moved on; community maintainers develop on the
developbranch. Per argilla-io/distilabel.
Connections 7
Outbound 6
scoped_to1depends_on2enables1used_by2Inbound 1
augments1Resources 5
Self-Instruct — the original generate→filter→finetune recipe (33% absolute SuperNI gain on GPT-3) that all current pipelines descend from.
Primary text of the outputs/naming clause — the exact wording governing what a Llama-distilled model must be called.
The standard open pipeline framework for synthetic generation + AI feedback, with Ollama/vLLM/llama.cpp backends for local-only runs.
GPU-accelerated dedup/quality-filter/synthesis pipeline (26.04, Apache-2.0) — current state of practice for corpus QC at scale.
Worked example of large-scale synthetic generation with real QC detail: prompt diversity strategy and 10-gram decontamination.