Synthetic Training Data Generation
Generating fine-tuning corpora with a teacher model over your own documents — the self-instruct/distillation pattern — then filtering hard and training a student on the survivors.
Summary
Generating fine-tuning corpora with a teacher model over your own documents — the self-instruct/distillation pattern — then filtering hard and training a student on the survivors.
It is the data-supply side of fine-tune-own-data: PEFT and the trainer frameworks (TRL, Axolotl, LLaMA-Factory) consume instruction/preference datasets, and synthetic generation is how those datasets get made without annotation budgets or shipping documents to a labeling vendor. With teacher inference on local vLLM/Ollama and corpora written as JSONL/Parquet to your own object storage, the entire loop stays sovereign.
- "Open model = free distillation" is wrong in both directions. DeepSeek-R1 (MIT) and Qwen3 (Apache-2.0) impose no output restrictions; Llama 4 allows it but forces the student's name to begin with "Llama"; Gemma's terms define a model trained on Gemma-generated synthetic data as a "Model Derivative" that inherits Gemma's use restrictions (even though "Google claims no rights in Outputs" themselves). Hosted APIs add contract terms — Gemini's API terms flatly prohibit using the service "to develop models that compete." Read the actual license of your teacher.
- Generation is the easy 20%; curation is the product. Self-Instruct's gains came after filtering invalid/similar examples; Cosmopedia (25B tokens from a Mixtral teacher) leaned on prompt diversity engineering plus 10-gram benchmark decontamination against MMLU/HellaSwag.
- Unfiltered synthetic data happily bakes in the teacher's errors and style collapse — dedup and quality classification (the NeMo Curator stages) are not optional at scale.
depends_onGeneral-Purpose LLM — the teacher; its license propagates constraints into your corpus and student.enablesSmall / Distilled Model — distillation over synthetic corpora is how task-specific small models get made.used_byHugging Face TRL / Axolotl — the trainers that consume the generated SFT/DPO datasets.
Definition
Using a strong "teacher" model to generate a fine-tuning corpus — instructions, answers, reasoning traces, preference pairs — often grounded in your own documents, then training a smaller "student" model on it. The shape traces back to Self-Instruct (Wang et al., Dec 2022): generate candidate examples from the model itself, filter out invalid and near-duplicate ones, fine-tune on what survives. Run with a local teacher (served via vLLM or Ollama), the whole loop — documents in, distilled model out — never leaves your hardware.
Hand-labeling instruction data is the expensive step in fine-tuning on data you control; a teacher model converts raw internal documents into training pairs at near-zero marginal cost. The trade-offs are quality (filtering and deduplication are where pipelines succeed or fail) and licensing: what you may legally do with teacher outputs depends entirely on the teacher's license, and the differences between open-weight licenses are material. The resulting corpora are ordinary JSONL/Parquet artifacts, typically versioned and staged on object storage alongside checkpoints.
Connections7
Outbound6
scoped_to1depends_on2enables1used_by2Inbound1
augments1Resources5
Self-Instruct — the original generate→filter→finetune recipe (33% absolute SuperNI gain on GPT-3) that all current pipelines descend from.
Primary text of the outputs/naming clause — the exact wording governing what a Llama-distilled model must be called.
The standard open pipeline framework for synthetic generation + AI feedback, with Ollama/vLLM/llama.cpp backends for local-only runs.
GPU-accelerated dedup/quality-filter/synthesis pipeline (26.04, Apache-2.0) — current state of practice for corpus QC at scale.
Worked example of large-scale synthetic generation with real QC detail: prompt diversity strategy and 10-gram decontamination.