Post Content
[[{“value”:”Can a 27 billion parameter model teach a 4 billion model to be as good as itself? Last time, RL got Qwen 4B from 52% to 58.4% and then hit a wall. This time I used Qwen 27B as a teacher and tried all three kinds of distillation on two DGX Sparks: training on the teacher’s answers, training on its probabilities, and on-policy distillation where the 27B grades every token the 4B writes.
Sponsored by Dex, the talent agent for senior engineers. Free for engineers: https://meetdex.link/promptengineering-1
Links
– Previous video (RL, the sandbox, the 58% wall): https://youtu.be/Pl_2TelxjdY
– Code: https://github.com/PromtEngineer/smoldataenvs-grpo-spark
– SmolDataEnvs (the tasks): https://huggingface.co/datasets/FineEnvs/SmolDataEnvs
– Thinking Machines, On-Policy Distillation: https://thinkingmachines.ai/blog/on-policy-distillation/
– Qwen3 Technical Report (strong-to-weak distillation): https://arxiv.org/abs/2505.09388
– Hinton, Vinyals, Dean (2015), Distilling the Knowledge in a Neural Network: https://arxiv.org/abs/1503.02531
– On-policy distillation papers: MiniLLM https://arxiv.org/abs/2306.08543 · GKD https://arxiv.org/abs/2306.13649
Chapters
0:00 Can Qwen 27B teach Qwen 4B?
0:25 What RL did last time, and the wall
1:47 The short answer
2:23 The setup: SmolDataEnvs and a strict checker
4:03 Why RL stopped helping
4:48 How distillation works
5:24 Why the teacher’s probabilities matter
6:17 Three ways to distill
8:10 Two DGX Sparks: teacher and student
8:53 Method 1: train on the teacher’s answers
09:55 Sponsor: Dex
11:00 Method 2: train on the teacher’s probabilities
12:14 Why the probabilities added nothing
13:32 The limit of learning from transcripts
13:55 Method 3: on-policy distillation
14:38 A real run, graded token by token
15:43 Did it break the 58% wall?
17:13 All six methods on one chart
18:12 What the 4B model already knows
19:41 The 4-tries test
20:47 Caveats
21:04 Two gotchas: tokenizers and vLLM memory
22:06 What should you use?
#distillation #qwen #llm #reinforcementlearning #localai #dgxspark”}]] Read More Prompt Engineering
#Promptengineering #AI