Post Content
[[{“value”:”Learn how to design, build, and train an advanced multimodal language model from scratch using modern techniques like mixture-of-experts, sparse attention, and reinforcement learning. This hands-on guide walks you through the entire lifecycle from tokenization and pre-training to post-training optimization and evaluation.
Video from @vukrosic
GitHub: https://github.com/vukrosic/glm-5.3-flash-from-scratch
Become AI researcher in 90 days: https://www.skool.com/become-ai-researcher-2669/about
Slides: https://github.com/vukrosic/glm-5.3-flash-from-scratch/blob/main/slides/slides.html
Blog: https://z.ai/blog/glm-5.3-flash
❤️ Support for this channel comes from our friends at Scrimba – the coding platform that’s reinvented interactive learning: https://scrimba.com/freecodecamp
⭐️ Chapters ⭐️
– 00:00 Introduction & What We’re Building
– 01:02 The Modern AI Researcher Role & Asking Research Questions
– 04:50 Tokenization & Byte-Level Vocabulary (Why Small Vocab Matters)
– 07:01 Embeddings & Transformer Forward Pass Overview
– 08:53 GLM-5.3 Architecture Overview & Model Specifications
– 10:25 Code Walkthrough: Embeddings & Token Representation
– 11:56 Manifold Constrained Hyperconnections (DeepSeek Residuals)
– 13:27 Output Projection & Weight Tying
– 14:27 RMSNorm & Normalization Layers
– 15:10 Positional Encodings (RoPE vs. NoPE) & Sparse Attention Indexer
– 18:30 Linear Attention (State-Space Memory) vs. Sparse Attention
– 20:46 Mixture of Experts (MoE) & Shared Experts
– 23:06 Adding Vision: Patch Embeddings & 2D RoPE
– 26:14 Pre-Training Pipeline, Loss & Optimization (AdamW)
– 28:05 Pre-Training Experiments: Data Diversity, Interleaving & Curriculums
– 30:27 Post-Training & Reinforcement Learning (RL) Setup
– 34:44 Designing Reward Functions & Group Relative Policy Optimization (GRPO)
– 37:37 Parameter-Efficient RL Updates & Freezing Layers
– 39:35 Evaluating RL Results: Task Gains & Regression Risks
– 40:47 RL Hyperparameter Experiments: Group Size, Temperature & Seeds
– 43:03 Summary & Advice for Aspiring AI Researchers
🎉 Thanks to our Champion and Sponsor supporters:
👾 @omerhattapoglu1158
👾 @goddardtan
👾 @akihayashi6629
👾 @kikilogsin
👾 @anthonycampbell2148
👾 @tobymiller7790
👾 @rajibdassharma497
👾 @CloudVirtualizationEnthusiast
👾 @adilsoncarlosvianacarlos
👾 @martinmacchia1564
👾 @ulisesmoralez4160
👾 @_Oscar_
👾 @jedi-or-sith2728
👾 @justinhual1290
—
Learn to code for free and get a developer job: https://www.freecodecamp.org
Read hundreds of articles on programming: https://freecodecamp.org/news”}]] Read More freeCodeCamp.org
#frecodecamp