I Rebuilt Qwen3 From Scratch and Pretrained It on a University Supercomputer

---

Opening

I reconstructed Qwen3-0.6B’s architecture component-by-component, built a 13-billion-token curated data pipeline, and pretrained the result on UNC’s Longleaf HPC cluster.

Quick thing about that first number, because I’d want it explained if I were you: the model’s called “0.6B,” but count the parameters and you get 751M. That’s not a mistake — it’s the price of a big vocabulary. Qwen3’s tokenizer knows 151,936 tokens, and each one needs a row in a 1024-dimensional embedding table. That’s 156M parameters right there, just to turn token IDs into vectors. My implementation doesn’t tie the output layer to that table — it learns a separate 156M projection back out to the vocabulary — so the two ends of the model together run about 312M parameters, roughly 40% of the total, before a single transformer layer does any thinking. The 28 layers in the middle? 440M. At this scale, the dictionary costs as much as the reasoning. Knowing that — knowing where the weight actually sits — is the difference between reading a model card and understanding a model.

---

Act 1 — Why

You can use a language model every day and understand almost nothing about it. I’d know — I did exactly that for a while. The request goes out, the text comes back, and the part in the middle stays a black box you’ve quietly agreed not to look inside. And using a thing is not the same as understanding it. At some point that gap started to bother me enough to do something about it.

So I rebuilt one. Not a toy, not a wrapper around someone else’s model — Qwen3’s architecture, written line by line, then trained on real data on real hardware. This is the story of that build, start to finish: the parts I got right, and what the thing is actually doing under all of it.

Two ideas hold up everything that follows, and if you’ve never poked around inside one of these models, they’re worth two minutes each. The first is the difference between writing rules and learning patterns — the gap between ordinary programming, where you spell out every step, and machine learning, where you show a pile of examples and let the patterns surface on their own. The second is next-token prediction, which is the specific trick this model learns: look at some text, then guess what comes next — not one answer, but a spread of probabilities across all ~152K tokens it knows. Everything in this project, from why I agonized over the training data to what that falling loss curve even represents, comes back to those two ideas.

If they’re not new to you, keep reading — the rest is engineering.

I didn’t build this because I wanted to use the technology; I can do that with an API key and ten minutes. I built it because I wanted to understand it at the level you only reach by writing each piece yourself and watching it fall over. I value knowing how a thing actually works, for its own sake — and this project is what that looks like when I take it seriously.

---

Act 2 — Architecture

Here’s the problem with claiming you rebuilt Qwen3: anyone can write a class called Qwen3 and fill it with attention-shaped code. How do you prove it’s actually the architecture and not just something that compiles?

You make it wear the real weights.

Before I trained anything of my own, I built Qwen3’s architecture by hand and loaded the genuine pretrained Qwen3-1.7B weights — Alibaba’s published model, ~3.4GB of them — straight into my from-scratch code. Not my training, their weights, my layers. If even one component was subtly wrong — a transpose in the wrong place, heads grouped incorrectly, RoPE rotating the wrong dimensions — the weights would land in the wrong slots and the output would be noise. Instead, I gave it a prompt and it generated clean, coherent text. That’s the proof. Every matrix multiply lines up with the reference implementation, because if it didn’t, nothing would come out the other end but garbage.

That validation lives in notebook 02. Once it passed, I knew the architecture was right — and I rebuilt it a second time, properly: not as one monolithic notebook class but as a small configurable library, every component its own module, sized down to the 0.6B configuration I actually wanted to train. That library is RQwen3. The rest of this section is what’s in it.

What changed since GPT-2, and what it costs

If you learned transformers from the GPT-2 paper, Qwen3’s block will look familiar in outline and different in every detail. None of these changes are free — each one trades something for something else, and the interesting part is the trade, not the novelty.

Component	GPT-2	RQwen3	What it trades
Normalization	LayerNorm	RMSNorm	Drops mean-centering and bias; keeps only a learned scale. Slightly less expressive, meaningfully cheaper — and in practice the centering wasn’t doing much.
Position	Learned table	RoPE	No position parameters at all; rotates Q and K by angle instead. Buys better length behavior, costs a little arithmetic per attention call.
Attention	Multi-head (MHA)	Grouped-query (GQA)	16 query heads share 8 key/value heads. Halves the KV cache at inference for a small quality hit — the whole point is cheaper generation.
FFN	GELU MLP	SwiGLU	A third projection acts as a learned gate. More parameters per layer, reliably lower loss for them.
Attention stability	—	QK-Norm	Normalizes queries and keys before RoPE. Pure addition — a few thousand parameters to stop attention scores from blowing up early in training.
Biases	Everywhere	None	Every linear layer is bias-free. Fewer parameters, a mild regularizing effect, and nothing of value lost.

The block itself lives in src/layers/block.py, and the whole model assembly is in src/models/rqwen3.py.

I’ll take the six that matter one at a time. The theme throughout: small models punish waste. Every choice here is, at bottom, about spending parameters and compute where they actually buy capability.

RMSNorm. LayerNorm centers activations (subtract the mean), scales them (divide by standard deviation), then applies a learned scale and shift. RMSNorm throws out half of that — no mean subtraction, no shift, just divide by the root-mean-square and apply a learned scale. One line: x * rsqrt(mean(x²) + ε) * weight. It turns out the centering was mostly ceremony; dropping it costs nothing measurable and saves a reduction op across every normalization in a 28-layer model. My implementation is parameterized by dimension, which matters in a second when it shows up somewhere unexpected.

RoPE. GPT-2 learns a position for every slot — a lookup table of “what position 5 feels like.” RoPE does something cleverer: it injects no position vectors at all and instead rotates the query and key vectors by an angle proportional to their position. The math has a beautiful property — when you later take the dot product of a query at position m and a key at position n, the result depends only on their relative distance m−n, never on the absolute positions. The model learns “three tokens back” instead of “position 5 attending to position 2.” Low dimensions rotate fast to encode local relationships, high dimensions rotate slowly for long-range ones. And it adds zero parameters — the rotation is pure arithmetic, computed on the fly.

GQA. This is the one that earns its place at inference time. In vanilla multi-head attention, every query head carries its own key and value heads — and during generation, those keys and values get cached for every token you’ve produced so far. That KV cache is often the memory bottleneck. GQA’s bet is that you don’t need a unique key/value per query: my 16 query heads are split into 8 groups, and each group shares one key/value head. Concretely, the K and V projections are half the width of Q, and at attention time each KV head is repeated to serve its two query heads. Half the KV cache, for a quality drop small enough that essentially every modern model has adopted it. The cost is borne where you can afford it (a hair of quality); the savings land where you can’t (memory during generation).

SwiGLU. GPT-2’s feed-forward is two matrices with a GELU in between. SwiGLU uses three: an “up” projection, a “gate” projection, and a “down” projection. The gate passes through a Swish activation and is multiplied elementwise into the up projection before the down projection brings it back to model width — the network learns to gate its own information flow. It’s more parameters per layer than a plain MLP (three matrices, not two), and it’s worth it: lower loss, consistently, which is why the whole field quietly switched.

QK-Norm. This is the Qwen3-specific detail that’s easy to miss, and it’s where that dimension-parameterized RMSNorm comes back. Right after projecting into query and key heads — and before RoPE rotates them — each head’s query and key vectors get their own RMSNorm, applied across the 128-dimensional head. Why bother? Early in training, attention scores can grow large enough that the softmax saturates and gradients die — “attention collapse.” Normalizing Q and K first keeps the scores in a sane range and the model trainable. It costs a few thousand parameters total and buys stability you’d otherwise pay for in failed runs.

No biases. Every linear layer in the model — the attention projections, the FFN, the output head — is declared bias-free. GPT-2 had biases everywhere out of habit. Modern practice dropped them: at scale they contribute almost nothing to capability, they add parameters, and removing them is mildly regularizing. It’s a small thing, but it’s the kind of small thing that signals the architecture was thought about rather than copied.

How a block goes together

Each of the 28 layers is the same shape: normalize, attend, add; normalize, feed-forward, add. Pre-norm, with residual connections carrying the original signal around each operation so gradients have a clean path back through all 28 layers:

x = x + Attention(RMSNorm(x))
x = x + SwiGLU(RMSNorm(x))

The block doesn’t hard-code its own components, though. It takes them as arguments — attention class, FFN class, norm class — defaulting to the Qwen3 set but swappable without touching the block itself. That’s not over-engineering for its own sake; it’s how a single MoE experiment becomes “pass MoEFeedForward to every other layer” instead of forking the whole model. The block is the seam the architecture flexes at.

The whole thing, sized

One config object drives every shape in the model — change a number here and the embedding, all 28 blocks, the attention heads, and the output head all resize consistently. This is the actual RQwen3 specification:

Setting	Value
d_model	1024
n_layer	28
num_heads	16
num_kv_heads	8 (2:1 GQA ratio)
head_dim	128
intermediate_size	3072
vocab_size	151,936
max_seq_len	2048
Total	~751M params

Embedding → 28 blocks → final norm → output head. Token IDs in, a probability distribution over 151,936 possible next tokens out. That’s the machine. The next problem was finding 13 billion tokens good enough to teach it.

---

Act 3 — Data

There’s a saying in machine learning that does most of the work of a semester: trash in, trash out. A model is a mirror of what you feed it. Show it sloppy, contradictory, low-effort text and it learns to produce sloppy, contradictory, low-effort text — faithfully, because faithful imitation is the only thing it knows how to do. Architecture gets the attention, but at this scale the data is the model. A clever block design can’t rescue a bad corpus; a good corpus forgives a lot.

That matters more, not less, for a small model. My 751M parameters see every token roughly once — one epoch, no second passes to average out the noise. A giant model trained on trillions of tokens can afford some garbage; the sheer volume drowns it. I couldn’t. Every token in my budget had to earn its place, which meant the central question of this whole project wasn’t “what architecture?” — I’d answered that — it was “what should this model read?”

So I started from the goal and worked backward. RQwen3 isn’t meant to be a general chatbot; it’s meant to be an educational assistant — something that can explain a statistics concept, walk through a CS problem, answer the kind of STEM question I’d actually ask. That goal isn’t decoration. It decided every source, every ratio, every filter below.

The mix, and why each piece is there

Six sources, about 13 billion tokens. Here’s the recipe, but the percentages aren’t the point — the reasons are:

Source	Tokens	Share	Why it’s in the mix
FineWeb-Edu	7.0 B	54%	The backbone: broad educational web text, pre-scored for quality.
Wikipedia	2.0 B	15%	Factual grounding — dense, edited, citation-backed.
OpenWebMath	1.5 B	12%	Math and statistics, formulas and worked examples. The STEM in STEM assistant.
StackExchange	1.0 B	8%	Question-and-answer format — literally the shape of the job.
peS2o*	1.0 B	8%	Open-access academic papers — formal explanation patterns.
Textbooks*	0.5 B	4%	A small dose of textbook-style structure, outsized effect.
Total	~13 B	100%	Source of truth: scripts/build_dataset.py.

FineWeb-Edu is the foundation — more than half the diet — because it’s web text that’s already been filtered for educational value, each page scored 0 to 5 by a classifier. The Llama and Phi teams both found the same thing at this scale: a smaller pile of higher-quality web data beats a larger pile of mediocre data for sub-1B models. More isn’t better. Better is better.

Wikipedia is there for grounding. An assistant that explains things confidently and wrongly is worse than useless, and Wikipedia is dense, well-edited, citation-backed prose — exactly the factual anchor I wanted. OpenWebMath is the deliberate STEM injection: LaTeX, proofs, statistical formulas, the actual texture of quantitative work, aimed squarely at the kinds of courses I care about. StackExchange earns its 8% on format alone — it’s question-then-answer, which is the precise interaction pattern an educational assistant lives in, sourced across programming, math, stats, and psychology. peS2o brings academic papers and the formal explanatory voice that comes with them. And textbooks, at just 4%, are there because the Phi-1.5 paper showed that even a little textbook-style data has an effect out of all proportion to its size — too much would dominate, but a pinch sharpens everything.

None of these ratios are guesses. They follow what Meta documented for Llama — roughly half web, a chunk of curated knowledge, the rest specialized — and they lean on the Doremi result that upweighting high-quality domains beyond their natural share pays off downstream. I wasn’t reinventing data mixing. I was applying what the people with real compute had already learned.

What “good” actually means

“Quality filtering” sounds vague until you have to write the rules, and writing the rules is where the understanding lives. For me “good data” cashed out as concrete, per-source decisions:

For FineWeb-Edu, keep only pages scoring 3 or higher and between 100 and 100,000 characters — high enough to drop the junk, not so high that I lose useful diversity; the length bounds chop off both content-free stubs and raw data dumps. Wikipedia loses its disambiguation pages and anything under 500 characters — neither teaches anything. OpenWebMath and the academic sources get length floors, because a real explanation is rarely twenty words long. StackExchange keeps the asker-selected answer when there is one, falling back to the highest-scored answer otherwise — either way, the community already voted on quality, so I let them. Then every source gets exact-duplicate removal by SHA-256 hashing each document, because the fastest way to waste a one-epoch token budget is to spend it reading the same paragraph twice.

Every one of those thresholds is a small judgment call, and making them was the part of this project I most wanted. This is what “I want to understand it at a low level” actually looks like in practice — not a grand insight, but a hundred small decisions about which document is worth a token and which isn’t, each one made on purpose instead of inherited from a default.

The honest follow-on question is whether each call was the right one — and the baseline run can’t answer that. A FineWeb-Edu score-threshold sweep (does 3 actually outperform 4?) and a source-share ablation (does the 8% StackExchange slot earn its place against more Wikipedia?) are both on the backlog for after the baseline finishes. This mix is the control group those experiments need.

Engineering for a cluster that will kill your job

There’s a gap between “I have good data” and “I can train on it for days on a shared supercomputer,” and closing that gap was its own piece of work.

I pre-tokenized everything ahead of time — converted all 13 billion tokens into flat binary files of raw token IDs, no headers, no compression, just numbers. (They have to be 32-bit, incidentally, for a small but telling reason: the vocabulary is 151,936 tokens, which overflows the 65,535 ceiling of a 16-bit integer. The big vocabulary that made the embedding table huge in Act 2 reaches out and doubles my dataset’s disk footprint here too — about 52GB. Choices echo.) Storing tokens this way lets me memory-map the files: the operating system pages data in and out on demand, so the model trains as if all 52GB were in memory without ever actually loading it there, and any chunk is reachable in constant time for proper shuffling.

The piece I’m most glad I built is resumability. A job on Longleaf gets a wall-time limit; when it’s up, the scheduler sends a termination signal and your process dies whether it’s ready or not. A naive training loop, restarted, would begin from a cold init — throwing away every gradient step it took before the kill. So each checkpoint records the full state I actually need to continue: weights, optimizer momentum, learning-rate schedule position, and a counter for how much data the run has consumed. The data loader reshuffles fresh each pass — it doesn’t seek back to the exact same token order — but the model picks up mid-sentence in the only sense that matters: weights, momentum, learning rate, all intact. The run continues, instead of starting the book over.

A number worth stating plainly

How much data is the right amount? This isn’t a vibe — there’s a result for it. The Chinchilla scaling work put compute-optimal training at roughly 20 tokens per parameter. For 751M parameters that’s about 15 billion tokens. My budget is 13 billion — a hair under optimal, close enough to be in the right regime and honest enough to admit it’s not exactly on the line. I’m stating that ratio out loud on purpose, because it’s the frame for reading the results later: I’m not training to convergence on infinite data, I’m spending a roughly compute-optimal budget once and watching what a small model does with it. That’s the experiment.

Thirteen billion carefully chosen tokens, tokenized, deduplicated, sharded, and resumable. The model was built. The data was ready. Next came the system to bring them together — and after that, the hardware to run it on, which turned out to be the hardest part of all.

---

Act 4 — Training System

A training loop is easy to write and hard to trust. The toy version fits in a tweet: for each batch, run the model, compute the loss, backpropagate, step the optimizer. That loop will train a model. It will also lose three days of progress the first time a shared cluster kills your job mid-step, silently re-read the same data after a restart, or quietly blow up because you decayed the weights you weren’t supposed to touch. The distance between “a loop that trains” and “a loop you can leave running for days on hardware you don’t control” is most of the actual work, and it’s the part nobody writes the tweet about.

So I didn’t write a script. I built two abstractions and let everything else hang off them.

CoreConfig: one number, everywhere

The first is almost embarrassingly simple: a single configuration object that defines every shape in the model. Hidden dimension, layer count, head counts, vocabulary size, the GQA ratio — all of it lives in one dataclass, and every component reads from it. The attention layer doesn’t hard-code 16 heads; it asks the config. The embedding doesn’t hard-code the vocabulary; it asks the config.

This sounds like bookkeeping until you’ve felt the alternative. A 28-layer transformer is a tower of shape dependencies — the head dimension has to divide the model dimension, the KV projections have to be half-width for a 2:1 GQA ratio, the output head has to match the vocabulary. Hard-code those in a dozen files and changing one number means hunting down eleven others, and the failure mode is a shape-mismatch error thrown somewhere deep in a forward pass after twenty minutes of setup. With one config driving everything, I change d_model from 1024 to 512 and the entire model — every layer, every projection, the optimizer groups, the data shapes — resizes consistently and correctly. That consistency is what made iteration survivable. It’s the difference between “try a smaller model” being a one-line experiment versus an afternoon of debugging.

TrainSession: the loop as a stateful object

The second abstraction is the training loop itself, wrapped into an object that holds its own state — model, optimizer, scheduler, step count, loss history, and how much data it’s consumed — rather than scattering that state across a function’s local variables. That design choice is what makes everything below possible, because a loop that can save and reload its own state is a loop you can interrupt. Here’s what it wraps, and why each piece is there.

The optimizer doesn’t decay everything. AdamW with weight decay is standard, but I apply that decay selectively: only to the big weight matrices — attention and feed-forward projections — and never to the one-dimensional parameters like the RMSNorm scales. The rule in code is literally “if it has fewer than two dimensions, or ‘norm’ is in its name, don’t decay it.” Why? Weight decay’s job is to keep large matrices from overfitting. A normalization layer’s learned scale is a single number per channel; shrinking it toward zero doesn’t regularize anything, it just fights the layer’s purpose and hurts stability. Most people inherit this rule from a tutorial. I wanted to know why it was there.

The learning rate warms up, then coasts down. A cosine schedule with linear warmup: the rate ramps from near-zero up to its peak over the first 500 steps, then follows a cosine curve back down across the full run. The warmup exists because a freshly initialized model is fragile — hit it with a full-size learning rate on step one and the early gradients can blow the weights out before they’ve organized into anything. Ramp up gently, and by the time you’re taking big steps the model can absorb them. The cosine decay is the mirror image at the other end: large updates early when there’s a lot to learn, progressively finer ones as the model settles.

Small batches pretend to be large ones. The L40S’s 48GB of VRAM can’t hold the batch size I actually want, so the loop accumulates gradients: it runs several small micro-batches, sums their gradients, and only then takes one optimizer step — dividing each micro-batch’s loss by the accumulation count so the total gradient comes out the same magnitude as one big batch would. The result is an effective batch of 128 examples — about 262,000 tokens per step — built from physical micro-batches of two, accumulated 64 times. The memory ceiling stops being a hard limit on batch size and becomes a knob I trade against step time. (The exact numbers — batch_size=2, grad_accum=64 — are where they are because of Act 5, not because I picked them up front. We’ll get there.)

It runs in bf16 on the GPU, and skips a piece of machinery most people keep. On CUDA, the forward pass runs under bfloat16 autocast — half the activation memory, and it uses the L40S’s BF16 tensor cores. Honest framing: bf16 (and the SDPA attention path the at-a-glance mentions) both arrived as the response to OOM. Act 5 tells that story; they’re production design now, but they didn’t start there. The detail I’m proud of is what’s missing: there’s no gradient scaler. Mixed-precision training usually needs one, because the older fp16 format has such a narrow numeric range that small gradients underflow to zero and have to be scaled up to survive. Bfloat16 trades precision for range and keeps fp32’s full dynamic range, so the underflow problem just doesn’t exist, and the scaler it requires is dead weight. On Apple Silicon, where I prototyped, the whole autocast path is a no-op and the loop runs in plain fp32 — same code, no special-casing, it just notices the hardware.

It shows its work. Every few hundred steps the loop stops and generates a sample from a fixed prompt, so I can read the model learning instead of squinting at a loss number. Watching “The theory of general relativity” decode into noise, then into word-shaped noise, then into something with grammar, is the most honest progress bar there is.

The part that matters on a cluster that kills your jobs

Longleaf gives every job a wall-time limit. When it’s up, SLURM doesn’t ask — it sends your process a SIGTERM and, a short grace period later, a SIGKILL that ends the conversation. A normal training loop, hit with that, simply dies: whatever happened since the last scheduled checkpoint is gone, and on restart the run begins again from that checkpoint with no memory of the partial progress.

Two pieces, working together, turn that hard stop into a pause.

First, the cluster entry point traps the SIGTERM. When the signal arrives, a handler fires that does one thing: it calls the session’s ordinary save_checkpoint, writing the full state — model, optimizer, scheduler, step, loss history — to disk, plus a weight snapshot tagged with the step for later forensics, then exits cleanly before the SIGKILL lands. There’s no special “emergency save” path, and that’s deliberate: the emergency checkpoint is just a normal checkpoint that happened to be triggered by a signal instead of a step counter. Same file, same format.

Second, on the next submission the script scans the checkpoint directory, finds the file with the highest step number, and resumes from it. The reason this closes cleanly is a small naming detail: the signal-triggered save writes the exact step_N.pt filename that the startup scanner is looking for. The dying job leaves a breadcrumb the next job is already searching for by name. Re-submit the same SLURM script — which is all auto-resume is — and the run picks up where the wall clock cut it off.

One honest caveat, because I’d rather state it than have a careful reader catch it: what resumes precisely is the model — weights, optimizer momentum, schedule position. The session also tracks a data_offset counter, but in the production setup the data loader reshuffles each pass, so that offset is a record of how much data has been consumed, not a seek back to the exact same examples in the exact same order. For pretraining, that’s the right tradeoff — what matters is that the model state continues seamlessly, not that token #4,000,001 is identical across a restart. I’d rather be precise about what the system guarantees than oversell it.

None of these pieces is clever in isolation. Selective weight decay, warmup, gradient accumulation, signal handling — each is a known technique. The system is the point: the thing you can configure once, launch, and trust to run for days, survive being killed, and pick itself back up. Every bullet above is something I only learned I needed after its absence bit me. That’s what the next section is about — all the ways the cluster bit me first.

---

Act 5 — Hardware

For three months this project ran on my laptop. Architecture work, weight inspection, the data pipeline, all the unit-test training — fine for proving the code is sound, hopeless for actually training 751M parameters on 13 billion tokens. To do the real run I needed real hardware — and it turned out I already had access to some.

Longleaf

Longleaf is UNC’s high-performance computing cluster: hundreds of nodes, and critically for me, partitions of NVIDIA A100s and L40S GPUs that any student can request time on. This is the part most undergraduates never find out about. You spend your degree running things on your own machine, and the whole time there’s a research supercomputer sitting behind a login node, free to use, waiting for a job script. These are data-center training GPUs — the kind of hardware the actual labs use — and UNC just has stacks of them available to anyone who asks. Finding that out felt like discovering a door in my own house I’d never opened.

The catch is that you don’t use a cluster the way you use a laptop. You don’t run your program; you write a job script describing what your program needs — how many GPUs, how much memory, how long, on which partition — hand it to a scheduler called SLURM, and wait in a queue with everyone else. SLURM decides when and where your job runs. And SLURM, it turns out, has a great many ways of telling you no.

Seven Ways SLURM Said No

I had the training code working. I had it tested locally. I figured moving it to the cluster was a formality — sync the files, submit the job, watch it run. The job died in seconds. So did the next one. What followed was the most educational week of the whole project, and not one minute of it was about machine learning.

The path was wrong before the model even loaded. My script computed the project root by walking up the directory tree, and I’d miscounted the levels — the cluster’s directory layout put my entry point three folders deep, not one, so every import failed before a single tensor existed.

The environment wasn’t the environment. Locally, my Python virtual environment was just active — I’d forgotten it was a thing I had to turn on. A SLURM job inherits none of your shell; it starts in a bare environment and runs exactly what you tell it, nothing more. The activation line was sitting commented out with a placeholder path.

Then came the cluster’s own vocabulary, which I had to learn one rejection at a time.

I asked for the wrong kind of GPU — gpu:a100:1, the obvious guess, which SLURM flatly refused because the real resource string on this cluster is gpu:nvidia_a100-pcie-40gb:1. You can’t guess that; you have to interrogate the scheduler with sinfo and read back exactly what it calls its own hardware.

Then I asked for the wrong queue entirely — I submitted to the partition named gpu, which sounds correct and is in fact a pool of ancient GTX 1080s with 8GB each; the A100s live on a separate partition I had to name explicitly.

Then the modules didn’t match — my setup script had installed one Python and CUDA version while my job script requested slightly different ones, the kind of minor Python-version mismatch that breaks everything and announces nothing.

Then the scheduler wanted a permission slip I didn’t know existed — jobs on the GPU partition require a specific quality-of-service flag, --qos=gpu_access, and without it submission is simply rejected.

Four separate refusals, each one a piece of cluster-specific knowledge that exists nowhere in my code and nowhere in my training — only in the cluster’s own configuration, readable only by asking it.

The seventh was the one that stung, because it wasn’t a typo or a config string — it was an assumption baked into the whole training loop. My memory budget was built for a smoke test, not for production. My pre-flight script had been green: it ran at seq_len=256, batch_size=2 and finished in 90 seconds. The production memory profile is at seq_len=2048, batch_size=4, and a smoke test that doesn’t exercise that profile doesn’t catch what’s lurking in it. The first real submission to the L40S production partition died in OOM during the forward pass. So did the next. And the one after that.

Three crashes, three stacked fixes — and each one is a piece of training engineering I should have shipped on day one but didn’t, because nothing on my laptop forced me to.

Fix 1: stop materializing the attention matrix. My naive (q @ k.T) * scale path keeps the full (B, H, T, T) score matrix in VRAM — about 1 GiB per layer at seq=2048, multiplied by 28 layers all sitting on the backward graph at once. Replacing it with F.scaled_dot_product_attention lets PyTorch dispatch to FlashAttention on the L40S’s Ada Lovelace cores, which never materializes that matrix at all. Attention memory drops from O(T²) to O(T). Past the first crash.

Fix 2: bf16 autocast. Wrap the forward pass and loss in torch.autocast(dtype=bfloat16). Half the activation memory, and the L40S’s BF16 tensor cores finally have something to do. No GradScaler — that’s a piece of fp16-era machinery for a numeric format with a narrower dynamic range; bf16 keeps fp32’s range and the scaler is dead weight. Past the second crash.

Fix 3: halve the micro-batch. Even with SDPA and bf16, the cross-entropy softmax over 151,936 logits has to stay in fp32 for numerical stability — and that softmax alone is around 5 GiB per micro-batch at batch_size=4. Bfloat16 autocast can’t shrink it; it’s a memory floor the architecture sets. Drop physical batch from 4 to 2, lift grad_accum_steps from 32 to 64, keep the effective batch nailed at 128. Past the third crash, and the run finally stayed up.

The pattern underneath all seven is the same, and it’s the thing I actually took away: code that runs perfectly on your machine has silently absorbed a hundred assumptions about its environment, and a cluster honors none of them. Every one of those assumptions has to be surfaced, named, and stated out loud before the job will run. That’s not a detour from the work. On a shared supercomputer, that is the work.

Making it repeatable

The other lesson was about friction. Each of those debugging cycles meant the same dance: sync code to the cluster, SSH in, submit, check the queue, pull the logs, read the failure, fix, repeat. Done by hand, typing each command, that loop is slow enough that you start avoiding it — and a debugging loop you avoid is a project that stalls. So I put the whole workflow behind a Makefile: make sync to push code, make submit to launch, make status to check the queue, make logs to tail output, make pull to bring back checkpoints. One word each. The point wasn’t elegance — it was removing every excuse not to iterate. Friction is the enemy of the experiment, and a Makefile is cheap insurance against your own reluctance.

It didn’t stop at seven

I’d love to tell you the seventh fix was the last one. It wasn’t — and that’s the honest shape of cluster work, so I’ll own it. When the A100 queue got long enough to bottleneck iteration, I added job variants targeting the cluster’s L40S GPUs as an alternative pool (which is how production ended up running there at all). When the dataset build itself turned out to be too large to finish inside a single job’s wall-time limit, I had to make the build resumable too — splitting it across jobs and writing a small tool to stitch the resulting shards back into one coherent dataset. And much later, deep into the production run, Sub 8 sat in the queue for 18.5 hours before it actually started — breaking the “submit at noon, resume tomorrow at noon” cadence I’d settled into, and quietly proving the auto-resume machinery handled real-world weirdness, not just the wall-clock cutoffs it was designed for. None of these were in the original plan. All of them are the kind of thing you only discover you need by hitting the wall in real time. Seven was never the final count; it was just where I stopped being surprised that there was a next one.

---

Act 6 — Results

Back in Act 1, I said the whole machine learns one trick: predict the next token. Pretraining is that trick, run at scale, for a very long time. The loss is a number that measures how surprised the model is by the actual next token — high when it has no idea what comes next, low when it saw it coming. Training is the slow business of making the model less surprised by real text. Everything in this section is a picture of that surprise going down.

The first proof it works at all

Before the cluster, before 13 billion tokens, I ran a tiny version on my laptop: 200 steps on TinyStories, a dataset of simple children’s stories, on Apple Silicon. It’s not a real pretraining run and was never meant to be — it’s a unit test for the whole training system. Does the loss actually fall? Do the checkpoints save and reload? Does the generation sample improve from one checkpoint to the next? It did, on all counts. I have the before-and-after saved: an untrained checkpoint whose weights are random noise, and a trained one 200 steps later, and the difference is visible right down in the weight distributions — notebook 04 lays the two histograms side by side and you can watch the random initialization organize itself into something with structure. That run proved the system was sound. What it couldn’t do was make a model that knows anything, because 200 steps on toy stories isn’t enough to learn the world. For that I needed the real run.

The real run, end-to-end

The run completed on 2026-06-19. Fifty thousand steps, 13 billion tokens, ten successful 24-hour SLURM submissions across 11 wall-clock days on UNC Longleaf’s L40S partition — plus three OOM crashes and two user-cancels on day one while the memory-wall fix from Act 5 was being worked out. The first nine submissions exited via the SIGTERM-and-resume machinery from Act 4; the tenth ended on natural max_steps exit (COMPLETED 0:0) and wrote checkpoints/final.pt. The thing I built to be trustworthy was trusted, repeatedly, by the only test that counts.

The model is the 751M-parameter from-scratch RQwen3 from Acts 2–4, trained on the curated 6-source corpus from Act 3 for almost exactly one full pass — which is the experiment I named earlier: spend a roughly Chinchilla-optimal budget (~20 tokens per parameter) once, and see what a small model does with carefully chosen data.

The loss tells the story:

random init   →  11.88
step  5,000   →   2.97
step 10,000   →   2.78
step 15,000   →   2.69
step 20,000   →   2.68
step 25,000   →   2.60
step 30,000   →   2.58
step 35,000   →   2.53
step 40,000   →   2.54    ← cosine tail flattens
step 45,000   →   2.50
step 50,000   →   2.5186  ← final

From 11.88 to 2.5186 — the model went from uniform confusion (a loss near 12 is roughly “every one of 152K tokens is equally likely”) to genuine predictive structure. In perplexity terms (the form that occasionally shows up in papers): 11.88 corresponds to a perplexity around 144,000 — effectively uniform over the vocabulary; 2.5186 puts perplexity at ≈ 12.4, meaning the model can narrow each next-token choice down to roughly a dozen options. The post-training stages are about turning “a dozen options” into “the right one.” And the shape of that curve is the interesting part, not just the endpoint. Look at the spacing: the first 5,000 steps do enormous work, then each subsequent 5,000 buys progressively less, and the last 15,000 just polish what’s already there. That’s not the run stalling — it’s exactly what a healthy training curve looks like. The easy patterns (basic grammar, common words, the shape of English) get learned fast and cheap; what remains is the long, slow grind of harder structure, bought one diminishing increment at a time. A reader who knows what they’re looking at can see from this curve alone that nothing’s broken — it’s bending the right way.

Is 2.5186 any good? A frame of reference

A loss number means nothing in isolation. Here’s the comparison that matters, for ~750M dense models trained from scratch and reported in the literature:

Reference model	Approx. final loss	Tokens trained on
GPT-2 large (774M)	≈ 2.85	WebText (2019)
Pythia-410M	≈ 2.85	300B (The Pile)
RQwen3 (this run, 751M)	2.5186	~13B (1 epoch)
Pythia-1B	≈ 2.60	300B (The Pile) — 23× more data
Qwen3-0.6B base	≈ 2.4	~5T — 400× more data

One honest caveat before reading too much into the table: cross-tokenizer loss comparison is loose. Qwen3’s tokenizer carries ~152K vocab; Pythia and GPT-2 both use ~50K. Each model is summing nats over a different unit, so a few hundredths in either direction shouldn’t be over-read. What carries the story is the rank ordering — near Pythia-1B, well above GPT-2 large, well below Qwen3-0.6B base — not the precise gap. Tokenizer-agnostic benchmarks (ARC, HellaSwag, MMLU) are how this gets settled cleanly, and those numbers are queued for the next chapter.

The headline I’d want a reader to take away: landing in Pythia-1B’s neighborhood with one-twentieth of Pythia-1B’s data is the result the carefully curated mix was supposed to buy. Below 2.4 would have been exceptional at this scale; above 3.0 would have suggested something was wrong; 2.5 is a clean, defensible number for the parameter budget and the compute. It’s also a quiet bit of evidence for the Act 3 thesis — that for sub-1B models, better tokens beat more tokens by enough that you can land near a peer trained on 23× the data.

The caveat I’d want stated explicitly, before anyone reads more into the loss than it can carry: this is a base model. It’s a next-token predictor, not an assistant. It will continue grammatical-but-mode-collapsing prompts, fabricate facts, and not follow instructions. Making it useful as the educational assistant Act 3 was aimed at is the job of supervised fine-tuning — the next chapter.

Watching it learn to think

The loss number is abstract. What it feels like is more legible in the samples. Every 500 steps the run stopped and generated from the same prompt, and reading those samples back in order is like watching something slowly wake up:

Step	What it generates
500	”the most important the most important the most important…” — stuck in a loop, has learned that some words are common and nothing else
6,000	”continuous, continuous, continuous…” — still looping, but on rarer words; the vocabulary is widening
10,000	”Max Planck developed general relativity in 1905” — a real, grammatical, confident sentence. It is also completely wrong (that was Einstein, in 1915), but it has learned the form of a fact
25,000	”expansion caused by gravitational attraction of stars and planets” — actual partial reasoning, concepts connected in a way that almost coheres
35,000	”expanding at 1000 km/s²” — specific numerical claims, with correctly-formatted units attached to wrong values
40,000	”The expansion is caused by the expansion of space itself…” — actually correct cosmology. Metric expansion of space — real GR, not motion-through-space. The biggest qualitative jump of the whole run
50,000	”The theory of general relativity is a theory of gravity, based on the idea that…” — final-form consolidation: confident, structured, on-topic

That progression is the entire thesis of the project in seven rows. The model doesn’t learn facts first and grammar later; it learns shape first — the rhythm of language, then the rhythm of a sentence, then the rhythm of an assertion — and only gradually fills that shape with anything true. At step 10,000 it confidently misattributes relativity, and that confident wrongness is more interesting than silence: it means the model has learned what a fact sounds like before it has learned any facts. The wrong values with right units at step 35,000 are the same story one level up. And then — and this is the moment I genuinely did not expect — at step 40,000, after the loss number had basically flattened, the prose started getting things right. “Caused by the expansion of space itself” is real general relativity. The model wasn’t done learning when the scalar loss had decided it was.

That’s the lesson I want to pull out, because it’s not the one I went in expecting: the loss number stopped moving long before the prose stopped improving. Between step 35K and step 50K, training loss bounces in a band from 2.50 to 2.54 — a flat line at the resolution of any plot you’d draw — and yet the cosmology in the samples goes from numerical nonsense with correct units to actually-correct GR. The scalar loss is a coarse summary of what the model is learning, not a complete one. Reading the samples is the only way to see the second derivative.

This is also what pretraining actually buys you — fluency, form, and (sometimes, at the tail) the beginnings of correctness — and it’s exactly why a second stage exists to layer instruction-following and reliable factual correctness on top, which the closing section of this Act picks up.

What’s next

The run is done; the project isn’t. Two pieces of honest work still owed:

Evaluation. A loss number compared to other loss numbers is a start, not a finish. The standard lm-evaluation-harness benchmarks — ARC, HellaSwag, MMLU — are how this base model gets numbers comparable to other models in their natural format, and those scores are the real test of whether the careful data curation in Act 3 bought measurable capability. I haven’t run them yet. final.pt is sitting on Longleaf’s /work filesystem; pulling it locally and standing up the eval harness is the next concrete task on the list, not a hypothetical. I’m flagging this rather than dressing it up: I’d rather you know it’s pending than think the absence of numbers means anything else.

My priors, written down before I have the results so they can be judged honestly when I do: HellaSwag should be the most defensible — it tests commonsense narrative completion, which is exactly what 13B tokens of FineWeb-Edu + Wikipedia ought to buy. MMLU is where I’m least confident; its hardest subjects (college-level math, formal logic, professional law) reward domain depth more than a 13B budget can plausibly deliver, and a small underperformance there is what I’d expect at this scale. ARC-Challenge is the most interesting bet, because OpenWebMath was an explicit ~12% slot precisely for reasoning-shaped data — if it lands respectably for a 751M model, that’s evidence the mix did something the loss number alone can’t show. When the numbers come in, those predictions get judged honestly against them.

Figure pending — eval results

lm-eval-harness results table — ARC / HellaSwag / MMLU, once eval is run.

Supervised fine-tuning. Pretraining bought fluency, not instruction-following; what the model has learned is to talk like a textbook, not to answer like one. The bridge from one to the other is SFT, and it’s already scaffolded in notebook 06 — the next chapter of the project. The real test of whether the curated data mix from Act 3 paid off lives there: when an instruction-tuned RQwen3 explains a stats concept side-by-side with a Qwen3-0.6B baseline that didn’t see the same data, the differences (or lack of them) tell you whether the choices in Act 3 mattered. The whole point of curating the corpus was that question. It gets answered next.

What I can say today is that the system worked, the loss curve bent the right way, the model is visibly learning to produce structured language with the occasional flash of correct content, and the whole thing ran itself to completion on free university hardware. That’s not a finished story. It’s the foundation under one — and the next chapters are queued.

---

Currently seeking: summer internships and undergraduate research opportunities in ML / data science. Repo: https://github.com/R-Theory/RQwen3 Reach: treese2028@gmail.com