Google's Gemma-2-2B-it, fine-tuned with QLoRA on the exact same legal/financial Q&A dataset as the from-scratch SLM-125M — grounded in a passage you supply, or from general knowledge.
| parameter | value | what it means |
|---|---|---|
| base model | Gemma-2-2B-it | Google's pretrained + instruction-tuned 2.64B-parameter model — not trained from scratch here |
| layers | 26 | number of stacked transformer blocks |
| hidden size | 2,304 | width of each token's internal representation |
| attention heads | 8 query / 4 KV (GQA) | grouped-query attention — fewer KV heads than query heads to reduce memory |
| native context length | 8,192 | Gemma-2-2B's own max context; this fine-tune uses a shorter 1,024-token window (below) |
| vocabulary size | 256,000 | SentencePiece tokenizer — 16x larger than the SLM's custom 16K BPE vocab |
| quantization | NF4, 4-bit | the frozen base model is loaded at 4-bit precision to fit fine-tuning on a single GPU |
| LoRA rank (r) | 16 | dimensionality of the small trainable adapter matrices injected into each layer |
| LoRA alpha | 32 | scaling factor applied to the adapter's contribution |
| LoRA dropout | 0.05 | regularization on the adapter layers only |
| target modules | 7 per layer | q/k/v/o attention projections + gate/up/down MLP projections get adapters; everything else stays frozen |
| trainable parameters | 20.8M (0.79%) | only the adapters train — the other 99.21% of the model is untouched |
QLoRA fine-tune on the identical 11,989-example dataset (9,980 grounded + 2,009 open-domain) used for the SLM-125M, with the identical 200-example held-out split (same shuffle seed) — the two models were never trained on, or tested against, different questions.
All 3 epochs were run and checkpointed to measure real eval_loss per epoch, rather than assume more training is better:
| epoch | eval_loss | |
|---|---|---|
| 1 | 0.537 | deployed — lowest loss |
| 2 | 0.576 | worse — early overfitting |
| 3 | 0.758 | worse still |
| learning rate | 2e-4 | cosine schedule, 3% warmup |
| effective batch size | 16 | per-device batch 4 × gradient accumulation 4 |
| max sequence length | 1,024 | fine-tuning tokenization cutoff (0 examples truncated at this length) |
Both models generated answers to the identical 200 held-out questions (never trained on by either model). A blinded LLM judge graded each answer independently against the reference answer — never told which model produced it.
| metric | n | SLM-125M | Gemma-2-2B |
|---|---|---|---|
| grounded accuracy | 155 | 51.6% | 92.9% |
| open-domain accuracy | 45 | 15.6% | 84.4% |
| grounding / faithfulness rate | 155 | 58.1% | 92.9% |
Gemma wins clearly on every axis — expected given the ~21x parameter gap and Google's much larger pretraining. This isn't a fair fight on raw quality, and isn't meant to be one: the more interesting comparison is 125M-from-scratch-and-fully-fine-tuned vs. 2.64B-pretrained-and-QLoRA-adapted, i.e. how much capability a lightweight adapter unlocks on a much bigger base versus building small from zero. Structural differences that make this not a clean isolated comparison: different tokenizers (16K custom BPE vs. 256K SentencePiece — raw loss/perplexity numbers on the two model-card sections above are not comparable to each other), full fine-tune vs. QLoRA adapter, and very different pretraining scale/data.
80% of random pulls come from the 45 held-out questions never used in training (a true generalization test, identical set to the SLM's); 20% come from the open-domain training questions (more variety, but the model may recall a memorized answer). Each pull is labeled below.
This is Google's Gemma-2-2B-it, adapted (not retrained from scratch) via a QLoRA adapter — only 0.79% of its parameters were actually trained. With a passage, it reads whatever text is in the passage box and answers from it, including saying "Not stated in the passage." when the answer genuinely isn't there. With no passage, it answers from general knowledge — both from Google's original pretraining and from the same open-domain Q&A examples the SLM was trained on.
Only the checkpoint with the lowest held-out loss (epoch 1) is deployed — later epochs measurably overfit on this small fine-tuning set, so "more training" was not better here.