Nov 2025
Optimising BERT for Question Answering
I wanted to know how small a BERT question-answering model could get before it noticeably got worse, and where exactly that line was.
Stack
BERT, PyTorch, Quantisation, Pruning, NLP
Links
Private source
Notes
Quantisation and a custom mask-enforced pruning path were the two levers, tested separately so I could tell which was actually doing the work. The interesting part was mapping compression against F1 closely enough to see exactly where the model starts to break instead of just noticing that it eventually does.
The numbers
- Quantisation
- Post-training quantisation reduced model size from 440 MB to 128 MB, a 69% reduction, with no more than 0.5% F1 loss.
- Pruning
- A custom mask-enforced pruning path reached 54.7% sparsity.
- Validation
- Profiled compression settings against the measured efficiency-versus-generalisation trade-off.