Oct 2025
Domain-Specific LLM Reasoning with PEFT
How little of a model can you touch and still move it? I trained only the bias terms, a rounding error's worth of parameters, and reasoning accuracy jumped anyway.
Stack
LLM, PEFT, BitFit, PyTorch, AI
Links
Private source
Notes
BitFit limits training to just the bias terms, which is about as small a nudge as fine-tuning gets. I ran it on clause-extracted health-insurance scenarios to see how far that small a nudge could move reasoning accuracy, and it moved more than I expected.
The numbers
- Method
- BitFit updated bias terms only, keeping the trained share at no more than 0.1% of parameters.
- Evaluation
- Reasoning accuracy increased from 47% to 73% on clause-extracted health-insurance scenarios.
- Scope
- Evaluation used clause-extracted health-insurance scenarios.