Case study

Fine-tuning was not enough: how a local 14B model reached practical parity with a single-pass frontier baseline

A field note on a local LLM POC for Playwright test automation: why fine-tuning collapsed, what recovered it, and how retrieval, critique, repair, and verification closed the gap.

Jun 18, 2026 · local-llm fine-tuning llm-evaluation playwright test-automation agents

Field note public-safe, no private data evidence for Albelito

I wanted to answer a specific question, not build a generic LLM demo:

Can a local model generate Playwright Page Objects and tests that follow the conventions of a real internal project?

Not just “similar-looking code”. Not just valid TypeScript. Code that respects patterns, internal APIs, wait strategies, assertions, and automation style.

The short answer from the POC was: yes, but fine-tuning alone was not enough.

The starting point

The local base model already knew how to write TypeScript. It could also imitate the broad shape of a Page Object.

But it failed where project automation actually gets hard: internal semantics, private APIs, wait logic, return values, and decisions that do not exist in public documentation.

That was the real gap.

The mistake: trusting loss instead of correctness

The first fine-tune was naive.

It looked promising because validation loss went down. In the actual evaluation, it collapsed: 8.8/100 correctness.

That became the most useful reminder from the whole POC: for verifiable coding tasks, loss can lie. Style can look right while the solution is broken.

The metric that mattered was correctness.

What worked

With distillation and more conservative training settings, the model improved to 55.6/100.

Better, but still not enough.

The real jump happened when the model stopped working alone:

fine-tuning to learn code shape and idiom
RAG over real examples to provide project context
a critic → repair loop to fix logic
verification to turn generation into iteration
escalation to a frontier model when needed

The POC stack was deliberately small:

Qwen2.5-Coder 14B as the local model
MLX LoRA for local fine-tuning on Apple Silicon
Claude Opus as the frontier baseline and distillation source
LangGraph / deepagents for the agentic harness
Playwright MCP to ground selectors in a live DOM
RAG over real project examples to reduce hallucinations around private APIs

With that harness, the system reached 75.9/100.

The single-pass frontier baseline scored 76.5/100.

The honest read

This does not mean the local model “beat” the frontier model.

The sample was small: 8 features, 1 judge. It needs to be repeated with 50+ features and multiple judges. Also, the frontier model inside the same harness would likely score higher.

The useful conclusion is different:

In narrow, verifiable domains, a local model can get very close when it is part of a good system.

The thesis

The product is not the fine-tune.

The product is the harness: context, retrieval, critique, repair, verification, and escalation.

Fine-tuning alone learns the shape.
Fine-tuning + harness starts to close the gap.

For test automation, that difference matters much more than a polished demo.

← All posts