Finetuning: what we need

The teacher should be a frontier model.

The finetune learns from the strongest model available (eg. Fable), not from the model the customer runs today. Why a frontier model:

• It has a better world model, so it can handle cases we never trained or tested for.
• It fills the gaps when the prompt isn't specific, and makes a sensible call where a smaller model would guess.

How we'd use it:

• Run it on the customer's real traffic for a while, even if they run Haiku or Opus today.
• Train the finetune on its outputs, plus the historical traces and reviewer feedback we already have.
• Specialised on one task and trained on frontier-level data, the finetune can beat the model the customer runs today.

We decide what counts as high quality data.

We know our customers' processes, so we want to define what a good output looks like, using our own signals:

• Evals that catch the errors we know about
• Code checks, eg. schema validation or business rules
• Human feedback from the people who review the agent's work
• More as we learn what matters

Only outputs that pass go into training. Better data is where the accuracy comes from.

The finetune has to beat the frontier model, including on inputs it hasn't seen.

Our customers switch when cost goes down and accuracy goes up. A finetune that is accurate on familiar inputs but slips on new ones fails that, so accuracy has to hold across all their traffic.

One approach we have in mind is a router in front of the finetune:

• For each request, check whether it belongs to the distribution the finetune was trained on.
• If it does, the finetune answers.
• If not, the frontier model answers, and the request becomes training data for the next round.

Happy to hear other ways of getting there.

A finetune only goes live if it beats the customer's current model.

That is the promise, so it has to be measured, not assumed.

• Before launch: benchmark every finetune against the model the customer runs today, on their own data.
• In production: keep comparing, eg. send 5% of requests to both.

We can pull the evidence through an API.

Our customers sign off on every model change. To earn their trust, we have to show them how the finetune performs against the frontier model, not just tell them. So we want to fetch a report through the API that we can put in front of them:

• Accuracy per field, finetune vs frontier, on the held-out set and on sampled live traffic
• How that changes over time, so a regression shows up before the customer notices it
• Example cases where the two disagree, with the reviewed answer
• Cost per task, and the share of traffic the finetune handles

It has to work without our customers defining anything.

Our customers won't write evals, so it has to work out of the box.

• Good defaults from the agent's output schema: exact match per field, a valid schema, null instead of a guess.
• On top of that, our own domain-specific evals that catch known errors.
• An API to send human feedback on individual outputs.

Accuracy regresses over time, so training never stops.

We see this with clients, for two reasons:

• Domain logic changes. A frontier model handles new data better than a finetune.
• Agent configs change: prompts, docs, data from integrations. A new prompt means new outputs, and the finetune has to follow.

When the config changes, the frontier model takes over until the finetune catches up.

We'd tag each request with a config version. A finetune is only valid for the version it was trained on, so:

• When the config changes, traffic goes back to the frontier model.
• A retrained finetune takes over once it passes the evals on the new version.
• The requests routed to the frontier model (see Serve) become training data for the next round.

It has to be a drop-in replacement for OpenAI.

Our system runs on OpenAI-style APIs, so switching should mean changing the base URL. That includes structured outputs with JSON schema, since many of the errors we see are format errors.

Serving should be priced per token, with many finetunes on one base model.

Like Fireworks. We'll have a finetune per agent, and sometimes per end client: one customer has around 50 client-specific configs. A dedicated GPU for each doesn't work.

Training should be billed by GPU minutes.

Like Modal. We retrain often, so training runs need to stay cheap.

Discounts for work that doesn't need an answer right away. Later

A lot of agent work is back office: incoming post, document runs, evals and retraining. It needs to be done today, not this second. We want to pay less for that, the way every major provider already charges less:

Turnaround	What we'd expect	What the market charges today
Real time	Standard price	Standard price
Within 4 hours	At least 30% off	Fireworks Spot: 30% off. Flex tiers at Google and OpenAI: 50% off, with minutes of delay
Within 24 hours	50% off	50% off batch at OpenAI, Anthropic, Google, AWS Bedrock, Fireworks and Together
Training, whenever capacity is free	At least 60% off on-demand GPU time	AWS Spot GPUs: up to 90% off. SageMaker managed spot training: up to 90%, 65–70% in AWS's own examples

We finetune to save cost, and we retrain often. Without these tiers, retraining and back-office volume eat into the savings.

You run the GPUs.

Uptime and GPU utilisation come with a lot of complexity, and we don't want to take that over. Our customers already accept finetuning providers like Fireworks.

Data stays in the EU, and our customers can approve you.

Our customers are German enterprises processing personal data. Our contracts need their written approval before we share any data with a new subprocessor, including historical traces. To get that approval, we need:

• EU hosting, with the processing and access locations stated
• A DPA and your subprocessor list
• Zero data retention, with clear security and deletion arrangements
• No rights to reuse our data or the models trained on it

We also want to be able to export the finetuned weights.

Versioned models we can audit end to end. Later

Customers in insurance and banking have to show auditors how each decision was made, internal auditors and external ones. With a finetune in the loop, that means following the whole trace for any output:

• Which model version answered, and which base model and training run it came from
• Which training data and quality checks went into that version
• The evidence report it passed before going live
• The config version, input and output of the request itself

We also need to:

• Keep old versions and their logs for as long as the customer's regulation requires
• Pin a customer to a version, and roll back to an earlier one

Under the EU AI Act, credit scoring and life and health insurance pricing are high-risk systems. They need automatic logs kept for at least six months, longer where financial-services law requires, and handed to regulators on request. Those duties apply from December 2027 (Article 12).