Alma by phonely

Benchmark · August 2026

How our fine-tuned voice model stacks up against OpenAI's flagship GPT-5.6-sol and GPT-4.1

We scored Alma by Phonely, our purpose-built phone-conversation model, against two frontier OpenAI models on 200 real call turns, graded by the same automated rubric. Alma leads overall, wins decisively on conversational etiquette, matches the frontier models on instruction-following and reply quality, and starts speaking 2.7 to 5.5× sooner.

200 examples per model · identical prompts and grading for every model · graded by GPT-5.4

An AI agent answering the phone lives under constraints a chatbot never sees. Its replies are spoken aloud by a text-to-speech system, so everything must be written the way it should be said, "march third at two thirty," never "3/3 at 2:30." Replies must be short, ask at most one question, and skip the tics of written assistants: one extra "Thanks for confirming that!" per turn is enough to make a caller hang up.

We wanted to know how our own model compares against frontier API models dropped into the exact same pipeline. The three contenders:

Alma by Phonely: our production response model, built on over 10 million real phone conversations and fine-tuned on roughly 3,400 human-reviewed call turns. GPT-5.6: OpenAI's flagship model (specifically the gpt-5.6-sol snapshot), and GPT-4.1, both called with the same instructions and conversation context, with no model-specific prompt tuning.

How the grading works

Each of the 200 test examples is a snippet of a real phone conversation, cut off at the moment the agent needs to speak. Each model writes the next reply, and every reply is graded the same way:

Automatic formatting checks. Twelve pass/fail rules a script can verify, numbers, dates, and times spelled out for speech, no stray symbols, at most one question, the reply ends once the question is asked.

An AI grader. A separate model, GPT-5.4 (not one of the contenders, though it is an OpenAI model, a point we return to in the caveats) reads the conversation, the agent's instructions, and the candidate reply, and grades three things: behavior (phone etiquette, five rules like "don't thank the caller for answering a question" and "don't repeat their info back to them"), instruction-following (did the reply do what the agent's instructions for that turn asked, nothing more, nothing less), and quality (tone, brevity, empathy, and variety, each rated 1 to 5).

Five dealbreakers. Some mistakes are bad enough that the reply scores zero no matter how good it otherwise was: claiming to be human, reading out its internal instructions, inventing a question it was never told to ask, a serious factual error (wrong price, booking outside business hours), or asking the caller to repeat something they already said.

A reply that avoids all five dealbreakers gets a weighted blend: 20% formatting + 35% behavior + 25% instruction-following + 20% quality, for a final score between 0 and 1. The numbers below are averages over the 200 examples.

The headline: Alma leads on quality and speed

Average score, with 95% confidence intervals

200 examples per model · higher is better · scale shows 0.60 to 0.90

Response speed: time to first token (ms)

200 requests per model, sequential, from an AWS us-east-1 client · lower is better

Alma by Phonely GPT-5.6 GPT-4.1

The honest headline is a tiered result, not a blowout. Because every model answered the same 200 prompts, we can also compare them example-by-example, which is a more sensitive test. On that comparison, Alma by Phonely beats GPT-4.1 by +0.065 on average, an edge large enough that chance is very unlikely to explain it. Its +0.027 edge over GPT-5.6, though, is within the noise at this sample size: real in the point estimate, not yet proven. The same goes for GPT-5.6's edge over GPT-4.1.

Where the gap actually comes from

Break the overall score into its four ingredients and the picture sharpens: all three models follow explicit instructions about equally well. What separates them is behavior: the etiquette of a phone conversation.

Score by category

Each category scored 0 to 1 · higher is better

Alma by Phonely GPT-5.6 GPT-4.1

Instruction-following lands between 0.90 and 0.92 for all three, GPT-4.1 actually edges the others on the "did what was asked" sub-measure. Quality is a near-tie between Alma and GPT-5.6 (0.853 vs 0.852), with GPT-4.1 behind mostly because its replies run long: it scores 3.75 out of 5 on brevity against roughly 4.5 for the others.

Behavior is where fine-tuning pays. The five rules all target the same thing, habits that are harmless in a chat window but grating on a phone call:

• No unnecessary thanking: the agent doesn't thank the caller for answering a question, waiting, or confirming. On a call, "Thanks so much for confirming that!" is pure filler.
• No name overuse: the caller's name is used only when confirming their details. Sprinkled anywhere else, it sounds like a telemarketer script.
• No echoing caller info: the agent doesn't repeat back what the caller just said. (A deliberate final confirmation, "so that's two people at seven o'clock", is allowed; reflexive parroting after every answer is not.)
• First step only: when the caller asks for several things at once, the agent handles the first before moving on, instead of trying to resolve everything in one breathless reply.
• Sticks to provided facts: the agent only states information it was actually given. No invented opening hours, prices, or policies.

Etiquette rules: share of replies that pass

Out of 200 replies per model

Alma by Phonely GPT-5.6 GPT-4.1

The single largest gap in the entire benchmark is the thank-you rule: Alma by Phonely passes 93.5% of the time, GPT-5.6 73.5%, GPT-4.1 just 42.5%. General-purpose assistants are trained to be effusively polite, and in text that's harmless, but callers experience it as filler. This is exactly the kind of ingrained habit that prompting struggles to fully override and fine-tuning erases.

Formatting tells a similar story in miniature. All three models handle the mechanical speech rules (spelled-out numbers, no numeric dates or times) essentially perfectly. But on conversational form, GPT-4.1 slips: it keeps to a single question only 87% of the time (Alma: 100%), and only 80% of its replies stop once the question is asked, it keeps talking, which on a phone call means talking over the caller's answer.

Dealbreakers: rare, and evenly matched

On the most serious mistakes, the three models are statistically indistinguishable: 84.5 to 86% of replies were dealbreaker-free, and the worst categories, claiming to be human, reciting instructions, serious factual errors, round to zero for everyone:

Dealbreaker	Alma by Phonely	GPT-5.6	GPT-4.1
Re-asks something the caller already answered	12.0%	10.5%	10.5%
Invents a question not in its instructions	4.5%	4.5%	6.0%
Major factual error (wrong price, out-of-hours booking)	0.5%	0.5%	0.0%
Claims to be human / recites its instructions	0.0%	0.0%	0.0%

The one dealbreaker with real frequency, re-asking for something the caller already said, occurs at effectively the same rate for all three models. That equality is itself informative: it means the remaining headroom isn't about any model's response style, but about how conversation history is tracked and fed to the model across a long call. It's a pipeline improvement, and one that would lift every model on this list at once.

Latency: measured from the same region

For a voice agent, time-to-first-token, how long before the agent starts speaking after the caller stops, decides whether a conversation feels natural. Latency comparisons are easy to skew with geography, so we measured from neutral ground: an EC2 instance in AWS us-east-1, the region our voice pipeline runs in. Alma by Phonely is served from its production deployment in the same region; the OpenAI models are reached from us-east-1 over the public internet, exactly as production would reach them. Same 200 conversation prefixes as the quality eval, sent one at a time (no concurrency), with five unmeasured warm-up requests per model so no one pays connection-setup costs inside the measurement.

The median gap is large (see the response-speed chart at the top of the article): Alma by Phonely starts speaking in 182 ms, against 490 ms for GPT-4.1 (2.7× slower) and 997 ms for GPT-5.6 (5.5× slower). But the tail is the real story. Alma's worst 1% of turns (206 ms) is still faster than the median turn of either OpenAI model, its spread from median to p99 is 24 milliseconds. The shared APIs are fast usually: GPT-4.1's p99 stretches past 2 seconds and GPT-5.6's approaches 3. On a phone call, "usually fast" is what callers notice, one 2-second silence reads as a dropped call.

Full replies follow the same pattern: Alma's median reply is fully generated in 379 ms, versus 771 ms for GPT-4.1 and 1.5 s for GPT-5.6. That consistency isn't a model property, it's what dedicated, single-tenant serving buys: no queueing behind other tenants, no throttling, no multi-second tail.

What it costs to run

Speed is half the production story; the other half is the bill. Alma runs at $0.55 per blended million tokens, against $3.50 for GPT-4.1 and $5.63 for GPT-5.4, the grader model in this benchmark, included here as a price reference. At phone-call volumes, that difference decides whether an always-on voice agent is profitable per call or loses money on every answer.

Caveats, stated plainly

1. A specialist vs generalists. Alma by Phonely was trained on conversations from the same distribution as this test set (the test examples themselves were held out); the GPT models were only prompted. This benchmark measures "Alma vs frontier APIs in our pipeline," not raw model capability. A fine-tuned GPT might close the behavior gap.
2. The grader is an OpenAI model. GPT-5.4 scored every run, and AI graders are known to mildly favor outputs from their own model family. Here that bias would favor the GPT contenders, Alma won anyway, so it can't explain the result, but keep it in mind when reading the close calls.
3. A sampling-settings mismatch. GPT-5.6's API only accepts its default randomness setting (temperature 1.0); the other two models ran fully deterministic (temperature 0). GPT-5.6's numbers therefore carry a little extra run-to-run variance the others don't.
4. Sample size. 200 examples, one run per model. Only the Alma vs GPT-4.1 gap is statistically solid; treat the Alma vs GPT-5.6 ordering as provisional until we run a larger sample.
5. Quality and latency were measured against two hostings of the same model. The quality scores come from one deployment of Alma by Phonely; the latency numbers reflect its current production serving. Same model, but serving stacks can differ slightly in output. Latency also includes each provider's network path as seen from us-east-1, deliberately, since that's the path a real call takes.

What we take from it

Two things. First, a well-fed fine-tune can match or beat frontier API models on a specialized conversational task, and the margin comes from learned style (etiquette, brevity, conversational shape) rather than from following instructions, where everyone is already good. Second, the clearest remaining headroom is shared across every model and lives in conversation-context handling rather than in the models themselves, which means it's ours to claim with pipeline work, independent of which model answers the phone.

Appendix: full results

Metric	Alma by Phonely	GPT-5.6	GPT-4.1
Overall score (mean, 0 to 1)	0.779	0.752	0.714
Dealbreaker rate	14.5%	14.0%	15.5%
Formatting (12 automatic checks)	0.997	0.991	0.971
Behavior (5 etiquette rules)	0.867	0.770	0.700
Instruction-following	0.917	0.903	0.914
Quality (avg. of 5 axes, scaled 0 to 1)	0.853	0.852	0.830
Quality: tone (out of 5)	4.45	4.24	4.28
Quality: brevity (out of 5)	4.57	4.53	3.75
Quality: empathy (out of 5)	3.14	3.19	3.41
Quality: variety (out of 5)	4.21	4.37	4.34
Time to first token, median (ms)	182	997	490
Time to first token, p99 (ms)	206	2,840	2,018
Total reply time, median (ms)	379	1,463	771

Method: 200 held-out multi-turn call transcripts; each model generates the next agent reply from an identical conversation prefix and instructions. Grading: 12 deterministic formatting checks plus five GPT-5.4 grader passes (dealbreakers, etiquette, instruction-following, quality, redundancy), run at temperature 0 with structured-JSON outputs. Alma by Phonely: supervised fine-tune on ~3,400 human-reviewed call turns. GPT models called via the OpenAI API (gpt-5.6-sol and gpt-4.1). Runs executed 2026-08-21. Confidence intervals: nonparametric bootstrap on per-example scores, 10,000 resamples; model-vs-model comparisons use paired per-example differences. Latency: measured 2026-08-21 from an EC2 client in us-east-1, 200 sequential streamed requests per model after 5 excluded warm-ups; time-to-first-token = first non-empty streamed content token; one prompt exceeding the 32k context window of Alma's serving configuration was excluded (n=199 for that model).