I gave both models the same 30 business tasks, three times each. Qwen finished with five more passes.
My agent’s tasks. The same setup. Not a universal ranking.
The cost gap was much biggerTypical completion: Claude 15.3s · Qwen 25.1s
Median time per test. The score gap is modest, not a proven quality advantage.
gap in recorded cost
per successful test
Spending on all 90 attempts divided by passes. Bars start at zero.
For my product, that was enough reason to keep testing lower-cost models.
These bills are incomplete, so both costs are minimums. The 21.5× gap describes recorded spending, not guaranteed savings. Scoring costs are separate.
Compare all six shortlisted models ↗Clearer instructions. Better tools. One check before answering.
Early run: 18 of 30 passed.
Final version: 81 of 90 passed across three runs.
Pass Fail · 24 of 30 tasks passed every time.
The model matters. So does everything I put around it.
Several changes were tested together, using questions I had already developed against. This shows progress on those tasks—not proof of which change caused it, or how it handles new tasks.
See what helped—and what didn’t ↗Compare the models, inspect any test, or follow the experiments step by step.
I measured success, cost and speed on the tasks I need my agent to handle.
Bars start at zero. A pass means the required answers and actions held up to review. Cost per pass = spending across all attempts ÷ passes, including failed attempts. ≥ marks an incomplete cost record; cost gaps are based on recorded spending, not guaranteed savings. The cost of scoring the answers is separate.
| Model | Tests passed | Run scores | Cost / attempt | Typical completion |
|---|
Execution includes the task model and operational helpers, with observed caching; grading is separate. “At least” marks a lower bound where costs were unknown. These are recorded experiment costs, not current price quotes. Provider route, harness and task selection affect results. Reused baseline rows are not new independent runs. Model names are the configured identifiers recorded by the experiment, not independently verified weight identities.
I screened ten selected workflows, then chose Gemini and Ling for additional runs. That selection means I can’t pool screening and confirmation results into an unbiased winner claim. A separate Claude SDK check passed 7/10 once; it is not the shared-loop Claude baseline.
My biggest takeaway: model choice, tool design and grading all mattered.
I’m encouraged by the result, but I wouldn’t generalize it beyond these tasks. I still need evidence on unseen questions, production behavior and voice latency.
I expected clearer guidance to help. Sometimes it did. Sometimes it just added more instructions.
My bar: the agent has to answer correctly and take the right actions. A plausible reply isn’t enough.
I’d enforce facts and permissions in code, and remove extra instructions when the model no longer needs them.
I can’t expect the agent to interpret a number if I haven’t made its unit clear. Accounts, people and account–program memberships are different. Separate stage counts don’t automatically form a conversion funnel.
I want tool responses to say exactly what happened. A draft isn’t saved. Queued isn’t delivered. An old “sending” event doesn’t prove another job is still pending.
The automatic grader was more forgiving than I wanted. Asking the model what tripped it up helped generate hypotheses; I still needed those checked against the actual inputs and tool results.
The agent found a sales leaderboard. The aggregate content tool was unavailable, so it stopped—without inspecting the individual creator records that could provide content evidence.
This failed all three final runs. I’d try clearer guidance on where to find evidence, or a better tool response. I haven’t tested that fix yet.
I want you to be able to question the result, with enough detail to see where it holds up and where it doesn’t.
$5.18 total for the final 90 trials. The full final development batch was $11.54 known plus $1.02 held for uncertain charges, against a $20 cap. Earlier batches are separate. Subscription-based review effort is not included.
All required turns, supported factual claims and correct durable action outcomes. Provider errors and unfinished answers count as failures. A correct table does not erase an unsupported conclusion.
I used model-assisted reviews checked against tool traces and local state, not blinded human judgments. The reviews allowed minor wording caveats and documented them.
I started with questions from real product demos for a creator-marketing operations agent. I used frozen, synthetic data, simulated outbound services, discovery and brief generation, and real action handlers with isolated durable state. For this public report, I’ve kept scenario names generic and paraphrased review notes. Raw conversations, customer records, credentials and private source code aren’t included.
I used the same 30 workflows during development and final confirmation. I didn’t use an unseen holdout for the final result. Twenty-four workflows passed 3/3, but three that passed the single DeepSeek baseline failed at least once later. A 90% average is not a no-regression guarantee, a confidence interval, or proof of channel parity. I hadn’t deployed this experiment to production at the time of this report.
The earlier model comparison, ten-workflow screen, Qwen guidance study, interrupted bundle and DeepSeek development used different scopes or treatments. The DeepSeek development sequence changed several things together; its gains cannot be assigned to one prompt sentence. A configuration-saving case also needed missing synthetic prerequisites, a real keyword-save capability and a narrowly documented grading correction. The matching prerequisite baseline still failed. I’m disclosing those changes because I can’t count them as pure model gains.
I used this research to frame the experiments. It informed the design; it doesn’t validate my results.
Selected reviewed experiments from September 11–13, 2026. Exact counts and cost values are retained in the download. Raw source artifacts remain private.