D-01 · model
From Gemini to GPT-6 Luna, decided by a benchmark
I picked Gemini for reliable tool calling (including the tool forced on the first step, which the “why?” panel depends on), low latency with reasoning set to minimal, and a cost of about one US cent per answer.
Then I replaced the qualitative choice with data: I ran the production agent with 7 models, measuring effectiveness, quality (G-Eval with 2 judges), latency and real cost, under a US$ 1.80 cap. GPT-6 Luna tied with Gemini within the confidence intervals and cost 26 times less. I kept Gemini while it had credit. When the Google credits ran out, on 29 Oct, I switched to Luna with one environment variable, after confirming it in the evals: 36/36.
7 models · 24 conversations per model · Luna 26× cheaper, same quality · Flash Lite as fallback
Benchmark report ↗
D-02 · search
Search that knows when an article does not apply
The help center is searched by meaning (embeddings) and by keywords. When the top result does not stand out from the second, the search returns nothing: a look-alike article leads the agent to a wrong answer, and asking is better.
I measured it on 50 questions written the way an employee would, including questions with no right article. A set of 17 was written only after the choice, to check it was not overfitted. On that set the overall score tied, and the report says so.
The diagnostic articles exposed the limit of embeddings: they find the topic, not whether the article applies. “I'd like a wireless keyboard” landed on the repair article, and “the fan is making noise” got “Computer won't turn on”. Search became two steps, as in Corrective RAG: hybrid search brings 5 candidates and a cheap model says which one applies, which one maybe does, or none. Before fixing it, I wrote 18 request × fault × not-covered questions and froze them in a commit. I compared rerankers, thresholds and a judge over the whole catalog; a threshold only worked at one exact value, so the categorical decision stayed.
81 questions: 56 → 78 right · wrong article: 22 → 0 · frozen set: 9/18 → 18/18 · US$ 0.00025 per search · without the judge, falls back to the margin
BUSCA.md ↗ research ↗
D-03 · guarantees
Rules in code, not in the prompt
The model decides what to do. Whatever must be guaranteed lives in tested code: priority from the impact × urgency matrix, deadlines in business hours, mandatory and atomic confirmation, reopening only within 7 days, and identity taken from the signed cookie, never from a model argument.
Which tools exist at each step is also decided in code, following the action-selector pattern. A malicious text can change what the model wants to do, but not what it is able to do. The prompt only handles how to talk.
step-policy.ts · domain/priority.ts · domain/business-time.ts · draft-service.ts
Guarantees table in the README ↗
D-04 · measurement
Honest evals
The evals check the database state, not only the text of the answer. Each attack runs 3 times, and passing 1 out of 3 counts as a failure. When Gemini returned a 503 inside the stream, an attack could “pass” without the model having answered; that now counts as a provider failure, reported separately.
The simulated employee (pass^k, as in τ-bench) is published with the result it gave, even when it is low. In the first round it found four real bugs, such as opening a ticket from a quick reply that was not a “yes”. Each one became a rule in code with a test, not a prompt tweak, and the next round was measured the same way.
database checked · attacks × 3 · provider failures kept apart · pass^1 0.72 → 0.93 and pass^3 0.50 → 0.80 after the fixes
docs/validacao ↗
