M. MachucaAI Solutions Developer
PT

← Portfolio · M-07

Case study · IT service desk

Farol Desk

An IT service-desk agent where every answer shows what it understood, what it did and why.

The employee describes the problem in their own words. The agent checks their tickets, searches the help center, opens or updates a ticket once they confirm, and calls an analyst when it needs to. Built as the technical challenge of a hiring process.

Try the demo ↗ Code ↗ 4:49 video ›

Public demo, in Portuguese: sign in with one click as Ana (employee) or Rafael (analyst), with made-up data. Up to 100 messages a day across all visitors.

The analyst console in Farol Desk, in Portuguese: the conversation queue with each one's signals, the handed-off conversation with the case summary, and the person's tickets.
From the app · the analyst console, with the queue and the conversation
unit tests, no LLM
573
E2E tests against the production build
48
evals with the real model
35/36
red team, mapped to OWASP
21/21

Tests on 11 Nov 2026 · evals on 03 Nov 2026 · every number with its source under Evidence

Farol Desk in 60 seconds

A fictional company of about 2,000 employees has an IT team buried in triage: tickets without context, duplicates and wrong priorities.

What the agent does
It talks to the employee, searches the help center, opens or updates a ticket once they confirm, and calls an analyst when it needs to. Every answer shows what it did and why.
The proof
35/36 evals with the real model, 21/21 in the red team and 573 tests without an LLM, each number with its source in the repository.
See it work
Public demo ↗ in Portuguese, one click as Ana or Rafael, or the 4:49 video, recorded before the rename.
My role
Architecture, rules, acceptance criteria and evidence. Coding agents wrote the code, in small verified increments.
Delivered
28 Oct 2026 · 176 commits
Stack
Strict TypeScript · Next.js 16 · Mastra · DeepSeek-V4-Flash (DeepInfra) · Drizzle + libSQL/Turso · Redux Toolkit · Vitest · Playwright
Code
Public on GitHub, with specs, validation reports and the demo video. The docs are in Portuguese.

The brief

What was asked

A fictional company of about 2,000 employees has an IT team buried in triage: tickets without context, duplicates, wrong priorities, and simple questions waiting in the queue.

The hiring process asked for a chat agent (Nexo Assist in the brief, now Farol Desk) that answers questions from the help center, looks up and opens tickets, and hands the conversation to an analyst when needed. On the other side, a console where the analyst takes over with the agent paused, then hands it back or closes it.

The part the brief itself calls the most important: being able to understand, in every answer, what the agent did and why.

Requirements

  • Strict TypeScript; Mastra as the agent framework; any LLM, with the choice explained.
  • Service-desk rules: priority from impact and urgency, deadlines in business hours, no duplicates, reopening within 7 days.
  • A simulated ticket service, separate from the agent, behaving like a legacy system that sometimes goes down.
  • Conversations, tickets and the state of each conversation surviving a restart.
  • Security and privacy (one person's tickets never shown to another) and the cost per conversation under control.
  • Runs locally with a few commands and follows the screen prototype.

01The product

Both sides of the desk

Screenshots of the app with the new brand and made-up data. The app is in Portuguese.

Farol Desk support screen: the employee says the monitor won’t turn on and she has a meeting at 3 pm; the agent replies with a draft hardware ticket, high priority, with Confirm, Edit details and Cancel buttons.
From the app · the draft ticket, before the employee confirms

For the employee

Solved without a queue

  • They write the way they talk. The agent searches the help center and asks “Did this solve it?”; the answer feeds the resolution indicator.
  • It finds a ticket without its number and warns when the deadline is at risk (75% of the business-hours deadline), showing the math.
  • It builds a draft with priority and deadline computed in code. Nothing is created before the person confirms; “Change details” recalculates on the server.
  • When three or more people report the same problem within 4 business hours, it raises the impact to “team” without revealing who reported it.
  • High frustration brings up a “Talk to an analyst” button. An explicit request, a critical case, an overdue ticket or doubt that persists hands the conversation over.

por quê?

Every answer opens its why

The first step of the model is forced to record its understanding, through the recordUnderstanding tool; that is where the panel comes from. Any decision can be audited in the interface itself and in the analyst console.

The “por quê?” (“why?”) panel on every answer shows:

  • intent
  • confidence
  • reason
  • tools, with input and output
  • rules that fired
  • model
  • tokens
  • time
The “por quê?” (“why?”) panel open on one of the agent's answers, in Portuguese: what it understood, the intent, the confidence, the reason, the actions it took, the system rules that fired and the turn's telemetry.
From the app · the “why?” of one answer, with its telemetry
The conversation report open in the analyst console, in Portuguese: what the person needs, what the agent already did and what is pending.
From the app · the conversation report in the console

For the analyst

Arriving with context

  • A queue with signals: high frustration, possible mass incident, reason for the handoff.
  • The conversation report: what the person needs, what the agent already did and what is pending. It is built from the history and the traces, without calling the model.
  • Takes over with the agent paused, hands it back or closes it. The database only accepts the change if the state is still the expected one, so analyst and agent never step on each other.
  • “Suggest a reply”: the model writes, the analyst reviews before sending.
  • Everything in real time over the SSE stream, with polling as the fallback. When an analyst takes over a conversation the agent was handling, the person's chat changes at once.

Indicators

Honest indicators

Confirmed resolution kept apart from presumed, not counting open tickets or people who came back with the same problem, and help-center gaps with personal data masked.

The Farol Desk indicators screen, in Portuguese: resolution by the agent, response time, estimated cost and resolution split into confirmed, presumed and the other outcomes.
From the app · indicators computed from the database
Hands holding a crumpled sheet inside an open office printer.
Equipment photo · Freeing up a paper jam, gratuit / FreeImagesLive · CC BY 3.0 · resized

A photo of the problem

From photo to analyst

The person sends a photo or screenshot of the problem. A low-cost vision model describes the image, and the agent only gets the description, treated as data and never as an instruction. Only the person and the analysts can see the image.

3 Nov test. In the local demo, Ana attached a photo of an open printer. The agent identified crumpled paper, asked about a message on the control panel and suggested checking the power. The power check was generic for the visible problem. Ana chose to leave the open machine alone and asked for help. The conversation moved to the analyst queue. The test ended there, with the repair still open.

View the actual chat reply (Portuguese) ↗

That screenshot of the reply predates the rename and shows the old look.

How the photo becomes text

The vision model reads the photo. The agent receives the description as conversation data. The animation is in Portuguese, predates the rename and still carries the Nexo Assist name.

02Inside

The system plan

The same structure as the repository's src/.

app/ · agent/ · infrastructure/ pages and routes, the Mastra agent, database and repositories
application/ use cases and ports
domain/ pure service-desk rules: no database, no network, no framework
Dependencies only point inwards. The domain knows nothing about Next, Mastra or the database.

  1. Layers

    Lean DDD. Dependencies only point inwards: app/, agent/ and infrastructure/ depend on application/, which depends on domain/. The domain knows nothing about Next, Mastra or the database, so priority, deadlines and reopening are tested without mocks. Infrastructure implements the application's ports, and a single file wires the pieces together (container.ts).

  2. Agent and model

    A Mastra agent with 11 tools. The first step is forced to record its understanding (intent, confidence and reason), and code decides which tools exist at each step. The model is DeepSeek-V4-Flash, through DeepInfra, with a daily and a total spend cap: when the cap runs out, the chat says so instead of spending. The trace and the telemetry in the analyst console say which model answered.

    • recordUnderstanding
    • searchHelpCenter
    • askDiagnosticCheck
    • listMyTickets
    • getTicket
    • draftNewTicket
    • draftTicketUpdate
    • confirmDraft
    • cancelDraft
    • offerQuickReplies
    • transferToAnalyst

  3. Data

    Drizzle on libSQL: SQLite in development, Turso when deployed. Conversations, messages with their trace, drafts and tickets survive a restart. The legacy ticket system is simulated by a decorator that fails 15% of calls; the service retries without creating duplicates. The search embeddings are cached in the database.

  4. Real time

    The chat receives each turn as NDJSON: steps, cards and text. The analyst console, and the chat while an analyst is attending, open an SSE stream. The server checks a cheap database signature every second and says when something changed. If the stream drops, the screen falls back to polling on its own.

  5. Optional

    OpenTelemetry tracing with the GenAI conventions, ready for Langfuse, and OpenID Connect SSO (PKCE, state, nonce, JWKS). Both ship turned off and were tested against local servers that follow the protocols, not against real Langfuse, Entra ID or Okta.

Delivery

From commit to live

Vercel, in production

The app is live on Vercel, with the database on Turso, as a public demo since 29 Oct 2026. Cost is capped in code: up to 100 messages a day across all visitors, per-person limits and a total ceiling, all reserved in the database before the model is called. The link-based access gate stays in the code: without the PUBLIC_DEMO switch, Vercel requires the link again.

Each release comes from a git archive of the latest commit, so nothing outside git is uploaded.

git · maindeploy-prod.ps1Vercel + Tursolive

Docker, CircleCI and Komodo

The Docker image (Node 24, non-root, database on a volume) is 486 MB and passed all 25 smoke-test checks: it migrates, seeds the sample data, logs in for real, checks permissions and restarts without losing data.

CircleCI runs types, lint, tests, E2E and the same smoke test, then publishes the image to Docker Hub. Komodo checks the digest of latest every 10 minutes and redeploys. That path has run since 29 Oct 2026: on every push to main with green CI, Komodo pulls the new image within 10 to 20 minutes. CI never publishes to Vercel.

git · mainCircleCIDocker HubKomodorunning

On its first run, on 28 Oct 2026, CircleCI caught an error my machine was hiding: the LayoutProps type that Next 16 generates in .next/ does not exist in a clean clone. The fix was to run next typegen before tsc. The image is on Docker Hub, and Komodo runs it only on my machine, at 127.0.0.1:3300.

Source: docs/specs/04-arquitetura.md · src/agent/tools/index.ts · docs/DEPLOY.md

03Decisions

What I chose, and why

As in the product, every decision has a “why?” (“por quê?” in the app): it opens what I understood, what I did and which rules stayed.

D-01model

From Gemini to GPT-6 Luna and DeepSeek, decided by measurement

The production model has been DeepSeek-V4-Flash, through DeepInfra, since 2 Nov. Each switch only happened after measuring the candidate in the evals.

7 models · 24 conversations per model · Luna 26× cheaper, same quality · since 2 Nov, DeepSeek-V4-Flash: 35/36 in the 3 Nov evals, for US$ 0.07

why?
Understood

I picked Gemini for reliable tool calling (including the tool forced on the first step, which the “why?” panel depends on), low latency with reasoning set to minimal, and a cost of about one US cent per answer. It was a qualitative choice.

Did

I replaced the qualitative choice with data: I ran the production agent with 7 models, measuring effectiveness, quality (G-Eval with 2 judges), latency and real cost, under a US$ 1.80 cap. GPT-6 Luna tied with Gemini within the confidence intervals and cost 26 times less.

I kept Gemini while it had credit. When the Google credits ran out, on 29 Oct, I switched to Luna with one environment variable, after confirming it in the evals of that time.

On 2 Nov the OpenRouter credit ran out and the public demo was left without a model. I switched to DeepSeek-V4-Flash, through DeepInfra, again after measuring: 35 of 36 in the 3 Nov eval round, for US$ 0.07 for the whole run.

Rules that stayed

With it came a spend ledger, with a daily and a total cap, so US$ 3 lasts at least three weeks. When the cap runs out, the chat says so instead of spending.

Benchmark report ↗

D-02search

Search that knows when an article does not apply

The help center is searched by meaning (embeddings) and by keywords, and a cheap model says whether the article it found applies. With no article that applies, asking is better than a wrong answer.

81 questions: 56 → 78 right · wrong article: 22 → 0 · frozen set: 9/18 → 18/18 · US$ 0.00025 per search · without the judge, falls back to the margin

why?
Understood

A look-alike article leads the agent to a wrong answer, and asking is better.

The diagnostic articles exposed the limit of embeddings: they find the topic, not whether the article applies. “I'd like a wireless keyboard” landed on the repair article, and “the fan is making noise” got “Computer won't turn on”.

Did

I measured it on 50 questions written the way an employee would, including questions with no right article. A set of 17 was written only after the choice, to check it was not overfitted. On that set the overall score tied, and the report says so.

Search became two steps, as in Corrective RAG: hybrid search brings 5 candidates and a cheap model says which one applies, which one maybe does, or none. Before fixing it, I wrote 18 request × fault × not-covered questions and froze them in a commit. I compared rerankers, thresholds and a judge over the whole catalog.

Rules that stayed

A threshold only worked at one exact value, so the categorical decision stayed: applies, maybe or none. Without the judge, search falls back to the margin: when the top result does not stand out from the second, it returns nothing.

BUSCA.md ↗research ↗

D-03guarantees

Rules in code, not in the prompt

The model decides what to do. Whatever must be guaranteed lives in tested code, and the prompt only handles how to talk.

step-policy.ts · domain/priority.ts · domain/business-time.ts · draft-service.ts

why?
Understood

A malicious text can change what the model wants to do, and a misread prompt must not be able to break a service-desk rule.

Did

Which tools exist at each step is decided in code, following the action-selector pattern: the model only picks among them. The malicious text changes what it wants to do, but not what it is able to do.

Rules that stayed

Priority from the impact × urgency matrix, deadlines in business hours, mandatory and atomic confirmation, reopening only within 7 days, and identity taken from the signed cookie, never from a model argument.

Guarantees table in the README ↗

D-04measurement

Honest evals

The evals check the database state, not only the text of the answer, and the result is published as it came out, even when it is low.

database checked · attacks × 3 · provider failures kept apart · pass^1 0.72 → 0.93 and pass^3 0.50 → 0.80 after the fixes

why?
Understood

When Gemini returned a 503 inside the stream, an attack could “pass” without the model having answered. And there were failures the fixed-case evals did not catch.

Did

I built the simulated employee (pass^k, as in τ-bench): a model plays the employee. In the first round it found four real bugs, such as opening a ticket from a quick reply that was not a “yes”. Each one became a rule in code with a test, not a prompt tweak, and the next round was measured the same way.

Rules that stayed

Each attack runs 3 times, and passing 1 out of 3 counts as a failure. Provider failures are counted separately. The simulated employee is published with the result it gave, even when it is low.

docs/validacao ↗

04Evidence

The numbers, with sources

Unit and E2E tests run without an LLM. Anything that depends on the model runs against the real model and has a report with the question, method, numbers and limits.

Unit and E2E tests on 11 Nov 2026. Model evals on 03 Nov 2026. Benchmarks and simulations carry their own dates.

unit tests
573
Vitest, no LLM: deadlines, priority, conversation states, privacy, duplicates, CSRF, OIDC against forged tokens, pass^k. There were 563 before the 11 Nov review.
README · npm test
E2E tests
48
Playwright against the production build, no LLM: login and SSO, handoff, report, indicators, real time, attached image, errors.
README · npm run test:e2e
evals with the real model
35/36
Intent, expected and forbidden tools, handoff, critical case and data leaks. Among them, 8/8 attacks, 6 scenarios taken from the simulated employee’s findings and 2 diagnostic-check scenarios. On 3 Nov, DeepSeek-V4-Flash passed 35 of 36 scenarios for US$ 0.07. In the time-clock incident it asked about a workaround and returned no ticket draft. The 2 Nov run shown in the video passed all 36.
docs/validacao/README.md
red team, mapped to OWASP
21/21
7 attacks × 3: injection, data leaks, acting without confirmation, system prompt, goal hijack. Indirect injection through a help-center article: 3/3.
docs/validacao/RED-TEAM.md
simulated employee
0.93 pass^1 · before 0.72
A model plays the employee, with facts it only shares when asked. Three rounds on 30 Oct, still with Gemini: 0.72 / 0.50 → 0.93 / 0.80 → 0.93 / 0.80, the third with one persona re-run after its fix. With 10 personas × 3, 28/30 conversations right; pass^3 0.50 → 0.80 (8/10 personas right in all 3 runs). The first round found real agent bugs, fixed with rules in code and tests, not prompt tweaks.
docs/validacao/SIMULACAO.md
help-center search
56 → 78 of 81 · 0 wrong article
Keywords against hybrid search with a relevance judge, on four sets; 18 request × fault × not-covered questions frozen before the fix: 9/18 → 18/18.
docs/validacao/BUSCA.md
SSE against polling
263–580 ms
Median with the stream, against 1,505–1,684 ms with the old polling. The E2E test fails if p95 goes over 2 s.
docs/validacao/TEMPO-REAL.md
suggested reply
9/9
3 conversations × 3, with no invented number or action in the analyst's draft.
docs/validacao/README.md
security reviews
5
Before and after deploying, 42 black-box checks against production, and a round with third-party audit skills. On 2 Nov, an external audit raised 6 risks (demo login left on alongside SSO, unverified e-mail, a reply after the handoff, duplicate ticket, daily cap and duplicate conversation), all fixed with tests. npm audit with no vulnerabilities. The 11 Nov review of the image attachment is in the next section.
docs/DEPLOY.md

05New · 11 Nov 2026

Review and a new name

On 11 Nov the project was renamed Farol Desk and got a new look. Only the brand changed: the repository, the demo URL and the Docker image keep the old name, so links already sent keep working. On the same day, a review checked the app end to end.

  1. Delivered as a technical challenge, under the name Nexo Assist.

  2. Demo opened to the public on Vercel. Image delivery (CircleCI → Docker Hub → Komodo) starts running.

  3. DeepSeek-V4-Flash in production, with a spend cap. External audit: 6 risks, fixed with tests.

  4. Image attachment in production. Evals: 35/36.

  5. End-to-end review and the new name: Farol Desk.

The review's questionDoes the app work end to end with the new brand, and what does a security review of the image attachment, live since 3 Nov, find?

What was checked

npm run typecheck · npm run lint
Clean.
npm test
573 tests in 77 files (up from 563).
npm run test:e2e
48/48 against the production build.
public demo · node scripts/checar-demo.mjs
Login, conversation API and case study responding.
real flow in the browser, with DeepSeek-V4-Flash
VPN error 809 → the right article and the Windows 10 or 11 question → impact and urgency → draft with High priority and the deadline compared with the 3 pm meeting → handoff → analyst console with the summary. No errors in the browser console.

Three defects, found and fixed

  1. E2E would not start on this machine.The test OIDC provider used port 3200, which is Grafana Tempo's default port; with the observability stack running in WSL, listen failed with EADDRINUSE while nothing showed up in Windows netstat. The E2E ports are now configurable (E2E_PORT, E2E_IDP_PORT) and the provider's default moved to 3210.
  2. The analyst's suggested reply bypassed the spend cap.It called the model without going through the dollar cap and without recording the cost. It now checks the cap first and records the cost from the tokens (two new tests).
  3. An ambiguous button in the diagnostics.“Já estava assim” (“It was already like that”), in reply to a yes-or-no question, did not say whether the check was right or wrong. It became “Já estava certo” (“It was already right”); the old label is still accepted in recorded conversations.

Security review of the image attachment

Done by a separate reviewer, read-only, over the image-attachment code, which had been in production since 3 Nov. Nothing critical, but the high finding applied to the public demo until this fix.

What was correct: access control on the image (the conversation's owner and an analyst in the same space; anyone else gets the same 404), type checked from the first bytes, the image description enters the history marked as untrusted data and cannot confirm a ticket, no secret in code or in an error response, migration 0009 is additive.

Findings: 1 high, 2 medium and 4 low
SeverityFindingFix
HighThe image limit was per conversation only. In the public demo, opening new conversations reset the count, and the database could be filled (up to 1 MB per message).Two atomic counters checked before writing: 10 images per person per day and DAILY_SCREENSHOT_LIMIT for the whole day (default 100). Three new tests.
MediumThe request body was read in full before the schema complained about its size. Outside Vercel, a POST of hundreds of MB would exhaust memory.parseBody counts the bytes as it reads and answers 413 once past the cap (100 KB on ordinary routes, 1.45 MB on the chat). Three new tests.
MediumThe image stayed stored even when the turn was later refused by another cap.The order did not change (the refused message is stored too, by design), but storage now has a daily cap, checked before writing.
LowThe image stayed in the browser cache for 1 hour, without checking the session.Cache-Control: private, no-cache and a CSP sandbox on the image response.
LowThe vision provider's error message went into the step the employee reads.A fixed reason in the conversation; the raw message only in the server log.
LowA failed image read did not enter the spend ledger.It enters as an estimate (US$ 0.0002).
LowTwo simultaneous requests could exceed the quota of 5 per conversation.Bounded by the day's atomic counters; the per-conversation quota is still enforced by counting.

Limits of this review

  • Image reading was not exercised with the real model in this round (only through tests with a test double and through E2E, which does not call the model).
  • The platform's body limit (Vercel) was not measured; the app's was unit-tested.
  • The embeddings and the relevance judge are still outside the spend ledger (less than US$ 0.001 per conversation).

Source, in Portuguese: docs/validacao/REVISAO-2026-11-11.md

06Demo video

From conversation to ticket, and from ticket to analyst

Carla sends a photo of her dark monitor and finds the loose HDMI cable during the checks. In a separate session, Ana sends a photo of a printer with jammed paper and asks for an analyst. Next, a time-clock failure becomes a ticket. The video runs 4:49. The monitor and time-clock scenes were recorded on 2 Nov. The printer conversation was tested on 3 Nov. The 3D sequence shows how a photo becomes text for the agent.

Recorded before the rename. The video shows the old name and look (“Nexo Assist”) and the “Já estava assim” button, which became “Já estava certo” on 11 Nov. The numbers shown in it come from the 2 Nov measurements.

H.264 · 1280×720 · 30 fps · 19.5 MB · plays only when you press play · Portuguese narration with a synthetic voice (Fish Audio) · original electric piano, bass and soft chords · original 1080p in the repository ↗

07Still open

What is left

  • What the recording revealed

    The time-clock ticket took six rounds of questions, and the handoff summary included a detail Carla had not reported. The third point, the “Já estava assim” button, was fixed on 11 Nov.

  • What the simulated employee caught

    Every bug from the three rounds became a rule in code, with tests: confirmation must be explicit (and a “Sim, é esse mesmo”, “yes, that’s the one”, that only identifies the ticket reopens nothing), a different problem gets a new ticket, Outlook is filed under E-mail, an analyst is offered only with a reason (a deadline that misses the need, an overdue ticket or frustration) and the workaround question comes before the priority. Re-run after the 29 Oct fixes, senha-sap and vpn-reuniao passed 3 out of 3, and so did the reopen eval. What remains is the model’s own variance: in the full suite on 29 Oct, one mass-incident scenario failed once and passed 3 out of 3 when repeated.

  • Limits of the 11 Nov review

    Image reading was not exercised with the real model in that round, only through tests with a test double and through E2E, which does not call the model. The platform's body limit (Vercel) was not measured; the app's was unit-tested.

  • Image delivery is local only

    The CircleCI → Docker Hub → Komodo path runs, but Komodo lives on my machine, on loopback only: it proves image delivery, it is not a second public environment. Production stays on Vercel, published by hand.

  • Small sample

    10 personas × 3 runs is too few to pin down a rate. The numbers show the direction, not a guarantee.

  • Spend cap

    The public demo has US$ 0.07 a day for the model. When it runs out, the chat says so and comes back the next day, with no fallback model. A conversation solved without an analyst costs about US$ 0.005 in model calls. Since 11 Nov the reply suggestion goes through the cap and records its cost; the embeddings and the relevance judge are still outside the spend ledger, at less than US$ 0.001 per conversation (cost report, in Portuguese ↗, scaling plan ↗).

  • Repeated calls

    Tracing showed the model calling the understanding tool several times in parallel on the first step. Only the first one counts, but the others spend tokens.

  • Evals outside CI

    They run by hand because they call the real model. The next step is running them in CI with a minimum threshold.

  • Real services

    SSO was only tested against a fake provider, and tracing against a local collector. Logging out does not revoke the JWT on the server, and the console needs a wide screen.

  • Material with the old brand

    The demo video, the image-reading animation and the screenshot from the printer test predate 11 Nov and still show the Nexo Assist name.

Open the demo and click “por quê?”.

Try the demo ↗ See the repository ↗ Contact on LinkedIn ↗ ← Back to the portfolio