LLM Evaluation: Evals, LLM-as-Judge and Observability for AI Apps

LLM evaluation is the practice of measuring whether an AI application gives good answers, takes the right actions and keeps doing so as prompts, models and data change. It combines a fixed set of test cases ("evals"), automated graders including LLM-as-a-judge, regression checks before every release, and tracing of what the system does in production.

It matters because LLM apps fail quietly. A prompt tweak that fixes one complaint can break ten other cases. A model upgrade can change tone, format or accuracy overnight. Without evals, you find out from customers.

This guide explains how to build an evaluation set, which graders to use, what research says about LLM judges, how to wire evals into your release process, and how to trace agents in production.

Why LLM apps need a different kind of testing

Traditional software tests check exact outputs: given this input, return that value. LLM outputs vary in wording even when they are correct, and correctness itself is often a judgement call. A summary can be accurate but miss the one detail the doctor needed. An agent can reach the right result through a different, equally valid path.

So LLM testing shifts from "is the output identical?" to "does the output meet the criteria?", and from a single pass/fail run to tracking scores across hundreds of cases over time.

The three layers of LLM quality

You need all three. Offline evals stop known problems coming back. Monitoring finds new problems. Human review keeps the graders honest.

Building an evaluation set

Start small and real. Anthropic's account of building its multi-agent research system notes that the team started with about 20 representative queries, because early changes have large effects that a small set detects easily. Waiting for a perfect thousand-case suite delays the feedback you need most.

Good sources of test cases:

• Real user requests from logs, de-identified.
• Every bug report and complaint, turned into a test case once fixed.
• Edge cases from domain experts: the tricky prescription, the ambiguous refund request.
• Adversarial inputs: prompt injection attempts, off-topic requests, abusive messages.
• Cases where the correct answer is "I don't know" or "let me transfer you".

For each case, record the input, any context documents, the expected outcome or criteria, and why the case exists. Grow the set to a few hundred cases as the product matures, and keep a separate held-out set so you do not tune prompts to the test.

Choosing graders

Use the cheapest grader that measures what you care about. If a field must be a valid date, check it with code, not a judge.

LLM-as-a-judge: what the research says

The idea took off with the 2023 paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, which found that a strong judge model agreed with human preferences more than 80% of the time, about the same rate at which humans agree with each other. The same paper documented biases that still matter:

• Position bias: in pairwise comparisons, judges tend to favour the answer shown first.
• Verbosity bias: judges tend to favour longer answers even when they are not better.
• Self-enhancement bias: judges may favour outputs from their own model family.

Practical rules for using judges well:

• Write a specific rubric with concrete criteria, not "rate the quality from 1 to 10".
• Ask for a short reason before the score, and a clear pass/fail alongside any numeric score.
• In pairwise comparisons, run both orders and only count consistent verdicts.
• Use a different model family for judging than for generating, where you can.
• Calibrate: have humans grade 50-100 cases and check the judge agrees before trusting it.
• Judge outcomes, not paths. For agents, check whether the final result is right rather than whether it followed your preferred steps.

Regression testing in your release process

Treat prompts, model versions and retrieval settings as code. Every change runs the eval suite, and the release is blocked if key scores drop.

1. Store prompts and configuration in version control.
2. Run the eval suite in CI on every pull request that touches them.
3. Compare against the last released version, not an absolute target, and show the diff of failing cases.
4. Set thresholds per metric: zero tolerance for safety failures, small tolerance for style.
5. Re-run the full suite whenever a model provider updates a model, even if your code did not change.
6. Track cost and latency in the same report; a "better" prompt that triples tokens is a business decision.

Observability: tracing what actually happens

Evals tell you how the system behaves on known cases. Tracing tells you what it did on real ones. For an agent, a single request can involve several model calls, retrievals and tool calls, and you need to see them as one tree with timings, token counts, costs and errors.

The emerging standard is OpenTelemetry's generative AI semantic conventions, which define spans and attributes for model calls, agents and tools, including operations such as invoking an agent and executing a tool. According to a May 2026 overview from Greptime, the conventions were still in Development status, and in June 2026 the project moved them into a dedicated repository so they can be versioned independently. Attribute names have changed along the way, so pin your instrumentation version and expect updates.

What to capture per request:

• The user and session (pseudonymised where possible).
• Each model call with model name, prompt version, tokens in and out, latency and cost.
• Each retrieval with query and document IDs returned.
• Each tool call with arguments, result status and duration.
• The final output and any user feedback.

For healthcare and other regulated settings, be deliberate about content. Logging full prompts and responses can mean storing patient data in a monitoring tool. Many teams log structure and metadata by default and keep content only in a restricted store with short retention.

The tool landscape

Check licences and hosting options before choosing. For Indian hospitals and banks, self-hosting or an Indian region often decides the shortlist.

Evaluating RAG systems and agents

RAG and agents need extra checks beyond final answer quality:

• Retrieval: did the right documents appear in the top results? Measure this separately; most bad RAG answers start with bad retrieval. Our post on GraphRAG, agentic RAG and long context covers retrieval design.
• Faithfulness: is every claim in the answer supported by the retrieved text?
• Tool use: were tool arguments valid, and were unnecessary calls avoided?
• Safety: did the agent refuse or escalate when it should have? Include injection tests from our prompt injection defence guide.
• Efficiency: steps, tokens and time per successful task.

Mistakes that make evals useless

• Testing only happy paths. If every case is a clean, typical request, the suite will pass while real users fail. Weight the set towards the hard cases.
• Letting the suite go stale. Products change. Review the set quarterly and retire cases that no longer reflect how the product is used.
• Tuning prompts to the test. If you iterate against the same cases for months, scores rise without real improvement. Keep a held-out set you look at rarely.
• One average score. An average hides the one category that collapsed. Report scores by category: safety, accuracy, format, refusal.
• No owner. Someone on the team must own the suite, review failures and decide thresholds, or it quietly stops being run.

A 30-day plan to get started

1. Week 1: collect 30-50 real cases, define pass criteria with the people who own the outcome, and add basic tracing.
2. Week 2: build code checks for structured fields and one calibrated LLM judge for free text.
3. Week 3: wire the suite into CI and block releases on safety regressions.
4. Week 4: start weekly human review of sampled production traces and turn findings into new cases.

Algon builds evaluation and tracing into every LLM development and RAG development project from the first sprint, because retrofitting it later costs far more. For teams running AI features in existing products, our managed engineering services can take on the ongoing evaluation work.

Conclusion

You cannot manage what you do not measure, and LLM apps make that painfully true. A modest, real evaluation set, a calibrated judge, regression checks in CI and good tracing will catch most problems before users do.

Start with 20-50 cases this week. The suite will grow with every incident, and within a few months it becomes the most valuable asset in your AI project.