Evaluating LLM Applications with Elena Samuylova as AI Judge

Understanding the Role of Evaluation in LLM-Based Applications

As Large Language Models (LLMs) become the backbone of modern AI-driven applications, the question of accurate evaluation becomes increasingly important. In a recent InfoQ podcast, Elena Samuylova, founder of Akibia, dives into the intricacies of evaluating LLM applications and how her team is turning large language models themselves into unbiased, repeatable evaluators.

With the rapid advancement of AI capabilities, the need for reliable evaluation methods has driven researchers and developers to explore non-traditional methods — including using LLMs to evaluate their own kind. This creates the opportunity for more scalable, aligned, and consistent metrics in production environments.

Why Traditional Evaluation Doesn’t Work for LLMs

Traditional software testing relies on inputs and outputs with expected results. However, evaluating generative AI content is more fluid and subjective, making it harder to assess whether an answer is just “right” or “wrong.” According to Elena Samuylova, current approaches tend to fall short because:

• LLMs generate open-ended responses, making binary evaluations ineffective.
• Human evaluation isn’t scalable and introduces inconsistency due to subjectivity.
• Prompt variability can change outputs significantly, making version control and reproducibility a challenge.

These challenges call for a more automated and intelligent form of evaluation — something LLMs themselves may be uniquely positioned to provide.

Akibia’s Unique Approach: AI-as-a-Judge

To address the complexity of evaluating LLM output, Samuylova’s team at Akibia is pioneering a framework called “AI-as-a-Judge”. The principle behind this strategy is to use language models — specifically optimized and aligned versions — to review, score, and explain the quality of other LLM outputs.

How Does It Work?

The AI-as-a-Judge framework operates by comparing LLM output against desired criteria using prompt-engineered evaluation agents. According to Samuylova, these evaluations go beyond mere accuracy and incorporate dimensions such as coherence, helpfulness, correctness, and stylistic alignment.

Each evaluation is designed to be:

• Transparent: Providing detailed rationales for scores.
• Repeatable: Ensuring the same input/output combination receives the same evaluation.
• Unbiased: Using alignment training to reduce subjectivity based on preference or language style.

Key Dimensions of LLM Evaluation

Evaluating an LLM-generated answer means judging on several criteria, not just correctness. Akibia’s methodology includes multiple evaluation dimensions:

• Correctness: Is the answer factually or logically accurate?
• Helpfulness: Is the response useful to the user?
• Coherence: Is the message internally consistent and organized?
• Style & Tone: Does the response match the intended audience and voice?
• Safety: Does the output avoid harmful or biased content?

By parsing out these dimensions, Akibia can deliver nuanced scoring that allows for targeted improvements in LLM behavior.

Using LLMs to Evaluate A/B Test Variants

Typically, A/B testing allows product teams to compare two versions of an interaction. In traditional development, analytics dictate which performs better. For LLM-driven applications (e.g., AI chat interfaces or co-pilots), this data is often messy or inconclusive.

Akibia uses LLMs-as-judges to review the outputs from both A and B model versions. The AI decides which version better meets the success criteria — and provides a textual rationale. This process is:

• Faster than collecting and analyzing human feedback.
• More scalable across a wide set of test cases and use contexts.
• Built for iteration, helping teams move quickly between prompt and model versions.

Case Study: Evaluating Different LLMs

According to Samuylova, clients often want to evaluate whether GPT-4 outperforms Claude, Mistral, or LLaMa in specific use cases. Using their AI judge framework, Akibia can score thousands of test cases across engines, allowing teams to:

• Benchmark performance across models.
• Identify model alignment to use-case goals.
• Support procurement decisions with data-backed evaluations.

Aligning Evaluation with Business Goals

One of the most insightful segments of the podcast reveals that developers must align LLM evaluation with real-world business metrics. Instead of focusing on general metrics like BLEU scores or word similarity, Elena suggests a use-case-specific approach.

Practical examples include:

• For an AI tutoring application, the LLM must prioritize helpfulness and clarity over style.
• If building a legal assistant, correctness and formality matter more than tone.
• For creative writing tools, coherence and style are often more important than factual precision.

This highlights the critical understanding that not all LLMs need to perform the same — their evaluation must be tailored to what success looks like in context.

Reducing Human Effort and Scaling AI Testing

Human evaluation of LLM outputs is time-consuming, expensive, and highly subjective. Akibia’s automated evaluation engine drastically reduces costs by removing the need for large-scale human assessment. Moreover, the ability to re-evaluate previously generated content opens doors to testing archived data with updated models and criteria.

Notable benefits include:

• Quick iteration across hundreds or thousands of variants.
• Historical insights into model drift or performance change over time.
• Simplified compliance and risk audits for enterprises using LLMs in regulated spaces.

Are LLMs Truly Reliable Evaluators?

One concern critics raise is whether LLMs can offer unbiased, reliable evaluations — especially when used to judge outputs from another LLM. Akibia addresses this concern by:

• Using specialized prompt engineering to minimize hallucination.
• Training evaluation agents on known ground-truth data to improve accuracy.
• Involving human reviewers in the loop during the calibration phase of evaluation models.

While the approach isn’t flawless, Elena notes that LLM-as-a-judge is far more consistent than scattered human assessments, especially at scale.

Integrating with Popular LLM Tools

Akibia’s AI evaluation capabilities are designed to integrate easily with common LLMOps platforms and frameworks. For engineering teams, this means seamless plug-ins for metrics dashboards, experiment tracking platforms, and CI/CD pipelines.

Through APIs or CLI integration, development teams can:

• Run regression tests during code or prompt changes.
• Benchmark different LLM providers pre-deployment.
• Log explanations of judgments for developer inspection.

The Future of LLM Evaluation

Looking ahead, Elena Samuylova sees a future where AI judges are incorporated into every stage of LLM development and operation — from pre-training alignment to model monitoring. By enabling automated and contextual evaluation, teams gain the ability to:

• Understand model behavior deeply without relying on constant human feedback.
• Build trust with stakeholders through transparent assessment reports.
• Operationalize evaluation as a continuously running service — not a one-time test.

Final Thoughts

Evaluating LLM applications isn’t as straightforward as grading a multiple-choice quiz — but with frameworks like AI-as-a-Judge, the industry is moving toward smarter, more reliable methodologies. Akibia’s work under Elena Samuylova’s leadership marks a significant leap forward in how enterprises can automate, scale, and trust their AI systems.

Whether you’re building a chatbot, code-writing assistant, or research summarizer, incorporating AI-assisted evaluation will be key to delivering consistent and high-quality user experiences.

For organizations looking to adopt LLMs, remember: the model only performs as well as how well you evaluate it. And with AI judging AI, a new era of machine-quality assurance is already underway.