How to Implement the LLM Arena-as-a-Judge Approach to Evaluate Large Language Model Outputs

In this tutorial, we will explore how to implement the LLM Arena-as-a-Judge approach to evaluate large language model outputs. Instead of assigning isolated numerical scores to each response, this method performs head-to-head comparisons between outputs to determine which one is better based on criteria you define, such as helpfulness, clarity, or tone.

Target Audience Analysis

The primary audience for this tutorial includes AI developers, data scientists, and business managers interested in enhancing customer service automation through AI. They often face challenges such as:

• Identifying effective evaluation methods for AI-generated outputs
• Ensuring the quality and reliability of large language model applications
• Integrating AI tools seamlessly into existing workflows

Their goals include optimizing AI performance, improving customer interactions, and achieving measurable business outcomes through technology. They prefer clear, concise communication paired with actionable insights that can be directly applied to their projects.

Installing the Dependencies

To begin, you’ll need API keys from both OpenAI and Google. Follow the instructions below to generate your keys:

• Google API Key: Visit this link to generate your key.
• OpenAI API Key: Go to this link to create a new key. If you’re a new user, you may need to add billing information and make a minimum payment of $5 to activate API access.

Since we’re using Deepeval for evaluation, the OpenAI API key is required.

Defining the Context

Next, we’ll define the context for our test case. Here’s the customer support scenario we’ll work with:

Dear Support,
I ordered a wireless mouse last week, but I received a keyboard instead.
Can you please resolve this as soon as possible?
Thank you,
John

Generating Model Responses

We will generate responses using OpenAI’s GPT-4.1 and Google’s Gemini 2.5 Pro. Here’s how:

import os
from getpass import getpass
os.environ["OPENAI_API_KEY"] = getpass('Enter OpenAI API Key: ')
os.environ['GOOGLE_API_KEY'] = getpass('Enter Google API Key: ')
    

Defining the Arena Test Case

Set up the ArenaTestCase to compare the outputs of the two models:

a_test_case = ArenaTestCase(
    contestants={
        "GPT-4": LLMTestCase(
            input="Write a response to the customer email above.",
            context=[context_email],
            actual_output=openAI_response,
        ),
        "Gemini": LLMTestCase(
            input="Write a response to the customer email above.",
            context=[context_email],
            actual_output=geminiResponse,
        ),
    },
)
    

Setting Up the Evaluation Metric

Define the ArenaGEval metric focusing on the quality of the support email:

metric = ArenaGEval(
    name="Support Email Quality",
    criteria=(
        "Select the response that best balances empathy, professionalism, and clarity. "
        "It should sound understanding, polite, and be succinct."
    ),
    evaluation_params=[
        LLMTestCaseParams.CONTEXT,
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    model="gpt-5",
    verbose_mode=True
)
    

Running the Evaluation

Finally, run the evaluation using the defined metric:

metric.measure(a_test_case)
    

Evaluation Results

The evaluation results indicated that GPT-4 outperformed Gemini in generating a support email that balanced empathy, professionalism, and clarity. GPT-4’s response was concise, polite, and action-oriented, effectively addressing the situation by:

• Apologizing for the error
• Confirming the issue
• Clearly outlining the next steps to resolve it

In contrast, Gemini’s response included multiple options and meta commentary, which diluted focus and reduced clarity. This illustrates GPT-4’s proficiency in delivering effective customer-centric communication.

Further Resources

Explore the GitHub Page for tutorials, codes, and notebooks. Follow us on Twitter for updates, and join our community on Reddit.