Evaluating LLM Features — Integrating LLM APIs

You can't improve what you don't measure. Build an eval set of realistic inputs (from logs, with consent and redaction) with expected outcomes. Grade with code checks where possible (exact match, schema valid, contains citation), LLM-as-judge with a rubric for open-ended quality (calibrate it against human labels), and human review for a sample. Run evals in CI on prompt/model changes and track scores over time; keep a held-out set to avoid overfitting prompts.

Grading methods

Grader	Good for	Cost	Caveat
Code (exact/regex/schema)	Classification, extraction, format	Free	Only objective criteria
LLM-as-judge + rubric	Helpfulness, tone, faithfulness	Tokens	Must calibrate; judge bias
Pairwise comparison	Prompt A vs B	Tokens	Position bias — randomize order
Human review	Gold labels, edge cases	Expensive	Slow; use sampling
Online metrics	Real impact (resolution rate, CSAT)	Instrumentation	Lagging, noisy

Python (pytest eval)

CASES = load_jsonl("evals/triage.jsonl")            # {"input": ..., "expected": {"category": ..., "urgency": ...}}

@pytest.mark.eval
def test_triage_accuracy():
    correct = 0
    for case in CASES:
        out = triage(case["input"])                  # calls the model with the current prompt version
        correct += out.category == case["expected"]["category"]
    accuracy = correct / len(CASES)
    record_metric("triage.accuracy", accuracy, prompt_id=PROMPT_ID, model=MODEL)
    assert accuracy >= 0.92                          # regression gate

Common pitfalls

Vibes-based prompt changes.
Evaluating on the same examples used to write the prompt.
Uncalibrated LLM judges.

Key points

Golden set from real traffic
Code graders first, judges with rubrics
Evals gate prompt/model changes

Further reading: Hamel Husain: Your AI Product Needs Evals

Practice questions

What is LLM-as-judge and what's its main risk?

Answer

Using a model with a rubric to grade outputs; it can be biased or inconsistent, so calibrate it against human labels and monitor agreement.

Why keep a held-out eval set?

Answer

Prompts tuned on the same examples can overfit; a held-out set shows whether improvements generalize.