Interview Q&A · 22 questions · 4 answered publicly
Evaluation Interview Questions
Ordered by how often they're asked. Sign in free to see every answer, rate your confidence and run timed mock interviews.
-
Most asked Evaluation leaf: What is a golden dataset?
Summary
Curated inputs with expected outputs/docs, versioned, covering edge cases.
Longer Answer
Curated inputs with expected outputs/docs, versioned, covering edge cases. Used for offline regression in CI.
Key Points
- Curated inputs with expected outputs/docs, versioned, covering edge cases
- Used for offline regression in CI
- Foundation
- Use the answer to connect the concept back to a realistic production or interview scenario for 'Evaluation leaf: What is a golden dataset?'
Comparison / Tradeoffs
Aspect Notes Core idea Curated inputs with expected outputs/docs, versioned, covering edge cases. Practical tradeoff Foundation. -
Most asked Evaluation leaf: Pros/cons of LLM-as-judge?
Summary
Scales grading with rubrics but can be biased/noisy.
Longer Answer
Scales grading with rubrics but can be biased/noisy. Calibrate with humans and use pairwise comparisons carefully.
Key Points
- Scales grading with rubrics but can be biased/noisy
- Calibrate with humans and use pairwise comparisons carefully
- Judging
- Use the answer to connect the concept back to a realistic production or interview scenario for 'Evaluation leaf: Pros/cons of LLM-as-judge?'
Comparison / Tradeoffs
Aspect Notes Core idea Scales grading with rubrics but can be biased/noisy. Practical tradeoff Judging. -
Most asked Evaluation leaf: What is faithfulness/groundedness?
Summary
Claims in the answer are supported by retrieved context.
Longer Answer
Claims in the answer are supported by retrieved context. Unfaithful fluent answers are still failures.
Key Points
- Claims in the answer are supported by retrieved context
- Unfaithful fluent answers are still failures
- Quality dimension
- Use the answer to connect the concept back to a realistic production or interview scenario for 'Evaluation leaf: What is faithfulness/groundedness?'
Comparison / Tradeoffs
Aspect Notes Core idea Claims in the answer are supported by retrieved context. Practical tradeoff Quality dimension. -
Most asked Evaluation leaf: Offline vs online evaluation tradeoffs?
Summary
Offline is fast/safe but may miss distribution shift; online is real but slower/riskier-use both.
Longer Answer
Offline is fast/safe but may miss distribution shift; online is real but slower/riskier-use both. Eval strategy.
Key Points
- Offline is fast/safe but may miss distribution shift; online is real but slower/riskier-use both
- Eval strategy
- Relevant areas: Ai, Evaluation
- Use the answer to connect the concept back to a realistic production or interview scenario for 'Evaluation leaf: Offline vs online evaluation tradeoffs?'
Comparison / Tradeoffs
Aspect Notes Core idea Offline is fast/safe but may miss distribution shift; online is real but slower/riskier-use both. Practical tradeoff Eval strategy. - Evaluation leaf: Why separate retrieval metrics from generation metrics?
- Evaluation leaf: Online metrics that matter?
- Evaluation leaf: Red teaming for LLM apps?
- Evaluation leaf: How do you detect regressions after a prompt change?
- Evaluation leaf: How do you evaluate agents?
- Evaluation leaf: How do you sample production traffic for review?
- Evaluation leaf: What is answer relevance?
- Evaluation leaf: How do you evaluate citation quality?
- Evaluation leaf: How do toxicity/safety classifiers fit?
- Evaluation leaf: What is calibration of confidence?
- Evaluation leaf: CI gate example for RAG PRs?
- Evaluation leaf: Inter-annotator agreement-why care?
- Evaluation leaf: Statistical significance in A/B for LLM features?
- Evaluation leaf: Synthetic eval data-when OK?
- Evaluation leaf: Tracking eval debt?
- Evaluation leaf: Multimodal eval challenges?
- How would you evaluate a production RAG or agent system beyond 'it looks good'?
- What guardrails would you put around an LLM-powered customer-facing feature?