Why evaluation needed its own system
LLM applications can appear better or worse across runs because retrieval candidates, generated language and model behaviour vary. A single demonstration is not enough to decide whether a prompt, model or retrieval change is safe to release.
Evaluation design
EvalStudio was conceived as a quality workspace connecting reusable datasets, repeatability analysis, automated scoring and regression review.
Golden setQuestions, expected evidence and review metadata.
ExperimentModel, prompt and retrieval configuration.
Repeated runsMeasure stability across executions.
ScoringMetrics plus LLM-assisted judgement.
Release gateCompare against a trusted baseline.
Metrics and signals
- Retrieval relevance: MRR, NDCG and hit rate.
- Repeatability: RMSE-style comparison across repeated result vectors.
- Grounding: whether claims are supported by retrieved evidence.
- Completeness: whether the answer covers the question’s essential parts.
- Citation quality: whether sources are present, traceable and aligned with claims.
- Operational signals: latency, failures, retries and token usage where available.
Regression workflow
A proposed model or prompt change is run against the same versioned dataset as the baseline. Results are compared by category, not only as one average. Failed examples become inputs for prompt, retrieval or ingestion diagnosis.
This makes evaluation a continuous engineering loop:
- Capture representative questions and known failures.
- Assign expected evidence or relevance labels.
- Run baseline and candidate configurations.
- Compare metrics, repeatability and qualitative judgements.
- Review regressions before release.
- Add newly discovered edge cases to the dataset.
What I learned
- An evaluation platform should make failures inspectable, not hide them behind one score.
- Question categories matter because improvements are rarely uniform.
- LLM-as-judge is useful as one signal, but it needs calibration and human review.
- Evaluation data requires product ownership: versioning, documentation and controlled change.