Quality Threshold
This is the minimum quality score you are willing to accept. Below this threshold, the response quality is not stable enough.
Current Threshold
80
Evals: Evaluation & Regression Verification
One of the most common illusions in LLM systems is "I just tried a few examples and it seems better." A truly reliable engineering process does not rely on gut feelings, but uses evaluation sets, thresholds, and version comparisons to continuously verify: what exactly did this change improve, and did it break anything elsewhere.
First define metrics
Then run version comparisons
Finally set thresholds to prevent regression
Switch evaluation sets to see how "pass" is determined
First select a task type, then adjust the thresholds. The page will instantly judge which versions can pass the line, and which ones, although higher in some metric, do not meet the overall deployment criteria.
Why Demo Trials Cannot Replace Evals
Too few samples
The few examples you manually test may happen to be those the model excels at, failing to cover the real online distribution.
Easily biased toward "amazing cases"
Humans easily remember a few brilliant successful outputs while ignoring infrequent but costly errors.
No unified threshold
Without preset thresholds, the team falls into debates of "I think it's fine" versus "I think it's not stable enough" every time.
Cannot detect regression
A change may improve one scenario while quietly degrading another. Without regression testing, it is hard to detect in time.
A Practical Evaluation Loop
1
Define tasks and metrics
First clarify what exactly to evaluate — accuracy, citation quality, safe refusal, or latency cost.
2
Prepare benchmark sets
Collect data covering real scenarios, avoiding looking only at a few pretty examples.
3
Version comparison
Compare every change against the baseline version, checking not only "did it get better" but also "did it get worse."
4
Set thresholds to block
Automatically block when key metrics fail to meet thresholds, preventing problematic versions from going live directly.
In a nutshell: The value of Evals is not just telling you "which version is better," but transforming "deploying based on gut feeling" into "deploying with evidence, thresholds, and regression protection."