Repository navigation
feat(scorers): award partial credit for goal_completion - #626
omkargaikwad23 wants to merge 6 commits into
Conversation
|
/gcbrun |
goal_completion scored 0 or 100, so an agent that completed two of a
plan's three outcomes looked identical to one that did nothing. The judge
now decomposes the conversation_plan into sub-goals and rules on each;
the score is the share met.
Three bugs surfaced along the way:
- The verdict was read from `response.split('\\n')[0]`, a split on a
literal backslash-n rather than a newline, so "first line" was the whole
response and any reasoning containing "PASS" scored 100.
- The prompt asked for escaped JSON, but the Gemini generator pipes
responses through sanitize_sql, which deletes backslashes and corrupts
the object. It now asks for strings that need no escapes.
- eval_service cast correct_results_count with int(), truncating a
fractional count of 1.67 to 1 and 0.67 to 0. The same truncation
already applied to every averaged metric.
The analyzer and both viewer fallbacks counted only exact 100s, which
would have rolled every partial score up as a failure; each result now
contributes its fraction of a pass. Prompt-level rollup keeps the
pessimistic min() across trials that every other metric uses.
BigQuery still rounds correct_results_count to INT64, as it does today
for every metric that reports a fractional count.
79b1758 to
b1b8b0e
Compare
omkargaikwad23
changed the title
|
As part of this change also introduce a new field where success/failure criteria can be mentioned and not part of conversation plan. Make it optional for now so its backward compatible |
should this be part of binary_rubric instead, and did you want explicit failure criteria as a separate list, or just success criteria? |
A separate field |
If a scenario sets success_criteria, the goal_completion judge rules on each criterion in place of sub-goals split from conversation_plan. The score is the share of criteria met. Scenarios without the field keep the current prompt unchanged. The simulated user does not see it.
2 participants
Summary
goal_completionscored only 0 or 100. An agent that met two of three outcomes scored the same as one that did nothing. The scorer now awards partial credit, and a new optional field holds the criteria to grade against.Partial credit
The judge rules on each goal as met or not met. The score is
100 × met / total, rounded to one decimal. For example, 2 of 3 goals met scores 66.7.The run summary counts each scenario as its fraction of a pass (
sum(scores) / 100), not only exact 100s:If a scenario runs more than once, it gets the score of its weakest run. This is the same
min()rule that every other metric uses. The analyzer,eval_servicesummary, dashboard, and trends viewer now show the fractional count.New optional field:
success_criteriaRequested in review. If a scenario sets
success_criteria, the judge rules on each criterion instead of splittingconversation_planinto sub-goals. The criteria stay out of the plan, so the simulated user does not see them.The field is backward compatible. If a scenario does not set it, the judge prompt is unchanged.
Bug fixes
response.split('\\n')[0], which splits on a literal backslash-n, not a newline. Any reasoning that contained "PASS" scored 100.sanitize_sql. The prompt now asks for strings that need no escapes.eval_serviceusedint()oncorrect_results_count. This truncated a fractional count, for example 1.67 to 1, for every averaged metric.Known limitations
correct_results_countto INT64, as it does for other fractional metrics. A fix needs a FLOAT64 schema migration.type: goal_completionstill counts only exact 100s in the summary.Test plan
evalbench/test/goalcompletionrate_test.py: partial scores, the'\\n'split regression,success_criteriascoring and fallback, analyzer rollupagent_grpc_proxy*_test.pyalso fail onmain.