Skip to content

[feat][evaluation] support async CustomRPC evaluators - #607

Draft
HearyShen wants to merge 14 commits into
mainfrom
codex/trae-work-async-evaluator
Draft

[feat][evaluation] support async CustomRPC evaluators#607
HearyShen wants to merge 14 commits into
mainfrom
codex/trae-work-async-evaluator

Conversation

@HearyShen

Copy link
Copy Markdown
Collaborator

Summary

Add asynchronous execution support for CustomRPC evaluators, primarily for long-running third-party evaluations.

  • add CustomRPCEvaluator.is_async and async_invoke_http_info
  • add EvaluationSPIService.AsyncInvokeEvaluator
  • generalize the evaluator async coordinator to Record -> Context -> Provider
  • add terminal CAS and duplicate/conflicting callback protection
  • add a ResumeReady scheduling handshake
  • preserve Agent evaluator async/debug metadata across callback races
  • reject builtin, cross-space, and legacy-protocol async CustomRPC evaluators

Compatibility

Existing CustomRPC evaluator versions remain synchronous by default. Agent evaluator async behavior and CustomRPCServer async eval-target behavior are covered by regression tests.

Tests

  • evaluation entity, converter, domain, application, and repository package tests
  • critical async coordinator, callback, Redis, and metadata race tests with -race
  • backend build: go build ./cmd

@HearyShen HearyShen changed the title feat(evaluation): support async CustomRPC evaluators [feat][evaluation] support async CustomRPC evaluators Aug 6, 2026
@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.33333% with 19 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...odules/evaluation/domain/service/evaluator_impl.go 88.54% 8 Missing and 7 partials ⚠️
...tion/infra/repo/evaluator/evaluator_record_impl.go 94.64% 2 Missing and 1 partial ⚠️
...aluation/domain/service/expt_run_item_turn_impl.go 94.44% 0 Missing and 1 partial ⚠️

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #607   +/-   ##
=======================================
  Coverage        ?   78.04%           
=======================================
  Files           ?      689           
  Lines           ?    83986           
  Branches        ?        0           
=======================================
  Hits            ?    65551           
  Misses          ?    14542           
  Partials        ?     3893           
Flag Coverage Δ
unittests 78.04% <93.33%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ation/application/convertor/evaluator/evaluator.go 92.17% <100.00%> (ø)
...luation/application/convertor/evaluator/openapi.go 96.55% <100.00%> (ø)
...modules/evaluation/application/eval_openapi_app.go 81.63% <100.00%> (ø)
...nd/modules/evaluation/application/evaluator_app.go 90.39% <100.00%> (ø)
...kend/modules/evaluation/domain/entity/evaluator.go 100.00% <100.00%> (ø)
...tion/domain/entity/evaluator_version_custom_rpc.go 100.00% <100.00%> (ø)
...ckend/modules/evaluation/domain/entity/expt_run.go 98.64% <ø> (ø)
backend/modules/evaluation/domain/entity/param.go 87.71% <ø> (ø)
...es/evaluation/domain/service/expt_run_item_impl.go 74.60% <100.00%> (ø)
...aluation/domain/service/expt_run_item_turn_impl.go 83.70% <94.44%> (ø)
... and 2 more

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e30f3dd...eaf6510. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant