Skip to content

[fix][evaluation] offline analysis mq - #612

Merged
VinCinx merged 10 commits into
mainfrom
fix/offline-analysis-mq
Aug 13, 2026
Merged

[fix][evaluation] offline analysis mq#612
VinCinx merged 10 commits into
mainfrom
fix/offline-analysis-mq

Conversation

@VinCinx

@VinCinx VinCinx commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

What type of PR is this?

Check the PR title

  • This PR title match the format: [<type>][<scope>] <description>. For example: [fix][backend] flaky fix
  • The description of this PR title is user-oriented and clear enough for others to understand.
  • Add documentation if the current PR requires user awareness at the usage level.
  • This PR is written in English. PRs not in English will not be reviewed.

(Optional) Translate the PR title into Chinese

(Optional) More detailed description for this PR(en: English/zh: Chinese)

en:
zh(optional):

(Optional) Which issue(s) this PR fixes

@VinCinx VinCinx changed the title fix: offline analysis mq [fix][evaluation] offline analysis mq Aug 11, 2026
@codecov

codecov Bot commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.91837% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...on/domain/service/expt_run_scheduler_event_impl.go 94.28% 2 Missing and 2 partials ⚠️

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #612      +/-   ##
==========================================
+ Coverage   78.16%   78.20%   +0.04%     
==========================================
  Files         693      693              
  Lines       84808    84885      +77     
==========================================
+ Hits        66288    66388     +100     
+ Misses      14577    14559      -18     
+ Partials     3943     3938       -5
Flag Coverage Δ
unittests 78.20% <95.91%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ules/evaluation/domain/service/expt_result_impl.go 72.99% <100.00%> (+0.35%) ⬆️
...es/evaluation/domain/service/expt_run_item_impl.go 80.92% <100.00%> (+6.07%) ⬆️
...d/modules/evaluation/domain/service/target_impl.go 84.96% <100.00%> (+0.31%) ⬆️
...on/domain/service/expt_run_scheduler_event_impl.go 81.31% <94.28%> (+1.68%) ⬆️

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39a1e85...2061173. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@VinCinx
VinCinx requested a review from xueyizheng August 11, 2026 10:58
xueyizheng
xueyizheng previously approved these changes Aug 12, 2026
View reviewed changes
@topjames666
topjames666 self-requested a review August 12, 2026 09:22
VinCinx and others added 8 commits August 13, 2026 11:59
@VinCinx @claude
fix(evaluation): 将 item-complete MQ 发送点后移到 scheduler 收口处
21f3b60 
下游离线分析收到 item-complete(success) 后立即反查 standard eval output,
却常读到 running/空。根因是发送时机早于读侧就绪: 链路A(CompleteItemRun)
只写了 run_log(result_state=Logged) 就发 MQ, 而读侧三张表(turn_result /
evaluator_result_ref / item_result)要等链路B(scheduler daemon 扫 Logged →
RecordItemRunLogs)在下一个 tick 才写, 竞态窗口约一个 tick(BOE 实测 ~22.8s)。

将 item-complete(success) 唯一发送点从链路A 后移到链路B 的 recordEvalItemRunLogs:
在 RecordItemRunLogs 写完读侧三张表 + 置 result_state=Resulted 之后再发, 此刻
下游反查必读到就绪结果, 竞态消除。

- 删除链路A CompleteItemRun 里的 PublishItemComplete 发送块, 仅保留写 Logged
- ExptSchedulerImpl 注入 itemCompletePublisher + exptItemRefRepo
- recordEvalItemRunLogs 循环外按归属集(expt_item_ref)分组批量补 EvalSetItem
  (ItemKey / per-item 版本), 循环内 State==Success 守卫后组装并发送
- per-item DatasetVersionID 取 expt_item_ref.EvalSetVersionID(多集非主集也正确),
  不用 ExptEvalItem.EvalSetVersionID(主集硬编码)
- 抽 buildItemCompleteEventFromScheduler 复用 buildItemCompleteEvent 语义, 单一实现
- 幂等靠 result_state Logged→Resulted 状态位; 下游按 (expt_id,expt_run_id,item_id) 去重
- 发送失败只 CtxWarn 不阻断

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VinCinx @claude
fix(evaluation): item-complete MQ 可靠投递(写表后发送、成功才置终态、失败下轮重发)
bdff1ff 
链路B item-complete 发送不可靠: 发失败只告警且 result_state 已置 Resulted 终态,
tick 下轮扫描(result_state=Logged)不再命中 → 消息永久丢失。

改为:
- 新增终态 ExptItemResultStateSent; Resulted 降级为"读侧已写待发送"
- tick 扫描条件纳入 Resulted(Logged→Logged|Resulted), 使待发送 item 下轮被重扫
- 发送成功才 MarkItemResultSent 翻 Sent; 失败留 Resulted 下轮重发
- 非成功行/开源 nil publisher 直接翻 Sent; 卡 Resulted 超 30min 强制翻 Sent + 告警
- 不重复写表: 靠 RecordItemRunLogs 既有幂等门(!=Logged 短路), 统计不重复累加
- sendItemComplete 成功打印完整 event JSON(便于验证与排障)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VinCinx @claude
test(evaluation): 补 item-complete 修复单测, 增量覆盖率达标(新增函数≥90%)
79dd036 
- Test_buildItemCompleteEvent_LinkAB_Equivalence: 钉死链路A/B组装一致契约
- Test_resolveItemCompleteMeta: 单集/多集/ref缺失/主集缺失/MGet失败/BatchGet失败 (0→100%)
- Test_MarkItemResultSent (0→100%)
- Test_findEvalSetForItem: nil/单集/多集命中不命中 (50→100%)
- Test_sendItemComplete_nilMeta: meta 缺失分支 (→100%)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VinCinx @claude
style(evaluation): fix gofmt formatting for item-complete files
678ac99 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@xueyizheng @claude @VinCinx
fix(evaluation): 修复 target stub 行导致 UpsertExptTurnResultFilter NPE 打挂实验
8642216 
fillExptTurnResultFilters 只判 map 命中就解引用
evalTargetOutput.EvalTargetRecord.EvalTargetOutputData.OutputFields,
撞上 buildTargetOutput 末尾构造的 stub(仅 ID/SpaceID/ItemID/TurnID/Status,
无 EvalTargetOutputData)即 NPE。panic 在调度器 goroutine 内被
HandleEventErr 捕获后直接把实验置 Failed。

线上 expt 7590117239516698626 即此故障: 某 turn 的 target record 因落库前
TOS 外传大字段超时而没进 MySQL, 但 target_result_id 已写进 expt_turn_result,
BatchGetRecordByIDs 查不到 → 走 stub 分支 → panic, status_message 就是
"panic occurred, reason=runtime error: invalid memory address..."。

补齐三层 nil 守卫(与 standard_eval_output.go / expt_export_impl.go 等 5 处
既有写法对齐), stub 行本就无 target 数据可填, 跳过即可。同时把重复的
四级链式解引用提成 outputData 局部变量。

补回归 UT: 去掉守卫即复现线上同款 nil pointer dereference。

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@xueyizheng @claude @VinCinx
fix(evaluation): ExecuteTarget 落库失败不再静默吞掉, 避免写下不存在的 target_result_id
aae83a3 
ExecuteTarget 的 defer 里 CreateEvalTargetRecord 失败是裸 return, err 仍是
进 defer 时的值 —— 执行成功场景下就是 nil。上层因此拿到「err=nil + record
非 nil」, 照常打印 call target success 并把 record.ID 写进 run_log /
expt_turn_result.target_result_id, 而 MySQL 里根本没这行。

典型触发: CreateEvalTargetRecord 先跑 SaveEvalTargetRecordData 外传 TOS
大字段, 上传超时即整条 record 都不落库(线上 upload_api.go:116
context deadline exceeded)。后果是该行 target 数据永久丢失, 且全链路
没有任何错误留痕 —— 排查时只看到 call target success。

改为: 落库失败即视为本次执行失败并抛出(与异步路径 asyncExecuteTarget 的
既有语义对齐), 交由 item 重试; 执行本身也失败时保留原始执行错误语义,
落库失败仅 CtxError 留痕。

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@VinCinx @claude
refactor(evaluation): revert reliable-delivery, send item-complete at…
d332e90 
… item start

按需求回退可靠投递机制(不引入 Sent 状态、不改 Resulted 语义、不动核心调度收敛):
- 删除 ExptItemResultStateSent / MarkItemResultSent / finalizeItemComplete /
  30min 兜底 / tick 扫描条件改动; result_state 恢复 Logged→Resulted(终态)。
- item-complete 发送移到 recordEvalItemRunLogs 循环每个 item 开头(RecordItemRunLogs 前),
  作为只读旁路: 仅发成功行, 发失败在 sendItemComplete 内 CtxWarn 后 return(仅本 item),
  不阻断落库/后续 item。是否真正投递仍由 producer 依空间开关+enable_analysis 判定。
- 不再消竞态(下游 defer 投递覆盖读侧就绪窗口), 目标为"成功行必发一次 MQ"(at-least-once)。
- 相应清理可靠投递专属单测, 保留组装/多集/发送旁路相关单测。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VinCinx @claude
fix(evaluation): item-complete 发送失败日志改为 CtxError
f0a1eac 
发送失败(publisher 返错)从 CtxWarn 提升为 CtxError, 便于告警发现丢发;
仍不阻断落库/后续 item(旁路语义不变)。同步注释。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VinCinx
VinCinx force-pushed the fix/offline-analysis-mq branch from 383b9c1 to f0a1eac Compare August 13, 2026 04:09
VinCinx and others added 2 commits August 13, 2026 13:26
@VinCinx @claude
fix(evaluation): 补 NewExptSchedulerSvc 单测调用参数, 修复 CI typecheck
08f5a76 
rebase 后 NewExptSchedulerSvc 新增 iItemCompletePublisher/iExptItemRefRepo 参数,
sandbox_agent_hourly_middleware_test.go 两处调用补足到 22 个固定参数。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VinCinx @claude
test(evaluation): 补 sendItemComplete publish 成功/失败 UT, 覆盖率 100%
2061173 
sendItemComplete 增量覆盖率 50%→100%: 补 publish 成功 / 失败(CtxError 不阻断) 两路径。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@VinCinx
VinCinx requested a review from xueyizheng August 13, 2026 06:00
@VinCinx
VinCinx merged commit 455deb8 into main Aug 13, 2026
16 checks passed
@VinCinx
VinCinx deleted the fix/offline-analysis-mq branch August 13, 2026 06:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xueyizheng xueyizheng xueyizheng approved these changes
@topjames666 topjames666 topjames666 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@VinCinx @xueyizheng @topjames666