RAG 盲測評分表
Blind RAG scorecard
CHINZY 編寫 · 2026-11-10。全部資料和答卷均為虛構,不代表任何專案業績。此檔案離線可用,不上傳評審結果。
Fictional documents and responses, not measured vendor performance. This worksheet runs offline and sends no review data.
1. 凍結資料 / Freeze the source set
- D2 · 有效手冊 / Current manual
- p2 §1:在 48 小時內登記 / Register within 48 hours.
p2 §2:附工單號 / Include the ticket number.
p3 §1:交服務檯 / Submit to the service desk.
p3 §2:保留回執 / Keep the receipt. - D1 · 已廢止 / Superseded
- p1 §1:原來由採購部接收 / The purchasing office used to receive submissions.
兩個版本都沒有退款時限。兩個候選方使用相同的資料、問題和讀取許可權。/ Neither version specifies a refund deadline. Both candidates use the same corpus, questions and permissions.
2. 對照兩份原始記錄 / Read both answer sheets
- Q1 何時登記?要附什麼? / When to register, and what to include?
兩組檢索均返回 / Both retrieve: D2 p2 §1, D2 p2 §2.
A: 48 小時內 [D2 p2 §1],附工單號 [D2 p2 §2] / Within 48 hours [D2 p2 §1]; include the ticket number [D2 p2 §2].
B: 48 小時內 [D2 p2 §1],附工單號 / Within 48 hours [D2 p2 §1]; include the ticket number. - Q2 交到哪裡?之後留什麼? / Where to submit, and what to keep?
A 檢索 / Retrieves: D2 p3 §1, D2 p3 §2. B 檢索 / Retrieves: D2 p3 §1, D1 p1 §1.
A: 交服務檯 [D2 p3 §1],保留回執 / Submit to the service desk [D2 p3 §1]; keep the receipt.
B: 交服務檯 [D2 p3 §1],保留回執 [D1 p1 §1] / Submit to the service desk [D2 p3 §1]; keep the receipt [D1 p1 §1]. - Q3 多久退款? / How long until a refund?
A: 手冊沒有規定,無法依據現有資料給出天數。/ The manual gives no deadline; I cannot give a number from these sources.
B: 七天。/ Seven days.
3. 逐項判定 / Mark each item
檢索以題為單位,只要前五個結果裡有該題至少一條標準證據即透過;引用以預定的表述為單位,位置與原文支援都正確才透過。缺引記失敗;還沒檢查記未測。拒答單列,不合成總分。
For retrieval, any one valid reference in the top five is a hit. For a claim, both the locator and the support must be correct. Mark a missing citation as failed; mark an unreviewed item as pending. Score abstention separately.
樣例校對 / Worked key: A = 2/2 · 3/4 · 1/1; B = 2/2 · 2/4 · 0/1(檢索、引用、拒答 / retrieval, citations, abstention)。
實際採購先用獲准使用的材料另行凍結題目、應引表述、允許的版本和閾值,儲存兩組原始輸出供複核。此練習只校準評分口徑。/ For a real pilot, freeze authorized test materials, expected claims, versions and thresholds before testing. Retain raw outputs. This exercise calibrates reviewers.