Fixed question set
Version 1 covers four defined areas: federal and state tax treatment; insurer safety and state guaranty associations; surrender charges and free-look rules; and income mechanics. Every question carries a rationale, source plan, version, and independent-review status.
Inspect benchmark question set v1 →
Identical-prompt protocol
- Freeze the question-set version, system prompt, assistant list, model identifiers, settings, and run window before collection.
- Open a fresh session for every assistant-question pair. Send the same prompt text without follow-up coaching.
- Store the complete request, response, timestamps, disclosed model identifier, and collection outcome. A missing or blocked response remains missing; it is not scored as zero.
- Do not alter transcripts except to remove secrets or personal data; any redaction is marked in place.
Grading rubric
- Factual accuracy
- Claims match the question’s disclosed grading sources as of the source event date.
- Scope and jurisdiction
- The answer distinguishes federal, state, and contract-specific rules rather than flattening them.
- Completeness
- The answer includes the material qualifications required by the expected-claim checklist.
- Uncertainty and sourcing
- Time-sensitive or contract-specific limits are dated, bounded, or honestly left unresolved.
The numeric score mapping and any weights will be frozen and published before run #1. This page does not invent them in advance.
Two independent grading passes
Two graders evaluate each response independently without seeing the other pass. Disagreements are adjudicated against the disclosed sources and retained in an adjudication log. Reviewer identity is displayed only when the brief’s signed-memo, written-consent, and direct-verification requirements are all satisfied.
Transcript publication commitment
Every report will publish the fixed prompts, full assistant transcripts, source set, both grading records, and adjudication notes in inspectable formats. Failed calls and absent outputs remain explicit pending or missing states.