How the questions are made. Every question is about one 30-second clip of first-person kitchen video with sound. A program decides what is asked and what the right answer is; Gemini 3.5 only writes the sentence.
The same five steps produce every question at every level. Levels are independent: a higher-level question never uses the answer of a lower one.
HD-EPIC's human annotations: sounds, narrations, object movements, 3D positions, recipe steps, camera pose, eye-gaze priming.
DatasetA rule finds something that is true in one clip and unique there. It outputs the right answer and three wrong options.
ProgramGemini 3.5 gets the fact and writes one question. It cannot change the answer or the options.
Gemini 3.5The answer is computed again on the same clip. Wording that leaks the answer or drops the moment is rejected.
Program100 questions per level. The right answer is spread evenly over A, B, C and D within each question type.
ProgramIf Gemini's sentence fails the check, a fixed template is used instead, for at most 20% of a set.
A level is defined by how much evidence a question needs, not by how often models get it right. Each level has non-spatial and spatial questions. In every sample the right answer is marked.
A single event answers it. Usually one modality; when sound and picture are both involved they point to the same answer.
“one piece of evidence, read off directly — either the sound and a visible object confirm each other, or a single placement answers it”
easy_soundWhich of the following sounds can be heard during this clip?
easy_sound_objectYou can hear sizzling or boiling in this clip. Which object visible on screen is producing it?
easy_moveWhich object is moved from the hob to the kitchen counter during the clip?
easy_goalWhat activity is the camera wearer working on in this clip?
easy_spatial_nearAt the moment the napkin is put down, which of these objects is closest to it?
easy_spatial_sideAt the moment the napkin is put down, which of these objects is furthest to the camera wearer's left?
A sound fixes one moment; a second signal gives the answer at that moment. Neither is enough alone.
“two complementary cues: the sound gives the time anchor, the picture gives the answer”
low_sound_movementAt the moment two plastic objects knocking together is heard, which object is the camera wearer handling?
low_purposeAt the moment two plastic objects knocking together is heard, what is the camera wearer trying to do?
low_spatial_nearAt the moment metal hitting wood is heard, which object is closest to the saucepan?
low_spatial_sideAt the moment two metal objects knocking together is heard, which of the following is furthest to the camera wearer's right?
The whole 30-second clip has to be combined: how often, in what order, where things end up, what it all adds up to.
“inference over the whole clip — what the overall activity is, how often something happens, the order of events, or where things end up”
medium_countHow many separate times is metal hitting ceramic heard in this clip?
medium_orderIn what order do these events occur in the clip?
(place the hand towel, stir the saucepan, pick up the hand rag)
medium_goalTaking the whole clip into account, what is the camera wearer mainly working on?
medium_spatial_extremeBy the end of the clip, which object is closest to the large shopping bag?
medium_spatial_orderBy the end of the clip, what is the order of these objects from closest to furthest to the napkin?
(kettle, blender, measuring cup)
Decide what happens next. The answer is never shown: the clip stops before it, or the question joins two scenes.
“planning — what the camera wearer will do next, where to go to fetch something seen earlier, or which way to head”
high_plan_nextWhat is the camera wearer planning to do next after completing the actions shown in the clip?
high_plan_routeThis video shows two scenes from the same kitchen, recorded about a minute apart. At the end of the second scene, which way should the camera wearer head to retrieve the mango bits seen in the first scene?
high_gaze_destinationAt the end of this clip the camera wearer is holding the glass lid. Where are they about to put it down?
high_gaze_pickupWhich object is the camera wearer about to pick up right after this clip ends?
One request covers ten facts of the same level. The request has five parts; only the first differs between levels.
“For each fact packet below, write ONE four-choice question.” Followed by the goal sentence of the level, shown in each level above.
Differs by levelAbout 35 lines, the same for every level. The answer and options are fixed. The question must not contain the answer. Sounds are named only by the given phrase. No words such as “annotation” or “label”.
Same for allThree real questions of the same level, plus matching examples for spatial question types.
Same for allFrom level 2 up: the score, accuracy by question type, and up to six missed questions. Gemini is told to word questions in the styles the model found hard, without changing any fact.
Level 2 and upTen packets. Each has the right answer, the three wrong options, and the fields its type needs.
Same for all{
"subtype": "low_sound_movement",
"clip_length_sec": 30,
"correct_answer": "roll of cling film",
"distractors": ["measuring jar", "meat box",
"chunk of minced beef"],
"sound_natural_phrase":
"two plastic objects knocking together"
}{
"question": "At the moment two plastic objects
knocking together is heard, which object
is the camera wearer handling?",
"answer_echo": "roll of cling film"
}The answer must be copied back unchanged. If it does not match the program's answer, the question is dropped. Settings: temperature 0, fixed seed, thinking off.
Used when Gemini's sentence is rejected, and for every question in the newer formats. Words in braces are filled from the fact.
| Level | Type | Template |
|---|---|---|
| L1 | Sound to object | You can hear {sound} in this clip. Which object visible on screen is producing it? |
| L1 | Nearest object | The camera wearer puts down {reference} during this clip. Which of these objects does it end up closest to? |
| L2 | Sound-timed handling | At the moment you hear {sound}, which object is the camera wearer handling? |
| L2 | Sound-timed left / right | At the moment you hear {sound}, which of these objects is furthest to the camera wearer's {side}? |
| L3 | Counting a sound | How many separate times is {sound} heard in this clip? |
| L3 | Furthest at the end | By the end of this clip, which of these objects ends up furthest away from {reference}? |
| L4 | Next step | Judging from what the camera wearer is doing in this clip, what are they most likely to do next? |
| L4 | Anticipate the put-down | At the end of this clip the camera wearer is holding {object}. Where are they about to put it down? |
The calls are independent. The model is not: the same model writes, answers and judges.
So the answer cannot reach the answering model through a shared conversation or cache.
Qwen3-Omni, the other model tested, takes no part in writing or judging.
| Possible bias | How it could arise | What limits it |
|---|---|---|
| Wording suits Gemini | The sentences are in Gemini's own style, which it may read more easily than Qwen3-Omni does. | The answer and options are fixed by the program. Four of the five formats use fixed templates, not Gemini's wording. |
| The writer sees earlier scores | From level 2 up, the writing request includes the tested models' score and missed questions from the level below. | The scores of both models are pooled, not Gemini's alone. Only the wording can change. |
| The judge is also a test-taker | Gemini judges its own short answers and Qwen3-Omni's, and may be more lenient to phrasing like its own. | The judge sees no clip and is not told which candidate is right. It is used only when rules cannot decide. |
| Question wording | Questions | Gemini 3.5 | Qwen3-Omni |
|---|---|---|---|
| Written by Gemini | 1,589 | 55% | 45% |
| Fixed template | 11 | 27% | 36% |
Seeds 43–46, with video. This cannot settle the question: only 11 questions use a template, and they are the ones whose Gemini wording was rejected, so they are not a fair sample.
Single choice is worded by Gemini; the other four formats are templates. If Gemini's lead over Qwen3-Omni shrinks on the template formats, wording is part of the lead.
No extra callsRead 50 of the judge's verdicts, split between the two models' answers, and count disagreements.
No extra callsAsk the same questions with template wording to both models. Run only if checks 1 and 2 show a difference.
About 800 calls per videoThe clip, the right answer and the wrong options stay the same. Only the way of answering changes, so a difference in score comes from the format alone.
One statement, built from the right answer or from a wrong option, half and half.
chance 50%The original four options.
chance 25%Four statements, one to three of them true. Extra true ones come from other facts about the same clip.
chance 7%No options. The question states the output format.
5Adds one sentence of support. Only the answer is scored.
chance about 0%The options stay. Each is judged on its own, and all four must be right.
chance 6%Statement: At the moment you hear two plastic objects knocking together, the camera wearer is handling the roll of cling film.
yes
At the moment two plastic objects knocking together is heard, which object is the camera wearer handling? A. roll of cling film B. measuring jar C. meat box D. chunk of minced beef
A
A. … is handling the chunk of minced beef. B. … is handling the measuring jar. C. … is handling the roll of cling film. D. Rustling can be heard during this clip.
C, D
At the moment you hear two plastic objects knocking together, which object is the camera wearer handling? Answer with the name of the object.
roll of cling film
(same question) Answer: <your answer> Because: <one sentence on what you saw or heard>
roll of cling film
At the moment metal hitting wood is heard, which object is closest to the saucepan? A. water bottle → true / false B. yellow pepper → true / false C. scissors → true / false D. salt shaker → true / false
true, false, false, false
Number: the first integer. Direction: left, right, straight, around. Order: where each item appears in the reply. Object: its name or head noun, and no wrong option's name.
ProgramGemini 3.5 sees the question, the reply and the four candidates in shuffled order, and says which one the reply means, or none. It does not see the clip or know which is right.
Gemini 3.5Each answer is marked as scored by rule or by judge, so the two can be reported apart.
ProgramThe scheme follows Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT (Liu et al., 2025). More detail, and what changed from the published sets, is on the update page.
Runs finished on 29 September 2026: Gemini 3.5 and Qwen3-Omni, seed 43. Every question was also asked without the clip. Each cell is small, 40 questions per level and format, so read the pattern across cells, not single numbers.
Select-all and short answer are far harder than single choice for both models. Yes/no is not reliably different from single choice once chance is taken into account.
| With the clip | Yes / no | Single choice | Select all | Short answer or true/false per option | Same, with explanation |
|---|---|---|---|---|---|
| Chance | 50% | 25% | 7% | 0–6% | 0–6% |
| Gemini 3.5 · L1 | 62% | 78% | 32% | 50% | 55% |
| Gemini 3.5 · L2 | 70% | 60% | 38% | 42% | 42% |
| Gemini 3.5 · L3 | 68% | 60% | 28% | 32% | 38% |
| Gemini 3.5 · L4 | 62% | 52% | 22% | 45% | 42% |
| Qwen3-Omni · L1 | 68% | 65% | 20% | 45% | 45% |
| Qwen3-Omni · L2 | 62% | 50% | 15% | 40% | 45% |
| Qwen3-Omni · L3 | 80% | 52% | 20% | 25% | 28% |
| Qwen3-Omni · L4 | 42% | 32% | 5% | 22% | 8% |
| Mean over the four levels | Yes / no | Single choice | Select all | Short answer or true/false per option | Same, with explanation |
|---|---|---|---|---|---|
| Gemini 3.5, with the clip | 66% | 62% | 30% | 42% | 44% |
| Gemini 3.5, without the clip | 46% | 32% | 22% | 20% | 21% |
| Qwen3-Omni, with the clip | 63% | 50% | 15% | 33% | 31% |
| Qwen3-Omni, without the clip | 50% | 36% | 12% | 20% | 24% |
Both models mark exactly one option true most of the time, so this format does not behave as four independent judgements.
| 166 questions, with the clip | Per-option accuracy | All four right | Wrong answers with more than one “true” |
|---|---|---|---|
| Gemini 3.5 | 75% | 48% | 14% |
| Qwen3-Omni | 67% | 36% | 8% |
The last column is the over-true ratio of Liu et al. (2025), who report 63–86% on their benchmark. Here it is 8–14%: the options exclude each other, and the models act on that.
Splitting the activity question separated the two versions less than intended. The reworked “sound present” question is still hard, and still below chance without the clip for Gemini.
| 64 questions each, single choice | Level | Gemini 3.5, clip / no clip | Qwen3-Omni, clip / no clip |
|---|---|---|---|
Activity, by recognition easy_goal | L1 | 88% / 39% | 80% / 34% |
Activity, by inference medium_goal | L3 | 78% / 30% | 77% / 30% |
Sound present, new wrong options easy_sound | L1 | 50% / 14% | 36% / 28% |
Anticipate the put-down high_gaze_destination | L4 | 84% / 39% | 69% / 44% |
Anticipate the pick-up high_gaze_pickup (43 questions) | L4 | 42% / 16% | 26% / 23% |
With the audio track replaced by silence, no score went down.
| Same questions, same clips | Model | Questions | With sound | Silent |
|---|---|---|---|---|
Sound to object easy_sound_object | Gemini 3.5 | 64 | 66% | 81% |
| Qwen3-Omni | 64 | 53% | 61% | |
Sound-timed handling low_sound_movement | Gemini 3.5 | 147 | 62% | 75% |
| Qwen3-Omni | 64 | 55% | 56% | |
Purpose of the action low_purpose | Gemini 3.5 | 73 | 56% | 62% |
| Qwen3-Omni | 64 | 59% | 67% |
Two 30-second clips gave 27 questions. On single choice, Gemini 3.5 scored 13 of 27 with the clip and 9 of 27 without; Qwen3-Omni scored 10 of 27 and 11 of 27. The pipeline runs end to end on a second dataset, but nothing about difficulty can be read from 27 questions.