A level is defined by how much evidence a question needs, not by how often models get it right. Each subtype is assigned to the level whose evidence structure it has, and the answer key is always computed by a program from annotations, never written by the question generator.

The two kinds of goal question

Both ask what activity the camera wearer is working on. They differ only in the wrong options.

• easy_goal (L1, recognition). The wrong options involve objects that are not in the clip. Recognising what is on screen is enough. Example: the answer is Prepare mango and a wrong option is Wash frying pan lid.
• medium_goal (L3, inference). Every wrong option shares objects with the right answer, so recognising objects cannot settle it; the sequence of actions has to. Example: Continue prepare puff pastry against Bake puff pastry and Filling Tarts.

The same idea on a second dataset

EPIC-Sounds has sound annotations only, with no objects, 3D or recipes. The levels are rebuilt there from sound evidence alone, on two 30-second clips (P02_133 15–45 s, P09_08 45–75 s), 27 questions in total.

There is no L4 on EPIC-Sounds: planning questions need annotations this dataset does not have.

Four things changed. Two subtypes were sitting in L3 without needing the whole window, one L1 subtype had wrong options that gave it away in the wrong direction, and the way a question is answered is now a variable of its own.

A problem found but not yet fixed

In easy_sound_object, low_sound_movement and the purpose questions, none of the wrong options appear in the clip. The question reduces to “which of these four is in the clip”, and the sound is not needed. A first check supports this: with the audio track replaced by silence, Gemini 3.5 scored 75% on low_sound_movement, against 76% with sound. The planned fix is to draw wrong options from other moments of the same clip.

Only if it still has one clear answer once the options are taken away. If it does, the options are dropped, the question states the output format it expects, and the reply is scored by rules first. If it does not, the options are kept and each one is judged true or false.

The conversion follows the scheme of Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT (Liu et al., 2025): sort questions by answer type, rewrite each type its own way, score by rules wherever possible, and judge each option true or false when a question cannot stand without its options.

Step 1 · Can the question stand without its options?

Step 2 · Rewrite the question and state the output format

Short-answer questions use fixed templates. Wording that depends on a list, such as “which of the following”, is rejected by the checker.

Step 3 · Score by rules, and use a judge only when rules cannot decide

When the rules give no verdict, for example “to clean up” for “to dry their hands”, Gemini 3.5 acts as judge. It sees only the question, the reply and the four known candidates in shuffled order, and returns which candidate the reply means, or none. It does not see the clip and is not told which candidate is right. Each row records whether it was scored by rule or by judge.

The five formats, on one question

Same clip, same right answer: roll of cling film.

Statement: At the moment you hear two plastic objects knocking together, the camera wearer is handling the roll of cling film.

Answer: <yes/no>

At the moment two plastic objects knocking together is heard, which object is the camera wearer handling?
A. roll of cling film
B. measuring jar
C. meat box
D. chunk of minced beef

A. … the camera wearer is handling the chunk of minced beef.
B. … the camera wearer is handling the measuring jar.
C. … the camera wearer is handling the roll of cling film.
D. Rustling can be heard during this clip.

At the moment you hear two plastic objects knocking together, which object is the camera wearer handling? Answer with the name of the object.

Answer: <your answer>

(same question as 4)

Answer: <your answer>
Because: <one sentence on what you saw or heard that supports your answer>

At the moment metal hitting wood is heard, which object is closest to the saucepan?
A. water bottle      A: <true/false>
B. yellow pepper     B: <true/false>
C. scissors          C: <true/false>
D. salt shaker       D: <true/false>

Why format 4b exists, and whether it is easier

It exists because some questions cannot be asked without their options. “Which object is closest to the saucepan” has no single answer a model could be expected to name: the candidates are the objects the annotators tracked, and the model cannot know which those are. Forcing such a question into a short answer would make the scoring unfair, so the options stay and only the way of answering changes.

One true/false judgement is easier than one multiple-choice pick: a guess is right half the time. The question as a whole is not. It counts as right only when all four judgements are right.

The same paper reports what models do in practice: once options are judged one at a time, the comparison between options is lost and models mark too many statements true. Among wrongly answered questions, 63–86% had more than one statement marked true, across seven models (Liu et al., 2025, Appendix D). The same ratio will be reported here.

Where the extra true statements in format 3 come from

• The right answer of the question itself is always one of the true statements.
• Further true statements come from other verified questions about the same 30-second window.
• If there are not enough of those, a statement about a sound that is present in the window is added.
• False statements come from the wrong options of the same questions, and from sounds that are absent.

How formats are compared

Chance differs by format, so scores are compared after correcting for it: (accuracy − chance) ÷ (1 − chance). Every format is also asked without the clip. Because the five formats share the same base question, each comparison is paired question by question.