Video QA question prompts

The exact question wordings and option sets used to probe vision-language models on short synthetic (Blender/Infinigen) videos. Templates are extracted from the generator's source; examples are verbatim from 4,903 generated scenes (80,099 gate-passed questions of the 22 live types). Each question is one multiple-choice item, asked in its own request (no shared context).

Prompt wrapper

Options are lettered A)…D) one per line, in the listed order. The model sees one of these:

Frames arm (24 frames, 480×270 PNG, sent as images after the text)

You are watching a short video, shown as {n} frames sampled uniformly in time (frame 1 is the start, frame {n} is the end).

Question: {question}

Options:
{options}

Reply with exactly one letter ({letters}). No other text.

Native-video arm

You are watching a short video.

Question: {question}

Options:
{options}

Reply with exactly one letter ({letters}). No other text.

Blind arm (no frames; measures text-only priors)

You will NOT be shown the video this question is about. Answer with your best guess anyway.

Question: {question}

Options:
{options}

Reply with exactly one letter ({letters}). No other text.

Format retry (appended once if the reply has no parseable letter)


Your previous reply was not a single option letter. Reply with ONE letter only.

Placeholders in the templates

Templates are shown as the Python f-strings in the generator. Placeholders are filled per scene:

Live question types (22)

existence · contact · count · stops · direction_reversal · visible_at_end · camera_motion · approach · spins · depth_order · correspondence · occlusion_event · size_comparison · lands · exit_side · speed_comparison · mover_count · separation_change · temporal_order · entry_side · landing_order · occlusion_recovery

existence 15,666 gate-passed questions in the corpus

Question template (questions.py q_existence)

f'Is there a {category} anywhere in the video?'

Options

['yes', 'no']

Option order: yes / no, always in this order.

Real examples (verbatim; certified answer highlighted)

Is there a ball anywhere in the video?
  1. yes
  2. no
riA_41392/existence/ball · certified answer B
Is there a ball anywhere in the video?
  1. yes
  2. no
riA_35170/existence/ball · certified answer A
Is there a cube anywhere in the video?
  1. yes
  2. no
riA_40775/existence/cube · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Is there a ball anywhere in the video?

Options:
A) yes
B) no

Reply with exactly one letter (A, B). No other text.

followed by the 24 frames as images.

contact 10,039 gate-passed questions in the corpus

Question template (questions.py q_contact)

f'Do the {da} and the {db} touch each other at any point in the video?'

Options

['yes', 'no']

Option order: yes / no, always in this order.

Real examples (verbatim; certified answer highlighted)

Do the green cube and the red cone touch each other at any point in the video?
  1. yes
  2. no
riA_40023/contact/red_cone_1-green_cube_2 · certified answer B
Do the red-and-purple striped cube and the purple cone touch each other at any point in the video?
  1. yes
  2. no
riA_30135/contact/purple_cone_0-red_purple_striped_cube_1 · certified answer B
Do the blue ball and the purple cone touch each other at any point in the video?
  1. yes
  2. no
riA_35362/contact/purple_cone_0-blue_ball_2 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Do the green cube and the red cone touch each other at any point in the video?

Options:
A) yes
B) no

Reply with exactly one letter (A, B). No other text.

followed by the 24 frames as images.

count 8,140 gate-passed questions in the corpus

Question template (questions.py q_count)

f'How many {plural} appear in the video?'

Options

count_options(n, qid)

Option order: numeric: four distinct counts containing the answer, sorted ascending; the answer's rank is drawn ~uniformly per question (count_options).

Real examples (verbatim; certified answer highlighted)

How many cylinders appear in the video?
  1. 1
  2. 2
  3. 3
  4. 4
riA_41987/count/cylinder · certified answer A
How many cylinders appear in the video?
  1. 0
  2. 1
  3. 2
  4. 3
riA_35967/count/cylinder · certified answer B
How many cones appear in the video?
  1. 0
  2. 1
  3. 2
  4. 3
riA_35786/count/cone · certified answer C
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: How many cylinders appear in the video?

Options:
A) 1
B) 2
C) 3
D) 4

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

stops 7,160 gate-passed questions in the corpus

Question template (questions.py q_stops)

f"The {manifest[name]['descriptor']} moves. Does it come to a stop before the video ends?"

Options

['yes', 'no']

Option order: yes / no, always in this order.

Real examples (verbatim; certified answer highlighted)

The red-and-orange striped cone moves. Does it come to a stop before the video ends?
  1. yes
  2. no
riA_35683/stops/red_orange_striped_cone_0 · certified answer B
The green-and-yellow striped cube moves. Does it come to a stop before the video ends?
  1. yes
  2. no
riA_41516/stops/green_yellow_striped_cube_1 · certified answer A
The orange ball moves. Does it come to a stop before the video ends?
  1. yes
  2. no
riA_41854/stops/orange_ball_1 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: The red-and-orange striped cone moves. Does it come to a stop before the video ends?

Options:
A) yes
B) no

Reply with exactly one letter (A, B). No other text.

followed by the 24 frames as images.

direction_reversal 5,336 gate-passed questions in the corpus

Question template (questions.py q_direction_reversal)

f"Does the {manifest[name]['descriptor']} change its direction of travel during the video?"

Options

['yes', 'no']

Option order: yes / no, always in this order.

Real examples (verbatim; certified answer highlighted)

Does the red-and-orange striped cone change its direction of travel during the video?
  1. yes
  2. no
riA_35683/direction_reversal/red_orange_striped_cone_0 · certified answer B
Does the green cylinder change its direction of travel during the video?
  1. yes
  2. no
riA_40318/direction_reversal/green_cylinder_0 · certified answer B
Does the blue-and-yellow striped cone change its direction of travel during the video?
  1. yes
  2. no
riA_36283/direction_reversal/blue_yellow_striped_cone_1 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Does the red-and-orange striped cone change its direction of travel during the video?

Options:
A) yes
B) no

Reply with exactly one letter (A, B). No other text.

followed by the 24 frames as images.

visible_at_end 5,001 gate-passed questions in the corpus

Question template (questions.py q_visible_at_end)

f"Is the {manifest[name]['descriptor']} still visible in the final moments of the video?"

Options

['yes', 'no']

Option order: yes / no, always in this order.

Real examples (verbatim; certified answer highlighted)

Is the yellow ball still visible in the final moments of the video?
  1. yes
  2. no
riA_41891/visible_at_end/yellow_ball_1 · certified answer B
Is the orange cylinder still visible in the final moments of the video?
  1. yes
  2. no
riA_32114/visible_at_end/orange_cylinder_0 · certified answer B
Is the yellow cube still visible in the final moments of the video?
  1. yes
  2. no
riA_41018/visible_at_end/yellow_cube_0 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Is the yellow ball still visible in the final moments of the video?

Options:
A) yes
B) no

Reply with exactly one letter (A, B). No other text.

followed by the 24 frames as images.

camera_motion 4,870 gate-passed questions in the corpus

Question template (questions.py q_camera_motion)

'How does the camera move during the video?'

Options

list(CAMERA_OPTIONS)
  1. static (not moving)
  2. orbiting around the scene
  3. moving straight toward or away from the scene
  4. turning in place (panning)

Option order: fixed set, always in this order.

Real examples (verbatim; certified answer highlighted)

How does the camera move during the video?
  1. static (not moving)
  2. orbiting around the scene
  3. moving straight toward or away from the scene
  4. turning in place (panning)
riA_31204/camera_motion · certified answer D
How does the camera move during the video?
  1. static (not moving)
  2. orbiting around the scene
  3. moving straight toward or away from the scene
  4. turning in place (panning)
riA_35670/camera_motion · certified answer D
How does the camera move during the video?
  1. static (not moving)
  2. orbiting around the scene
  3. moving straight toward or away from the scene
  4. turning in place (panning)
riA_31212/camera_motion · certified answer D
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: How does the camera move during the video?

Options:
A) static (not moving)
B) orbiting around the scene
C) moving straight toward or away from the scene
D) turning in place (panning)

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

approach 4,380 gate-passed questions in the corpus

Question template (questions.py q_approach)

f"Compared with the start of the video, where is the {manifest[name]['descriptor']} at the end?"

Options

list(APPROACH_OPTIONS)
  1. closer to the camera
  2. farther from the camera
  3. about the same distance
  4. it is never visible

Option order: fixed set, always in this order.

Real examples (verbatim; certified answer highlighted)

Compared with the start of the video, where is the orange ball at the end?
  1. closer to the camera
  2. farther from the camera
  3. about the same distance
  4. it is never visible
riA_30872/approach/orange_ball_2 · certified answer B
Compared with the start of the video, where is the purple ball at the end?
  1. closer to the camera
  2. farther from the camera
  3. about the same distance
  4. it is never visible
riA_35740/approach/purple_ball_1 · certified answer C
Compared with the start of the video, where is the yellow cylinder at the end?
  1. closer to the camera
  2. farther from the camera
  3. about the same distance
  4. it is never visible
riA_42180/approach/yellow_cylinder_3 · certified answer C
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Compared with the start of the video, where is the orange ball at the end?

Options:
A) closer to the camera
B) farther from the camera
C) about the same distance
D) it is never visible

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

spins 3,839 gate-passed questions in the corpus

Question template (questions.py q_spins)

f"Does the {manifest[name]['descriptor']} rotate (spin or tip over) during the video?"

Options

['yes', 'no']

Option order: yes / no, always in this order.

Real examples (verbatim; certified answer highlighted)

Does the blue-and-purple striped cone rotate (spin or tip over) during the video?
  1. yes
  2. no
riA_40459/spins/blue_purple_striped_cone_1 · certified answer B
Does the red-and-blue striped cone rotate (spin or tip over) during the video?
  1. yes
  2. no
riA_41256/spins/red_blue_striped_cone_0 · certified answer B
Does the purple-and-orange striped ball rotate (spin or tip over) during the video?
  1. yes
  2. no
riA_36872/spins/purple_orange_striped_ball_3 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Does the blue-and-purple striped cone rotate (spin or tip over) during the video?

Options:
A) yes
B) no

Reply with exactly one letter (A, B). No other text.

followed by the 24 frames as images.

depth_order 2,803 gate-passed questions in the corpus

Question template (questions.py q_depth_order)

f'Which object is closer to the camera {when}: the {da} or the {db}?'

Options

_shuffled([f'the {da}', f'the {db}', 'they are about the same distance away', 'neither is visible'], qid)

Option order: fixed set, order shuffled deterministically per question id.

Real examples (verbatim; certified answer highlighted)

Which object is closer to the camera in the middle of the video: the green cube or the red ball?
  1. the green cube
  2. the red ball
  3. neither is visible
  4. they are about the same distance away
riA_31113/depth_order/red_ball_1-green_cube_2@12 · certified answer B
Which object is closer to the camera in the middle of the video: the blue cylinder or the green-and-purple striped cylinder?
  1. the green-and-purple striped cylinder
  2. they are about the same distance away
  3. neither is visible
  4. the blue cylinder
riA_35552/depth_order/blue_cylinder_1-green_purple_striped_cylinder_3@12 · certified answer A
Which object is closer to the camera in the middle of the video: the blue cube or the green cone?
  1. neither is visible
  2. the blue cube
  3. they are about the same distance away
  4. the green cone
riA_30669/depth_order/green_cone_0-blue_cube_4@12 · certified answer D
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Which object is closer to the camera in the middle of the video: the green cube or the red ball?

Options:
A) the green cube
B) the red ball
C) neither is visible
D) they are about the same distance away

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

correspondence 2,399 gate-passed questions in the corpus

Question template (questions.py q_correspondence)

f'Look at the object that is furthest to the {side} at the very start of the video. Where is that same object at the end of the video?'

Options

list(CORR_OPTIONS)
  1. in the left third of the frame
  2. in the middle third of the frame
  3. in the right third of the frame
  4. no longer visible

Option order: fixed set, always in this order.

Real examples (verbatim; certified answer highlighted)

Look at the object that is furthest to the right at the very start of the video. Where is that same object at the end of the video?
  1. in the left third of the frame
  2. in the middle third of the frame
  3. in the right third of the frame
  4. no longer visible
riA_30411/correspondence · certified answer D
Look at the object that is furthest to the left at the very start of the video. Where is that same object at the end of the video?
  1. in the left third of the frame
  2. in the middle third of the frame
  3. in the right third of the frame
  4. no longer visible
riA_36257/correspondence · certified answer B
Look at the object that is furthest to the right at the very start of the video. Where is that same object at the end of the video?
  1. in the left third of the frame
  2. in the middle third of the frame
  3. in the right third of the frame
  4. no longer visible
riA_40780/correspondence · certified answer C
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Look at the object that is furthest to the right at the very start of the video. Where is that same object at the end of the video?

Options:
A) in the left third of the frame
B) in the middle third of the frame
C) in the right third of the frame
D) no longer visible

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

occlusion_event 1,892 gate-passed questions in the corpus

Question template (questions.py q_occlusion_event)

f'Consider the {da} and the {db}. During the video, which one passes in front of the other, hiding part of it?'

Options

_shuffled([f'the {da}', f'the {db}', neither, both], qid)

Option order: fixed set, order shuffled deterministically per question id.

Real examples (verbatim; certified answer highlighted)

Consider the blue cube and the blue-and-green striped cube. During the video, which one passes in front of the other, hiding part of it?
  1. the blue-and-green striped cube
  2. both: each passes in front of the other at different times
  3. neither: they never overlap on screen
  4. the blue cube
riA_36692/occlusion_event/blue_green_striped_cube_0-blue_cube_1 · certified answer A
Consider the purple-and-orange striped cylinder and the yellow cylinder. During the video, which one passes in front of the other, hiding part of it?
  1. the purple-and-orange striped cylinder
  2. the yellow cylinder
  3. both: each passes in front of the other at different times
  4. neither: they never overlap on screen
riA_30409/occlusion_event/yellow_cylinder_0-purple_orange_striped_cylinder_1 · certified answer D
Consider the blue-and-yellow striped ball and the green cube. During the video, which one passes in front of the other, hiding part of it?
  1. both: each passes in front of the other at different times
  2. neither: they never overlap on screen
  3. the green cube
  4. the blue-and-yellow striped ball
riA_35091/occlusion_event/green_cube_2-blue_yellow_striped_ball_3 · certified answer D
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Consider the blue cube and the blue-and-green striped cube. During the video, which one passes in front of the other, hiding part of it?

Options:
A) the blue-and-green striped cube
B) both: each passes in front of the other at different times
C) neither: they never overlap on screen
D) the blue cube

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

size_comparison 1,456 gate-passed questions in the corpus

Question template (questions.py q_size_comparison)

f'Which is the bigger object: the {da} or the {db}?'

Options

_shuffled([f'the {da}', f'the {db}', same, 'neither is visible'], qid)

Option order: fixed set, order shuffled deterministically per question id.

Real examples (verbatim; certified answer highlighted)

Which is the bigger object: the blue-and-yellow striped cube or the red cube?
  1. the red cube
  2. they are about the same size
  3. the blue-and-yellow striped cube
  4. neither is visible
riA_36640/size_comparison/red_cube_1-blue_yellow_striped_cube_4 · certified answer C
Which is the bigger object: the blue cone or the green cube?
  1. they are about the same size
  2. the blue cone
  3. neither is visible
  4. the green cube
riA_35471/size_comparison/blue_cone_0-green_cube_3 · certified answer D
Which is the bigger object: the green cube or the yellow cube?
  1. the yellow cube
  2. they are about the same size
  3. neither is visible
  4. the green cube
riA_41227/size_comparison/yellow_cube_0-green_cube_1 · certified answer D
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Which is the bigger object: the blue-and-yellow striped cube or the red cube?

Options:
A) the red cube
B) they are about the same size
C) the blue-and-yellow striped cube
D) neither is visible

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

lands 1,396 gate-passed questions in the corpus

Question template (questions.py q_lands)

f"The {manifest[name]['descriptor']} is falling. Does it land on the ground before the video ends?"

Options

['yes', 'no']

Option order: yes / no, always in this order.

Real examples (verbatim; certified answer highlighted)

The yellow-and-orange striped cylinder is falling. Does it land on the ground before the video ends?
  1. yes
  2. no
riA_41846/lands/yellow_orange_striped_cylinder_0 · certified answer A
The blue ball is falling. Does it land on the ground before the video ends?
  1. yes
  2. no
riA_31757/lands/blue_ball_0 · certified answer A
The blue cylinder is falling. Does it land on the ground before the video ends?
  1. yes
  2. no
riA_40970/lands/blue_cylinder_0 · certified answer A
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: The yellow-and-orange striped cylinder is falling. Does it land on the ground before the video ends?

Options:
A) yes
B) no

Reply with exactly one letter (A, B). No other text.

followed by the 24 frames as images.

exit_side 1,216 gate-passed questions in the corpus

Question template (questions.py q_frame_edge)

f'The {d} leaves the picture during the video. Through which edge of the frame does it leave?'

Options

list(EDGE_OPTIONS)
  1. the left edge
  2. the right edge
  3. the top edge
  4. the bottom edge

Option order: fixed set, always in this order.

Real examples (verbatim; certified answer highlighted)

The green ball leaves the picture during the video. Through which edge of the frame does it leave?
  1. the left edge
  2. the right edge
  3. the top edge
  4. the bottom edge
riA_40797/exit_side/green_ball_1 · certified answer A
The blue cone leaves the picture during the video. Through which edge of the frame does it leave?
  1. the left edge
  2. the right edge
  3. the top edge
  4. the bottom edge
riA_40433/exit_side/blue_cone_0 · certified answer B
The yellow cylinder leaves the picture during the video. Through which edge of the frame does it leave?
  1. the left edge
  2. the right edge
  3. the top edge
  4. the bottom edge
riA_41323/exit_side/yellow_cylinder_3 · certified answer A
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: The green ball leaves the picture during the video. Through which edge of the frame does it leave?

Options:
A) the left edge
B) the right edge
C) the top edge
D) the bottom edge

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

speed_comparison 1,166 gate-passed questions in the corpus

Question template (questions.py q_speed_comparison)

f'Which moves faster over the video: the {da} or the {db}?'

Options

_shuffled([f'the {da}', f'the {db}', same, 'neither of them moves'], qid)

Option order: fixed set, order shuffled deterministically per question id.

Real examples (verbatim; certified answer highlighted)

Which moves faster over the video: the red cube or the blue ball?
  1. they move at about the same speed
  2. the red cube
  3. neither of them moves
  4. the blue ball
riA_35696/speed_comparison/blue_ball_1-red_cube_3 · certified answer B
Which moves faster over the video: the green-and-orange striped cylinder or the orange cylinder?
  1. the orange cylinder
  2. the green-and-orange striped cylinder
  3. neither of them moves
  4. they move at about the same speed
riA_30408/speed_comparison/orange_cylinder_0-green_orange_striped_cylinder_1 · certified answer B
Which moves faster over the video: the green ball or the red-and-yellow striped cylinder?
  1. neither of them moves
  2. they move at about the same speed
  3. the red-and-yellow striped cylinder
  4. the green ball
riA_41946/speed_comparison/green_ball_0-red_yellow_striped_cylinder_2 · certified answer C
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Which moves faster over the video: the red cube or the blue ball?

Options:
A) they move at about the same speed
B) the red cube
C) neither of them moves
D) the blue ball

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

mover_count 1,077 gate-passed questions in the corpus

Question template (questions.py q_mover_count)

'How many of the objects change position during the video? (An object that only turns in place without moving does not count.)'

Options

count_options(n, qid)

Option order: numeric: as count (count_options).

Real examples (verbatim; certified answer highlighted)

How many of the objects change position during the video? (An object that only turns in place without moving does not count.)
  1. 0
  2. 1
  3. 2
  4. 4
riA_20137/mover_count · certified answer C
How many of the objects change position during the video? (An object that only turns in place without moving does not count.)
  1. 2
  2. 3
  3. 4
  4. 5
riA_30629/mover_count · certified answer A
How many of the objects change position during the video? (An object that only turns in place without moving does not count.)
  1. 0
  2. 1
  3. 2
  4. 3
riA_40342/mover_count · certified answer C
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: How many of the objects change position during the video? (An object that only turns in place without moving does not count.)

Options:
A) 0
B) 1
C) 2
D) 4

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

separation_change 851 gate-passed questions in the corpus

Question template (questions.py q_separation_change)

f'Compared with the start of the video, how are the {da} and the {db} positioned relative to each other at the end?'

Options

list(SEP_OPTIONS)
  1. closer together
  2. farther apart
  3. about the same distance apart
  4. they are never both visible

Option order: fixed set, always in this order.

Real examples (verbatim; certified answer highlighted)

Compared with the start of the video, how are the blue cube and the red-and-green striped cylinder positioned relative to each other at the end?
  1. closer together
  2. farther apart
  3. about the same distance apart
  4. they are never both visible
riA_41179/separation_change/red_green_striped_cylinder_2-blue_cube_3 · certified answer B
Compared with the start of the video, how are the blue cube and the yellow ball positioned relative to each other at the end?
  1. closer together
  2. farther apart
  3. about the same distance apart
  4. they are never both visible
riA_42173/separation_change/blue_cube_0-yellow_ball_2 · certified answer B
Compared with the start of the video, how are the purple cylinder and the red ball positioned relative to each other at the end?
  1. closer together
  2. farther apart
  3. about the same distance apart
  4. they are never both visible
riA_41876/separation_change/purple_cylinder_2-red_ball_3 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Compared with the start of the video, how are the blue cube and the red-and-green striped cylinder positioned relative to each other at the end?

Options:
A) closer together
B) farther apart
C) about the same distance apart
D) they are never both visible

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

temporal_order 632 gate-passed questions in the corpus

Question template (questions.py q_temporal_order)

f'Which starts moving first: the {da} or the {db}?'

Options

_shuffled([f'the {da}', f'the {db}', same, 'neither of them moves'], qid)

Option order: fixed set, order shuffled deterministically per question id.

Real examples (verbatim; certified answer highlighted)

Which starts moving first: the yellow ball or the blue-and-purple striped ball?
  1. the yellow ball
  2. the blue-and-purple striped ball
  3. neither of them moves
  4. they start moving at the same time
riA_20022/temporal_order/blue_purple_striped_ball_1-yellow_ball_2 · certified answer D
Which starts moving first: the orange cube or the orange cylinder?
  1. the orange cylinder
  2. they start moving at the same time
  3. neither of them moves
  4. the orange cube
riA_42150/temporal_order/orange_cylinder_2-orange_cube_3 · certified answer B
Which starts moving first: the blue-and-orange striped cylinder or the red ball?
  1. neither of them moves
  2. they start moving at the same time
  3. the red ball
  4. the blue-and-orange striped cylinder
riA_40846/temporal_order/blue_orange_striped_cylinder_1-red_ball_3 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Which starts moving first: the yellow ball or the blue-and-purple striped ball?

Options:
A) the yellow ball
B) the blue-and-purple striped ball
C) neither of them moves
D) they start moving at the same time

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

entry_side 545 gate-passed questions in the corpus

Question template (questions.py q_frame_edge)

f'The {d} is not visible at the start but appears later. Through which edge of the frame does it enter?'

Options

list(EDGE_OPTIONS)
  1. the left edge
  2. the right edge
  3. the top edge
  4. the bottom edge

Option order: fixed set, always in this order.

Real examples (verbatim; certified answer highlighted)

The red-and-purple striped cylinder is not visible at the start but appears later. Through which edge of the frame does it enter?
  1. the left edge
  2. the right edge
  3. the top edge
  4. the bottom edge
riA_36767/entry_side/red_purple_striped_cylinder_1 · certified answer B
The green cone is not visible at the start but appears later. Through which edge of the frame does it enter?
  1. the left edge
  2. the right edge
  3. the top edge
  4. the bottom edge
riA_35614/entry_side/green_cone_0 · certified answer B
The blue cube is not visible at the start but appears later. Through which edge of the frame does it enter?
  1. the left edge
  2. the right edge
  3. the top edge
  4. the bottom edge
riA_36692/entry_side/blue_cube_1 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: The red-and-purple striped cylinder is not visible at the start but appears later. Through which edge of the frame does it enter?

Options:
A) the left edge
B) the right edge
C) the top edge
D) the bottom edge

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

landing_order 129 gate-passed questions in the corpus

Question template (questions.py q_landing_order)

f'Both the {da} and the {db} are falling. Which one lands on the ground first?'

Options

_shuffled([f'the {da}', f'the {db}', same, neither], qid)

Option order: fixed set, order shuffled deterministically per question id.

Real examples (verbatim; certified answer highlighted)

Both the green-and-yellow striped cube and the purple cube are falling. Which one lands on the ground first?
  1. they land at the same time
  2. neither lands before the video ends
  3. the green-and-yellow striped cube
  4. the purple cube
riA_30387/landing_order/purple_cube_0-green_yellow_striped_cube_1 · certified answer D
Both the green cube and the yellow cube are falling. Which one lands on the ground first?
  1. neither lands before the video ends
  2. they land at the same time
  3. the green cube
  4. the yellow cube
riA_35720/landing_order/green_cube_0-yellow_cube_1 · certified answer D
Both the yellow-and-purple striped cube and the red-and-orange striped cube are falling. Which one lands on the ground first?
  1. they land at the same time
  2. the red-and-orange striped cube
  3. neither lands before the video ends
  4. the yellow-and-purple striped cube
riA_36151/landing_order/yellow_purple_striped_cube_2-red_orange_striped_cube_3 · certified answer D
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Both the green-and-yellow striped cube and the purple cube are falling. Which one lands on the ground first?

Options:
A) they land at the same time
B) neither lands before the video ends
C) the green-and-yellow striped cube
D) the purple cube

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

occlusion_recovery 106 gate-passed questions in the corpus

Question template (questions.py q_occlusion_recovery)

f"At some point the {manifest[name]['descriptor']} is completely hidden behind another object. Does it come back into view before the video ends?"

Options

['yes', 'no']

Option order: yes / no, always in this order.

Real examples (verbatim; certified answer highlighted)

At some point the green cone is completely hidden behind another object. Does it come back into view before the video ends?
  1. yes
  2. no
riA_31448/occlusion_recovery/green_cone_2 · certified answer A
At some point the green-and-orange striped cube is completely hidden behind another object. Does it come back into view before the video ends?
  1. yes
  2. no
riA_41851/occlusion_recovery/green_orange_striped_cube_1 · certified answer A
At some point the red cylinder is completely hidden behind another object. Does it come back into view before the video ends?
  1. yes
  2. no
riA_30719/occlusion_recovery/red_cylinder_3 · certified answer A
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: At some point the green cone is completely hidden behind another object. Does it come back into view before the video ends?

Options:
A) yes
B) no

Reply with exactly one letter (A, B). No other text.

followed by the 24 frames as images.

Alternative wordings (probes)

Neutral-wording A/B arm (same options and order)

lands rewording probe (same question id and answer letter)

Retired question types (3)

motion_direction 2 gate-passed questions in the corpus

Retired: camera parallax certifies the wrong direction on screen (riA_10076: the certified 'right' is entirely camera motion; riA_10131: opposite to the object)

Question template (questions.py q_motion_direction)

f"In which direction on screen does the {manifest[name]['descriptor']} mainly move over the video?"

Options

['left', 'right', 'up', 'down']

Option order: fixed set, always in this order.

Real examples (verbatim; certified answer highlighted)

In which direction on screen does the yellow cylinder mainly move over the video?
  1. left
  2. right
  3. up
  4. down
riA_2001/motion_direction/yellow_cylinder_1 · certified answer B
In which direction on screen does the blue cylinder mainly move over the video?
  1. left
  2. right
  3. up
  4. down
riA_2001/motion_direction/blue_cylinder_0 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: In which direction on screen does the yellow cylinder mainly move over the video?

Options:
A) left
B) right
C) up
D) down

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

relative_position 1 gate-passed questions in the corpus

Retired: ambiguous for a human: diagonal layouts, and 'on screen' vs 'in the world' left/right/above/below read differently

Question template (questions.py q_relative_position)

f'Where is the {da} relative to the {db} on screen {when}?'

Options

list(RELPOS_OPTIONS)
  1. to the left of it
  2. to the right of it
  3. above it
  4. below it

Option order: fixed set, always in this order.

Real examples (verbatim; certified answer highlighted)

Where is the blue cylinder relative to the yellow cylinder on screen at the end of the video?
  1. to the left of it
  2. to the right of it
  3. above it
  4. below it
riA_2000/relative_position/blue_cylinder_0-yellow_cylinder_1@23 · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: Where is the blue cylinder relative to the yellow cylinder on screen at the end of the video?

Options:
A) to the left of it
B) to the right of it
C) above it
D) below it

Reply with exactly one letter (A, B, C, D). No other text.

followed by the 24 frames as images.

scene_slide 1 gate-passed questions in the corpus

Retired: unintuitive for a human to answer (which way the background slides)

Question template (questions.py q_scene_slide)

'The camera moves during the video. As it does, does the scene as a whole slide to the left or to the right across the screen?'

Options

['to the left', 'to the right']

Option order: fixed set, always in this order.

Real examples (verbatim; certified answer highlighted)

The camera moves during the video. As it does, does the scene as a whole slide to the left or to the right across the screen?
  1. to the left
  2. to the right
riA_2001/scene_slide · certified answer B
Full prompt as sent (frames arm, first example)
You are watching a short video, shown as 24 frames sampled uniformly in time (frame 1 is the start, frame 24 is the end).

Question: The camera moves during the video. As it does, does the scene as a whole slide to the left or to the right across the screen?

Options:
A) to the left
B) to the right

Reply with exactly one letter (A, B). No other text.

followed by the 24 frames as images.