Evaluation Stages

AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs

"Can Omni-MLLMs reasoning like humans from sight and sound"?

Eval Stage 1: Perception

amic.wav

Audio Multi-instance Classification (AMIC)

Question: Please identify the categories of sound-emitting objects in the given audio and how many different instances of each category are making sounds.

Answer: {"keyboard": "1", "dog": "2"}

Visual Multi-instance Classification (VMIC)

Question: Please identify the categories of visible objects in the given image and how many different instances of each category are visible.

Answer: { "horse": "2", "man": "3" }

avl.mp4

Audio-Visual Localization (AVL)

Question: Please generate the bounding-box of sound-emitting instances in the given image with the help of the given audio.

Answer: {"dog_1": [40.0, 169.0, 143.0, 90.0], "dog_2": [423.0, 67.0, 177.0, 92.0]}

avm.mp4

Audio-Visual Matching (AVM)

Question: Are the contexts of audio and visual content matching?

Answer: No.

Eval Stage 2: Understanding

Visual-ref Audio Retrieval (VAR)

Question: Based on the given image as a reference, identify which sounds in the provided audio list contain the objects corresponding to the image.

Answer: [2, 4, 5]

var.mp4

Audio-ref Visual Retrieval (AVR)

Question: Based on the given audio as a reference, identify which images in the provided image list contain the objects corresponding to the sound.

Answer: [1, 2, 6, 7]

2AD-0-onOTM.wav

avr.mp4

Audio-Visual Captioning (AVC)

Question: Based on the given audio as a reference, identify which images in the provided image list contain the objects corresponding to the sound.

Answer: A black cat is playing with water in the sink while a man and woman talk quietly in the background.

avc.mp4

Eval Stage 3: Reasoning

avh.mp4

Audio-ref Visual Hallucination (AVH)

Question: Is the talking people visible in the video?

Answer: No.

vah.mp4

Visual-ref Audio Hallucination (VAH)

Question: Is the man making sound in the audio?

Answer: No.

avqa.mp4

Audio-Visual Question Answering (AVQA)

Question: Is the instrument on the left more rhythmic than the instrument on the right?

Answer: Yes .

avlg.mp4

Audio-Visual Language Grounding (AVLG)

Question: Segment the object in the given framse based on the given text reference. Reference: The object making the loudest sound.

Answer: { "frame_0": [ 452, 198, 433, 520 ], "frame_1": [ 318, 146, 433, 574 ], "frame_2": [ 368, 123, 456, 597 ], "frame_3": [ 354, 120, 455, 600 ], "frame_4": [ 342, 101, 416, 579 ], "frame_5": [ 317, 79, 619, 574 ], "frame_6": [ 208, 104, 890, 568 ], "frame_7": [ 199, 9, 995, 711 ], "frame_8": [ 114, 21, 870, 699 ], "frame_9": [ 0, 54, 571, 666 ] }

Eval Stage 4: Primitive Sensation

Audio Sensation Question Answering (ASQA)

Question: Please compare the given two sound segments, which one has a softer volume?

Answer: Last.

asqa.mp4

Visual Sensation Question Answering (VSQA)

Question: What is the surface roughness of the rotating object?

Answer: Rough.

vsqa.mp4

Audio-Visual Sensation Question Answering (AVSQA)

Question: which object is making the louder sound?

Answer: Pyramid.

avsqa.mp4

Page updated

Google Sites

Report abuse