Scene 5: Kitchen B

A: toaster

B: light switch [electric outlet]

C: envelope [napkin on microwave]

D: light switch

E: microwave oven [microwave]

F: door [microwave door]

G: bottle [glass sauce bottle]

H: picnic basket [drying rack]

I: soap dispenser

J: bottle [plastic bottle with blue vanilla flavor]

K: mug [dry mug]

L: sink

M: frying pan [dirty pan in sink]

N: mug [dirty mug in sink]

O: countertop

P: waste container

Q: cupboard

R: plastic bag [trashbag]

Here we provide a scene from our real scene planning evaluation, and one example task for it. We provide the object detections and labels from OWL-ViT, which is what our planner has access to, but we provide more precise labels in square brackets. We provide color-coded bounding boxes for the object detections that are relevant for the example task. For each version of the LLM planner, we provide its output, including possible interaction with a VLM, and the generated plan. Outputs generated by the VLMs are in orange, and we mark correct and incorrect plans with green and red respectively.

Task 31: Bring me the glass object.

No VLM

Thought: There is no specific glass object mentioned in the list, but bottles are commonly made of glass. I will assume that one of the bottles (G or J) is the glass object.

Plan:

1. Go to object G

2. Pick up object G

3. Bring to human object G

4. Done

InstructBLIP

Thought: I can ask which objects are made of glass out of all the objects.

Question about [A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R]: Is this object made of glass?

Answer:

A: No (0.67), Yes (0.25), Unknown (0.04)

B: No (0.75), Yes (0.16), Unknown (0.06)

C: No (0.58), Yes (0.32), Unknown (0.06)

D: No (0.7), Yes (0.24), Unknown (0.03)

E: Yes (0.56), No (0.34), Unknown (0.05)

F: Yes (0.75), No (0.2), Bottle (0.01)

G: Yes (0.88), No (0.1), Glass (0.01)

H: No (0.71), Yes (0.22), Unknown (0.04)

I: No (0.87), Yes (0.1), Plastic (0.01)

J: Yes (0.48), No (0.43), Unknown (0.04)

K: No (0.64), Yes (0.33), Unknown (0.01)

L: Yes (0.6), No (0.32), Unknown (0.04)

M: Yes (0.5), No (0.45), Unknown (0.02)

N: No (0.49), Yes (0.46), Unknown (0.02)

O: Yes (0.58), No (0.32), Unknown (0.05)

P: No (0.84), Yes (0.12), Unknown (0.03)

Q: Yes (0.66), No (0.29), Unknown (0.02)

R: No (0.69), Yes (0.24), Unknown (0.04)

Thought: The object with the highest Yes value for being made of glass is G. I will bring object G to the human.

Plan:

1. Go to object G

2. Pick up object G

3. Bring to human object G

4. Done

PG-InstructBLIP (ours)

Thought: I can ask which object is made of glass out of all the objects.

Question about [A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R]: Is this object made of glass?

Answer:

A: No (0.84), Yes (0.12), Unknown (0.02)

B: No (0.96), Yes (0.02), Unknown (0.0)

C: No (0.97), Yes (0.01), Nothing (0.0)

D: No (0.94), Yes (0.04), Unknown (0.01)

E: No (0.74), Yes (0.21), Glass (0.01)

F: Yes (0.6), No (0.34), Glass (0.03)

G: Yes (0.95), No (0.04), Glass (0.0)

H: No (0.93), Yes (0.05), Unknown (0.01)

I: No (0.93), Yes (0.05), Plastic (0.01)

J: No (0.68), Yes (0.24), Plastic (0.05)

K: Yes (0.89), No (0.11), Glass (0.0)

L: No (0.94), Yes (0.04), Metal (0.01)

M: No (0.9), Yes (0.08), Glass (0.0)

N: Yes (0.73), No (0.25), Glass (0.01)

O: No (0.88), Yes (0.08), Unknown (0.01)

P: No (0.95), Yes (0.03), Unknown (0.0)

Q: No (0.94), Yes (0.04), Unknown (0.0)

R: No (0.87), Yes (0.09), Unknown (0.01)

Thought: The glass object is the one with the highest Yes value. The glass object is G.

Plan:

1. Go to object G

2. Pick up object G

3. Bring to human object G

4. Done

Page updated

Google Sites

Report abuse