Generated Code Analysis
Generated Code Analysis
Embedded Files
LLM generated code error type
Spatial error: misunderstand spatial relations leading to miscalculations
Error in Calculating Relative Position
Neglecting Collision Volume
...
Constraint violation: violate the given constraint or condition
Violation of Skill Capability Limits
Fabrication of Falsified Data
...
Logical error:
Error in Execution Order
Misjudgment of Conditions
...
Errors Observed in LLM-Generated Code
Analysis
Analysis
The semantic planner plays a important role in accurately understanding tasks and generating complete task plans. What is not easily reflected in the statistics is that some code generated by the coder-only approach may be incomplete, making errors occurrence number reduced, while in reality, they may be more severe.
The branching structure not only enhances the ability to handle more complex environments and improves code reusability, but it also provides feasibility checks to ensure the safe execution of strategies.
We believe that only one planner is sufficient, as task analysis and planning are not entirely independent tasks. Forcefully separating them into two parts can introduce unnecessary information transfer stages, potentially leading to the loss of details.
Inappropriate summarization and excessive levels of information transfer may lead to forgetfulness and conflict. We observed that in RoboTool, analyzer and planner sometimes have differing opinions. By the time the process reaches the parameter calculator and coder stages, the input lacks environmental descriptions and consists only of two conflicting summaries, leading to insufficient grounds for determining who is correct, even failure to generate complete code.
Understanding spatial relationships remains quite challenging for LLMs, especially in terms of considering collision volumes and calculating relative directions. An parameter calculator is crucial for accurately computing the geometric information required for robotic tasks, we found that providing calculation examples is really helpful.
The mixed usage of variable and constant information on the same attribute may confuse the LLM and reduce the accuracy of parameter calculations.
Instance analysis
Instance analysis
Next, we will detail the performance of each variant across the four specific tasks. For each case, we provide a heatmap of ten code samples on the right side, where the colors indicate the number of errors as shown in the legend. Additionally, the dropdown menu includes code examples generated by each variant, using the same color scheme to represent various types of errors, with green highlighting the correct code that is emphasized.
Light switching
Light switching
Error Description
Error Description
S+C (without parameter calculator)
The target position for climbing should be the desired standing state on top of the box, which requires accurately calculating the surface based on the box's sizes, while also accounting for the robot's height.
It is challenging to calculate the correct target position for pushing the box in front of the stairs.
Forgot to climb the box after pushing it.
C (coder only)
Similar spatial error and Logical error with S+C.
RoboTool
Similar climbing target position miscalculation. However, RoboTool typically makes this mistake when the step height is too high. It forcefully computes a result within the climbing range, attempting to climb even when it surpasses its capability.
Ours S+P+C
Ours S+P+C
# Step 1: Check the height of the lower stair
if lowerstair_size[2] <= 0.24:
# Step 2: Climb the lower stair
lower_stair_position = get_position("lower stair")
target_position = np.array(
[
lower_stair_position[0],
lower_stair_position[1],
lowerstair_size[2] + robot_size[2] / 2,
]
)
climb_to_position(target_position)
else:
# Step 3: Check if the box can be used as an additional step to climb the lower stair
if box_size[2] <= 0.24 and (lowerstair_size[2] - box_size[2]) <= 0.24:
# Step 5: Move the box in front of the lower stair
box_position = get_position("moveable box")
lower_stair_position = get_position("lower stair")
position_in_front_of_lower_stair = np.array(
[
lower_stair_position[0]
- (lowerstair_size[0] / 2 + box_size[0] / 2),
lower_stair_position[1],
0,
]
)
push_to_position(box_position, position_in_front_of_lower_stair)
# Step 6: Climb onto the box
box_position = get_position("moveable box") # Update box position
target_position = np.array(
[box_position[0], box_position[1], box_size[2] + robot_size[2] / 2]
)
# print("climb_target_pos", target_position)
climb_to_position(target_position)
# Step 7: Climb the lower stair from the box
lower_stair_position = get_position("lower stair")
target_position = np.array(
[
lower_stair_position[0],
lower_stair_position[1],
lowerstair_size[2] + robot_size[2] / 2,
]
)
climb_to_position(target_position)
else:
raise Exception("Task_Unsolvable_Error")
# Step 4: Check the height of the higher stair relative to the lower stair
if (higherstair_size[2] - lowerstair_size[2]) <= 0.24:
# Step 8: Climb the higher stair
higher_stair_position = get_position("higher stair")
target_position = np.array(
[
higher_stair_position[0],
higher_stair_position[1],
higherstair_size[2] + robot_size[2] / 2,
]
)
climb_to_position(target_position)
else:
raise Exception("Task_Unsolvable_Error")
# Step 9: Stand up to reach the button
stand_up()
# Step 10: Check if the button is within reach
button_position = get_position("button")
if 0.55 <= (button_position[2] - higherstair_size[2]) <= 0.70:
# Step 11: Press the button
hand_touch_position(button_position)
else:
raise Exception("Task_Unsolvable_Error")
# Step 12: Sit down
sit_down()
S+C
S+C
# Step 1: Feasibility Check for Climbing the Lower Stair
if lowerstair_size_z <= 0.24:
# Step 2: Climb the Lower Stair
lower_stair_position = get_position('lower stair')
climb_to_position(lower_stair_position)
else:
# Additional Steps for Using the Box
box_position = get_position('moveable box')
# Calculate target_position for the box to form an additional step
target_position_for_box = lower_stair_position - np.array([0, 0, box_size_z])
push_to_position(box_position, target_position_for_box)
# Step B: Climb onto the Box
box_position = get_position('moveable box') # Update box position
climb_to_position(box_position)
# Step 2: Climb the Lower Stair (Re-evaluated)
lower_stair_position = get_position('lower stair')
climb_to_position(lower_stair_position)
# Step 3: Feasibility Check for Climbing the Higher Stair
if higherstair_size_z - lowerstair_size_z <= 0.24:
# Step 4: Climb the Higher Stair
higher_stair_position = get_position('higher stair')
climb_to_position(higher_stair_position)
else:
raise Exception('Task_Unsolvable_Error')
# Step 5: Feasibility Check for Pressing the Button
button_position = get_position('button')
if 0.55 <= (button_position[2] - higher_stair_position[2]) <= 0.70:
# Step 6: Stand Up and Press the Button
stand_up()
hand_touch_position(button_position)
else:
raise Exception('Task_Unsolvable_Error')
# Step 7: Sit Down
sit_down()
RoboTool
RoboTool
# Climb to the lower stair
lower_stair_position = get_position('lower_stair')
target_position_lower_stair = np.array([1.81, 0.0, 0.4])
climb_to_position(target_position_lower_stair)
# Climb to the higher stair
# Assuming the higher stair's details are provided similarly, and we have calculated the target position for climbing to it
higher_stair_position = get_position('higher_stair')
# For example, if the calculated target position for the higher stair is [1.81, 0.0, 0.64], then:
target_position_higher_stair = np.array([1.81, 0.0, 0.64])
climb_to_position(target_position_higher_stair)
# Stand up on two legs
stand_up()
# Press the button by hand touching the position
# Assuming the button's position is provided and we have calculated the target position for hand touching it
button_position = get_position('button')
# For example, if the calculated target position for hand touching the button is [1.81, 0.0, 0.6], then:
target_position_button = np.array([1.81, 0.0, 0.6])
hand_touch_position(target_position_button)
# Sit down with four legs on the floor
sit_down()
C
C
# Step 1: Determine if the movable box is needed
lower_stair_position = get_position('lower stair')
higher_stair_position = get_position('higher stair')
box_position = get_position('moveable box')
# Calculate if the robot can climb the lower stair without the box
if lowerstair_size_z > 0.24:
# Check if the box can be used as a step
if (lowerstair_size_z - box_size_z) <= 0.24:
# Step 2: Move to the box and push it in front of the lower stair
push_to_position(box_position, lower_stair_position - np.array([0, 0, box_size_z]))
# Update box position after moving it
box_position = get_position('moveable box')
# Step 3: Climb the lower stair
climb_to_position(lower_stair_position + np.array([0, 0, lowerstair_size_z]))
# Step 4: Climb the higher stair
climb_to_position(higher_stair_position + np.array([0, 0, higherstair_size_z]))
# Step 5: Stand up on the higher stair
stand_up()
# Step 6: Press the button
button_position = get_position('button')
hand_touch_position(button_position)
# Step 7: Sit down
sit_down()
Elevator ride
Elevator ride
Error Description
Error Description
S+C (without parameter calculator)
It didn't account for the collision volume between the robot and the wall, attempting to move directly underneath the button.
A few codes exhibit the issue of skills transition, like attempting to perform the bipedal hand_touch operation without first executing the stand in a quadrupedal state.
C (coder only)
Similarly, it ignore the collision volumes between the robot and the wall.
Attempting to walk directly to the center of a fixed high platform.
The feasibility check for the climb only considered the lower bound, merely confirming that the climb height is greater than 0.
Ignored the provided robot size and fabricated a distance parameter.
RoboTool
Miscalculate the target position of climbing.
The lack of a branching structure significantly limits its generalization and reusability, requiring the code to be regenerated for each floor requirement.
In the face of this complex scenario, some samples generated only incomplete code.
Forget to sit down.
Ours S+P+C
Ours S+P+C
# Step 1: Check the height of the platform
platform_position = get_position("platform")
if platform_position[2] > 0.24:
raise Exception("Task_Unsolvable_Error")
# Step 2: Climb onto the platform
target_position = np.array(
[
platform_position[0],
platform_position[1],
platform_position[2] + robot_size[2] / 2,
]
)
climb_to_position(target_position)
# Step 3: Determine which button to press (up or down)
if current_floor < target_floor:
button_to_press = "button_up"
else:
button_to_press = "button_down"
# Step 4: Move to the button's position
button_position = get_position(button_to_press)
target_position = np.array(
[
button_position[0] - (robot_size[0] / 2 + 0.1),
button_position[1],
robot_size[2] / 2,
]
) # Assuming button size is negligible for x-axis adjustment
walk_to_position(target_position)
# Step 5: Check if the button is within reach
button_position = get_position(
button_to_press
) # Re-fetch the position to ensure accuracy
if not (0.55 <= button_position[2] - platform_position[2] <= 0.70):
raise Exception("Task_Unsolvable_Error")
# Step 6: Stand up and press the button
stand_up()
hand_touch_position(button_position)
sit_down()
# Step 7: Enter the elevator
elevator_position = get_position("elevator")
target_position = np.array(
[elevator_position[0], elevator_position[1], robot_size[2] / 2]
)
walk_to_position(target_position)
# Step 8: Move to the floor button's position inside the elevator
floor_button_position = get_position(f"button_{target_floor}")
target_position = np.array(
[
floor_button_position[0] - (robot_size[0] / 2 + 0.1),
floor_button_position[1],
robot_size[2] / 2,
]
)
walk_to_position(target_position)
# Step 9: Check if the floor button is within reach
floor_button_position = get_position(
f"button_{target_floor}"
) # Re-fetch the position to ensure accuracy
if not (0.55 <= floor_button_position[2] - elevator_position[2] <= 0.70):
raise Exception("Task_Unsolvable_Error")
# Step 10: Stand up and press the floor button
stand_up()
hand_touch_position(floor_button_position)
sit_down()
S+C
S+C
# Step 1: Check the height of the platform
platform_position = get_position('platform')
if platform_position[2] > 0.24:
raise Exception("Task_Unsolvable_Error")
# Step 2: Climb onto the platform
climb_to_position(get_position('platform'))
# Step 3: Determine which button to press (up or down)
if current_floor < target_floor:
button_to_press = 'button_up'
elif current_floor > target_floor:
button_to_press = 'button_down'
else:
# Task is already completed if current_floor == target_floor
print("Task already completed")
exit()
# Step 4: Move to the correct button
walk_to_position(get_position(button_to_press))
# Step 5: Press the button
stand_up() # Robot needs to stand up to reach the button
hand_touch_position(get_position(button_to_press))
sit_down() # Sit down after pressing the button
# Step 6: Enter the elevator
walk_to_position(get_position('elevator'))
# Step 7: Press the floor button inside the elevator
stand_up() # Stand up to reach the floor button
hand_touch_position(get_position(f'button_{target_floor}'))
sit_down() # Sit down after selecting the floor
# Step 8: Exit the elevator upon reaching the target floor
# Assuming the elevator automatically opens and reaches the target floor
walk_to_position(get_position('elevator'))
RoboTool
RoboTool
# Step 1: Climb onto the Platform
platform_position = get_position("platform")
# The calculated target position for climbing onto the platform
climb_target_position = np.array([2.6, 0.0, 0.11])
climb_to_position(climb_target_position)
# Step 2: Walk to the Button_up
button_up_position = get_position("button_up")
# robot_size = get_size('Robot')
# Calculating the target position to walk in front of the Button_up
walk_target_position_button_up = np.array(
[button_up_position[0] - (robot_size[0] / 2 + 0.1 / 2), 0.0, 0]
)
walk_to_position(walk_target_position_button_up)
# Step 3: Hand Reach to Press Button_up
# The robot stands up to extend its reach
stand_up()
# The calculated target position to hand touch the Button_up
hand_touch_target_position_button_up = np.array([button_up_position[0], 0.0, 0.789])
hand_touch_position(hand_touch_target_position_button_up)
# After pressing the button, the robot sits down
sit_down()
# Step 4: Walk into the Elevator
elevator_position = get_position("elevator")
# Calculating the target position to walk into the elevator
walk_target_position_elevator = np.array([elevator_position[0] - 0.8 / 2, -0.9, 0])
walk_to_position(walk_target_position_elevator)
# Step 5: Hand Reach to Press Button_3
# The robot stands up again to reach the button inside the elevator
stand_up()
button_3_position = get_position("button_3")
# Calculating the target position to hand touch the Button_3
hand_touch_target_position_button_3 = np.array([button_3_position[0], -0.9, 0.789])
hand_touch_position(hand_touch_target_position_button_3)
# The robot completes the task by sitting down
sit_down()
C
C
# Step 1: Climb onto the platform if needed
platform_position = get_position('platform')
walk_to_position(platform_position)
if platform_position[2] > 0: # Check if the platform is above the ground level
climb_to_position(platform_position)
# Step 2: Walk to the call button (up or down depending on the target floor)
if target_floor > current_floor:
button_position = get_position('button_up')
else:
button_position = get_position('button_down')
# Adjust the target position to stand in front of the button
# Assuming the button is fixed on the wall and robot needs to face the wall to press the button
button_position_to_stand = button_position - np.array([0.5, 0, 0]) # Adjust 0.5m away from the wall
walk_to_position(button_position_to_stand)
# Step 3: Stand up and press the call button
stand_up()
hand_touch_position(button_position)
sit_down()
# Step 4: Walk into the elevator
elevator_position = get_position('elevator')
# Adjust the target position to stand inside the elevator, considering the robot size
elevator_position_to_stand = elevator_position + np.array([elevator_size[0]/4, 0, 0]) # Adjust to be inside
walk_to_position(elevator_position_to_stand)
# Step 5: Press the floor button inside the elevator
floor_button_position = get_position(f'button_{target_floor}')
# Adjust the target position to stand in front of the floor button
floor_button_position_to_stand = floor_button_position - np.array([0.5, 0, 0]) # Adjust 0.5m away from the wall
walk_to_position(floor_button_position_to_stand)
# Step 6: Stand up and press the floor button
stand_up()
hand_touch_position(floor_button_position)
sit_down()
Package delivery
Package delivery
Error Description
Error Description
S+C (without parameter calculator)
Similar neglection of the collision volumes between the robot and the wall.
C (coder only)
It even miscalculated the doorbell Feasibility check.
It directly set the walking target to the box's position and did not update it after the box's position was disturbed.
RoboTool
Spatial errors are mainly concentrated around incorrect positioning in front of the button, similar to the first point in S+C.
There is a sample was incomplete where only the box-pushing action was performed.
Ours S+P+C
Ours S+P+C
# Step 1: Check feasibility to ring the doorbell
doorbell_position = get_position("doorbell")
if not (0.55 <= doorbell_position[2] <= 0.70):
raise Exception("Task_Unsolvable_Error")
# Step 2: Move to the doorbell
target_position_x = doorbell_position[0] - (
door_size[0] / 2 + robot_size[0] / 2
)
target_position_y = doorbell_position[1]
target_position_z = robot_size[2] / 2 # Robot is on the ground
walk_to_position(
np.array([target_position_x, target_position_y, target_position_z])
)
# Step 3: Stand up
stand_up()
# Step 4: Ring the doorbell
# Assuming the doorbell's position hasn't changed, no need to call get_position again
hand_touch_position(doorbell_position)
# Immediately sit down after ringing the doorbell
sit_down()
# Step 5: Wait for the door to open
# This step involves no specific action from the robot but assumes a human will open the door after the doorbell is rung.
# Step 6: Move to the package
package_position = get_position(
"package"
) # Get the most updated position of the package
walk_to_position(package_position)
# Step 7: Push the package to the door
door_position = get_position(
"door"
) # Get the most updated position of the door
target_position_x = door_position[0] + (door_size[0] / 2 + package_size[0] / 2)
target_position_y = door_position[1]
target_position_z = package_position[2] # Since it's moving on the ground
push_to_position(
package_position,
np.array([target_position_x, target_position_y, target_position_z]),
)
# Step 8: Move the package inside
# Assuming the package needs to be pushed further inside beyond the door threshold
package_position = get_position(
"package"
) # Get the most updated position of the package after the previous push
target_position_x += 0.5 # Push further inside by 0.5 meters
push_to_position(
package_position,
np.array([target_position_x, target_position_y, target_position_z]),
)
S+C
S+C
# Step 1: Check feasibility to ring the doorbell
doorbell_position = get_position("doorbell")
# Assuming ground level is z=0, check if the doorbell's height is within reach
if 0.55 <= doorbell_position[2] <= 0.70:
# Step 2: Move to the doorbell
# Calculate the target position in front of the doorbell, maintaining the same z-level
target_position_in_front_of_doorbell = np.array(
[doorbell_position[0], doorbell_position[1],
0]
)
walk_to_position(target_position_in_front_of_doorbell)
# Step 3: Stand up
stand_up()
# Step 4: Ring the doorbell
hand_touch_position(doorbell_position)
# Step 5: Sit down
sit_down()
# Assuming Step 6 (waiting for the door to be opened) is handled externally
# Step 7: Push the package to the door
package_position = get_position("package")
door_position = get_position("door")
# Calculate a target position just inside the door for the package
# Assuming the door opens towards the inside and we need to place the package just beyond the threshold
target_position_for_package = np.array(
[door_position[0] - (door_size_x / 2 + package_size_x / 2),
door_position[1], 0]
)
push_to_position(package_position, target_position_for_package)
# Step 8: Move the package inside
# Assuming we might need to adjust the position slightly further inside
final_target_position_for_package = np.array(
[target_position_for_package[0] - 0.5, target_position_for_package[1], 0]
)
package_position = get_position("package") # Get the most updated position
push_to_position(package_position, final_target_position_for_package)
else:
print("Task unsolvable with current capabilities. Doorbell out of reach.")
RoboTool
RoboTool
# Step 1: Push the Package Closer to the Door
# Get the most updated position of the package
package_position = get_position('package')
# Calculate the target position for the package to be pushed to, closer to the door
target_position_package = np.array([3.08 - 0.05/2 - 0.4/2, -0.2, 0.125])
# Push the package to the calculated position
push_to_position(package_position, target_position_package)
# Get the most updated position of the package after pushing
package_position = get_position('package')
# Step 2: Walk to the Doorbell's Position
# Calculate the target position to walk to in front of the doorbell
target_position_doorbell_walk = np.array([3.018, 0.0 + 0.06/2 + 0.4/2, 0])
# Walk to the calculated position in front of the doorbell
walk_to_position(target_position_doorbell_walk)
# Step 3: Stand Up to Reach the Doorbell
# Stand up to be able to reach the doorbell
stand_up()
# Step 4: Ring the Doorbell
# Calculate the target position to touch the doorbell
target_position_doorbell_touch = np.array([3.018, 0.0, 0.58])
# Touch the doorbell position to ring it
hand_touch_position(target_position_doorbell_touch)
# Step 5: Push the Package Inside the Door
# Get the most updated position of the package again
package_position = get_position('package')
# Calculate the target position for the package to be inside the door
target_position_package_inside = np.array([3.08 + 0.05/2 + 0.4/2, -0.2, 0.125])
# Push the package to the calculated position inside the door
push_to_position(package_position, target_position_package_inside)
# Get the most updated position of the package after pushing it inside
package_position = get_position('package')
C
C
# Step 1: Move to the package
package_position = get_position("package") # Get the current position of the package
walk_to_position(package_position) # Move to the package's position
# Step 2: Push the package closer to the door
# Assuming we need to leave some space for the robot to move in front of the package to push it towards the door
# We calculate a new target position for the package that is closer to the door but leaves room for the robot
door_position = get_position("door") # Get the current position of the door
# Calculate a target position for the package that is near the door but leaves space for the robot to maneuver
target_package_position = np.array(
[
door_position[0] - robot_size[0] - package_size_x / 2.0,
door_position[1],
package_position[2],
]
)
push_to_position(
package_position, target_package_position
) # Push the package to the calculated position
# Step 3: Move to the doorbell
doorbell_position = get_position("doorbell") # Get the current position of the doorbell
walk_to_position(doorbell_position) # Move to the doorbell's position
# Step 4: Ring the doorbell
# Check if the doorbell's height is within the reach of the robot's hand when standing up
if (
doorbell_position[2] - robot_size[2] >= 0.55
and doorbell_position[2]-robot_size[2] <= 0.70
):
stand_up() # Stand up to reach the doorbell
hand_touch_position(doorbell_position) # Ring the doorbell
sit_down() # Sit down back on four legs after ringing the doorbell
else:
# If the doorbell is not within reach, the robot might need to adjust its position or the task cannot be completed as described
print("Doorbell is not within reach. Adjust position or cannot complete task.")
# Assuming a human opens the door after the doorbell is rung
# Step 5: Push the package inside through the door
# After the door is opened, move back to the package
walk_to_position(target_package_position) # Move back to the package's new position
# Calculate a new target position for the package that is inside the door
inside_door_target_position = np.array(
[
door_position[0] + door_size_x / 2.0 + package_size_x / 2.0,
door_position[1],
door_position[2],
]
)
push_to_position(
target_package_position, inside_door_target_position
) # Push the package inside through the door
Bridge Building
Bridge Building
Error Description
Error Description
S+C (without parameter calculator)
Similar miscalculation of climbing target position, which ignore the robot's height and use center position as height.
Use the size to calculate the position.
C (coder only)
It attempts to push a box on the ground while it is on the low platform, which means push an object that is not on the same plane as itself.
Forget to climb the movable box before climbing the high platform.
RoboTool
It incorrectly calculated the target position for pushing the box between the two platforms.
Push the box that is not on the same plane as itself. , similar to first point of C
Ours S+P+C
Ours S+P+C
# Step 1: Check the position of the movable box
movable_box_position = get_position("movable box")
# Step 2: Push the movable box to the gap between the low and high platforms
low_platform_position = get_position("low platform")
high_platform_position = get_position("high platform")
target_position_x = (low_platform_position[0] + high_platform_position[0]) / 2
target_position_y = (low_platform_position[1] + high_platform_position[1]) / 2
target_position_z = 0 # Box is on the ground
target_position = np.array([target_position_x, target_position_y, target_position_z])
push_to_position(movable_box_position, target_position)
movable_box_position = get_position("movable box") # Update box position after pushing
# Step 3: Check the feasibility of climbing onto the low platform
if low_platform_size[2] > 0.24:
raise Exception("Task_Unsolvable_Error")
# Step 4: Climb onto the low platform
target_position = np.array(
[
low_platform_position[0],
low_platform_position[1],
low_platform_size[2] + robot_size[2] / 2,
]
)
climb_to_position(target_position)
# Step 5: Check the feasibility of climbing onto the movable box from the low platform
if box_size[2] - low_platform_size[2] > 0.24:
raise Exception("Task_Unsolvable_Error")
# Step 6: Climb onto the movable box
movable_box_position = get_position("movable box") # Ensure the position is up-to-date
target_position = np.array(
[movable_box_position[0], movable_box_position[1], box_size[2] + robot_size[2] / 2]
)
climb_to_position(target_position)
# Step 7: Check the feasibility of climbing onto the high platform from the movable box
if high_platform_size[2] - box_size[2] > 0.24:
raise Exception("Task_Unsolvable_Error")
# Step 8: Climb onto the high platform
target_position = np.array(
[
high_platform_position[0],
high_platform_position[1],
high_platform_size[2] + robot_size[2] / 2,
]
)
climb_to_position(target_position)
# Step 9: Task completion
print("Task completed. The robot has successfully reached the high platform.")
S+C
S+C
# Step 1: Obtain the Position of the Moveable Box
box_position = get_position("moveable box")
# Step 2: Push the Moveable Box to the Gap Between the Low and High Platforms
# Assuming the gap is directly between the low and high platforms along the x-axis
# and considering the size of the box to calculate the target position accurately
target_position_for_box = box_position.copy()
target_position_for_box[0] = (
low_platform_size_x + (high_platform_size_x - low_platform_size_x) / 2
)
push_to_position(box_position, target_position_for_box)
# Step 3: Obtain the Position of the Low Platform
low_platform_position = get_position("low platform")
# Step 4: Climb onto the Low Platform
# Assuming we have the height of the robot and can calculate the feasibility based on it
if low_platform_position[2] - robot_size[2] <= 0.24:
climb_to_position(low_platform_position)
else:
raise Exception("Task_Unsolvable_Error")
# Step 5: Walk to the Edge of the Low Platform Nearest to the Moveable Box
# Assuming the edge of the low platform nearest to the moveable box is towards the center of the gap
target_position_on_low_platform = low_platform_position.copy()
target_position_on_low_platform[0] = (
low_platform_position[0] + (low_platform_size_x / 2) - (robot_size[0] / 2)
)
walk_to_position(target_position_on_low_platform)
# Step 6: Climb onto the Moveable Box
# Update the box position after pushing it
box_position = get_position("moveable box")
if box_position[2] - low_platform_position[2] <= 0.24:
climb_to_position(box_position)
else:
raise Exception("Task_Unsolvable_Error")
# Step 7: Obtain the Position of the High Platform
high_platform_position = get_position("high platform")
# Step 8: Climb onto the High Platform
if high_platform_position[2] - box_position[2] <= 0.24:
climb_to_position(high_platform_position)
else:
raise Exception("Task_Unsolvable_Error")
RoboTool
RoboTool
# Step 1: Climb onto the Low Platform
low_platform_position = get_position("Low_platform")
low_platform_size = get_size("Low_platform")
# Calculating the target position for climbing onto the low platform
target_position_low_platform = np.array(
[
low_platform_position[0],
low_platform_position[1],
low_platform_position[2] + robot_size[2] / 2,
]
)
climb_to_position(target_position_low_platform)
# Step 2: Push the Movable Box to a Position Adjacent to the Low Platform
movable_box_position = get_position("Movable_box")
movable_box_size = get_size("Movable_box")
# Calculating the target position for the movable box
target_position_movable_box = np.array(
[
low_platform_position[0] + low_platform_size[0] / 2 + movable_box_size[0] / 2,
low_platform_position[1],
0.183,
]
)
push_to_position(movable_box_position, target_position_movable_box)
# Update the position of the movable box after pushing
movable_box_position = get_position("Movable_box")
# Step 3: Climb from the Low Platform onto the Movable Box
# Calculating the target position for climbing onto the movable box
target_position_on_box = np.array(
[
movable_box_position[0],
movable_box_position[1],
movable_box_position[2] + robot_size[2] / 2,
]
)
climb_to_position(target_position_on_box)
# Step 4: Climb from the Movable Box onto the High Platform
high_platform_position = get_position("High_platform")
high_platform_size = get_size("High_platform")
# Calculating the target position for climbing onto the high platform
target_position_high_platform = np.array(
[
high_platform_position[0],
high_platform_position[1],
high_platform_position[2] + robot_size[2] / 2,
]
)
climb_to_position(target_position_high_platform)
C
C
# Step 1: Get the position of the movable box.
box_position = get_position("moveable box")
# Step 2: Push the movable box to a position adjacent to the low platform.
# Assuming we have the positions of the low platform and the box, we calculate the target position for the box.
# This target position is right next to the low platform, on the side closest to the high platform.
low_platform_position = get_position("low platform")
box_target_position_near_low_platform = np.array(
[
low_platform_position[0] + (low_platform_size_x / 2 + box_size_x / 2),
low_platform_position[1],
0,
]
) # Adjust the x-coordinate to place the box next to the low platform.
push_to_position(box_position, box_target_position_near_low_platform)
# Step 3: Climb onto the low platform.
# The target position is on the low platform, so we adjust the z-coordinate to the height of the low platform.
climb_target_position_on_low_platform = np.array(
[low_platform_position[0], low_platform_position[1],
low_platform_size_z]
)
climb_to_position(climb_target_position_on_low_platform)
# Step 4: Push the movable box to a position adjacent to the high platform.
# First, get the updated position of the box since it might have changed after the previous push.
box_position = get_position("moveable box")
high_platform_position = get_position("high platform")
box_target_position_near_high_platform = np.array(
[
high_platform_position[0] - (high_platform_size_x / 2 + box_size_x / 2),
high_platform_position[1],
0,
]
) # Adjust the x-coordinate to place the box next to the high platform, on the side closest to the low platform.
push_to_position(box_position, box_target_position_near_high_platform)
# Step 5: Climb onto the high platform from the low platform using the movable box.
# The target position is on the high platform, so we adjust the z-coordinate to the height of the high platform.
climb_target_position_on_high_platform = np.array(
[high_platform_position[0], high_platform_position[1],
high_platform_size_z]
)
climb_to_position(climb_target_position_on_high_platform)
The full environment description could be found at google drive
RoboTool Prompts
RoboTool Prompts
Analyzer
Analyzer
You are a quadrupedal robot in a 3D world. You need to provide an answer in the following format in order to finish a task:
Answer Template:
<start of analysis>
<end of analysis>
<start of analysis summarization>
The first key feature is ...
The second key feature is ...
The third key feature is ...
The fourth ...
The fifth ...
<end of analysis summarization>
Constraints you must follow:
* You can push an object only if it meets all the following three conditions simultaneously: first, the robot is not on the surface of the object; second, the robot and the object are on the same surface and thrid, the surface belongs to an immovable object.
* When you are on the top surface of an object, you cannot move across a gap to another object unless the bounding boxes of these two objects are adjacent. For example, to compute the gap between two objects' bounding boxes in the x-direction: Assume the object1 is at 1.0 with size 0.5 along x-axis, and the object2 is at 2.0 with size 0.2 along x-axis. The gap can be computed by: object2_center_pos - object2_size/2 - object1_center_pos - object1_size/2 = 2.0 - 0.5/2 - 1.0 - 0.2/2 = 2.0 - 0.25 - 1.0 - 0.1 = 1.75 - 1.0 - 0.1 = 0.75 - 0.1 = 0.65. Similarly, you can compute the gap in the y-direction. Two objects are considered adjacent if the gap is 0.
- You can only climb up or climb down a platform with a height no larger than 0.24m in one step.
- When you stand up on two legs, your left hand can reach target with height in range [0.55m, 0.70m].
Remember:
* You must provide the key feature that affect the task feasibility, its value and the related constraints between <start of analysis summarization> and <end of analysis summarization>.
* Show the calculation processes and show the intermediate calculation steps clearly between <start of analysis> and <end of analysis>.
* You can only walk to a target position or push an object to a target position.
* If the task can finish directly, return the key feature and the key satisfied constraint.
* You must climb up onto the corresponding surface before pushing an object.
In the following, I will provide you the task description.
Planner
Planner
You are a quadrupedal robot on the ground in a 3D world. You must follow the following answer template:
Given a object list [OBJECT1, OBJECT2, ...]
<start of analysis>
[OBJECT1]: ... The robot can use this object as a ...
[OBJECT2]: ... The robot can use this object as a ...
[OBJECT3]: ... The robot ...
[Key Feature and Constraints]:
[Abstract Plan]: First, the robot will ... Now the robot is on the surface of [OBJECT]. The [OBJECT] is [movable/immovable]. Then, ... Now the robot is on the surface of [OBJECT]. Finally, ...
<end of analysis>
Calculation Examples:
* The x-y gap between two objects' bounding boxes in the x direction: The object1 is at 1.0 with size 0.5 along x-axis. The object2 is at 2.0 with size 0.2 along x-axis. The gap is object2_center_pos - object2_size/2 - object1_center_pos - object1_size/2 = 2.0 - 0.5/2 - 1.0 - 0.2/2 = 2.0 - 0.25 - 1.0 - 0.1 = 1.75 - 1.0 - 0.1 = 0.75 - 0.1 = ....
Rules for analysis:
* You must only choose [OBJECT] from the object list.
* You must think and describe each object's physical properties, affordances, and roles in completing the task.
* You must know that each object occupies a bounding box with size provided.
* You must pay attention to the key features in the [Description] and show all the related constraints in [Key Feature and Constraints].
* You must not ignore any constraints when reasoning [Key Feature and Constraints].
Rules for [Abstract Plan]:
* You must think about how to use [OBJECT]s to finish the task.
* You must think about the change of the surfaces the robot is standing on after each step.
* You can only push an object to a target position or walk to a target position.
* Do not walk to a [OBJECT] before pushing the [OBJECT], since the push_to_position skill already handles walking to the [OBJECT].
* You must think step by step and show the thinking process, including what objects you want to use, how to move them, in what order and cause what effects.
* You must stictly satisfy all the constraints in [Constraints you must follow] and ensure the feasibility of the plan.
* You do not need to provide exact numerical values in the [Abstract Plan].
* If you need to push an object, you must think of how to get to the same surface as that object, and explicitly mention that in the [Abstract Plan]. If you need to walk to an intermediate position, you must mention that in the [Abstract Plan].
* You must perform the skills one at a time.
* You shouldn't call walk_to_position before push_to_position, because push_to_position is already consists of walk to a proper place in order to interact with the object.
* You need to sit down right after you call hand touch position in order to obtain a stable pose and avoid falling.
* You don't need to walk to a position in order to move an object, you can simply call push_to_position.
* To walk to the top of [OBJECT], you should walk to the xy center of [OBJECT].
Constraints you must follow:
* You can push an object only if it meets all the following two conditions simultaneously: first, You are on the surface of an immovable object or ground; second, you and the object are on top of the same surface, and you are not on top of the object it is pushing.
* If you want to move from one object to another object, you must make sure that 1) the bounding boxes of the two objects are in contact along one axis or the bounding boxes overlap along one axis. 2) the target surface must be 0.12 meters or less above the surface you are currently on.
I will provide you with [Task Description] in the following, and you will respond with the analysis. You must complete the task successfully using the objects provided. You must make the abstract plan use as few steps as possible.
You are a quadrupedal robot that can move in 3D space. You have a skill set containing the following skills:
- walk_to_position(target_position): walk with four legs to the target_position in the same x-y plane as the starting position. Note that you can only move to places with the same height.
- climb_to_position(target_position): Climb to a platform higher or lower than the robot, and let the robot's center reach target_position. The height difference of the two consecutive platforms is less than 0.24m. Note that if the height difference is large than 0.24m, this skill cannot be performed.
- push_to_position(object_position, target_position): Walk toward a moveable object whose center's position is object_position, and push it till the object's center's position is target_position.
- stand_up(): stand up on its two hind legs. This enables the robot to touch higher positions.
- sit_down(): transit from bipedal standing to sitting down with four legs on the ground.
- hand_touch_position(target_position): touch target_position with your left hand with two legs standing on the ground. Note that you can only touch positions with height in range [0.55m, 0.75m] relative to your standing platform. Note the difference between the target's height and the target's height relative to the robot's standing platform.
Rules for detailed plan:
* You must plan according to the analysis.
* You must use existing skills.
* You must make each plan step only call one skill at once and be as atomic as possible.
* You must get the updated [OBJECT]'s position again if [OBJECT] has moved since the last 'get_position([OBJECT])'.
* You must strictly follow the constraints.
* Do not walk to a [OBJECT] before pushing the [OBJECT], since the push_to_position skill already handles walking to the [OBJECT].
- You must perform the skills one at a time.
- You shouldn't call walk_to_position before push_to_position, because push_to_position is already consists of walk to a proper place in order to interact with the object.
- You need to sit down right after you call hand touch position in order to obtain a stable pose and avoid falling.
- You don't need to walk to a position in order to move an object, you can simply call push_to_position.
- To walk to the top of [OBJECT], you should walk to the xy center of [OBJECT].
Example answers for plan:
<start of plan>
* Use the [SKILL] to [SINGLE_TASK].
<end of plan>
Calculator
Calculator
This part is to calculate the 3D target positions.
Common Rules:
* Calculate step by step and show the calculation process between <start of description> and <end of description>.
- Return the 3D target position for functions 'push_to_position', 'climb_to_position', 'walk_to_position', 'hand_reach_position' used in the plan.
* You must not assume any position and directly query the updated position of the objects.
* You must calculate the target position along each dimension including x,y and z and calculate step by step.
* The "get_position" and "get_size" do not need target positions. Return a space character between <start of answer> and <end of answer>.
* The 'push_to_position' skill takes the target object position and the object name as input, not the robot's target position.
* The 'walk_to_position' skill takes the target 3D position as input. Note that the z position does not matter since the skill itself will handle the climbing. You can fill in 0 for the z position.
* When calculating the target position for the 'push_to_position' skill, you must consider the spatial relationship between the objects. For instances, the target position of [OBJECT1] may be dependent on the position of [OBJECT2] and [OBJECT3].
* You can assume that the robot size is (robot_size[0], robot_size[1], robot_size[2]).
Some tips on calculation:
- The height of the top surface of an object is object_position_z + object_size_z / 2. If the object is on the ground, then the height of the top surface is simply object_size_z.
- When the robot is on top of something, then the robot's height is the height of the object's top surface plus robot_size[2]/2. Note the the difference with the case where an object is on another object.
- When the robot is on the ground, the height of the robot center is robot_size[2]/2, not 0.
- When the robot walk to an object, it means that the robot move to next the object, not on the object. Please first reason the direction of the robot with respect to the robot, then determine the the target position of the robot with special care of this direction.
- When two objects are next to each other, the distance between their centers along an axis is half of the sum of their sizes along that axis.
- When the robot is next to or in front of an object, the distance between the robot's center and the object's center along an axis is half of the sum of the robot's size and the object's size along that axis. If the object is perpendicular to any axis, then the axis which align the robot's center and the object's center is that axis.
- Note the difference between height in world coordinate system, and the relative height to a certain platform. To calculate the relative height, you need to use the height in world coordinate system substract the height of the platform.
- Don't omit anything! You should output executable codes.
Example 1:
<Current Step>: Use the "push_to_position" to push 'box' to the middle of object_1 and object_2 in x-axis.
<start of description>
* The target position can be computed by using values obtained by "get_position" from the last step.
<end of description>
<start of answer>
The 3D target position is [(object_1_pos[0] + object_2_pos[0])/2, object_2_pos[1], object_2_pos[2]].
<end of answer>
Example 2:
<Current Step>: Use the "push_to_position" to push object_1 next to object_2.
<start of description>
* Since object_3 is occupying the adjacent positive x direction of object_2, we can only push object_1 to the negative x direction of the object_2. The target position along the x-axis is object_2_pos[0] - object_2_size[0]/2 - object_1_size[0]/2.
<end of description>
<start of answer>
The 3D target position is [object_2_pos[0] - object_2_size[0]/2 - object_1_size[0]/2, object_2_pos[1], object_1_pos[2]].
<end of answer>
Example 3:
<Current Step>: Use the "walk_to_position" or "climb_to_position" to walk or climb on the top of [OBJECT].
<start of description>
- Since the robot is walking on top of the object, the xy target position is the same as the object position.
- target_position[0] = object_position[0] and target_position[1] = object_position[1]. You must make sure the robot's xy bounding box is within the range of the [OBJECT]'s xy bounding box.
- The target position along the z axis is the object_size[2] + robot_size[2]/2.
<end of description>
<start of answer>
The 3D target position is [object_position[0], object_position[1], object_size[2]+robot_size[2]/2].
<end of answer>
In the following, you will see the plan and must follow the rules.
Coder
Coder
You are a quadrupedal robot.
The robot has a skill set: ['walk_to_position', 'climb_to_position', 'push_to_position', 'get_position', 'get_size', 'stand_up', 'sit_down', 'hand_touch_position'].
You have a description of the plan to finish a task. We want you to turn the plan into the corresponding program with following functions:
```
def get_position(object_name):
return object_position
```
get_position returns the 3D position of the object's center. The center position is located in the middle of the object.
```
def get_size(object_name):
return object_size
```
object_size is the physical properties for the object.
- quadrupedal walk/climb
```
def walk_to_position(target_position): # walk to the target_position in the same x-y plane as the starting position. Note that you can only move to places with the same height. target_position is a 3d numpy array.
```
```
def climb_to_position(target_position): # Climb to a platform higher or lower than the robot and reach target_position, where the height difference is less than 0.24m. Note that if the height difference is large than 0.24m, this skill cannot be performed. target_position is a 3d numpy array.
```
```
def push_to_position(object_position, target_position): # Walk toward a moveable object whose center's position is object_position, and push it till its center's position is target_position. target_position is a 3d numpy array.
```
```
def stand_up(): # stand up on its two hind legs. This enables the robot to touch higher positions.
```
```
def sit_down(): # transit from bipedal standing to sitting down with four legs on the ground.
```
```
def hand_touch_position(target_position): # touch target_position with your left hand with two legs standing on the ground. target_position is a 3d numpy array. Note that you can only touch positions with height relative to your standing platform in range [0.55m, 0.75m]. The target's height relative to the your standing platform is calculated by the height in world coordinate system substract the height of the platform.
```
Example answer code:
```
# python
import numpy as np # import numpy because we are using it below
# Always get a position of an object with the 'get_position' function before trying to move to an object.
box_position = get_position('box')
```
Rules:
* Always format the code in code blocks.
* You must not leave unimplemented code blocks in your response.
* You must not leave undefined variables in your response.
* The only allowed library is numpy. Do not import or use any other library. If you use np, be sure to import numpy.
* If you are not sure what value to use, just use your best judge. Do not use None for anything.
* Do not define new functions, and only use existing functions.
* If you want to interact with a movable [OBJECT], you must get the most updated position of an object with the 'get_position' function right before you call other functions.
* If you need to use the position of any [OBJECT], you must get the most updated position of an object with the 'get_position' function right before you call other functions or do other calculations.
- For later convenience, you may first get all positions of all objects, and renew their position everytime you need.
- For functions 'walk_to_position', 'climb_to_position', 'push_to_position' and 'hand_touch_position', you should use the target position later provided in calculation results.
- For function 'hand_touch_position', note the standing platform. When do feasibility check, you should use the target position height relative to the standing platform.
- After you move an object, you must call 'get_position' function to get the most updated position of the object.
- You should calculate the target positions every time you use 'walk_to_position', 'climb_to_position', 'push_to_position' and 'hand_touch_position'. Don't mix with previous calculations.
In the following, I provide you the description and you respond with the code.
Take aways
Take aways
Thank you very much for your interest in our work. Here are some of our suggestions on how to prompt a robotics LLM agent.
Teach the LLM to use branching structures to solve problems.
Appropriately providing examples to guide the LLM, especially calculation examples.
Avoid excessive information transfer nodes to prevent forgetfulness.
Avoid providing both variable and constant information on the same attribute.
Page updated
Google Sites
Report abuse