Project

NTU–A*STAR Joint Project · WP 2.2

Spatially-Grounded Perception and Planning for Embodied AI Agents

Build a reusable, planner-ready spatial layer that bridges perception and planning for manufacturing robots.

Project objectives

Connect perception to planning. Build a reusable spatial layer that organizes the objects, places and relationships a manufacturing robot needs to plan its actions.
Support long-horizon, safe mobile manipulation. Combine navigation and pick-and-place in dynamic, semi-structured shop-floor environments.
Validate reusability. Evaluate two manufacturing demos and transfer across at least two robot embodiments or layouts with minimal reconfiguration.

Application 1: electric meter testing and calibration

Task. Pre-shipment testing and calibration in an electric-meter manufacturing and assembly plant.
Challenge. Reliable tray handling and precise rack insertion despite uncertain depth and object boundaries.

A mobile robot with two RealMan RM65 arms and custom grippers collects a loaded tray from an external roller station, installs two pairs of meters along 60-degree guide channels, moves the empty tray to the rear deck and returns it to the station.

The column-foot joint turns the mast and upper body together. Lift, arm, column and base movements run sequentially. Mission, Interaction and 3D Object Graphs follow the task; the head RGB-D view and both wrist-mounted D405 views show the same operation from their rigid camera mounts.

Simulated Pipeline

Dual RealMan RM65 arms · Custom grippers · Simulation

Recursive self-improvement (RSI)

The intermediate RSI learning process improves the meter-insertion policy through execution feedback. The first graph requests insertion before the lift settles. Adding a wait lets the lower pair complete; adding wrist observation and lateral alignment then resolves the upper-row offset. Three runs of the same task model complete 0, 2 and 4 target meters.

Real-world deployment & open challenges

The real robot can complete the basic tray-handling and meter-insertion workflow, though execution is still relatively slow. Perception remains a challenge: uncertain object boundaries, noisy depth estimates and residual scene points can affect grasping and alignment.

Real robot transferring a tray beside the production-line test racks — Tray handling · 2×

Combined diagnostics: 2D and 3D meter bounding boxes on the left; boundary overlays and depth variation maps on the right — The 2D and 3D bounding boxes locate the meter, but uncertain boundaries and depth variation make its exact extent unclear, complicating grasp placement and clearance checks.

Real robot picking meters from a tray and inserting them into the test rack — Meter loading · 5×

RGB, depth and point-cloud cross-sections showing residual environment points inside the meter volume — Residual points can remain inside the meter’s volume after it is removed from the scene model. A planner may treat these points as obstacles and reject a feasible grasp.

Application 2: planetary gearbox assembly

Task. Collect components, assemble a planetary gearbox and fasten its cover.
Challenge. Align parts within tight clearances while maintaining stable grasps and coordinating both arms.

A mobile dual-arm robot explores the workcell, builds an object graph and selects a task-dependent chain of objects. It collects gear components from separate tables, assembles the gearbox, retrieves a wall-mounted driver, fastens the cover and returns the tool.

Ranger Mini 3.0 · Dual PiPER arms · Simulation

Contact simulation

Carrier and cover insertion

Both stages reach their seated positions.

Remaining manipulation challenges

Grasp slip, incomplete insertion and limited screw travel.

Application 3: biological sample handling

Task. Prepare biological samples through washing, mixing, centrifugation, aliquoting and storage.
Challenge. Handle small tubes accurately, track sample identity and coordinate parallel steps with equipment readiness.

A Ranger Mini mobile base with a lifting carriage and two PiPER arms carries out a 15-step workflow across five laboratory stations: materials storage, liquid handling, mixing, centrifugation and finished-sample storage. It collects and identifies sample S01, adds wash solution, mixes and centrifuges the sample. It then retains the pellet—the material collected at the bottom of the tube—mixes it back into liquid and prepares two labeled aliquots, or measured portions, S01-A and S01-B, before inspection, storage and cleanup.

The Mission Graph organizes the work order and its dependencies. The Object Graph connects samples, equipment and the room; the Interaction Graph tracks current actions, grasps and support relationships. Target-tube preparation runs alongside centrifugation. Unloading becomes available only after preparation is complete, the rotor has stopped and the lid is unlocked.