Language-guided grasping in clutter
A unified pipeline turns a plain-language instruction into an executable grasp by joining language grounding, perception, and aerial manipulation.
IEEE/RSJ IROS · International Conference on Intelligent Robots and Systems
A Unified Framework for Aerial Grasping in Cluttered Environments
- 1 Robotics Research Center, IIIT Hyderabad, India
- 2 Department of Computer Science, University of Manchester, UK
TL;DR A language-guided aerial manipulator that actively explores a cluttered scene, predicts 6-DoF grasp candidates, and selects a collision-safe grasp to execute — running in real time on onboard hardware.
01 — Overview
Abstract
Grasping objects with a flying robot is hard when the workspace is cluttered. Objects sit behind one another, and the drone risks colliding with nearby obstacles as it reaches in. Most existing aerial grasping systems aim for the geometric center of an object, assume the object is already clearly visible, and treat perception, grasp selection, and control as separate problems.
AeroGrab brings these pieces into a single pipeline. The user gives a natural-language instruction such as “grasp the silver bottle on the white table.” The system locates the target, then flies around it to collect better viewpoints. For each viewpoint, a grasp network proposes several 6-DoF grasp candidates. Every candidate is checked by a collision-aware feasibility module that reasons about the full body of the drone and arm — not just the gripper. The best safe grasp is then executed using standard minimum-snap trajectory generation and an adaptive controller.
In real cluttered scenes, this combination of active viewing and feasibility-aware selection produces robust, reliable grasps, while staying light enough to run on a small onboard computer.
02 — Why it matters
The problem
Aerial manipulators can reach places ground robots cannot — high shelves, gaps, and spaces that are otherwise hard to access. But to grasp something useful, a flying robot has to do several things well at once: find the target, choose a stable place to grip it, and reach in without hitting anything.
Two common simplifications break down in the real world. First, aiming for the center of an object works for compact, uniform shapes, but everyday objects are best grasped at functional regions — a handle, a rim, an edge — not their centroid. Second, many systems assume the target is clearly visible. In clutter, objects are partially hidden, so whether a good grasp even exists depends heavily on where the camera is looking.
These issues are usually handled in isolation. AeroGrab argues they should be solved together: the robot should move specifically to reveal feasible grasps, and it should judge each grasp by whether the whole platform can safely fly to it.
- Centroid grasping Misses functional grasp regions on real objects.
- Static viewpoint Occlusion hides feasible grasps from a fixed pose.
- Arm-only checks Ignore collisions with the drone body during approach.
- Separate stages Perception, grasping, and control rarely act as one loop.
03 — Core idea
Look in order to grasp, grasp only where it is safe
Instead of treating where to look, where to grasp, and is it safe to fly there as separate steps, AeroGrab couples them. The drone changes its viewpoint to expose feasible grasps, and it scores every candidate by whether the full aerial manipulator — body, arm, and gripper — can reach it without hitting the surrounding clutter.
Active perception for viewpoint-dependent grasps
The robot autonomously repositions around a scene to improve how well it sees the target and to expose feasible grasp approaches.
Collision-aware 6-DoF feasibility
A ranking module scores grasps by full-body approach clearance, reachability, and clutter, selecting a safe grasp while actively checking for collisions.
04 — How it works
The pipeline, end to end
The input is a single RGB-D stream and one natural-language command. The output is a collision-safe grasp the robot actually flies to and executes. Here is the path from one to the other.
-
01
Language
An LLM splits the command into a scene prompt and a target prompt.
-
02
Search & explore
Yaw-scan to find the scene, then orbit it to reveal the occluded target.
-
03
Grasp synthesis
A hybrid CNN-ViT proposes many 6-DoF grasp candidates on the target.
-
04
Feasibility
Filter by reachability and full-body collisions; score each candidate's safety.
-
05
Decide
If the best score clears a threshold, execute; otherwise keep exploring.
-
06
Execute & lift
Track a minimum-snap trajectory, close the gripper, and lift clear of clutter.
Language-guided target localization
The command — for example “grasp the green bottle from the steel table” — is decomposed by an LLM into a scene prompt (“steel table”) and a target prompt (“green bottle”). SAM-3 segments these in real time and keeps the masks consistent across frames, giving a stable visual reference the rest of the pipeline can rely on.
Active viewpoint exploration
The robot first yaw-scans to find the scene, estimates its 3D centroid, then orbits around it. The orbit radius adapts so the whole scene stays inside the camera's field of view. As the drone moves, SAM-3 keeps updating the masks until the target becomes visible — at which point the task shifts from searching to grasping.
6-DoF grasp synthesis
A cropped region of interest around the target is fed to a hybrid CNN-ViT network trained on GraspNet-1Billion. It predicts a set of 6-DoF grasps, each with a translation, orientation, gripper width, and confidence score. Because these annotations were made for fixed-base arms, aerial-specific filters follow.
Aerial feasibility filtering
Two physical constraints reshape the candidate set. Gravity-aligned approach discards grasps that would require reaching up from beneath a surface. Stability-aware scoring penalizes grasps far from the object's center — which create larger moment arms at lift-off — without removing them, so they remain available when clutter rules out safer options.
Collision-aware safety scoring
Candidates are transformed to the world frame and checked with inverse kinematics for reachability. For reachable grasps, a full-body approach trajectory is generated and evaluated in a batched collision check: a fast KD-tree pass skips empty regions, then the GPU tests whether scene points fall inside the robot's convex hulls along the path. The visual score is then down-weighted by this collision penalty as a soft constraint.
Decision logic & execution
The grasp with the highest safety-weighted score wins. If even the best score is below the feasibility threshold, the robot does not fail — it keeps orbiting the target until a collision-free corridor appears. Once a safe grasp is found, a minimum-snap trajectory and an adaptive controller carry the robot in, the gripper closes, and the object is lifted clear.
Why the soft score A hard collision rule would reject many grasps that only graze clutter — often unavoidable in dense scenes. The penalty drives genuinely unsafe grasps to zero while keeping minor, grazing approaches on the table.
05 — System flow
Architecture overview
Perception and language run partly offboard; all time-critical planning and control run onboard the aerial platform.
06 — Experiments
What we tested, and how
Each scenario targets a distinct failure mode of aerial manipulation. In every trial the robot receives a language query, searches for the target, selects a feasible grasp, and executes it while keeping track of the target under viewpoint changes and occlusion.
Tabletop clutter
“Grasp the grey case with yellow latches on the white table.”
Dense tabletop clutter with partially overlapping objects. Tests grasp selection and a collision-feasible approach in near-field clutter.
Window / aperture-constrained
“Grasp the silver bottle on the white table inside the room.”
The target must be reached through a narrow window, enforcing strict base clearance and only partial visibility of the object.
Shelf / reachability & stability
“Grasp the grey-and-white triangular pipe inside the shelf.”
A multi-tier shelf with restricted access in depth and height. Tests reachability and stability constraints at the edge of the workspace.
Hardware platform
- FrameCustom 450 mm quadrotor · 2.5 kg takeoff (no payload)
- MotorsFlyCat 5010 750 KV BLDC · 12″ propellers
- Flight ctrlCUAV X7+ running PX4 · ROS 2 Humble (PX4 DDS)
- OnboardNVIDIA Orin Nano Super companion computer
- CameraZED 2i stereo RGB-D
- Arm4-DoF (4R) serial · Dynamixel XH430-W350
- OffboardRTX 4090 (24 GB) + Ryzen 9 · segmentation & viz only
- TrackingOptiTrack motion-capture for pose ground truth
Simulation
- EngineGazebo Harmonic · high-fidelity
- DynamicsPX4 Software-In-The-Loop (SITL)
- Arm4-DoF via ros2_control in torque mode
- SensorOAK-Lite RGB-D model
- Sim-to-realGaussian depth noise (σ = 0.02 m) + dropout
Metrics
- GA ↓Grasp Accuracy — distance from predicted to ideal grasp center (cm).
- SGL ↓Search-to-Grasp Latency — end-to-end mission time (s).
- CFGR ↑Collision-Free Grasp Rate — full grasps completed cleanly (%).
- OSF ↓Object Search Failures — target not found (%).
- CIF ↓Collision-Induced Failures — grasp lost to a collision (%).
07 — Results
What the numbers show
80%
collision-free grasp rate
on hardware (tabletop clutter)
on hardware (tabletop clutter)
41 ms
perception-to-plan latency
per planning cycle
per planning cycle
16.8 s
average end-to-end
episode time
episode time
Against a centroid-based baseline and a traditional 6-DoF aerial baseline under identical cluttered setups, AeroGrab holds a much higher collision-free grasp rate as clutter density increases. Centroid methods can post a low grasp-error number by trivially targeting the object center, but that simplification collapses in dense clutter; the 6-DoF baseline, lacking active viewing and tight collision checking, struggles to even find occluded targets.
Baseline comparison — tabletop, hardware (N = 30)
↓ lower is better · ↑ higher is better
| Method | Clutter | GA (cm) ↓ | CFGR ↑ | OSF ↓ | CIF ↓ |
|---|---|---|---|---|---|
| Bauer (CoRL '25) | Sparse | 1.2 ± 0.4 | 43.33% | –a | 56.67% |
| Ramon-Soria (Eng '20) | Sparse | 4.6 ± 0.5 | 56.67% | 20.00% | 23.33% |
| AeroGrab (Ours) | Sparse | 2.2 ± 0.6 | 90.00% | 3.33% | 6.67% |
| Bauer (CoRL '25) | Dense | 2.4 ± 0.5 | 10.00% | –a | 90.00% |
| Ramon-Soria (Eng '20) | Dense | 5.5 ± 0.4 | 23.33% | 43.33% | 33.33% |
| AeroGrab (Ours) | Dense | 3.1 ± 0.6 | 80.00% | 10.00% | 10.00% |
a OSF not reported for Bauer; target assumed already visible.
AeroGrab across all three scenarios
| Scenario | GA (cm) ↓ | SGL (s) ↓ | CFGR ↑ | OSF ↓ | CIF ↓ |
|---|---|---|---|---|---|
| Hardware runs (N = 30) | |||||
| 1 · Tabletop | 3.1 ± 0.6 | 16.8 | 80.0% | 10.0% | 10.0% |
| 2 · Window | 3.8 ± 0.6 | 10.5 | 73.33% | 10.0% | 16.67% |
| 3 · Shelf | 4.9 ± 0.4 | 15.0 | 70.0% | 16.67% | 13.33% |
| Simulation runs (N = 50) | |||||
| 1 · Tabletop | 1.2 ± 0.2 | 6.8 | 90.0% | 4.0% | 6.0% |
| 2 · Window | 2.1 ± 0.4 | 4.7 | 86.0% | 6.0% | 8.0% |
| 3 · Shelf | 1.9 ± 0.3 | 5.0 | 80.0% | 8.0% | 12.0% |
Architecture ablation — hardware (N = 30)
Latency = perception-to-plan compute per cycle (excludes motion)
| Variant | Latency (ms) ↓ | GA (cm) ↓ | CFGR ↑ | OSF ↓ | CIF ↓ |
|---|---|---|---|---|---|
| No obstacle awareness | 24 ± 6 | 5.2 ± 0.3 | 33.33% | 13.33% | 53.34% |
| No real-time update (open-loop) | 41 ± 10 | 4.5 ± 1.1 | 46.66% | 33.34% | 20.00% |
| CPU-only collision | 145 ± 12 | 3.8 ± 0.5 | 53.34% | 23.33% | 23.33% |
| Full pipeline (Ours) | 41 ± 12 | 3.1 ± 0.6 | 80.00% | 10.00% | 10.00% |
The ablation makes the trade-offs concrete. Dropping obstacle awareness is fastest but collapses reliability — many high-confidence grasps are simply not safe once full-body clearance is enforced. Running open-loop lets state drift go uncorrected during the approach. A CPU-only collision check is accurate but too slow to keep up with changing geometry. Only with all three — collision-aware feasibility, fast updates, and closed-loop perception — does the full pipeline reach the best overall result.
08 — Demo
Video
Real-world grasps across tabletop, window, and shelf scenarios.
09 — Citation
BibTeX
If you find this work useful, please consider citing it.
@inproceedings{singh2025aerograb,
title = {AeroGrab: A Unified Framework for Aerial Grasping in Cluttered Environments},
author = {Singh, Shivansh Pratap and Nair, Naveen and Ujjwal, Samaksh and
Mishra, Sarthak and Patil, Soham and Yadav, Rishabh Dev and Roy, Spandan},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year = {2026}
}
10 — Acknowledgements & contact
Acknowledgements
This work is partly supported by the Edge-AI-Di.Vision project from Qualcomm Technologies and the UASAT project sponsored by MeitY, India. The authors acknowledge the use of large language models (Gemini and ChatGPT) solely for refining the linguistic flow of the manuscript.