Tac-Man: Simulation Studies

In Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects, we present a prior-free strategy for articulated object manipulation leveraging tactile sensors. This approach has been validated through extensive real-world experiments, and here we present our simulation studies for further large-scale validation.

This repository hosts the code for the simulation studies as detailed in our paper. For a comprehensive understanding, please consider reading the full paper.

Preparing Environment

Install Omniverse Isaac Sim

Installation of Omniverse Isaac SimOur simulations run on NVIDIA Isaac Sim, which is part of the NVIDIA Omniverse platform. Ensure that you install the Isaac Sim 2022.2.1 for compatibility with our code. We are actively working on supporting later versions.

Post-installation, the simulation environment can be initiated using the provided Python script. It is advisable to use an alias, as suggested in the official documentation:

alias omni_python='~/.local/share/ov/pkg/isaac_sim-2022.2.1/python.sh'

Next, we need a few additional packages:

omni_python -m pip install trimesh tqdm bezier 

Download Pre-Processed Martnet-Mobility / GAPartNet Data

We evaluate our methods on a subset of objects from the PartNet-Mobility dataset. We also use annotations from GAPartNet for initializing grasps.

The datasets have been pre-processed to include extra details (see Appx. C of our paper.). To access the pre-processed data, please fill out the GAPartNet Dataset Application Form to comply with GAPartNet's terms and conditions. Once completed, download data from the link below and extract the data into data/gapartnet/.

Data link: Google Drive.

Download Playboards

Please download the data from the link below and unzip the contents to data/playboards/

Data link: Google Drive.

Start Simulation

For objects from PartNet-Mobility, please run:

omni_python run_partnet.py
            --idx       # Test case index, 0 by default.
            --obj_scale # object scaling, 0.5 by default
            --x_offset  # x-axis offset of robot base, 0.75 by defualt
            --y_offset  # y-axis offset of robot base, 0.0  by defualt
            --z_offset  # z-axis offset of robot base, 0.0  by defualt
            --headless  # run simulation in headless mode
            --capture   # record the simulation with Replicator
            --device    # computing device for transformation and states, "cuda" by default

For each case, we have pre-annotated grasps and robot base positions as initialization data in data/gapartnet/grasps.json.

For playboards with Bezier trajectories, run:

omni_python run_playboard.py
            --n_ctrl    # amount of control points of the Bezier curve
            --id        # test case index (under each `n_ctrl`)
            --x_offset  # x-axis offset of robot base, 0.5 by defualt
            --y_offset  # y-axis offset of robot base, 0.0 by defualt
            --z_offset  # z-axis offset of robot base, 0.0 by defualt
            --inverse   # run the trajectory in reverse
            --headless  # run simulation in headless mode
            --capture   # record the simulation with Replicator
            --device    # computing device for transformation and states, "cuda" by default

Important note

Setting the Base Positions for Fixed Manipulator

We find that Correct positioning of the robot base is critical to avoid joint limits during interactions. This problem is caused by limited workspace and could be solved by using mobile manipulators rather than fixed robot arms with limited workspace. For simplicity, we provide proper robot placements in data/..., with which the robot should be able to finish the manipulation. If it is stuck at some point, retry with an adjusted base position.

Mimicing Gripper Compliance with Rigid Body Simulation

Due to the some bugs in the deformable body simulation within Isaac Sim, we use a simple heuristic lock_joint_drive(...) and release_joint_drive(...) to mimic gripper compliance. Please refer to Appx. B of our paper.

Paralleled Environment

While the codebase is designed with parallel evaluation in mind, the implementation is not yet complete. We aim to introduce this feature in future updates.

Known Issues

1. Some URDF cannot be imported properly when launching run_partnet.py using Isaac Sim 2023.

Contact and Reference

If you encounter issues or have questions, please open a GitHub issue or contact Yuyang ([email protected]).

If you find our research beneficial, we appreciate citations to our paper:

@article{zhao2024tac,
  title={Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects},
  author={Zhao, Zihang and Li, Yuyang and Li, Wanlin and Qi, Zhenghao and Ruan, Lecheng and Zhu, Yixin and Althoefer, Kaspar},
  journal={arXiv preprint arXiv:2403.01694},
  year={2024}
}