{"RequestId":"23b87c36-e02b-48e3-bb73-ab41c8b71c45","Code":200,"Message":"success","Data":{"isTop":null,"relatedPaperId":[101799],"nexa":"{}","Id":143990,"Namespace":"X-Humanoid","Name":"RoboMIND","CreatedBy":"Cherrytest","ChineseName":"","License":"apache-2.0","Description":"","Visibility":5,"Type":4,"Owner":"X-Humanoid","UserDefineTags":"real-world,robotic manipulation,Manipulation,Embodied AI,robotics,multi-embodiment,VLA,teleoperation data,dual arm","Likes":5,"Downloads":485593,"Size":null,"ReadmeContent":"# [RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation](https://x-humanoid-robomind.github.io/)\n\n[![License](https://img.shields.io/badge/License-Apache_2.0-yellow.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Project Page](https://img.shields.io/badge/Project%20Page-RoboMIND-blue.svg)](https://x-humanoid-robomind.github.io/)\n[![Dataset](https://img.shields.io/badge/Dataset-flopsera-000000.svg)](https://www.beaicloud.com/datasets/datasetDetail?path=%2Fdata-detail%2F21181956226031626&type=open)\n[![Hugging Face](https://img.shields.io/badge/Hugging_Face-RoboMIND-000000.svg)](https://huggingface.co/datasets/x-humanoid-robomind/RoboMIND)\n[![arXiv](https://img.shields.io/badge/arXiv-2412.13877-red.svg?style=flat-square)](https://arxiv.org/abs/2412.13877)\n\nAccepted by [Robotics: Science and Systems (RSS) 2025](https://roboticsconference.org/program/papers/152/).\n\n# \uD83D\uDCE2 ANNOUNCEMENT | RoboMIND V2.0 Release\nWe're excited to announce the release of **RoboMIND V2.0**!     \n\n\uD83D\uDD17 [Access the complete RoboMIND V2.0 collection on ModelScope](https://modelscope.cn/collections/X-Humanoid/RoboMIND20)\n\n## \uD83D\uDCBE Overview of RoboMIND \uD83D\uDCBE\n<img src=\"./static/images/piechart_new.png\" border=0 width=100%>\n\n### \uD83E\uDD16 Composition of RoboMIND \uD83E\uDD16\nWe present RoboMIND (Multi-embodiment Intelligence Normative Dataset and Benchmark for Robot Manipulation), a comprehensive dataset featuring 107k real-world demonstration trajectories spanning 479 distinct tasks and involving 96 unique object classes.\n\nThe RoboMIND dataset integrates teleoperation data from multiple robotic embodiments, comprising 52,926 trajectories from the Franka Emika Panda single-arm robot, 19,152 trajectories from the Tien Kung humanoid robot, 10,629 trajectories from the AgileX Cobot Magic V2.0 dual-arm robot, and 25,170 trajectories from the UR-5e single-arm robot.\n\nRoboMIND provides researchers and developers with an invaluable resource for advancing robotic learning and automation technologies by encompassing a broad spectrum of task types and diverse object categories. This dataset stands out for its substantial scale and exceptional quality, ensuring its effectiveness and reliability in practical applications.\n\n### \uD83D\uDD0E Distribution of Trajectory Lengths \uD83D\uDD0E\nDifferent robotic embodiments exhibit distinct trajectory length distributions. Franka and UR robots typically feature shorter trajectories with fewer than 200 timesteps, making them ideal for training fundamental manipulation skills. In contrast, Tien Kung and AgileX robots generally demonstrate longer trajectories exceeding 500 timesteps, which makes them better suited for training long-horizon tasks and complex skill combinations.\n\n\n### \uD83D\uDE80 Task Categories \uD83D\uDE80\nBased on natural language descriptions and considering factors such as object size, usage scenarios, and operational skills, we classify the dataset tasks into six major categories: 1) Articulated Manipulations (Artic. M.).  2) Coordination Manipulations (Coord. M.).  3) Basic Manipulations (Basic M.). 4) Multiple Object Interactions (Obj. Int.). 5) Precision Manipulations (Precision M.). 6) Scene Understanding (Scene U.)\nBeyond basic manipulations, the dataset includes numerous complex tasks, providing rich data support for training generalized robotic policies.\n\n\n### \uD83D\uDCAA Diversity of Objects \uD83D\uDCAA\nThe dataset encompasses 96 distinct object categories. In kitchen scenarios, it includes common foods like strawberries, eggs, bananas, and pears, as well as complex adjustable appliances such as ovens and bread makers. In domestic settings, the dataset features both rigid objects like tennis balls and deformable objects like toys. Office and industrial scenarios include small objects requiring precise control, such as batteries and gears. This diverse object range enhances dataset complexity and supports training versatile manipulation policies applicable across various environments.\n\n\n<img src=\"./static/images/Distribution_new.png\" border=5 width=95%>\n\n\n## \uD83D\uDCC1 Data Description \uD83D\uDCC1\nBuilding high-quality robotic training datasets is crucial for developing end-to-end embodied AI models with strong generalization capabilities. An ideal dataset should cover diverse scenarios, task types, and robotic embodiments, enabling models to adapt to different environments and reliably execute various tasks. Our team has constructed a large-scale, real-world robotic learning dataset that records interaction data during long-horizon task execution in complex environments, supporting the training of models with general manipulation capabilities.\n\nBelow is a partial directory structure example showing two training trajectories and two validation trajectories for a single task using the Franka robot:\n\n```\n.\n|-- h5_agilex_3rgb\n|-- h5_franka_1rgb\n|   |-- bread_in_basket\n|   |   `-- success_episodes\n|   |       |-- train\n|   |       |   |-- 1014_144602\n|   |       |   |   `-- data\n|   |       |   |       `-- trajectory.hdf5\n|   |       |   |-- 1014_144755\n|   |       |   |   `-- data\n|   |       |   |       `-- trajectory.hdf5\n|   |       |-- val\n|   |       |   |-- 1014_144642\n|   |       |   |   `-- data\n|   |       |   |       `-- trajectory.hdf5\n|   |       |   |-- 1014_151731\n|   |       |   |   `-- data\n|   |       |   |       `-- trajectory.hdf5\n|-- h5_franka_3rgb\n|-- h5_simulation\n|-- h5_tienkung_gello_1rgb\n|-- h5_tienkung_xsens_1rgb\n|-- h5_ur_1rgb\n```\n\n## \uD83D\uDDC3️ HDF5 File Format \uD83D\uDDC3️\n\nPlease refer to [all_robot_h5_info.md](./static/all_robot_h5_info.md).\n\nDue to equipment maintenance, 675 trajectories in the h5_franka_3rgb folder only contain image data from the left and right cameras. \n\nFor the specific data paths, please refer to [franka_3rgb_2cam_paths.md](./static/franka_3rgb_2cam_paths.md).\n\nIn the simulation data, the acquisition frequency of the camera and the robotic arm is approximately 1:4. Additionally, the depth image is not available temporarily.\n\n## \uD83E\uDDF0 Task Language Instructions \uD83E\uDDF0 \n\nWe have provided corresponding language instructions for each task [RoboMIND_instr.csv](./static/RoboMIND_v1_2_instr.csv)。\n\n## \uD83D\uDCCA Example of Data Usage \uD83D\uDCCA\n\nPlease refer to [Quick_Start.ipynb](./quick_start.ipynb).\n\nPlease note:\n\n1. For h5_franka_3rgb, h5_franka_1rgb, h5_ur_1rgb, and h5_franka_fr3_dual, the image channel order is BGR.\n2. For all other robotic embodiments, the image channel order is RGB.\n\n```python\nif sensor_type == 'rgb_images':\n    # These embodiments image data are recorded in BGR\n    if cur_embodiments in ['h5_franka_3rgb', 'h5_franka_1rgb', 'h5_ur_1rgb', 'h5_franka_fr3_dual']:\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    # Other embodiments image data are recorded in RGB  \n    else:\n        img = img\n```\n## \uD83D\uDEE0️ Dataset Utilities & Scripts\n\nTo facilitate data validation and processing, we maintain an official utility repository on GitHub: **[RoboMIND-dataset-utils](https://github.com/Open-X-Humanoid/RoboMIND-dataset-utils)**. \n\nThis toolkit currently provides:\n* **Data Quality Validation:** Scripts to perform full scans across the `benchmark1_0/1_1/1_2` data and generate detailed CSV reports.\n* **End-Effector Pose Recalculation:** For specific data subsets (`h5_ur_1rgb`, `h5_simulation`, `h5_sim_franka_3rgb`), we provide forward kinematics scripts to recalculate accurate end-effector poses directly from `joint_position` data and the corresponding URDF models.\n\nWe highly recommend utilizing these scripts if your VLA training or evaluation pipeline strictly relies on end-effector poses.\n\n## \uD83D\uDCD6 Version Update \uD83D\uDCD6\n\n### Version 1.1 & 1.2\n\nCompared to Version 1.0, we further expanded the dataset, which now includes 107K trajectories, 479 tasks, and covers 96 different object classes.\n\nIn version 1.2, we added 10 tasks of Upright_Cup data to Version 1.1, including 1 real-world task and 9 tasks from the digital twin environment. The goal of these 10 tasks is to flip a mug, but they involve different environmental settings, such as the range of mug placement, table textures, and mug appearances.\n\nThe frame-level fine-grained language instruction annotation data has been updated. Please refer to [language_description_annotation_json](https://huggingface.co/datasets/x-humanoid-robomind/RoboMIND/tree/main/static/language_description_annotation_json)\n\nFor more HDF5 file formats, please refer to [all_robot_h5_info_v1.2.md](./static/all_robot_h5_info_v1.2.md).\n\n### Version 1.0\n\nThe initial version of RoboMIND contains 55K trajectories, and 279 tasks, and involves 69 different object classes.\n\n## \uD83D\uDCDD Citation \uD83D\uDCDD\nIf you find RoboMIND helpful in your research, please consider citing:\n\n```\n@inproceedings{wu2025robomind,\n              title={Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation},\n              author={Wu, Kun and Hou, Chengkai and Liu, Jiaming and Che, Zhengping and Ju, Xiaozhu and Yang, Zhuqin and Li, Meng and Zhao, Yinuo and Xu, Zhiyuan and Yang, Guang and others},\n              booktitle={Robotics: Science and Systems (RSS) 2025}, \n              year={2025},\n              publisher={Robotics: Science and Systems Foundation}, \n              url={https://www.roboticsproceedings.org/rss21/p152.pdf} \n}\n```\n## Reference Document ##\nFor the input and output when training the model, please refer to [robomind.yaml](./static/robomind.yaml).\n\n## \uD83D\uDDE8️ Discussions \uD83D\uDDE8️\nIf you're interested in RoboMIND, welcome to join our WeChat group for discussions.\n\n<img src=\"./static/images/qrcode.jpg\" border=0 width=30%>","AlreadyStar":false,"GmtCreate":1757920083,"GmtModified":1791740443,"Tags":null,"Status":1,"FullName":"X-Humanoid","Organization":{"Id":2922,"GmtCreated":1762302783,"GmtModified":1762303353,"Name":"X-Humanoid","FullName":"X-Humanoid","Type":1,"Avatar":"https://resources.modelscope.cn/avatar/ffa840ba-c4f2-448e-8bdb-12aba7aa11cb.png","CreatedBy":"jydong01","GithubAddress":"https://github.com/Open-X-Humanoid","Description":"[\"root\",{},[\"p\",{},[\"span\",{\"data-type\":\"text\"},[\"span\",{\"data-type\":\"leaf\"},\"欢迎来到北京人形机器人旗下开源组织-开工造物开源社区，让我们一起探索具身智能的无限可能。\"]]],[\"p\",{},[\"span\",{\"data-type\":\"text\"},[\"span\",{\"data-type\":\"leaf\"},\"\"]]],[\"p\",{\"ind\":{\"left\":0},\"list\":{\"listId\":\"lz4h3m7ietb\",\"level\":0,\"isTaskList\":false,\"isOrdered\":false,\"listStyleType\":\"SCIR_ECIR_SREC\",\"symbolStyle\":{},\"listStyle\":{\"format\":\"bullet\",\"text\":\"●\",\"align\":\"left\"}}},[\"span\",{\"data-type\":\"text\"},[\"span\",{\"data-type\":\"leaf\"},\"北京人形官网：https://x-humanoid.com/\"]]],[\"p\",{\"ind\":{\"left\":0},\"list\":{\"listId\":\"lz4h3m7ietb\",\"level\":0,\"isTaskList\":false,\"isOrdered\":false,\"listStyleType\":\"SCIR_ECIR_SREC\",\"symbolStyle\":{},\"listStyle\":{\"format\":\"bullet\",\"text\":\"●\",\"align\":\"left\"}}},[\"span\",{\"data-type\":\"text\"},[\"span\",{\"data-type\":\"leaf\"},\"天工造物开源社区官网：https://opensource.x-humanoid-cloud.com/\"]]],[\"p\",{\"ind\":{\"left\":0},\"list\":{\"listId\":\"lz4h3m7ietb\",\"level\":0,\"isTaskList\":false,\"isOrdered\":false,\"listStyleType\":\"SCIR_ECIR_SREC\",\"symbolStyle\":{},\"listStyle\":{\"format\":\"bullet\",\"text\":\"●\",\"align\":\"left\"},\"extraData\":{},\"isChecked\":false,\"hideSymbol\":false}},[\"span\",{\"data-type\":\"text\"},[\"span\",{\"data-type\":\"leaf\"},\"联系我们：opensource@x-humanoid.com\"]]]]","DatasetCount":null},"UsedFor":null,"CertificationMark":0,"LastUpdatedTime":1784523290,"FromSite":"maas","SourcePlatform":null,"topIndex":null,"IsFlex":0,"StorageSize":12278969805241,"RelateArxivId":["2412.13877"],"Avatar":null,"ProtectedMode":2,"ApprovalMode":1,"ApplyMeta":null,"NEXA":{"Source":null,"DatasetCover":null,"ScientificField":null,"SubScientificField":null,"Catalogues":[]}},"PageNumber":null,"PageSize":null,"TotalCount":null}