Dual-arm tactile VLA manipulation with π0.5-DROID基于 π0.5-DROID 的双机械臂 VTLA 操作系统
Investigating how tactile feedback can inform a robot’s next action during pick-and-place tasks.研究机器人在抓取、放置物体时，如何利用触觉信息决定下一步动作。
Building a dual-Franka platform that brings tactile information into vision-language-action learning for tabletop manipulation.我正在搭建双 Franka Panda 操作平台，把触觉信息接入视觉—语言—动作模型，用于抓取、放置等桌面任务。
- Role项目角色
- Project Developer · VTLA Systems项目开发者 · VTLA 系统
- Period项目时间
- Jun 2026–present
- Support项目资助
- NVIDIA Developer Program · 8× H100 training accessNVIDIA Developer Program · 8 张 H100 训练资源
Build & progress具体工作与进展
What is in place目前做到哪一步
Teleoperation & demonstrations遥操与操作示教
Built the leader–follower setup with two Franka Panda arms and UMI grippers for demonstration collection.搭建两台 Franka Panda 与 UMI 夹爪组成的主从平台，用于采集操作示教。
Completed已完成
Model adaptation模型适配
Fine-tuned the existing π0.5-DROID model on robot task data using the available H100 training resources.基于机器人任务数据，使用 H100 训练资源微调已有 π0.5-DROID 模型。
Completed已完成
Local inference本地推理
Deployed the adapted model on an RTX 5090 for local inference.将适配后的模型部署到 RTX 5090，完成本地推理。
Completed已完成
Tactile integration触觉接入
Developing how tactile information enters the manipulation policy for pick-and-place tasks.继续开发触觉信息接入操作策略的方法，面向抓取、放置等桌面任务。
In progress进行中
01 / problem问题
Why a robot needs touch after contact机器人接触物体后，为什么还需要触觉
A vision-language-action model can plan from what it sees and hears, but manipulation also depends on what happens at contact. This project asks how tactile feedback should enter the policy and change the next action.视觉—语言—动作模型可以根据画面和指令规划动作，但真正拿起物体后，接触是否稳定、夹取是否合适，都需要触觉来判断。这个项目研究触觉应当怎样进入策略，并影响下一步动作。
02 / contribution工作
My contribution我负责的工作
I own the system path across teleoperation, data collection, model adaptation and local deployment.我负责从遥操、数据采集、模型适配到本地部署的整条系统链路。
- Built a leader–follower teleoperation system with two Franka Panda arms and UMI grippers.搭建两台 Franka Panda 与 UMI 夹爪组成的主从遥操系统。
- Fine-tuned π0.5-DROID with eight H100 GPUs provided through the NVIDIA Developer Program.使用 NVIDIA Developer Program 提供的 8 张 H100 微调 π0.5-DROID。
- Deployed the adapted model locally on an RTX 5090; tactile integration remains under development.在 RTX 5090 上完成适配模型的本地推理部署；触觉接入仍在开发中。
03 / approach方法
Demonstrate, train, deploy示教、训练、部署
The workflow starts with leader–follower demonstrations, uses the collected task data to fine-tune π0.5-DROID, and serves the adapted model locally on an RTX 5090. The next research step is to incorporate tactile feedback into the manipulation policy.先通过主从机械臂采集操作示教，再用任务数据微调 π0.5-DROID，最后把模型部署到本地 RTX 5090 进行推理。接下来的研究重点是把触觉反馈接入操作策略。
- TacTip tactile sensors replace the original fingertips of the Franka Panda gripper.在 Franka Panda 原生夹爪上，以 TacTip 触觉传感器替换原指尖。
- A ZED 2i observes the scene externally; an Intel RealSense D435 provides the wrist-mounted view.外部 ZED 2i 观察场景，腕部 Intel RealSense D435 提供近距离视角。
- Tactile feedback integration with the VLA policy is in development.触觉反馈与 VLA 策略的接入仍在开发中。
04 / outcome结果
Current system state目前进展
The project has connected teleoperated demonstration, model training and local robot inference in one working pipeline. Tactile integration for tabletop manipulation is now under active development.目前已经打通遥操示教、模型训练和本地机器人推理的完整流程，正在继续完善触觉信息在桌面操作任务中的接入方式。
Evidence论文与材料