This guide provides step-by-step instructions for setting up the GRID teleoperation interface using an Apple Vision Pro VR device to control a robot.

​
Prerequisites

Before proceeding with the setup, ensure the following requirements are met:
  • Install and configure GRID Enterprise, initializing it with the Isaac Sim workflow to enable VR-based teleoperation.
  • Download and install TrackingStreamer on Apple Vision Pro from the App Store. This app streams head, wrist, and hand tracking data to GRID.
  • Ensure that both the system running GRID and the Apple Vision Pro are connected to the same wireless network for seamless data streaming.

​
Configuring the IP Address for Streaming

To establish proper communication between the Apple Vision Pro and GRID, follow these steps:
  1. Open the TrackingStreamer application on the Apple Vision Pro and note the displayed IP address.
  2. Update the IP address in the agent configuration. Example configuration:
agent_cfg:
  - teleop:
      type: "SE3"
      device: "AVP"
      ip: 192.168.254.206

​
Operator Calibration

Calibration is necessary to align the operator’s wrist pose with the robot’s end effector, ensuring precise teleoperation. This process is performed once, with the robot stationary and visual aids in simulation guiding the alignment.

​
Calibration Steps

  1. Enable the calibration by setting calibrate: True in agent_cfg.yaml configuration.
  2. Start a session to visualize the scene, robot, and workspace: 
    session start teleop_calib rbfranka_sbfarm_visionpro.json
    • This session displays two transforms in simulation, tracking wrist poses for both hands.
    • Workspace bounds are also visualized.
  3. Observe the transformation needed to align the wrist with the robot’s end effector and note the workspace bounds.
  4. Stop the session: 
    session stop teleop_calib
  5. Update the bounds and transform parameters in the agent_cfg.yaml configuration in the following manner.
agent_cfg:
  - teleop:
      type: "SE3"
      device: "AVP"
      test: False
      ip: 192.168.254.206
      calibrate: True
      bounds:
        min: [-0.1, -0.6, -0.8]
        max: [0.6, 0.6, 1.2]
      transform:
        pos: [0.0, 0, -1.0]
        rot: [1, 0, 0, 0]

​
Teleoperation Validation

Once calibration is complete, unfreeze the robot and test teleoperation.

​
Validation Steps

  1. Set calibrate: False in agent_cfg.yaml configuration.
  2. Start the session to validate teleoperation: 
    session start teleop rbfranka_sbfarm_visionpro.json
open teleop viz
  3. Open the visualization interface in a browser and navigate to this link.
The operator can now control the robot in simulation using arm and hand movements.