This guide demonstrates how to create a robot that can detect specific objects using computer vision and navigate towards them. The example works with both simulated and real robots.

​
Overview

The detect and navigate example combines:

  • Object detection using OWLv2 model
  • Robot movement control
  • Real-time camera feed processing
  • Velocity-based navigation

​
Core Components

​
Detection Model

We use the OWLv2 model for object detection:

from grid.model.perception.detection.owlv2 import OWLv2
det_model = OWLv2()

​
Box Center Calculation

This function calculates the center point of detected objects and checks if they’re too close:

def get_box_center(box, image_size):
    x_min, y_min, x_max, y_max = box
    # Check if object is too close (covers >38.5% of image)
    if image_size[0] * image_size[1] * .385 < (x_max-x_min) * (y_max-y_min):
        return -1, -1
    x_center = (x_min + x_max) / 2
    y_center = (y_min + y_max) / 2
    return x_center, y_center

​
Velocity Calculation

This function determines how the robot should move based on the target’s position:

def get_velocity(target, image_size):
    # Calculate normalized offsets from image center
    image_center_x = image_size[0] / 2
    image_center_y = image_size[1] / 2
    offset_x = target[0] - image_center_x
    offset_y = target[1] - image_center_y
    
    # Convert to normalized velocities
    norm_offset_x = offset_x / image_center_x
    angular_velocity = max(min(-norm_offset_x, 2), -2)
    
    return (
        Velocity(0.8, 0, 0),  # Linear velocity
        Velocity(0, 0, angular_velocity)  # Angular velocity
    )

​
Main Control Loop

The main loop continuously:

  1. Captures images from the robot’s camera
  2. Detects the target object
  3. Calculates required movement
  4. Controls the robot’s motion
def main_loop():
    while True:
        rgb = agent.getImage()
        if rgb is not None and rgb.data is not None:
            # Detect object
            boxes, scores, labels = det_model.detect_object(
                rgbimage=rgb.data[:, :, :3], 
                text_prompt=obj
            )
            
            if boxes is not None and len(boxes) > 0:
                # Calculate movement based on detection
                mid_x, mid_y = get_box_center(boxes[0], [rgb.data.shape[1], rgb.data.shape[0]])
                if mid_x == -1 and mid_y == -1:
                    agent.moveByVelocity(Velocity(0, 0, 0), Velocity(0, 0, 0))
                    return
                linear_vel, angular_vel = get_velocity([mid_x, mid_y], [rgb.data.shape[1], rgb.data.shape[0]])
            else:
                # Search for object by rotating
                linear_vel = Velocity(0, 0, 0)
                angular_vel = Velocity(0, 0, 2)
                
            agent.moveByVelocity(linear_vel, angular_vel)
        time.sleep(.5)

​
Robot Setup

​
Simulated Robot (Isaac Quadruped)

For simulation environments, use the Isaac Quadruped configuration:

from grid.robot.locomotion.isaac_locomotion import IsaacLocomotion

agent = IsaacLocomotion()
agent.run()

​
Real Robot (Go2)

For physical robot deployment, use the Go2 configuration:

from grid.robot.locomotion.go2 import Go2Real
agent = Go2Real("wlp0s20f3")  # Replace with your network interface

​
Running the Example

  1. First, ensure you have GRID installed and set up properly.

  2. Choose your robot type:

SIM = False  # For real robot
# or
SIM = True   # For simulated robot
  1. Define your target object:
obj = "blue dustbin"  # For real robot (or any object in your environment)
# or
obj = "forklift"  # For simulated robot
  1. Run the main loop:
main_loop()

The robot will start searching for the specified object, and once detected, it will navigate towards it while maintaining a safe distance.

Remember to stop the robot if it gets too close to the target object or if you need to interrupt the operation.