GRID Docs | General Robotics

Complete Chapter 2 before this experiment. A non-blocking mission keeps running on the robot while your Python program does other work. Here you will sample a camera and ask GRID cloud to detect an object. These detections are observations, not motion commands or proof that an inspection passed. Download this guide as a notebook

1. Connect and localize

Start in a project folder with a GRID development session:

grid skill run --dev --robot YOUR_ROBOT

Keep that CLI process running. Use its workspace .venv Python environment (or select that kernel if you use the downloadable notebook). Run the snippets in order in one Python session. Set your registered robot name and use the numbered selectors. The robot must have navigation enabled.

from grid_nexus_client import make_robot
from grid_nexus_client.nav import make_nav

ROBOT = "YOUR_ROBOT"
robot = make_robot(ROBOT)
nav = make_nav(robot)
state = nav.status()
print(state)
if state.mission_running:
    raise RuntimeError("A mission is already running; coordinate with its operator first")

Choose a completed map and a saved mission on it. Start and verify the robot’s location in the viewer before attempting the observation loop.

MAP_PAGE = 1
maps = [m for m in nav.list_maps(page=MAP_PAGE, limit=100) if m.status == "COMPLETED"]
if not maps:
    raise RuntimeError("No completed maps on this page. Check the account or MAP_PAGE.")
for number, item in enumerate(maps, 1):
    print(number, item.name, item.map_id)
choice = input("Map number: ").strip()
if not choice.isdecimal() or not 1 <= int(choice) <= len(maps):
    raise ValueError("Choose a listed map number.")
MAP_ID = maps[int(choice) - 1].map_id

Choose the saved mission you reviewed in the editor. Change MISSION_PAGE if needed.

MISSION_PAGE = 1
missions = nav.list_paths(MAP_ID, page=MISSION_PAGE, limit=100)
if not missions:
    raise RuntimeError("No saved missions on this page. Check MISSION_PAGE or save one in the editor.")
for number, item in enumerate(missions, 1):
    print(number, item.name, item.path_id)
choice = input("Reviewed mission number: ").strip()
if not choice.isdecimal() or not 1 <= int(choice) <= len(missions):
    raise ValueError("Choose a listed mission number.")
MISSION = missions[int(choice) - 1].path_id

nav.start(map_id=MAP_ID, timeout_s=90)
nav.view()
home = nav.localize(timeout_s=30)
print(home)

2. Set the observation task

Choose an RGB camera, such as front_left on Vision60 or head_camera_left on R1Pro. Start with OBJECT = "door". The client uses credentials from the CLI session. Each run saves its images and records in a new folder.

import json
import math
import os
import time
from datetime import datetime, timezone
from pathlib import Path

import numpy as np
from IPython.display import display
from PIL import Image, ImageDraw

from grid_cortex_client import CortexClient
from grx.viewer import RemoteLiveSession

CAMERA = input("RGB camera name: ").strip()
OBJECT = "door"
MAX_RUN_S = 300
INTERVAL_S = 1
cortex = CortexClient(timeout=20)
url = os.environ.get("GRID_TELEMETRY_URL", "").strip()
telemetry = RemoteLiveSession(url) if url else None
output_dir = Path("observations-" + datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ"))
output_dir.mkdir()
print("Saved evidence:", output_dir.resolve())

Support code — run once; reading is optional

This helper reads a camera image, calls the detector, and saves evidence. It also logs images and counts to the Telemetry UI when available.

def observe_once(folder):
    stamp = datetime.now(timezone.utc)
    name = stamp.strftime("%Y%m%dT%H%M%S%fZ")
    record = {
        "time_utc": stamp.isoformat(),
        "map_id": MAP_ID,
        "camera": CAMERA,
        "object": OBJECT,
        "image": None,
        "boxes": [],
        "scores": [],
        "outcome": "error",
    }
    try:
        frame = robot.getImage(CAMERA)
        if frame is None:
            raise RuntimeError("Camera returned no image.")
        pixels = frame.decode()
        if pixels.dtype != np.uint8 or pixels.ndim != 3 or pixels.shape[2] != 3 or not pixels.size:
            raise ValueError("Select a nonempty uint8 RGB camera.")
        image = Image.fromarray(pixels)
        image.save(folder / (name + ".jpg"))
        record["image"] = name + ".jpg"
        if pixels.mean() <= 2 and pixels.std() <= 2:
            raise RuntimeError("Image is nearly black. Check the camera and lighting.")
        pose = nav.pose()
        record["robot_pose_near_read"] = {"x": pose.x, "y": pose.y, "yaw": pose.yaw}
        found = cortex.run("owlv2", image_input=image, prompt=OBJECT, box_threshold=0.25)
        if len(found["boxes"]) != len(found["scores"]):
            raise ValueError("Detector box and score counts differ.")
        for box, score in zip(found["boxes"], found["scores"]):
            if len(box) != 4 or not all(math.isfinite(float(v)) for v in [*box, score]):
                raise ValueError("Detector returned invalid coordinates or scores.")
            x1, y1, x2, y2 = box
            bounds = [
                max(0, math.floor(x1)),
                max(0, math.floor(y1)),
                min(image.width, math.ceil(x2)),
                min(image.height, math.ceil(y2)),
            ]
            if bounds[2] > bounds[0] and bounds[3] > bounds[1]:
                record["boxes"].append(bounds)
                record["scores"].append(float(score))
        if telemetry is not None:
            telemetry.log("inspection/rgb", pixels, kind="image")
            telemetry.log("inspection/detection_count", np.array([len(record["boxes"])], dtype=np.float32))
        record["outcome"] = "detections" if record["boxes"] else "no_detections"
        return record
    except BaseException as error:
        record["error"] = f"{type(error).__name__}: {error}"
        raise
    finally:
        (folder / (name + ".json")).write_text(json.dumps(record, indent=2))

3. Check one image while stationary

Confirm the full image shows the correct camera and surroundings. Zero detections is valid. A dark, stale, blurred, or wrong-camera image is not useful evidence. The helper checks for missing or nearly black images; you must check the rest.

preflight_ok = False
if nav.status().mission_running:
    raise RuntimeError("Finish the active mission before the stationary check.")
preview_dir = output_dir / "stationary-check"
preview_dir.mkdir(exist_ok=True)
preview = observe_once(preview_dir)
display(Image.open(preview_dir / preview["image"]))
print(preview["scores"])
preflight_ok = input("Type READY if the camera and object are correct: ") == "READY"
preflight_settings = (CAMERA, OBJECT)

4. Run the mission and detect objects

This cell moves the robot after RUN. It calls the camera and detector in sequence. Slow requests reduce the sampling rate. blocking=False lets the notebook continue while the robot executes the mission.

if not preflight_ok or preflight_settings != (CAMERA, OBJECT):
    raise RuntimeError("Approve a stationary check with these settings first.")
state = nav.status()
if state.state != "ready" or state.map_id != MAP_ID or state.mission_running:
    raise RuntimeError("Nav must be ready on this map with no active mission.")
if not all(math.isfinite(v) and v > 0 for v in (MAX_RUN_S, INTERVAL_S)):
    raise ValueError("Time limits must be positive and finite.")
if input("Run the reviewed mission and observe? Type RUN: ") != "RUN":
    raise RuntimeError("Mission not started")
mission_dir = output_dir / datetime.now(timezone.utc).strftime("mission-%Y%m%dT%H%M%S%fZ")
mission_dir.mkdir()
deadline = time.monotonic() + MAX_RUN_S
try:
    nav.run_mission(MISSION, blocking=False)
    while nav.status().mission_running:
        if time.monotonic() >= deadline:
            raise TimeoutError("Observation time limit reached.")
        record = observe_once(mission_dir)
        print(record["time_utc"], len(record["boxes"]), OBJECT)
        time.sleep(INTERVAL_S)
finally:
    nav.cancel_mission()
print(nav.result())

The finally block requests cancellation after completion, errors, or interruption, including a failure during mission start. Confirm that the physical robot stopped. Sensor, inference, and telemetry requests can delay cancellation. The time limit is checked between samples; it is not a hard stop deadline. Use the e-stop if needed. Inspect nav.result() and nav.events() after an error. A mission that ended did not necessarily succeed. Review saved samples even if navigation failed.

5. Review a saved image

The folder contains full images, detector boxes and scores, errors, and robot poses near each image read. Those poses are not object locations or synchronized capture poses. Select a sample to see its full image and numbered detection boxes.

RESULTS_FOLDER = mission_dir  # Or Path("an-earlier-mission-folder").
records = sorted(RESULTS_FOLDER.glob("*.json"))
for number, file in enumerate(records):
    record = json.loads(file.read_text())
    print(number, record["time_utc"], record["outcome"], len(record["boxes"]))
if not records:
    raise RuntimeError("No samples saved in this folder.")
SAMPLE = int(input("Sample number to inspect: "))
if not 0 <= SAMPLE < len(records):
    raise ValueError("Choose a listed sample number.")
selected = json.loads(records[SAMPLE].read_text())
if not selected["image"]:
    raise RuntimeError(selected.get("error", "No image for this sample."))
image = Image.open(RESULTS_FOLDER / selected["image"])
display(image)  # Always inspect the full image.
annotated = image.copy()
draw = ImageDraw.Draw(annotated)
for number, box in enumerate(selected["boxes"]):
    draw.rectangle(box, outline="red", width=3)
    draw.text((box[0], box[1]), str(number), fill="red")
display(annotated)
print(selected.get("error", ""))

6. Ask about one detection

Select a box that actually contains the object. Ask a question about the crop, then compare the answer with the full image. The answer is saved beside its sample reference. Change the question if you change the object. A crop can hide context or a false detection.

if not selected["boxes"]:
    raise RuntimeError("No detections in this sample. Choose another image.")
BOX = int(input("Box number: "))
if not 0 <= BOX < len(selected["boxes"]):
    raise ValueError("Choose a numbered box.")
QUESTION = "Is this door open, closed, or uncertain? Explain the visible evidence."
crop = image.crop(tuple(selected["boxes"][BOX]))
display(crop)
answer = cortex.run("qwen_vl", image_input=crop, prompt=QUESTION)
print(answer["output"])
answer_file = RESULTS_FOLDER / ("answer-" + datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") + ".txt")
answer_file.write_text(
    json.dumps(
        {
            "sample": records[SAMPLE].name,
            "box": selected["boxes"][BOX],
            "question": QUESTION,
            "answer": answer["output"],
        },
        indent=2,
    )
)

Repeated sightings are not unique objects. No detections does not prove absence. A model answer does not verify an inspection and never changes the robot’s route. The same camera and model calls work when the robot is stationary.

7. Finish

Run cleanup even after an error. Collect results first. If stopping cannot be confirmed, use the physical stop. Review camera images before sharing the folder.

nav.shutdown()
robot.shutdown()

cortex.close()

Get Started

Simulation

Navigation

Deployment

FAQ

Support

Chapter 3 — Detect objects while navigating

1. Connect and localize

2. Set the observation task

3. Check one image while stationary

4. Run the mission and detect objects

5. Review a saved image

6. Ask about one detection

7. Finish

​1. Connect and localize

​2. Set the observation task

​3. Check one image while stationary

​4. Run the mission and detect objects

​5. Review a saved image

​6. Ask about one detection

​7. Finish

1. Connect and localize

2. Set the observation task

3. Check one image while stationary

4. Run the mission and detect objects

5. Review a saved image

6. Ask about one detection

7. Finish