> ## Documentation Index
> Fetch the complete documentation index at: https://docs.generalrobotics.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Chapter 3 — Detect objects while navigating

> Run a mission in the background while sampling images and GRID cloud detections.

Complete [Chapter 2](/navigation/planning) before this experiment.

A non-blocking mission keeps running on the robot while your Python program
does other work. Here you will sample a camera and ask GRID cloud to detect an
object. These detections are observations, not motion commands or proof that
an inspection passed.

[Download this guide as a notebook](/assets/notebooks/navigation/observation.ipynb)

## 1. Connect and localize

Start in a project folder with a [GRID development session](/get-started/first-skill):

```bash theme={null}
grid skill run --dev --robot YOUR_ROBOT
```

Keep that CLI process running. Use its workspace `.venv` Python environment
(or select that kernel if you use the downloadable notebook). Run the snippets
in order in one Python session. Set your registered robot name and use the numbered selectors.
The robot must have navigation enabled.

```python theme={null}
from grid_nexus_client import make_robot
from grid_nexus_client.nav import make_nav

ROBOT = "YOUR_ROBOT"
robot = make_robot(ROBOT)
nav = make_nav(robot)
state = nav.status()
print(state)
if state.mission_running:
    raise RuntimeError("A mission is already running; coordinate with its operator first")
```

Choose a completed map and a saved mission on it. Start and verify the robot's
location in the viewer before attempting the observation loop.

```python theme={null}
MAP_PAGE = 1
maps = [m for m in nav.list_maps(page=MAP_PAGE, limit=100) if m.status == "COMPLETED"]
if not maps:
    raise RuntimeError("No completed maps on this page. Check the account or MAP_PAGE.")
for number, item in enumerate(maps, 1):
    print(number, item.name, item.map_id)
choice = input("Map number: ").strip()
if not choice.isdecimal() or not 1 <= int(choice) <= len(maps):
    raise ValueError("Choose a listed map number.")
MAP_ID = maps[int(choice) - 1].map_id
```

Choose the saved mission you reviewed in the editor. Change `MISSION_PAGE` if needed.

```python theme={null}
MISSION_PAGE = 1
missions = nav.list_paths(MAP_ID, page=MISSION_PAGE, limit=100)
if not missions:
    raise RuntimeError("No saved missions on this page. Check MISSION_PAGE or save one in the editor.")
for number, item in enumerate(missions, 1):
    print(number, item.name, item.path_id)
choice = input("Reviewed mission number: ").strip()
if not choice.isdecimal() or not 1 <= int(choice) <= len(missions):
    raise ValueError("Choose a listed mission number.")
MISSION = missions[int(choice) - 1].path_id
```

```python theme={null}
nav.start(map_id=MAP_ID, timeout_s=90)
nav.view()
home = nav.localize(timeout_s=30)
print(home)
```

## 2. Set the observation task

Choose an RGB camera, such as `front_left` on Vision60 or `head_camera_left` on R1Pro.
Start with `OBJECT = "door"`. The client uses credentials from the CLI session.
Each run saves its images and records in a new folder.

```python theme={null}
import json
import math
import os
import time
from datetime import datetime, timezone
from pathlib import Path

import numpy as np
from IPython.display import display
from PIL import Image, ImageDraw

from grid_cortex_client import CortexClient
from grx.viewer import RemoteLiveSession

CAMERA = input("RGB camera name: ").strip()
OBJECT = "door"
MAX_RUN_S = 300
INTERVAL_S = 1
cortex = CortexClient(timeout=20)
url = os.environ.get("GRID_TELEMETRY_URL", "").strip()
telemetry = RemoteLiveSession(url) if url else None
output_dir = Path("observations-" + datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ"))
output_dir.mkdir()
print("Saved evidence:", output_dir.resolve())
```

<Accordion title="Support code — run once; reading is optional">
  This helper reads a camera image, calls the detector, and saves evidence.
  It also logs images and counts to the [Telemetry UI](/cli/nav-telemetry) when available.

  ```python theme={null}
  def observe_once(folder):
      stamp = datetime.now(timezone.utc)
      name = stamp.strftime("%Y%m%dT%H%M%S%fZ")
      record = {
          "time_utc": stamp.isoformat(),
          "map_id": MAP_ID,
          "camera": CAMERA,
          "object": OBJECT,
          "image": None,
          "boxes": [],
          "scores": [],
          "outcome": "error",
      }
      try:
          frame = robot.getImage(CAMERA)
          if frame is None:
              raise RuntimeError("Camera returned no image.")
          pixels = frame.decode()
          if pixels.dtype != np.uint8 or pixels.ndim != 3 or pixels.shape[2] != 3 or not pixels.size:
              raise ValueError("Select a nonempty uint8 RGB camera.")
          image = Image.fromarray(pixels)
          image.save(folder / (name + ".jpg"))
          record["image"] = name + ".jpg"
          if pixels.mean() <= 2 and pixels.std() <= 2:
              raise RuntimeError("Image is nearly black. Check the camera and lighting.")
          pose = nav.pose()
          record["robot_pose_near_read"] = {"x": pose.x, "y": pose.y, "yaw": pose.yaw}
          found = cortex.run("owlv2", image_input=image, prompt=OBJECT, box_threshold=0.25)
          if len(found["boxes"]) != len(found["scores"]):
              raise ValueError("Detector box and score counts differ.")
          for box, score in zip(found["boxes"], found["scores"]):
              if len(box) != 4 or not all(math.isfinite(float(v)) for v in [*box, score]):
                  raise ValueError("Detector returned invalid coordinates or scores.")
              x1, y1, x2, y2 = box
              bounds = [
                  max(0, math.floor(x1)),
                  max(0, math.floor(y1)),
                  min(image.width, math.ceil(x2)),
                  min(image.height, math.ceil(y2)),
              ]
              if bounds[2] > bounds[0] and bounds[3] > bounds[1]:
                  record["boxes"].append(bounds)
                  record["scores"].append(float(score))
          if telemetry is not None:
              telemetry.log("inspection/rgb", pixels, kind="image")
              telemetry.log("inspection/detection_count", np.array([len(record["boxes"])], dtype=np.float32))
          record["outcome"] = "detections" if record["boxes"] else "no_detections"
          return record
      except BaseException as error:
          record["error"] = f"{type(error).__name__}: {error}"
          raise
      finally:
          (folder / (name + ".json")).write_text(json.dumps(record, indent=2))


  ```
</Accordion>

## 3. Check one image while stationary

Confirm the full image shows the correct camera and surroundings. Zero detections
is valid. A dark, stale, blurred, or wrong-camera image is not useful evidence.
The helper checks for missing or nearly black images; you must check the rest.

```python theme={null}
preflight_ok = False
if nav.status().mission_running:
    raise RuntimeError("Finish the active mission before the stationary check.")
preview_dir = output_dir / "stationary-check"
preview_dir.mkdir(exist_ok=True)
preview = observe_once(preview_dir)
display(Image.open(preview_dir / preview["image"]))
print(preview["scores"])
preflight_ok = input("Type READY if the camera and object are correct: ") == "READY"
preflight_settings = (CAMERA, OBJECT)
```

## 4. Run the mission and detect objects

This cell moves the robot after RUN. It calls the camera and detector in sequence.
Slow requests reduce the sampling rate. `blocking=False` lets the notebook continue
while the robot executes the mission.

```python theme={null}
if not preflight_ok or preflight_settings != (CAMERA, OBJECT):
    raise RuntimeError("Approve a stationary check with these settings first.")
state = nav.status()
if state.state != "ready" or state.map_id != MAP_ID or state.mission_running:
    raise RuntimeError("Nav must be ready on this map with no active mission.")
if not all(math.isfinite(v) and v > 0 for v in (MAX_RUN_S, INTERVAL_S)):
    raise ValueError("Time limits must be positive and finite.")
if input("Run the reviewed mission and observe? Type RUN: ") != "RUN":
    raise RuntimeError("Mission not started")
mission_dir = output_dir / datetime.now(timezone.utc).strftime("mission-%Y%m%dT%H%M%S%fZ")
mission_dir.mkdir()
deadline = time.monotonic() + MAX_RUN_S
try:
    nav.run_mission(MISSION, blocking=False)
    while nav.status().mission_running:
        if time.monotonic() >= deadline:
            raise TimeoutError("Observation time limit reached.")
        record = observe_once(mission_dir)
        print(record["time_utc"], len(record["boxes"]), OBJECT)
        time.sleep(INTERVAL_S)
finally:
    nav.cancel_mission()
print(nav.result())
```

The `finally` block requests cancellation after completion, errors, or interruption,
including a failure during mission start. Confirm that the physical robot stopped.
Sensor, inference, and telemetry requests can delay cancellation. The time limit
is checked between samples; it is not a hard stop deadline. Use the e-stop if needed.

Inspect `nav.result()` and `nav.events()` after an error. A mission that ended did
not necessarily succeed. Review saved samples even if navigation failed.

## 5. Review a saved image

The folder contains full images, detector boxes and scores, errors, and robot poses
near each image read. Those poses are not object locations or synchronized capture poses.
Select a sample to see its full image and numbered detection boxes.

```python theme={null}
RESULTS_FOLDER = mission_dir  # Or Path("an-earlier-mission-folder").
records = sorted(RESULTS_FOLDER.glob("*.json"))
for number, file in enumerate(records):
    record = json.loads(file.read_text())
    print(number, record["time_utc"], record["outcome"], len(record["boxes"]))
if not records:
    raise RuntimeError("No samples saved in this folder.")
SAMPLE = int(input("Sample number to inspect: "))
if not 0 <= SAMPLE < len(records):
    raise ValueError("Choose a listed sample number.")
selected = json.loads(records[SAMPLE].read_text())
if not selected["image"]:
    raise RuntimeError(selected.get("error", "No image for this sample."))
image = Image.open(RESULTS_FOLDER / selected["image"])
display(image)  # Always inspect the full image.
annotated = image.copy()
draw = ImageDraw.Draw(annotated)
for number, box in enumerate(selected["boxes"]):
    draw.rectangle(box, outline="red", width=3)
    draw.text((box[0], box[1]), str(number), fill="red")
display(annotated)
print(selected.get("error", ""))
```

## 6. Ask about one detection

Select a box that actually contains the object. Ask a question about the crop,
then compare the answer with the full image. The answer is saved beside its sample reference.
Change the question if you change the object. A crop can hide context or a false detection.

```python theme={null}
if not selected["boxes"]:
    raise RuntimeError("No detections in this sample. Choose another image.")
BOX = int(input("Box number: "))
if not 0 <= BOX < len(selected["boxes"]):
    raise ValueError("Choose a numbered box.")
QUESTION = "Is this door open, closed, or uncertain? Explain the visible evidence."
crop = image.crop(tuple(selected["boxes"][BOX]))
display(crop)
answer = cortex.run("qwen_vl", image_input=crop, prompt=QUESTION)
print(answer["output"])
answer_file = RESULTS_FOLDER / ("answer-" + datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") + ".txt")
answer_file.write_text(
    json.dumps(
        {
            "sample": records[SAMPLE].name,
            "box": selected["boxes"][BOX],
            "question": QUESTION,
            "answer": answer["output"],
        },
        indent=2,
    )
)
```

Repeated sightings are not unique objects. No detections does not prove absence.
A model answer does not verify an inspection and never changes the robot's route.
The same camera and model calls work when the robot is stationary.

## 7. Finish

Run cleanup even after an error. Collect results first. If stopping cannot be
confirmed, use the physical stop. Review camera images before sharing the folder.

```python theme={null}
nav.shutdown()
robot.shutdown()
```

```python theme={null}
cortex.close()
```


## Related topics

- [Navigation through GRID CLI](/navigation/overview.md)
- [Chapter 2 — Plan your own routes](/navigation/planning.md)
- [Chapter 1 — Run your first mission](/navigation/first-mission.md)
- [Flexiv Rizon Arm](/python-api/flexiv-rizon/flexivrizon.md)
- [Hand](/python-api/robot-interface/hand.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.