> ## Documentation Index
> Fetch the complete documentation index at: https://docs.generalrobotics.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# SAM3
> Segment Anything 3 — text, point, or box prompted segmentation
Segment objects using exactly one prompt type per call: text, points, or boxes.
SAM3 offers two client methods:
* **`run()`** — Returns a single union mask. Use when you only need the combined segmentation.
* **`run_with_detections()`** — Returns the union mask plus per-instance masks, bounding boxes, and confidence scores. Use when you need individual object information (e.g. counting objects, filtering by confidence, or processing instances separately). Only available for **text** and **box** prompts — point prompts have no detection semantics.
## Parameters
RGB image as file path, URL, PIL Image, or numpy array.
Text prompt describing objects to segment. Exclusive with `points`/`boxes`. Also accepted as `text_prompt`.
List of `[x, y]` point coordinates. Exclusive with `text`/`boxes`. Requires `labels`. Also accepted as `prompts`.
List of `[x0, y0, x1, y1]` box coordinates. Exclusive with `text`/`points`. Requires `labels`.
Required for `points`/`boxes`. `1` = foreground, `0` = background.
Optional HTTP timeout.
## Returns — `run()`
`np.ndarray` — Binary union mask of shape `(H, W)` with dtype `uint8`. All matched instances are OR'd into a single mask. Foreground `255`, background `0`.
## Returns — `run_with_detections()`
`dict` with keys:
* `union_mask` — Combined binary mask `(H, W)`, dtype `uint8` (same as `run()` output)
* `masks` — List of per-instance binary masks, each `(H, W)` dtype `uint8`
* `boxes` — List of `[x0, y0, x1, y1]` bounding boxes (text/box prompts only)
* `scores` — List of confidence scores (sorted descending)
For point prompts, `masks`, `boxes`, and `scores` will be empty lists.
## Example Output — `run()`
## Example — `run()`
Use `run()` when you only need the combined mask — e.g. masking a region, computing area, or passing to a downstream model.
```python theme={null}
from grid_cortex_client import CortexClient
import numpy as np
from PIL import Image
client = CortexClient()
img = Image.open("cats.jpg") # 640x480 RGB
# Text prompt — returns a single union mask
mask = client.run(model_id="sam3", image_input=img, text="cat")
print(mask.shape, mask.dtype)
# (480, 640) uint8
fg = np.count_nonzero(mask)
print(f"foreground: {fg} pixels ({fg / mask.size * 100:.1f}%)")
# foreground: 107198 pixels (34.9%)
# Points prompt — click on center of image
points_mask = client.run(
model_id="sam3",
image_input=img,
points=[[320, 240]],
labels=[1],
)
fg = np.count_nonzero(points_mask)
print(f"foreground: {fg} pixels ({fg / points_mask.size * 100:.1f}%)")
# foreground: 137017 pixels (44.6%)
# Boxes prompt — box around the right cat
boxes_mask = client.run(
model_id="sam3",
image_input=img,
boxes=[[347, 26, 639, 369]],
labels=[1],
)
fg = np.count_nonzero(boxes_mask)
print(f"foreground: {fg} pixels ({fg / boxes_mask.size * 100:.1f}%)")
# foreground: 107299 pixels (34.9%)
```
## Example Output — `run_with_detections()`
## Example — `run_with_detections()`
Use `run_with_detections()` when you need per-instance information — e.g. counting objects, filtering by confidence, or processing each instance separately. Works with **text** and **box** prompts only.
```python theme={null}
from grid_cortex_client import CortexClient
import numpy as np
from PIL import Image
client = CortexClient()
img = Image.open("cats.jpg") # 640x480 RGB
# Text prompt with per-instance detections
result = client.run_with_detections(model_id="sam3", image_input=img, text="cat")
print(result.keys())
# dict_keys(['union_mask', 'boxes', 'scores', 'masks'])
print(result["union_mask"].shape, result["union_mask"].dtype)
# (480, 640) uint8
print(f"instances found: {len(result['masks'])}")
# instances found: 2
for i, (mask, box, score) in enumerate(zip(result["masks"], result["boxes"], result["scores"])):
fg = np.count_nonzero(mask)
print(f" instance {i}: score={score:.4f}, box={box}, pixels={fg}")
# instance 0: score=0.9327, box=[14.58, 55.12, 315.22, 473.21], pixels=50090
# instance 1: score=0.9252, box=[347.68, 26.83, 638.91, 368.76], pixels=57108
# Filter by confidence
high_conf = [s for s in result["scores"] if s > 0.9]
print(f"high-confidence instances: {len(high_conf)}")
# high-confidence instances: 2
# Boxes prompt with per-instance detections
result = client.run_with_detections(
model_id="sam3",
image_input=img,
boxes=[[14, 55, 315, 473], [347, 26, 639, 369]],
labels=[1, 1],
)
print(f"instances: {len(result['masks'])}")
# instances: 2
for i, (mask, box, score) in enumerate(zip(result["masks"], result["boxes"], result["scores"])):
fg = np.count_nonzero(mask)
print(f" instance {i}: score={score:.4f}, box={box}, pixels={fg}")
# instance 0: score=0.9905, box=[347.76, 26.36, 638.77, 370.08], pixels=57268
# instance 1: score=0.9903, box=[15.29, 55.16, 314.74, 472.38], pixels=49950
```