> ## Documentation Index
> Fetch the complete documentation index at: https://docs.generalrobotics.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# sam3
> Segment an image with SAM-3 using exactly one prompt type
Segment an image with SAM-3 using exactly one prompt type.
## Parameters
RGB image as file path, URL, PIL Image, or numpy array.
Text prompt describing objects to segment (exclusive with points/boxes).
List of \[x, y] point coordinates (exclusive with text/boxes).
List of \[x0, y0, x1, y1] box coordinates (exclusive with text/points).
Required for points/boxes. 1=foreground, 0=background.
Optional HTTP timeout in seconds.
## Returns
np.ndarray: Binary segmentation mask as numpy array (H, W) with dtype uint8.
Foreground pixels are 255, background pixels are 0.
## Example
```python theme={null}
from grid_cortex_client import CortexClient, ModelType
import numpy as np
from PIL import Image
client = CortexClient()
image = np.array(Image.open("cat.jpg"))
# Text prompt
mask = client.run(ModelType.SAM3, image_input=image, text="cat")
# Point prompts
mask = client.run(
ModelType.SAM3, image_input=image,
points=[[320, 240]], labels=[1]
)
# Box prompts
mask = client.run(
ModelType.SAM3, image_input=image,
boxes=[[100, 120, 520, 480]], labels=[1]
)
```
SAM3 offers two client methods:
* **`run()`** — Returns a single union mask. Use when you only need the combined segmentation.
* **`run_with_detections()`** — Returns the union mask plus per-instance masks, bounding boxes, and confidence scores. Use when you need individual object information (e.g. counting objects, filtering by confidence, or processing instances separately). Only available for **text** and **box** prompts — point prompts have no detection semantics.
## `run_with_detections()`
Same call signature as `run()`. Returns a `dict` with keys:
* `union_mask` — Combined binary mask `(H, W)`, dtype `uint8` (same as `run()` output)
* `masks` — List of per-instance binary masks, each `(H, W)` dtype `uint8`
* `boxes` — List of `[x0, y0, x1, y1]` bounding boxes (text/box prompts only)
* `scores` — List of confidence scores (sorted descending)
For point prompts, `masks`, `boxes`, and `scores` will be empty lists.
## Example Output — `run()`
Use `run()` when you only need the combined mask — e.g. masking a region, computing area, or passing to a downstream model.
```python theme={null}
from grid_cortex_client import CortexClient
import numpy as np
from PIL import Image
client = CortexClient()
img = Image.open("cats.jpg") # 640x480 RGB
# Text prompt — returns a single union mask
mask = client.run(model_id="sam3", image_input=img, text="cat")
print(mask.shape, mask.dtype)
# (480, 640) uint8
fg = np.count_nonzero(mask)
print(f"foreground: {fg} pixels ({fg / mask.size * 100:.1f}%)")
# foreground: 107198 pixels (34.9%)
# Points prompt — click on center of image
points_mask = client.run(
model_id="sam3",
image_input=img,
points=[[320, 240]],
labels=[1],
)
fg = np.count_nonzero(points_mask)
print(f"foreground: {fg} pixels ({fg / points_mask.size * 100:.1f}%)")
# foreground: 137017 pixels (44.6%)
# Boxes prompt — box around the right cat
boxes_mask = client.run(
model_id="sam3",
image_input=img,
boxes=[[347, 26, 639, 369]],
labels=[1],
)
fg = np.count_nonzero(boxes_mask)
print(f"foreground: {fg} pixels ({fg / boxes_mask.size * 100:.1f}%)")
# foreground: 107299 pixels (34.9%)
```
## Example Output — `run_with_detections()`
Use `run_with_detections()` when you need per-instance information — e.g. counting objects, filtering by confidence, or processing each instance separately. Works with **text** and **box** prompts only.
```python theme={null}
from grid_cortex_client import CortexClient
import numpy as np
from PIL import Image
client = CortexClient()
img = Image.open("cats.jpg") # 640x480 RGB
# Text prompt with per-instance detections
result = client.run_with_detections(model_id="sam3", image_input=img, text="cat")
print(result.keys())
# dict_keys(['union_mask', 'boxes', 'scores', 'masks'])
print(result["union_mask"].shape, result["union_mask"].dtype)
# (480, 640) uint8
print(f"instances found: {len(result['masks'])}")
# instances found: 2
for i, (mask, box, score) in enumerate(zip(result["masks"], result["boxes"], result["scores"])):
fg = np.count_nonzero(mask)
print(f" instance {i}: score={score:.4f}, box={box}, pixels={fg}")
# instance 0: score=0.9327, box=[14.58, 55.12, 315.22, 473.21], pixels=50090
# instance 1: score=0.9252, box=[347.68, 26.83, 638.91, 368.76], pixels=57108
# Filter by confidence
high_conf = [s for s in result["scores"] if s > 0.9]
print(f"high-confidence instances: {len(high_conf)}")
# high-confidence instances: 2
# Boxes prompt with per-instance detections
result = client.run_with_detections(
model_id="sam3",
image_input=img,
boxes=[[14, 55, 315, 473], [347, 26, 639, 369]],
labels=[1, 1],
)
print(f"instances: {len(result['masks'])}")
# instances: 2
for i, (mask, box, score) in enumerate(zip(result["masks"], result["boxes"], result["scores"])):
fg = np.count_nonzero(mask)
print(f" instance {i}: score={score:.4f}, box={box}, pixels={fg}")
# instance 0: score=0.9905, box=[347.76, 26.36, 638.77, 370.08], pixels=57268
# instance 1: score=0.9903, box=[15.29, 55.16, 314.74, 472.38], pixels=49950
```
## Related topics
- [sam3](/models/cortex/sam3.md)
- [CortexClient](/python-api/grid-cortex-client/cortexclient.md)