How to Unlock 5 Vision Skills with the Moondream Cloud API

Most multimodal demos show one flashy result and then move on. A caption here, a detection there, maybe a segmentation mask if you are lucky. What they often do not show is whether the same model can stay coherent across very different tasks on the same image.

That is exactly what makes Moondream interesting.

Instead of treating visual understanding as one generic chat interface, Moondream exposes a small set of grounded skills: caption, query, detect, point, and segment. That design is much more useful in real applications because downstream code rarely wants prose alone. It wants descriptions, yes, but it also wants boxes, points, structured answers, and masks.

In this post, we will use a single image of three children in costume and walk through all five of those skills using the same prompts and outputs as the companion notebook, Moondream3.ipynb. The goal is not to make Moondream look magical. It is to show what the API is actually good at, where it is already practical, and where you still need a bit of cleanup around the result.

What You Will Learn

• How Moondream’s skill-based API differs from a generic vision chatbot.
• How to use caption, query, detect, point, and segment on the same image.
• Why a single image is enough to reveal whether a model’s understanding is consistent.
• Which results are immediately useful in products, and which ones still need light post-processing.
• How the companion notebook and the blog post line up section by section.

Why This Demo Works

This is a good LearnOpenCV-style demo for a simple reason: it uses one image, but asks the model to do five very different things with it.

That matters because a lot of vision models look impressive when you isolate the easiest capability. Captioning alone is not enough. Detection alone is not enough. Segmentation alone is not enough. A stronger test is whether the model can describe the image, answer a structured question about it, localize objects, identify finer target points, and then produce a mask for a specific subject without losing track of the scene.

That is what this walkthrough is built to show.

The prompts are intentionally practical rather than academic:

The Demo Image

Here is the image we use throughout the article and the notebook:

It is a deceptively useful test image.

There are multiple people, costume semantics, small accessories, visible gestures, and one clear subject we can isolate with segmentation. That makes it richer than a toy benchmark image while still being easy to interpret visually.

How to Get a Moondream API Key

Before you run the companion notebook or the API snippets in this post, you need a Moondream Cloud API key.

The current official flow is:

1. Open the Moondream Cloud Console.
2. Sign in with Google or GitHub.
3. Land on the API keys page and create a new key.
4. Copy the key and store it in an environment variable before running the notebook.

export MOONDREAM_API_KEY="your_api_key_here"

According to Moondream’s official quickstart, there is a free Cloud tier and you can get started without a credit card. That makes the cloud path a very easy way to try the model before deciding whether you want a more local or production-focused deployment later.

Minimal Cloud API Setup

The companion notebook uses the Moondream Cloud API with requests. The exact notebook has more display helpers, but the core interaction is this simple:

import os
import requests

API_BASE = "
HEADERS = {
    "Content-Type": "application/json",
    "X-Moondream-Auth": os.environ["MOONDREAM_API_KEY"],
}

def call_moondream(endpoint, payload):
    response = requests.post(
        f"{API_BASE}{endpoint}",
        headers=HEADERS,
        json=payload,
        timeout=120,
    )
    response.raise_for_status()
    return response.json()

In the notebook, the image is first downloaded from LearnOpenCV, then converted into a data URL so the same notebook works reliably without depending on the API server to fetch a third-party URL directly.

That detail is worth keeping. It makes the notebook much more reproducible.

Caption: Three Levels of Detail

Captioning is still the fastest way to get a feel for a vision model, but Moondream’s caption skill gets more interesting when you ask for different lengths instead of one generic answer.

for length in ["short", "normal", "long"]:
    result = call_moondream(
        "/caption",
        {
            "image_url": API_IMAGE_URL,
            "length": length,
            "stream": False,
        },
    )
    print(length, result["caption"])

Here are the actual outputs from the notebook run:

[SHORT]
Three children in superhero costumes stand in a room: one in a Spiderman suit,
another in a Superman costume, and a girl in a white fairy costume with a tiara.

[NORMAL]
Three children stand in a hallway wearing superhero costumes. On the left,
Spider-Man in a red and blue suit with black gloves and web pattern. In the
center, Superman in a blue and red suit with a yellow belt. On the right, a
fairy in a white dress and tiara with translucent wings. The background includes
a white door, a lamp on a dark table, and a light-colored wall.

The short caption is already usable as alt text or a compact scene summary. The normal caption adds more environment and costume detail without wandering too far. The long caption in the notebook goes even further, describing each child individually and expanding the background description.

That range is useful in practice:

• Short captions work well for indexing, previews, or accessibility summaries.
• Normal captions are a nice default for product descriptions or scene overviews.
• Long captions are better when you want denser narration for documentation or dataset enrichment.

This is one of Moondream’s strengths. It does not just “caption the image.” It gives you a controllable descriptive interface.

Query: Structured Scene Understanding

The query skill is where Moondream starts feeling less like a caption generator and more like a model you can wire into downstream logic.

Instead of asking a vague question like “What is in this image?”, the notebook uses a structured prompt:

query_prompt = (
    "Describe each child from left to right as JSON with keys "
    "costume_theme, standout_accessories, hand_pose, and smile."
)

query_result = call_moondream(
    "/query",
    {
        "image_url": API_IMAGE_URL,
        "question": query_prompt,
    },
)

Returned JSON
{
  "costume_theme": "Spiderman and Superman",
  "standout_accessories": [
    "Spiderman: Red and blue suit with web pattern",
    "Superman: Blue and red suit with 'S' logo",
    "Child: Superman: Blue and red suit with 'S' logo"
  ],
  "hand_pose": "Spiderman: Left hand raised, Superman: Right hand raised",
  "smile": "Child 1: Smiling, Child 2: Smiling, Child 3: Smiling"
}

This is a very instructive result because it is useful, but not perfect.

The good part is that the model follows the structured-output request and returns JSON-like content that is easy to parse. That is already better than asking for a prose description and trying to reverse-engineer structure afterward.

The limitation is that the answer compresses the scene too aggressively. It merges costume themes, duplicates Superman, and does not really preserve the left-to-right breakdown we asked for.

That is exactly the kind of honest result worth showing in a LearnOpenCV post. Moondream is clearly capable of structured first-pass extraction, but if your application needs strict per-person records, you should still validate or post-process the output. In other words, this is a strong automation primitive, not magic.

Detect: Boxes for People and Costume Details

The detect skill is where Moondream becomes much easier to use in real products. Instead of only describing what is present, it returns normalized bounding boxes for the object you ask for.

In the notebook, we deliberately mix one broad class with two smaller costume-related targets:

detect_people = call_moondream("/detect", {"image_url": API_IMAGE_URL, "object": "person"})
detect_wings = call_moondream("/detect", {"image_url": API_IMAGE_URL, "object": "fairy wings"})
detect_tiara = call_moondream("/detect", {"image_url": API_IMAGE_URL, "object": "tiara"})

Here is the exact figure from the companion notebook:

This result is strong for two reasons.

First, the person detections are clean and intuitive. The model finds all three children.

Second, the accessory detections are much more interesting than another generic person demo. Detecting fairy wings and tiara shows that the interface is open-vocabulary enough to support application-level concepts instead of only a narrow fixed class list.

That is exactly the kind of thing you want in workflows like:

• screenshot understanding
• shopping and catalog analysis
• themed photo organization
• costume and apparel tagging
• downstream cropping for second-stage models

Point: Fine-Grained Localization

Detection gives you a box. Pointing gives you a target location.

That makes point a better fit when you care about a specific part or feature rather than the full spatial extent of an object.

The notebook uses two prompts:

point_hands = call_moondream("/point", {"image_url": API_IMAGE_URL, "object": "raised hand"})
point_logo = call_moondream("/point", {"image_url": API_IMAGE_URL, "object": "superman logo"})

Here is the rendered notebook figure:

This is a nice example of where point and detect serve different roles:

• detect answers “where is the object footprint?”
• point answers “where should I target or mark this concept?”

The raised hand result is especially practical because it returns four points, one for each raised hand in the image. The superman logo result is even more interesting because it localizes a small, specific visual detail rather than a whole person.

This is useful for:

• UI automation targets
• gesture recognition pipelines
• symbolic or logo localization
• interactive overlays
• robotics pick or attention points

Segment: Turning a Box into a Mask

Segmentation is the most spatially precise capability in the set, but the important part of this demo is not just that a mask comes back. It is how the mask is requested.

The notebook first reuses the person detections, then picks the rightmost child, and only then calls segment:

rightmost_person = max(
    detect_people["objects"],
    key=lambda obj: (obj["x_min"] + obj["x_max"]) / 2,
)

segment_person = call_moondream(
    "/segment",
    {
        "image_url": API_IMAGE_URL,
        "object": "person",
        "spatial_refs": [[
            rightmost_person["x_min"],
            rightmost_person["y_min"],
            rightmost_person["x_max"],
            rightmost_person["y_max"],
        ]],
    },
)

That is an important pattern. In real systems, segmentation often works best as a second-stage refinement step after a broader localization step.

Here is the exact segmentation visualization from the notebook companion:

This is a very compelling result because it turns the pipeline into something operational:

1. detect all people
2. choose the subject of interest
3. refine that region into a mask

That workflow is useful for:

• subject cutouts
• photo editing
• selective effects
• region-specific reasoning
• grounding language references to one instance instead of many

It is also a better story than showing segmentation in isolation. The combination of detect plus segment is what makes the capability feel product-ready.

Why This API Design Matters

The real takeaway from this post is not just that Moondream can do five tasks. It is that the tasks are exposed in a way that maps naturally onto what engineers actually build.

A lot of multimodal systems force everything through one text interface. Moondream’s skill-based surface is more useful because it reduces the amount of interpretation your application needs to do after the model answers.

That is why the same image can support very different product behaviors:

• a captioning workflow for accessibility
• a structured query workflow for metadata
• a detection workflow for overlays
• a pointing workflow for interaction
• a segmentation workflow for editing or compositing

Companion Notebook

This article is designed to pair directly with Moondream3.ipynb.

The notebook uses the same image, the same prompts, and the same Moondream Cloud API skills shown in this post. That makes the relationship between the two very straightforward:

• the blog post explains why each demo matters
• the notebook shows how to run each step end to end

If you want to teach, present, or adapt these examples for another image, the notebook is the better starting point. If you want the higher-level product and vision-model perspective, this article is the better overview.

Together, they make a strong “read it, then run it” LearnOpenCV package.

Conclusion

Moondream is compelling not because it does one thing especially well, but because it gives you a compact set of grounded visual skills that work together.

On a single image, we used it to generate captions at different detail levels, produce structured scene output, localize broad objects and small accessories, mark fine target points, and segment one specific subject. That is a lot of practical surface area from one model and one API.

Just as importantly, the results are honest. Captioning and localization look strong. Segmentation is quite compelling when used as a refinement step. Structured querying is useful, but still benefits from downstream validation when you want strict schemas.

That is exactly the kind of tradeoff worth showing in a LearnOpenCV post. Not hype, not dismissal, just a clear view of what the model can already do and how to use it well.

To see these skills powering a real application, check out our guide on building an AI agent using a VLM.