Overview

Scheduled threshold checks over your telemetry, your warehouse SQL, or a model’s records, firing actions when a metric breaches. Define them with defineAlert.

An alert is a scheduled check over your workspace. On each tick it computes one metric over a slice of your data, compares it to a threshold, and, when the threshold is breached, fires its actions, each as its own run. You define one with defineAlert.

What it measures is up to the scope: span telemetry (every tool run, play batch and agent message emits spans), your play and tool runs end to end, a raw SQL query against the orchestration store or your data warehouse, or the records of a model.

Define an alert

alerts/error-spike.ts

import { defineAlert } from "@cargo-ai/cdk";

import { sentinel } from "../agents/sentinel";
import { slack } from "../connectors/slack";
import { enrich } from "../tools/enrich";

export const errorSpike = defineAlert("error-spike", {
  description: "Error rate of the enrich workflow",
  schedule: { type: "cron", cron: "@every 5m" }, // when the check runs
  scope: { kind: "spans", workflow: enrich }, // which spans it watches
  threshold: { metric: "errorRate", operator: "gte", value: 10 }, // when it breaches
  actions: [
    {
      ref: slack.actions.postMessage,
      config: {
        channelId: "C0123456789",
        format: "markdown",
        body: "Enrich error rate hit {{event.value}}% — {{event.spansUrl}}",
      },
    },
    { ref: sentinel, config: { prompt: "Investigate {{alert.url}}" } },
  ],
});

schedule takes a 5-field cron expression or an @every interval (@every 5m, @every 1h30m), evaluated in UTC. An alert evaluates at most once a minute — every tick scans your spans, so sub-minute intervals are rejected. In the CDK enabled defaults to true, so a deployed alert is armed; set it to false to deploy one without arming it. folder files the alert under an "alert"-kind folder.

Scope: what it watches

The scope’s kind names the data source. Each source pairs with its own metric menu, so a metric can never be asked of a source that can’t compute it.

Spans scope

kind: "spans" watches span telemetry directly. Use any subset of these filters; omitted fields don’t narrow. They mirror the Spans view.

Field	Narrows to
workflow	one play or tool’s workflow: a handle, or workflowRef(uuid)
parentAgent	the spans of runs one agent spawned
nodeKind	native, connector, tool, or agent nodes
integration	one integration slug
connector	one connector
action	one action slug
tool	one tool node
agent	one agent node
executionTitleOrErrorMessage	spans whose title or error message contains the text (case-insensitive)
executionStatuses	pending, success, and/or error
userUuid	runs started by one user

parentAgent and agent are different filters. parentAgent matches spans of runs an agent started — the agent-trigger case. agent matches an agent node running inside the watched spans.

workflow takes a play/tool handle or workflowRef(uuid) — not toolRef or agentRef. Spans are keyed by the workflow behind a tool, not by the tool’s own uuid, so passing the wrong kind of reference is rejected at deploy time rather than producing an alert that silently matches nothing.

Runs scope

kind: "runs" watches your play and tool runs end to end, rather than the individual node executions a spans scope sees. A run is one record travelling through a workflow, so count counts records processed and duration measures a record’s whole journey, not one node’s.

export const stalledEnrichment = defineAlert("stalled-enrichment", {
  schedule: { type: "cron", cron: "@every 15m" },
  scope: { kind: "runs", workflow: enrich, statuses: ["error"] },
  threshold: { metric: "count", operator: "gte", value: 25 },
  actions: [{ ref: sentinel, config: { prompt: "Investigate {{alert.url}}" } }],
});

Field	Narrows to
workflow	one play or tool’s workflow: a handle, or workflowRef(uuid). Omitted watches every play and tool
statuses	runs in these statuses (idle, pending, running, success, error, cancelling, cancelled, skipped)
recordTitleOrErrorMessage	runs whose record title or error message contains the text (case-insensitive)
releaseUuid	runs of one release
userUuid	runs started by one user

Ad-hoc runs, the ones an agent spawns outside a play or a tool, are never counted, whatever the filters. They live on the workspace’s default workflow, which this source excludes, so an agent’s own activity can’t drown out the workflows you are watching. Use a spans scope with parentAgent to alert on an agent instead.

Records scope

kind: "records" watches the same play and tool work as a runs scope, counted once per record instead of once per attempt. Each record is held in its latest state, so a record that failed and was later re-run into success stops counting as failed, and idle or skipped runs never appear at all. Reach for it when you care about how your records are doing rather than how your executions are doing.

export const unresolvedRecords = defineAlert("unresolved-records", {
  schedule: { type: "cron", cron: "@every 1h" },
  scope: { kind: "records", workflow: enrich },
  threshold: { metric: "errorRate", operator: "gte", value: 5 },
  actions: [{ ref: sentinel, config: { prompt: "Records still failing" } }],
});

Field	Narrows to
workflow	one play or tool’s workflow: a handle, or workflowRef(uuid). Omitted watches every play and tool
statuses	records in these statuses (pending, running, success, error, cancelling, cancelled)
titleOrErrorMessage	records whose title or error message contains the text (case-insensitive)
releaseUuid	records of one release
userUuid	records started by one user

Retries are the difference between the two sources. Three failed attempts at one record are three failed runs but one failed record, and if a fourth attempt succeeds, the record counts as a success while the three failed runs remain. Alert on runs to catch a flaky workflow; alert on records to catch work that never got done. Ad-hoc runs are excluded here too.

SQL scopes

Two sources run raw SQL and use its result as the value. The query computes the value itself, so the threshold carries only the comparison:

• kind: "orchestrationQuery" runs read-only ClickHouse SQL over your orchestration data: the spans, runs, batches, and records tables, automatically scoped to your workspace.
• kind: "storageQuery" runs read-only SQL over your data warehouse, referencing models as <dataset>.<model> exactly as the model SQL editor does.

export const slowNights = defineAlert("slow-nights", {
  schedule: { type: "cron", cron: "0 * * * *" },
  scope: {
    kind: "orchestrationQuery",
    query: `select count(*) from spans
            where execution_status = 'error'
              and execution_started_at > now() - interval 1 hour`,
  },
  threshold: { operator: "gte", value: 50 },
  actions: [
    { ref: sentinel, config: { prompt: "Error spike: {{event.value}}" } },
  ],
});

A warehouse query looks the same, against your own tables:

export const missingEmails = defineAlert("missing-emails", {
  schedule: { type: "cron", cron: "0 * * * *" },
  scope: {
    kind: "storageQuery",
    query: `select 100.0 * sum(case when email is null then 1 else 0 end)
                 / nullif(count(*), 0)
            from crm.contacts`,
  },
  threshold: { operator: "gte", value: 20 },
  actions: [{ ref: sentinel, config: { prompt: "Contact data decayed" } }],
});

Query scopes are not windowed for you: Cargo runs the query exactly as written. An orchestration query should be self-windowing (as the interval 1 hour above does), otherwise every tick evaluates your whole history. A warehouse query sees your models as they stand, which is usually what you want: it is a statement about the data, not about a time slice.

The value is the first column of the first row, and it must be numeric. A non-number produces an error event. A query that returns no rows or a NULL counts as an empty window instead: the tick is recorded as healthy with value: null and nothing fires. That is what an aggregate over no rows gives you, and what a rate query returns for 0 / 0 in an idle window, so an empty window can never quietly read as a real 0 and breach an lte threshold.

See Querying orchestration data for the columns of each orchestration table, the ClickHouse idioms that differ from PostgreSQL, and the limits a query runs under; see Querying models for the warehouse side.

A warehouse query never waits for a model to refresh. An alert runs on a cron, and blocking a tick on a sync would leave it measuring a moment that has passed by the time the sync lands.

Model scope

kind: "model" watches the records of one model, optionally narrowed by a filter, the same filter a segment is built from. Unlike every other source, a model is measured as it stands rather than over the evaluation window:

export const decayingContacts = defineAlert("decaying-contacts", {
  schedule: { type: "cron", cron: "0 9 * * *" },
  scope: {
    kind: "model",
    model: contacts,
    filter: {
      conjonction: "and",
      groups: [
        {
          conjonction: "and",
          conditions: [{ kind: "string", columnSlug: "email", operator: "isNull" }],
        },
      ],
    },
  },
  threshold: { metric: "recordsShare", operator: "gte", value: 30 },
  actions: [{ ref: sentinel, config: { prompt: "{{event.value}}% missing" } }],
});

Threshold: when it breaches

operator is gte (breach at or above value) or lte (breach at or below). Apart from the SQL scopes, where the query is the metric, metric says what is measured. It is required, since it decides what the value means, and a metric that supports aggregations requires one too.

Spans, runs and records share a metric menu; the scope decides what is being counted, one node execution, one record’s whole journey, or one record:

metric	Value	aggregation
errorRate	failed as a percentage (0–100) of the window’s finished rows	none
duration	duration in seconds, over finished rows only	avg, p50, p95, p99
credits	credits consumed by the window’s rows	sum, avg, p95
count	number of spans / runs / records in the window	none

count with lte is a dead-man’s switch: an empty window really evaluates to 0, so silence breaches. { metric: "count", operator: "lte", value: 0 } tells you a workflow stopped running at all. The other metrics treat an empty window as nothing to judge, not as zero — and for errorRate and duration, “empty” means no finished rows, so a window of runs that are all still going is not judged yet either.

A model has its own four, about the records themselves and about how fresh they are:

metric	Value
recordsCount	records matching the scope
recordsShare	matching records as a percentage (0–100) of the model’s total
freshness	minutes since the model last emitted records
syncDuration	duration in seconds of the model’s last finished sync

recordsShare requires the scope’s filter: a share of everything is always 100, so the pair is rejected at save time rather than firing on a constant.

Actions: what fires on breach

Each action becomes its own run, exactly like a play’s healthAlertActions. An action is a connector action, an agent, or a tool, and its config is the input it runs with:

actions: [
  { ref: slack.actions.postMessage, config: { channelId: "C0…", body: "…" } },
  { ref: sentinel, config: { prompt: "…" }, waitUntilFinished: true },
  { ref: enrich, config: {} },
]

An agent’s config is typed already — every agent takes the same { prompt, output? }, so a misspelled key is an editor error in the literal above with nothing to import.

A connector action or a tool has an input of its own, and a bare object literal leaves it unchecked. Wrap it in alertConnectorAction / alertToolAction to have TypeScript check config against the real thing:

import { alertConnectorAction, alertToolAction, defineAlert } from "@cargo-ai/cdk";

actions: [
  alertConnectorAction({
    ref: slack.actions.postMessage,
    config: { channelId: "C0…", body: "Error rate {{event.value}}%" },
  }),
  alertToolAction({ ref: enrich, config: { domain: "acme.com" } }),
]

Misspelled and missing fields become editor errors instead of deploy failures, and every field still accepts a {{ … }} template string — so a numeric input can be bound to {{event.value}}. Where the two differ is the source of the schema: a connector action’s comes from cargo-ai cdk types, so an integration you haven’t synced keeps the loose object, while a tool’s comes from its own defineWorkflow input and needs no sync — but a toolRef(uuid) names a tool you didn’t author here, so that one stays loose. Both are helpers rather than the type of actions because TypeScript can’t infer a per-element type through an array literal, the same reason agentConnectorTrigger is one.

config is interpolated against the firing under two roots — alert is what you configured, event is what this firing measured — so an action can say what happened:

Variable	Value
{{alert.name}} / {{alert.uuid}} / {{alert.url}}	the alert that fired
{{event.value}}	the computed value, rounded to two decimals
{{event.threshold}}	the threshold it was compared against
{{event.operator}}	gte or lte
{{event.windowStart}} / {{event.windowEnd}}	the evaluated window, as ISO timestamps
{{event.spansUrl}}	link to the workspace’s Spans view

The evaluation window

An alert does not re-scan a fixed lookback on every tick. Each evaluation covers the time since the previous one, so windows are contiguous and never overlap and every span is judged exactly once:

• Window start — where the last evaluation ended. The very first evaluation starts from the moment the alert was last saved.
• Window end — slightly behind now, by an allowance for span indexing lag. Spans that land late are picked up by the next tick instead of being missed.

This is why the cron is the window size: @every 5m means each evaluation judges roughly the last five minutes.

The window applies to the spans, runs and records sources. A SQL scope windows itself (or doesn’t), and a model scope is a statement about the records as they stand right now; for those two, the cron is only how often the question gets asked.

Actions fire at most once per window. Before firing, an alert atomically claims its window; if a retry or an overlapping tick already claimed it, nothing is recorded and nothing fires. Actions spawn runs that spend credits and can take real action, so a duplicate is worse than a rare miss — and a sustained breach is detected again on the next tick anyway.

Disabled alerts are skipped, and every evaluation — breach or not — records an event.

From the CLI

cargo-ai observability alert list
cargo-ai observability alert get <alert-uuid>

cargo-ai observability alert create \
  --name "Enrich error rate" \
  --cron "@every 5m" \
  --scope '{"kind":"spans","workflowUuid":"<uuid>"}' \
  --threshold '{"metric":"errorRate","operator":"gte","value":10}'

cargo-ai observability alert create \
  --name "Contacts stopped syncing" \
  --cron "0 * * * *" \
  --scope '{"kind":"model","modelUuid":"<uuid>"}' \
  --threshold '{"metric":"freshness","operator":"gte","value":180}'

cargo-ai observability alert update --uuid <alert-uuid> --enabled false
cargo-ai observability alert remove <alert-uuid>

create also takes --actions (a JSON array), --description and --folder. On update, passing none to --description or --folder clears the field, while omitting the flag leaves it untouched.

Unlike defineAlert, an alert created through the API or the CLI starts disabled — arm it with alert update --uuid <uuid> --enabled true once its scope and threshold look right.

Try a scope and threshold before you commit to it — preview computes the value now and reports whether it would breach, without firing anything:

cargo-ai observability alert preview \
  --scope '{"kind":"spans","workflowUuid":"<uuid>"}' \
  --threshold '{"metric":"duration","aggregation":"p95","operator":"gte","value":30}' \
  --window-minutes 60

--window-minutes (default 60) is the lookback for spans, runs and records scopes; a query scope windows itself and a model scope is measured as it stands, so the flag doesn’t affect either.

Using the UI

See Using the UI to build an alert visually, with a live preview of the spans it matches, and Events for reading an alert’s history.

An alert built in the UI can be brought back into code: cargo-ai cdk pull writes it as a defineAlert under alerts/, with its scope, threshold and actions referencing the other resources it was pulled alongside.