> ## Documentation Index > Fetch the complete documentation index at: https://docs.getcargo.ai/llms.txt > Use this file to discover all available pages before exploring further. # Overview > Scheduled threshold checks over your telemetry, your warehouse SQL, or a model's records, firing actions when a metric breaches. Define them with defineAlert. An **alert** is a scheduled check over your workspace. On each tick it computes one metric over a slice of your data, compares it to a threshold, and, when the threshold is breached, fires its actions, each as its own run. You define one with `defineAlert`. What it measures is up to the scope: [span telemetry](/deploy/monitoring#spans-and-traces) (every tool run, play batch and agent message emits spans), your play and tool **runs** end to end, a raw **SQL** query against the orchestration store or your data warehouse, or the records of a **model**. ## Define an alert ```ts alerts/error-spike.ts theme={null} import { defineAlert } from "@cargo-ai/cdk"; import { sentinel } from "../agents/sentinel"; import { slack } from "../connectors/slack"; import { enrich } from "../tools/enrich"; export const errorSpike = defineAlert("error-spike", { description: "Error rate of the enrich workflow", schedule: { type: "cron", cron: "@every 5m" }, // when the check runs scope: { kind: "spans", workflow: enrich }, // which spans it watches threshold: { metric: "errorRate", operator: "gte", value: 10 }, // when it breaches actions: [ { ref: slack.actions.postMessage, config: { channelId: "C0123456789", format: "markdown", body: "Enrich error rate hit {{event.value}}% — {{event.spansUrl}}", }, }, { ref: sentinel, config: { prompt: "Investigate {{alert.url}}" } }, ], }); ``` `schedule` takes a 5-field cron expression or an `@every` interval (`@every 5m`, `@every 1h30m`), evaluated in UTC. An alert evaluates **at most once a minute** — every tick scans your spans, so sub-minute intervals are rejected. In the CDK `enabled` defaults to `true`, so a deployed alert is armed; set it to `false` to deploy one without arming it. `folder` files the alert under an `"alert"`-kind [folder](/folders/overview). ## Scope: what it watches The scope's `kind` names the data source. Each source pairs with its own metric menu, so a metric can never be asked of a source that can't compute it. ### Spans scope `kind: "spans"` watches span telemetry directly. Use any subset of these filters; omitted fields don't narrow. They mirror the **Spans** view. | Field | Narrows to | | ------------------------------ | ----------------------------------------------------------------------- | | `workflow` | one play or tool's workflow: a handle, or `workflowRef(uuid)` | | `parentAgent` | the spans of runs one agent spawned | | `nodeKind` | `native`, `connector`, `tool`, or `agent` nodes | | `integration` | one integration slug | | `connector` | one connector | | `action` | one action slug | | `tool` | one tool *node* | | `agent` | one agent *node* | | `executionTitleOrErrorMessage` | spans whose title or error message contains the text (case-insensitive) | | `executionStatuses` | `pending`, `success`, and/or `error` | | `userUuid` | runs started by one user | `parentAgent` and `agent` are different filters. `parentAgent` matches spans of runs an agent *started* — the agent-trigger case. `agent` matches an agent *node* running inside the watched spans. `workflow` takes a play/tool handle or `workflowRef(uuid)` — not `toolRef` or `agentRef`. Spans are keyed by the workflow behind a tool, not by the tool's own uuid, so passing the wrong kind of reference is rejected at deploy time rather than producing an alert that silently matches nothing. ### Runs scope `kind: "runs"` watches your play and tool **runs** end to end, rather than the individual node executions a spans scope sees. A run is one record travelling through a workflow, so `count` counts records processed and `duration` measures a record's whole journey, not one node's. ```ts theme={null} export const stalledEnrichment = defineAlert("stalled-enrichment", { schedule: { type: "cron", cron: "@every 15m" }, scope: { kind: "runs", workflow: enrich, statuses: ["error"] }, threshold: { metric: "count", operator: "gte", value: 25 }, actions: [{ ref: sentinel, config: { prompt: "Investigate {{alert.url}}" } }], }); ``` | Field | Narrows to | | --------------------------- | --------------------------------------------------------------------------------------------------------------- | | `workflow` | one play or tool's workflow: a handle, or `workflowRef(uuid)`. Omitted watches every play and tool | | `statuses` | runs in these statuses (`idle`, `pending`, `running`, `success`, `error`, `cancelling`, `cancelled`, `skipped`) | | `recordTitleOrErrorMessage` | runs whose record title or error message contains the text (case-insensitive) | | `releaseUuid` | runs of one release | | `userUuid` | runs started by one user | Ad-hoc runs, the ones an agent spawns outside a play or a tool, are never counted, whatever the filters. They live on the workspace's default workflow, which this source excludes, so an agent's own activity can't drown out the workflows you are watching. Use a spans scope with `parentAgent` to alert on an agent instead. ### Records scope `kind: "records"` watches the same play and tool work as a runs scope, counted once per **record** instead of once per attempt. Each record is held in its latest state, so a record that failed and was later re-run into success stops counting as failed, and idle or skipped runs never appear at all. Reach for it when you care about how your records are doing rather than how your executions are doing. ```ts theme={null} export const unresolvedRecords = defineAlert("unresolved-records", { schedule: { type: "cron", cron: "@every 1h" }, scope: { kind: "records", workflow: enrich }, threshold: { metric: "errorRate", operator: "gte", value: 5 }, actions: [{ ref: sentinel, config: { prompt: "Records still failing" } }], }); ``` | Field | Narrows to | | --------------------- | -------------------------------------------------------------------------------------------------- | | `workflow` | one play or tool's workflow: a handle, or `workflowRef(uuid)`. Omitted watches every play and tool | | `statuses` | records in these statuses (`pending`, `running`, `success`, `error`, `cancelling`, `cancelled`) | | `titleOrErrorMessage` | records whose title or error message contains the text (case-insensitive) | | `releaseUuid` | records of one release | | `userUuid` | records started by one user | Retries are the difference between the two sources. Three failed attempts at one record are three failed runs but one failed record, and if a fourth attempt succeeds, the record counts as a success while the three failed runs remain. Alert on runs to catch a flaky workflow; alert on records to catch work that never got done. Ad-hoc runs are excluded here too. ### SQL scopes Two sources run raw SQL and use its result as the value. The query computes the value itself, so the threshold carries only the comparison: * `kind: "orchestrationQuery"` runs read-only **ClickHouse SQL** over your orchestration data: the `spans`, `runs`, `batches`, and `records` tables, automatically scoped to your workspace. * `kind: "storageQuery"` runs read-only SQL over your **data warehouse**, referencing models as `.` exactly as the model SQL editor does. ```ts theme={null} export const slowNights = defineAlert("slow-nights", { schedule: { type: "cron", cron: "0 * * * *" }, scope: { kind: "orchestrationQuery", query: `select count(*) from spans where execution_status = 'error' and execution_started_at > now() - interval 1 hour`, }, threshold: { operator: "gte", value: 50 }, actions: [ { ref: sentinel, config: { prompt: "Error spike: {{event.value}}" } }, ], }); ``` A warehouse query looks the same, against your own tables: ```ts theme={null} export const missingEmails = defineAlert("missing-emails", { schedule: { type: "cron", cron: "0 * * * *" }, scope: { kind: "storageQuery", query: `select 100.0 * sum(case when email is null then 1 else 0 end) / nullif(count(*), 0) from crm.contacts`, }, threshold: { operator: "gte", value: 20 }, actions: [{ ref: sentinel, config: { prompt: "Contact data decayed" } }], }); ``` Query scopes are **not** windowed for you: Cargo runs the query exactly as written. An orchestration query should be self-windowing (as the `interval 1 hour` above does), otherwise every tick evaluates your whole history. A warehouse query sees your models as they stand, which is usually what you want: it is a statement about the data, not about a time slice. The value is the **first column of the first row**, and it must be numeric. A non-number produces an `error` event. A query that returns no rows or a `NULL` counts as an empty window instead: the tick is recorded as `healthy` with `value: null` and nothing fires. That is what an aggregate over no rows gives you, and what a rate query returns for `0 / 0` in an idle window, so an empty window can never quietly read as a real `0` and breach an `lte` threshold. See [Querying orchestration data](/reference/orchestration-query) for the columns of each orchestration table, the ClickHouse idioms that differ from PostgreSQL, and the limits a query runs under; see [Querying models](/models/querying) for the warehouse side. A warehouse query never waits for a model to refresh. An alert runs on a cron, and blocking a tick on a sync would leave it measuring a moment that has passed by the time the sync lands. ### Model scope `kind: "model"` watches the records of one [model](/models/overview), optionally narrowed by a `filter`, the same filter a [segment](/models/segments) is built from. Unlike every other source, a model is measured **as it stands** rather than over the evaluation window: ```ts theme={null} export const decayingContacts = defineAlert("decaying-contacts", { schedule: { type: "cron", cron: "0 9 * * *" }, scope: { kind: "model", model: contacts, filter: { conjonction: "and", groups: [ { conjonction: "and", conditions: [{ kind: "string", columnSlug: "email", operator: "isNull" }], }, ], }, }, threshold: { metric: "recordsShare", operator: "gte", value: 30 }, actions: [{ ref: sentinel, config: { prompt: "{{event.value}}% missing" } }], }); ``` ## Threshold: when it breaches `operator` is `gte` (breach at or above `value`) or `lte` (breach at or below). Apart from the SQL scopes, where the query is the metric, `metric` says what is measured. It is required, since it decides what the value means, and a metric that supports aggregations requires one too. **Spans, runs and records** share a metric menu; the scope decides what is being counted, one node execution, one record's whole journey, or one record: | `metric` | Value | `aggregation` | | ----------- | ---------------------------------------------------------------- | -------------------------- | | `errorRate` | failed as a percentage (0–100) of the window's **finished** rows | none | | `duration` | duration in seconds, over finished rows only | `avg`, `p50`, `p95`, `p99` | | `credits` | credits consumed by the window's rows | `sum`, `avg`, `p95` | | `count` | number of spans / runs / records in the window | none | `count` with `lte` is a dead-man's switch: an empty window really evaluates to `0`, so **silence breaches**. `{ metric: "count", operator: "lte", value: 0 }` tells you a workflow stopped running at all. The other metrics treat an empty window as nothing to judge, not as zero — and for `errorRate` and `duration`, "empty" means no *finished* rows, so a window of runs that are all still going is not judged yet either. A **model** has its own four, about the records themselves and about how fresh they are: | `metric` | Value | | -------------- | ------------------------------------------------------------- | | `recordsCount` | records matching the scope | | `recordsShare` | matching records as a percentage (0–100) of the model's total | | `freshness` | minutes since the model last emitted records | | `syncDuration` | duration in seconds of the model's last finished sync | `recordsShare` requires the scope's `filter`: a share of *everything* is always 100, so the pair is rejected at save time rather than firing on a constant. ## Actions: what fires on breach Each action becomes its own run, exactly like a play's `healthAlertActions`. An action is a connector action, an agent, or a tool, and its `config` is the input it runs with: ```ts theme={null} actions: [ { ref: slack.actions.postMessage, config: { channelId: "C0…", body: "…" } }, { ref: sentinel, config: { prompt: "…" }, waitUntilFinished: true }, { ref: enrich, config: {} }, ] ``` An **agent**'s `config` is typed already — every agent takes the same `{ prompt, output? }`, so a misspelled key is an editor error in the literal above with nothing to import. A **connector action** or a **tool** has an input of its own, and a bare object literal leaves it unchecked. Wrap it in `alertConnectorAction` / `alertToolAction` to have TypeScript check `config` against the real thing: ```ts theme={null} import { alertConnectorAction, alertToolAction, defineAlert } from "@cargo-ai/cdk"; actions: [ alertConnectorAction({ ref: slack.actions.postMessage, config: { channelId: "C0…", body: "Error rate {{event.value}}%" }, }), alertToolAction({ ref: enrich, config: { domain: "acme.com" } }), ] ``` Misspelled and missing fields become editor errors instead of deploy failures, and every field still accepts a `{{ … }}` template string — so a numeric input can be bound to `{{event.value}}`. Where the two differ is the source of the schema: a connector action's comes from [`cargo-ai cdk types`](/get-started/project-layout), so an integration you haven't synced keeps the loose object, while a tool's comes from its own `defineWorkflow` input and needs no sync — but a `toolRef(uuid)` names a tool you didn't author here, so that one stays loose. Both are helpers rather than the type of `actions` because TypeScript can't infer a per-element type through an array literal, the same reason `agentConnectorTrigger` is one. `config` is interpolated against the firing under two roots — `alert` is what you configured, `event` is what this firing measured — so an action can say what happened: | Variable | Value | | ----------------------------------------------------- | ------------------------------------------- | | `{{alert.name}}` / `{{alert.uuid}}` / `{{alert.url}}` | the alert that fired | | `{{event.value}}` | the computed value, rounded to two decimals | | `{{event.threshold}}` | the threshold it was compared against | | `{{event.operator}}` | `gte` or `lte` | | `{{event.windowStart}}` / `{{event.windowEnd}}` | the evaluated window, as ISO timestamps | | `{{event.spansUrl}}` | link to the workspace's Spans view | ## The evaluation window An alert does **not** re-scan a fixed lookback on every tick. Each evaluation covers the time since the previous one, so windows are contiguous and never overlap and every span is judged exactly once: * **Window start** — where the last evaluation ended. The very first evaluation starts from the moment the alert was last saved. * **Window end** — slightly behind now, by an allowance for span indexing lag. Spans that land late are picked up by the next tick instead of being missed. This is why the cron is the window size: `@every 5m` means each evaluation judges roughly the last five minutes. The window applies to the **spans**, **runs** and **records** sources. A SQL scope windows itself (or doesn't), and a model scope is a statement about the records as they stand right now; for those two, the cron is only how often the question gets asked. Actions fire **at most once** per window. Before firing, an alert atomically claims its window; if a retry or an overlapping tick already claimed it, nothing is recorded and nothing fires. Actions spawn runs that spend credits and can take real action, so a duplicate is worse than a rare miss — and a sustained breach is detected again on the next tick anyway. Disabled alerts are skipped, and every evaluation — breach or not — records an [event](/alerts/events). ## From the CLI ```bash theme={null} cargo-ai observability alert list cargo-ai observability alert get cargo-ai observability alert create \ --name "Enrich error rate" \ --cron "@every 5m" \ --scope '{"kind":"spans","workflowUuid":""}' \ --threshold '{"metric":"errorRate","operator":"gte","value":10}' cargo-ai observability alert create \ --name "Contacts stopped syncing" \ --cron "0 * * * *" \ --scope '{"kind":"model","modelUuid":""}' \ --threshold '{"metric":"freshness","operator":"gte","value":180}' cargo-ai observability alert update --uuid --enabled false cargo-ai observability alert remove ``` `create` also takes `--actions` (a JSON array), `--description` and `--folder`. On `update`, passing `none` to `--description` or `--folder` clears the field, while omitting the flag leaves it untouched. Unlike `defineAlert`, an alert created through the API or the CLI starts **disabled** — arm it with `alert update --uuid --enabled true` once its scope and threshold look right. Try a scope and threshold before you commit to it — `preview` computes the value now and reports whether it would breach, without firing anything: ```bash theme={null} cargo-ai observability alert preview \ --scope '{"kind":"spans","workflowUuid":""}' \ --threshold '{"metric":"duration","aggregation":"p95","operator":"gte","value":30}' \ --window-minutes 60 ``` `--window-minutes` (default `60`) is the lookback for spans, runs and records scopes; a query scope windows itself and a model scope is measured as it stands, so the flag doesn't affect either. ## Using the UI See [Using the UI](/alerts/using-ui) to build an alert visually, with a live preview of the spans it matches, and [Events](/alerts/events) for reading an alert's history. An alert built in the UI can be brought back into code: `cargo-ai cdk pull` writes it as a `defineAlert` under `alerts/`, with its scope, threshold and actions referencing the other resources it was pulled alongside.