Monitoring a tool

Inspect runs, pull metrics, re-queue failures, and set alerts from the CLI — or the UI

Every run a tool produces is queryable from the CLI, so you can monitor health, compute error rates, and re-queue failures from a terminal or CI job. The same data backs the Records and Metrics views in the UI.

Inspect runs from the CLI

cargo-ai orchestration tool list                       # → workflowUuid

# List recent runs, optionally filtered by status
cargo-ai orchestration run list \
  --workflow-uuid <uuid> \
  --statuses error \
  --created-after 2025-01-12

# Drill into one run — inputs, node-by-node outputs, timings, errors
cargo-ai orchestration run get <run-uuid>

Run statuses are running, success, error, and cancelled.

Metrics

Pull per-node execution counts and credit usage over a date range:

cargo-ai orchestration run get-metrics \
  --workflow-uuid <uuid> \
  --created-after 2025-01-12 \
  --created-before 2025-01-19

{
  "runMetrics": [
    {
      "nodeUuid": "enrich-node-uuid",
      "totalExecutionsCount": 1000,
      "successExecutionsCount": 950,
      "errorExecutionsCount": 48,
      "cancelledExecutionsCount": 2,
      "creditsUsedCount": 450
    }
  ]
}

Error rate per node = errorExecutionsCount / totalExecutionsCount (here 48 / 1000 = 4.8%).

Re-queue failed runs

Download the failed runs, extract their record IDs, and re-run only those:

# 1. Download failures
cargo-ai orchestration run download \
  --workflow-uuid <uuid> \
  --statuses error \
  --created-after 2025-01-12 > failed_runs.json

# 2. Re-run just the failed records
RECORD_IDS=$(jq '[.[].recordId]' failed_runs.json)
cargo-ai orchestration batch create \
  --workflow-uuid <uuid> \
  --data "{\"kind\":\"recordIds\",\"recordIds\":$RECORD_IDS}"

Before re-queuing, check whether failures are transient (rate limits) or permanent (invalid data). Fix permanent issues first — retrying won’t help.

From the UI

The tool editor’s Records view lists the same runs with status filters (by batch, version, status, date, node outcome); click a run for the node-by-node breakdown. The Metrics view charts success rate, latency (p50/p95/p99), volume, and error distribution, grouped by batch, version, or date. Failed runs can be retried individually or in bulk from Records.

Alerts

Send alerts to your own systems with a webhook. Cargo POSTs a payload like this on failures:

{
  "event": "tool.execution.failed",
  "tool_id": "tool_abc123",
  "tool_name": "Company Enrichment",
  "execution_id": "exec_xyz789",
  "error": "Rate limit exceeded",
  "timestamp": "2024-01-15T10:30:00Z"
}

Slack and email notifications (failure thresholds, success-rate drops, unusual volume) are configured under Settings → Notifications.

Troubleshooting

Tool execution times out

Break complex tools into smaller ones, add timeouts to slow external calls, reduce batch sizes, or use async execution for long-running operations.

Rate limit errors from integrations

Only connector nodes (kind: "connector") are rate-limited. Add delays between calls, distribute load, reduce concurrency, or check the integration’s documented limits.

Data not found in enrichment

Verify the input format (e.g. domain without https://), confirm the record exists in the provider’s database, and add fallback providers or validation before the enrichment step.

CRM write failures

Confirm the record exists first, check field permissions and required fields, validate data types against CRM field types, and watch for duplicate-detection rules.

Best practices

• Set alerts before production — don’t wait for users to report issues.
• Review metrics regularly — spot trends before they become incidents.
• Fix root causes — investigate repeated failures instead of blindly retrying.
• Version changes — correlate performance shifts with deploys.