Alerts

Alerts monitor job executions and lifecycle events, then route matching events to notification channels. Outcron supports two complementary notification models:

• Alert rules — condition-based evaluation (e.g. "fire after 3 consecutive failures"). Rules emit durable rule.fired events when their conditions match.
• Event subscriptions — pub/sub on job, execution, schedule, healthcheck, and rule.fired transitions (e.g. "notify on every execution.failed for this job"). Subscriptions own the immediate notification channels.

Subscriptions and escalation steps share the same channel types and durable delivery pipeline. You can also acknowledge events, silence matchers during maintenance, and escalate unacked alerts to secondary channels.

Alert Rules

An alert rule defines:

• Job — which job to monitor (required).
• Schedule — optionally limit monitoring to a specific schedule.
• Condition type — what triggers the alert.
• Threshold — the trigger value.
• Cooldown — minimum minutes between firings (default: 60).

Condition Types

Window-Based Conditions

The error_rate and duration_threshold conditions use a sliding evaluation window:

• Window minutes — how far back to look when evaluating the condition.
• Sample minimum — minimum number of executions required before the condition is evaluated. Prevents false alerts during low-traffic periods.

Cooldown

After an alert fires, the cooldown period prevents the same rule from firing again immediately. This avoids notification floods during extended outages.

Rules do not deliver notifications directly. To notify when a rule fires, create an event subscription for rule.fired and attach the email, webhook, chat, or incident channels to that subscription.

Event Subscriptions

Subscriptions route lifecycle and rule-fired events to channels without defining a threshold condition. Useful when you want to know about every success, every failure, every config change, or every alert rule firing.

• Event types — one or more of:
  • execution.started, execution.succeeded, execution.failed, execution.timeout, execution.cancelled
  • execution.retry_exhausted, execution.blocked_by_policy, execution.no_overlap_skipped, execution.long_running
  • healthcheck.healthy, healthcheck.unhealthy
  • rule.fired
  • job.created, job.updated, job.deleted
  • schedule.misfired, schedule.paused, schedule.resumed
• Filter by job or filter by schedule — scope the subscription so you don't receive notifications for every job in the org.
• Rate limit per minute — drop excess matches for the current UTC minute. Set to 0 for unlimited.

Subscriptions go through a durable outbox: emitted events are persisted before matching, so restarting the server between emit and delivery never loses notifications.

Notification Channels

Outcron supports seven channel types. All credentials are encrypted at rest and masked in API responses.

Email

Standard SMTP delivery. Config: a list of recipient addresses.

Webhook

POSTs a JSON payload to an HTTPS URL. Optional HMAC-SHA256 signing secret — if set, each request carries an X-Outcron-Signature header like t=<unix>,v1=<hex>.

Slack

Incoming Webhook URL only. Create one in Slack under Apps → Incoming Webhooks or a custom app's webhook configuration. The URL must use https://hooks.slack.com/.... Payload uses Block Kit with title, summary, and a "View in Outcron" button.

Microsoft Teams

Workflow-based webhook URL only (Power Automate's "Post to a channel when a webhook request is received" template). Accepted URLs must use Microsoft-controlled workflow hosts such as *.logic.azure.com, *.logic.azure.us, *.logic.azure.cn, or *.api.powerplatform.com. Legacy Office 365 connector URLs (outlook.office.com/webhook/...) are deprecated by Microsoft and will be rejected at validation time.

Discord

Discord webhook URL only. Outcron posts an embed-style message with the alert summary and key event details.

PagerDuty

Events API v2 with the integration routing key. Fire → trigger with a stable dedup key per event. Outcron acknowledgement propagates as event_action=acknowledge. Outcron does not auto-resolve PagerDuty incidents — ack means "someone is handling this," not "the condition cleared."

Optional severity_override maps all events from this channel to a specific severity (critical, error, warning, info). Without an override, severity is derived from the condition type or lifecycle event type.

Opsgenie

Alert API v2 with an API key. Configure the region (us or eu) to match your Opsgenie account. Fire → POST /v2/alerts with alias = AlertEvent.id. Ack → POST /v2/alerts/{alias}/acknowledge. Same no-auto-resolve rule as PagerDuty.

Optional team routes the alert to a specific Opsgenie team.

Acknowledgement

Once an alert event fires, anyone with alerts:write can acknowledge it:

• From the UI: open the event detail, write an optional note, click Acknowledge.
• From the CLI: outctl alerts events ack <event-id> --note "investigating".
• From the API: POST /orgs/{orgID}/alerts/events/{eventID}/ack.

Ack fields (acked_by_user_id, acked_at, ack_note) are recorded on the event. If the event has PagerDuty or Opsgenie channels attached, the ack propagates to those vendors as acknowledge (never resolve).

Unack (/unack) clears the ack locally. It does not send a counter-signal to the vendors — their incidents stay in the "acknowledged" state until they resolve on their own.

Ack and unack are write operations. If the organization is billing-locked, recover billing first; alert history remains readable, but acknowledgement changes are blocked until the lock clears.

Silences

Silences suppress alert firing for a bounded time window. Useful during planned maintenance.

A silence is defined by:

• Reason and expires_at (or duration_seconds at create time).
• One or more matchers, which AND together:
  • match_source_kind — rule or subscription.
  • match_source_id — a specific alert rule ID or subscription ID.
  • match_job_id / match_schedule_id — scope to one job or schedule.
  • match_source_event_type — scope to one lifecycle event type.

At least one matcher is required. An empty silence would suppress every event in the org and is rejected at validation.

Suppressed fires are logged to alert_silence_hits so you can audit what was muted. View hits via Silences → [silence] → hits in the UI or outctl alerts silences hits <silence-id>.

Escalation Policies

An escalation policy is a named sequence of steps. Each step has:

• after_minutes — how long after the event triggered before this step fires.
• Channels — the policy step owns its channels; they don't overlap with subscription channels.

When an alert source has an escalation policy configured and an event fires:

1. Subscription channels deliver immediately.
2. A background job evaluates unacked events every minute. When a step becomes due, its channels get a fresh delivery through the existing outbox.
3. Acknowledging the event halts further steps. No additional deliveries fire once acked_at is set.

Each (event, step) pair is tracked in alert_event_escalations with a unique constraint so a server restart mid-evaluation won't duplicate deliveries.

Alert Events

When an alert fires, it creates an alert event. Each event captures:

• Source — usually the matching subscription; older rule-owned events may show rule.
• Source event type — for subscription events, the matched event slug (e.g. execution.failed or rule.fired).
• Event type — fired (real alert) or test (manual test).
• Source snapshot — the subscription configuration at firing time; rule.fired event details also include the rule snapshot.
• Delivery status — tracks whether notifications were sent successfully.
• Ack fields — acked_by_user_id, acked_at, ack_note when the event has been acknowledged.

Event Delivery States

Failed deliveries are retried with exponential backoff.

Alert event history is retained for 90 days by default.

Testing Alerts

Use the Test button on a rule to queue a synthetic rule.fired event without waiting for the real condition to trigger. Matching subscriptions handle delivery, and resulting events are labeled with event type test. Subscriptions have an equivalent Test action that synthesizes one of their selected event types and sends through that subscription's channels.

Permissions

All alert surfaces (rules, subscriptions, silences, escalation policies, event ack) sit under alerts:read and alerts:write.