Alerts & Channels

AgentKavach alerts notify you when an agent approaches or exceeds its budget. You configure threshold/channel pairs — when spend crosses a threshold percentage, the corresponding channel fires. Channels include email, Slack, PagerDuty, webhooks, and a kill switch.

How Alerts Work #

Each alert is a threshold + channel pair. The threshold is a float between 0.0 and 1.0 representing the percentage of budget consumed (e.g., 0.70 = 70%).

After every LLM call, AgentKavach checks the current usage against the budget. If usage crosses a threshold, the associated channel fires. Alerts are evaluated in ascending threshold order, so lower thresholds fire first.

Evaluable Budget Types

Alerts are evaluated independently for each budget type. When an agent has multiple budget types configured (e.g., cost + tokens + duration), each type is checked separately against the threshold percentage. The deduplication key is (agent, channel, threshold_pct, budget_type), so the same alert can fire once per budget type.

Budget TypeAlert Message Format
cost$12.50 / $50.00 (25.0%)
tokens_total500,000 tokens / 1,000,000 tokens
tokens_input300,000 tokens / 600,000 tokens
tokens_output200,000 tokens / 400,000 tokens
calls5,000 calls / 10,000 calls
duration45,000 ms / 86,400,000 ms

A 5-minute cooldown prevents duplicate alerts. Once a channel fires for a given threshold, it will not fire again for 5 minutes even if subsequent calls keep crossing the same threshold. This prevents alert storms during high-throughput runs.

ℹ️ Multiple channels per threshold

You can attach multiple channels to the same threshold. For example, send both an email and a Slack message at 80% spend.

ChannelType Enum #

The ChannelType enum defines the available alert channels:

ValueDescription
EMAILSend an email alert via Resend API
SLACKPost a Block Kit message to a Slack webhook
PAGERDUTYTrigger a PagerDuty incident via Events API v2
WEBHOOKPOST a JSON payload to any URL with optional HMAC signing
KILLInvoke the kill callback and block all subsequent LLM calls
python
from agentkavach.alerts import AlertRule, ChannelType

Email Channel #

Sends an email alert using the Resend API. The Resend API key is managed server-side by AgentKavach (via the RESEND_API_KEY environment variable) — you do not need to provide it. Emails are sent from no-reply@agentkavach.com. You can provide a custom HTML template with variable interpolation. Thresholds can apply to any budget type (cost, tokens, calls, duration, loops), not just budget percentage.

ParameterTypeRequiredDefaultDescription
channel_typeChannelTypeYesMust be ChannelType.EMAIL.
thresholdfloatYesBudget percentage (0.0 to 1.0) at which this alert fires.
tostrYesRecipient email address. Use comma-separated values for multiple recipients.
templatestrNoBuilt-in templateCustom HTML email template. Supports template variables like {agent_name}, {pct}, {spent}, etc.
python
from agentkavach.alerts import AlertRule, ChannelType

# Basic email alert at 80% spend
email_alert = AlertRule(
    channel_type=ChannelType.EMAIL,
    threshold=0.80,
    to="platform-team@example.com",
)

# Email with custom template
email_custom = AlertRule(
    channel_type=ChannelType.EMAIL,
    threshold=0.90,
    to="oncall@example.com,cto@example.com",
    template="""
    <h2>AgentKavach Alert: {agent_name}</h2>
    <p>Agent <strong>{agent_name}</strong> has used {pct}% of its {period} budget.</p>
    <p>Spent: ${spent} / ${budget} (${remaining} remaining)</p>
    <p>Severity: {severity}</p>
    <p><a href="{dashboard_url}">View Dashboard</a></p>
    """,
)

Slack Channel #

Posts a rich Block Kit message to a Slack channel via an incoming webhook. The message includes agent name, spend details, and a link to the dashboard.

ParameterTypeRequiredDefaultDescription
channel_typeChannelTypeYesMust be ChannelType.SLACK.
thresholdfloatYesBudget percentage (0.0 to 1.0) at which this alert fires.
webhook_urlstrYesSlack incoming webhook URL. Must start with https://.
templatestrNoBuilt-in Block Kit messageCustom plain-text message template. Supports template variables.
python
from agentkavach.alerts import AlertRule, ChannelType

slack_alert = AlertRule(
    channel_type=ChannelType.SLACK,
    threshold=0.70,
    webhook_url="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
)

Getting a Slack Webhook URL

  1. Go to api.slack.com/apps and click Create New App.
  2. Choose From scratch and name it (e.g., "AgentKavach Alerts").
  3. In the left sidebar, click Incoming Webhooks.
  4. Toggle Activate Incoming Webhooks to On.
  5. Click Add New Webhook to Workspace.
  6. Select the channel where alerts should be posted.
  7. Copy the webhook URL (starts with https://hooks.slack.com/services/).

PagerDuty Channel #

Triggers a PagerDuty incident using the Events API v2. The incident is created with critical severity and includes agent name, spend details, and budget information in the custom details.

ParameterTypeRequiredDefaultDescription
channel_typeChannelTypeYesMust be ChannelType.PAGERDUTY.
thresholdfloatYesBudget percentage (0.0 to 1.0) at which this alert fires.
routing_keystrYesPagerDuty Events API v2 integration/routing key.
templatestrNoBuilt-in summaryCustom summary template for the PagerDuty incident. Supports template variables.
python
from agentkavach.alerts import AlertRule, ChannelType

pagerduty_alert = AlertRule(
    channel_type=ChannelType.PAGERDUTY,
    threshold=0.95,
    routing_key="e93facc04764012d7bfb002500d5d1a6",
)

Getting a PagerDuty Routing Key

  1. Log in to PagerDuty and navigate to Services.
  2. Select or create the service for AgentKavach alerts.
  3. Go to the Integrations tab.
  4. Click Add Integration and select Events API v2.
  5. Copy the Integration Key (this is your routing key).

Webhook Channel #

Sends a POST request with a JSON payload to any URL. Optionally signs the request body with HMAC-SHA256 for verification. This is the most flexible channel — use it to integrate with any system that accepts webhooks.

ParameterTypeRequiredDefaultDescription
channel_typeChannelTypeYesMust be ChannelType.WEBHOOK.
thresholdfloatYesBudget percentage (0.0 to 1.0) at which this alert fires.
urlstrYesThe URL to POST the JSON payload to.
secretstrNoNoneHMAC-SHA256 secret for signing. When set, the signature is sent in the X-AgentKavach-Signature header.
templatestrNoBuilt-in JSON payloadCustom template for the JSON body. Supports template variables.
python
from agentkavach.alerts import AlertRule, ChannelType

webhook_alert = AlertRule(
    channel_type=ChannelType.WEBHOOK,
    threshold=0.80,
    url="https://api.example.com/agentkavach-alerts",
    secret="whsec_your_signing_secret_here",
)

Webhook Payload

The default JSON payload sent to your webhook endpoint:

json
{
  "event": "budget.threshold_crossed",
  "agent_name": "research-bot",
  "threshold": 0.80,
  "pct": 82.5,
  "spent": 4.13,
  "budget": 5.00,
  "remaining": 0.87,
  "period": "daily",
  "severity": "warning",
  "timestamp": "2025-01-15T14:30:00Z"
}

HMAC Signature Verification

When a secret is provided, AgentKavach computes an HMAC-SHA256 signature of the raw JSON body and sends it in the X-AgentKavach-Signature header. Verify it on your server:

python
import hmac
import hashlib

def verify_signature(body: bytes, signature: str, secret: str) -> bool:
    expected = hmac.new(
        secret.encode(),
        body,
        hashlib.sha256,
    ).hexdigest()
    return hmac.compare_digest(f"sha256={expected}", signature)

Kill Switch #

The kill switch is a special channel that terminates the agent when the budget threshold is crossed. When triggered, it invokes the on_kill callback, sets an internal _killed flag, and all subsequent LLM calls raise BudgetExceededError immediately without making any API calls.

ParameterTypeRequiredDefaultDescription
channel_typeChannelTypeYesMust be ChannelType.KILL.
thresholdfloatYesBudget percentage (0.0 to 1.0) at which the kill switch fires. Typically set to 1.0 (100%).
python
from agentkavach import AgentKavach, Budget
from agentkavach.alerts import AlertRule, ChannelType

def emergency_stop(agent_name: str, spent: float, budget: float):
    """Called when the kill switch fires."""
    print(f"KILLED: {agent_name} spent {spent:.2f} of {budget:.2f}")
    # Clean up resources, save state, notify team, etc.

guard = AgentKavach(
    agent_name="expensive-agent",
    api_key="cg_...",
    budget=Budget(daily=10.00),
    channels=[
        AlertRule(channel_type=ChannelType.KILL, threshold=1.0),
    ],
    on_kill=emergency_stop,
)

🚨 Kill is permanent for the run

Once the kill switch fires, the agent instance is permanently blocked. There is no way to resume — you must create a new AgentKavach instance. This is by design to prevent runaway spend.

Dashboard Stop Button #

The Stop button in the dashboard triggers the kill switch for an agent. When clicked:

  1. A "kill" alert event is recorded in the database
  2. If the agent's SDK has an on_kill callback registered, it will be invoked on the next LLM call attempt
  3. All subsequent guard.create() calls raise BudgetExceededError

⚠️ Stop button does not terminate processes

The Stop button does not directly terminate a running process. It sets the killed state that the SDK checks. If your agent does not define an on_kill callback, the button will mark the agent as killed but the process itself continues running until it tries to make another LLM call (which will then fail).

To ensure the Stop button works as expected, always define an on_kill callback:

python
def emergency_stop(agent_name: str, spent: float, budget: float):
    """Called when the kill switch fires via the dashboard Stop button."""
    print(f"KILLED: {agent_name} spent {spent:.2f} of {budget:.2f}")
    # Save state, release resources, notify team
    agent.save_checkpoint("killed_by_dashboard")
    sys.exit(1)

guard = AgentKavach(
    agent_name="my-agent",
    api_key="cg_...",
    budget=Budget.daily(50),
    on_kill=emergency_stop,  # Required for Stop button to halt the process
)

Complete Multi-Channel Example #

A production setup with escalating alerts across multiple channels:

python
from agentkavach import AgentKavach, Budget
from agentkavach.alerts import AlertRule, ChannelType

def on_kill(agent_name, spent, budget):
    print(f"Agent {agent_name} killed. Spent {spent:.2f}/{budget:.2f}")

guard = AgentKavach(
    agent_name="production-agent",
    api_key="cg_...",
    budget=Budget(daily=50.00, monthly=1000.00),
    channels=[
        # 50% — early warning via email
        AlertRule(
            channel_type=ChannelType.EMAIL,
            threshold=0.50,
            to="platform-team@example.com",
        ),

        # 70% — escalate to Slack
        AlertRule(
            channel_type=ChannelType.SLACK,
            threshold=0.70,
            webhook_url="https://hooks.slack.com/services/T.../B.../xxx",
        ),

        # 80% — dual alert: email + Slack
        AlertRule(
            channel_type=ChannelType.EMAIL,
            threshold=0.80,
            to="oncall@example.com",
        ),
        AlertRule(
            channel_type=ChannelType.SLACK,
            threshold=0.80,
            webhook_url="https://hooks.slack.com/services/T.../B.../xxx",
        ),

        # 95% — page the on-call engineer
        AlertRule(
            channel_type=ChannelType.PAGERDUTY,
            threshold=0.95,
            routing_key="e93facc04764012d7bfb002500d5d1a6",
        ),

        # 100% — kill the agent
        AlertRule(
            channel_type=ChannelType.KILL,
            threshold=1.0,
        ),
    ],
    on_kill=on_kill,
)

Template Variables Reference #

All channel templates support the following variables, interpolated using Python string formatting:

VariableExampleDescription
{agent_name}research-botName of the agent that triggered the alert
{pct}82.5Current spend as a percentage of budget
{spent}4.13Total amount spent so far (USD)
{budget}5.00Total budget limit (USD)
{remaining}0.87Remaining budget (USD)
{period}dailyBudget period type (daily, monthly, or total)
{level}warningAlert level based on threshold (info, warning, critical)
{severity}criticalAlias for level, used in PagerDuty integration
{resets_at}2025-01-16T00:00:00ZWhen the budget resets (daily/monthly budgets only)
{dashboard_url}https://app.agentkavach.dev/agents/research-botDirect link to the agent in the AgentKavach dashboard

Environment Variables for Channels #

Channels can read credentials from environment variables so you do not have to hardcode them in your source code:

ChannelEnv VarDescription
EmailRESEND_API_KEYServer-side Resend API key (managed by AgentKavach, not user-provided)
SlackAGENTKAVACH_SLACK_WEBHOOK_URLDefault Slack incoming webhook URL
PagerDutyAGENTKAVACH_PAGERDUTY_ROUTING_KEYDefault PagerDuty Events API v2 routing key
WebhookAGENTKAVACH_WEBHOOK_URLDefault webhook endpoint URL
WebhookAGENTKAVACH_WEBHOOK_SECRETDefault HMAC signing secret for webhooks
DashboardAGENTKAVACH_DASHBOARD_URLBase URL for dashboard links in alert templates (defaults to https://app.agentkavach.dev)

⚠️ Never commit secrets

Store all API keys, webhook URLs, and signing secrets in environment variables or a .env file. Never hardcode them in your source code or commit them to version control.