Alerts & Channels
AgentKavach alerts notify you when an agent approaches or exceeds its budget. You configure threshold/channel pairs — when spend crosses a threshold percentage, the corresponding channel fires. Channels include email, Slack, PagerDuty, webhooks, and a kill switch.
How Alerts Work #
Each alert is a threshold + channel pair. The threshold is a float between 0.0 and 1.0 representing the percentage of budget consumed (e.g., 0.70 = 70%).
After every LLM call, AgentKavach checks the current usage against the budget. If usage crosses a threshold, the associated channel fires. Alerts are evaluated in ascending threshold order, so lower thresholds fire first.
Evaluable Budget Types
Alerts are evaluated independently for each budget type. When an agent has multiple budget types configured (e.g., cost + tokens + duration), each type is checked separately against the threshold percentage. The deduplication key is (agent, channel, threshold_pct, budget_type), so the same alert can fire once per budget type.
| Budget Type | Alert Message Format |
|---|---|
cost | $12.50 / $50.00 (25.0%) |
tokens_total | 500,000 tokens / 1,000,000 tokens |
tokens_input | 300,000 tokens / 600,000 tokens |
tokens_output | 200,000 tokens / 400,000 tokens |
calls | 5,000 calls / 10,000 calls |
duration | 45,000 ms / 86,400,000 ms |
A 5-minute cooldown prevents duplicate alerts. Once a channel fires for a given threshold, it will not fire again for 5 minutes even if subsequent calls keep crossing the same threshold. This prevents alert storms during high-throughput runs.
ℹ️ Multiple channels per threshold
ChannelType Enum #
The ChannelType enum defines the available alert channels:
| Value | Description |
|---|---|
EMAIL | Send an email alert via Resend API |
SLACK | Post a Block Kit message to a Slack webhook |
PAGERDUTY | Trigger a PagerDuty incident via Events API v2 |
WEBHOOK | POST a JSON payload to any URL with optional HMAC signing |
KILL | Invoke the kill callback and block all subsequent LLM calls |
from agentkavach.alerts import AlertRule, ChannelTypeEmail Channel #
Sends an email alert using the Resend API. The Resend API key is managed server-side by AgentKavach (via the RESEND_API_KEY environment variable) — you do not need to provide it. Emails are sent from no-reply@agentkavach.com. You can provide a custom HTML template with variable interpolation. Thresholds can apply to any budget type (cost, tokens, calls, duration, loops), not just budget percentage.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
channel_type | ChannelType | Yes | — | Must be ChannelType.EMAIL. |
threshold | float | Yes | — | Budget percentage (0.0 to 1.0) at which this alert fires. |
to | str | Yes | — | Recipient email address. Use comma-separated values for multiple recipients. |
template | str | No | Built-in template | Custom HTML email template. Supports template variables like {agent_name}, {pct}, {spent}, etc. |
from agentkavach.alerts import AlertRule, ChannelType
# Basic email alert at 80% spend
email_alert = AlertRule(
channel_type=ChannelType.EMAIL,
threshold=0.80,
to="platform-team@example.com",
)
# Email with custom template
email_custom = AlertRule(
channel_type=ChannelType.EMAIL,
threshold=0.90,
to="oncall@example.com,cto@example.com",
template="""
<h2>AgentKavach Alert: {agent_name}</h2>
<p>Agent <strong>{agent_name}</strong> has used {pct}% of its {period} budget.</p>
<p>Spent: ${spent} / ${budget} (${remaining} remaining)</p>
<p>Severity: {severity}</p>
<p><a href="{dashboard_url}">View Dashboard</a></p>
""",
)Slack Channel #
Posts a rich Block Kit message to a Slack channel via an incoming webhook. The message includes agent name, spend details, and a link to the dashboard.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
channel_type | ChannelType | Yes | — | Must be ChannelType.SLACK. |
threshold | float | Yes | — | Budget percentage (0.0 to 1.0) at which this alert fires. |
webhook_url | str | Yes | — | Slack incoming webhook URL. Must start with https://. |
template | str | No | Built-in Block Kit message | Custom plain-text message template. Supports template variables. |
from agentkavach.alerts import AlertRule, ChannelType
slack_alert = AlertRule(
channel_type=ChannelType.SLACK,
threshold=0.70,
webhook_url="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
)Getting a Slack Webhook URL
- Go to api.slack.com/apps and click Create New App.
- Choose From scratch and name it (e.g., "AgentKavach Alerts").
- In the left sidebar, click Incoming Webhooks.
- Toggle Activate Incoming Webhooks to On.
- Click Add New Webhook to Workspace.
- Select the channel where alerts should be posted.
- Copy the webhook URL (starts with
https://hooks.slack.com/services/).
PagerDuty Channel #
Triggers a PagerDuty incident using the Events API v2. The incident is created with critical severity and includes agent name, spend details, and budget information in the custom details.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
channel_type | ChannelType | Yes | — | Must be ChannelType.PAGERDUTY. |
threshold | float | Yes | — | Budget percentage (0.0 to 1.0) at which this alert fires. |
routing_key | str | Yes | — | PagerDuty Events API v2 integration/routing key. |
template | str | No | Built-in summary | Custom summary template for the PagerDuty incident. Supports template variables. |
from agentkavach.alerts import AlertRule, ChannelType
pagerduty_alert = AlertRule(
channel_type=ChannelType.PAGERDUTY,
threshold=0.95,
routing_key="e93facc04764012d7bfb002500d5d1a6",
)Getting a PagerDuty Routing Key
- Log in to PagerDuty and navigate to Services.
- Select or create the service for AgentKavach alerts.
- Go to the Integrations tab.
- Click Add Integration and select Events API v2.
- Copy the Integration Key (this is your routing key).
Webhook Channel #
Sends a POST request with a JSON payload to any URL. Optionally signs the request body with HMAC-SHA256 for verification. This is the most flexible channel — use it to integrate with any system that accepts webhooks.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
channel_type | ChannelType | Yes | — | Must be ChannelType.WEBHOOK. |
threshold | float | Yes | — | Budget percentage (0.0 to 1.0) at which this alert fires. |
url | str | Yes | — | The URL to POST the JSON payload to. |
secret | str | No | None | HMAC-SHA256 secret for signing. When set, the signature is sent in the X-AgentKavach-Signature header. |
template | str | No | Built-in JSON payload | Custom template for the JSON body. Supports template variables. |
from agentkavach.alerts import AlertRule, ChannelType
webhook_alert = AlertRule(
channel_type=ChannelType.WEBHOOK,
threshold=0.80,
url="https://api.example.com/agentkavach-alerts",
secret="whsec_your_signing_secret_here",
)Webhook Payload
The default JSON payload sent to your webhook endpoint:
{
"event": "budget.threshold_crossed",
"agent_name": "research-bot",
"threshold": 0.80,
"pct": 82.5,
"spent": 4.13,
"budget": 5.00,
"remaining": 0.87,
"period": "daily",
"severity": "warning",
"timestamp": "2025-01-15T14:30:00Z"
}HMAC Signature Verification
When a secret is provided, AgentKavach computes an HMAC-SHA256 signature of the raw JSON body and sends it in the X-AgentKavach-Signature header. Verify it on your server:
import hmac
import hashlib
def verify_signature(body: bytes, signature: str, secret: str) -> bool:
expected = hmac.new(
secret.encode(),
body,
hashlib.sha256,
).hexdigest()
return hmac.compare_digest(f"sha256={expected}", signature)Kill Switch #
The kill switch is a special channel that terminates the agent when the budget threshold is crossed. When triggered, it invokes the on_kill callback, sets an internal _killed flag, and all subsequent LLM calls raise BudgetExceededError immediately without making any API calls.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
channel_type | ChannelType | Yes | — | Must be ChannelType.KILL. |
threshold | float | Yes | — | Budget percentage (0.0 to 1.0) at which the kill switch fires. Typically set to 1.0 (100%). |
from agentkavach import AgentKavach, Budget
from agentkavach.alerts import AlertRule, ChannelType
def emergency_stop(agent_name: str, spent: float, budget: float):
"""Called when the kill switch fires."""
print(f"KILLED: {agent_name} spent {spent:.2f} of {budget:.2f}")
# Clean up resources, save state, notify team, etc.
guard = AgentKavach(
agent_name="expensive-agent",
api_key="cg_...",
budget=Budget(daily=10.00),
channels=[
AlertRule(channel_type=ChannelType.KILL, threshold=1.0),
],
on_kill=emergency_stop,
)🚨 Kill is permanent for the run
Dashboard Stop Button #
The Stop button in the dashboard triggers the kill switch for an agent. When clicked:
- A "kill" alert event is recorded in the database
- If the agent's SDK has an
on_killcallback registered, it will be invoked on the next LLM call attempt - All subsequent
guard.create()calls raiseBudgetExceededError
⚠️ Stop button does not terminate processes
on_kill callback, the button will mark the agent as killed but the process itself continues running until it tries to make another LLM call (which will then fail).To ensure the Stop button works as expected, always define an on_kill callback:
def emergency_stop(agent_name: str, spent: float, budget: float):
"""Called when the kill switch fires via the dashboard Stop button."""
print(f"KILLED: {agent_name} spent {spent:.2f} of {budget:.2f}")
# Save state, release resources, notify team
agent.save_checkpoint("killed_by_dashboard")
sys.exit(1)
guard = AgentKavach(
agent_name="my-agent",
api_key="cg_...",
budget=Budget.daily(50),
on_kill=emergency_stop, # Required for Stop button to halt the process
)Complete Multi-Channel Example #
A production setup with escalating alerts across multiple channels:
from agentkavach import AgentKavach, Budget
from agentkavach.alerts import AlertRule, ChannelType
def on_kill(agent_name, spent, budget):
print(f"Agent {agent_name} killed. Spent {spent:.2f}/{budget:.2f}")
guard = AgentKavach(
agent_name="production-agent",
api_key="cg_...",
budget=Budget(daily=50.00, monthly=1000.00),
channels=[
# 50% — early warning via email
AlertRule(
channel_type=ChannelType.EMAIL,
threshold=0.50,
to="platform-team@example.com",
),
# 70% — escalate to Slack
AlertRule(
channel_type=ChannelType.SLACK,
threshold=0.70,
webhook_url="https://hooks.slack.com/services/T.../B.../xxx",
),
# 80% — dual alert: email + Slack
AlertRule(
channel_type=ChannelType.EMAIL,
threshold=0.80,
to="oncall@example.com",
),
AlertRule(
channel_type=ChannelType.SLACK,
threshold=0.80,
webhook_url="https://hooks.slack.com/services/T.../B.../xxx",
),
# 95% — page the on-call engineer
AlertRule(
channel_type=ChannelType.PAGERDUTY,
threshold=0.95,
routing_key="e93facc04764012d7bfb002500d5d1a6",
),
# 100% — kill the agent
AlertRule(
channel_type=ChannelType.KILL,
threshold=1.0,
),
],
on_kill=on_kill,
)Template Variables Reference #
All channel templates support the following variables, interpolated using Python string formatting:
| Variable | Example | Description |
|---|---|---|
{agent_name} | research-bot | Name of the agent that triggered the alert |
{pct} | 82.5 | Current spend as a percentage of budget |
{spent} | 4.13 | Total amount spent so far (USD) |
{budget} | 5.00 | Total budget limit (USD) |
{remaining} | 0.87 | Remaining budget (USD) |
{period} | daily | Budget period type (daily, monthly, or total) |
{level} | warning | Alert level based on threshold (info, warning, critical) |
{severity} | critical | Alias for level, used in PagerDuty integration |
{resets_at} | 2025-01-16T00:00:00Z | When the budget resets (daily/monthly budgets only) |
{dashboard_url} | https://app.agentkavach.dev/agents/research-bot | Direct link to the agent in the AgentKavach dashboard |
Environment Variables for Channels #
Channels can read credentials from environment variables so you do not have to hardcode them in your source code:
| Channel | Env Var | Description |
|---|---|---|
RESEND_API_KEY | Server-side Resend API key (managed by AgentKavach, not user-provided) | |
| Slack | AGENTKAVACH_SLACK_WEBHOOK_URL | Default Slack incoming webhook URL |
| PagerDuty | AGENTKAVACH_PAGERDUTY_ROUTING_KEY | Default PagerDuty Events API v2 routing key |
| Webhook | AGENTKAVACH_WEBHOOK_URL | Default webhook endpoint URL |
| Webhook | AGENTKAVACH_WEBHOOK_SECRET | Default HMAC signing secret for webhooks |
| Dashboard | AGENTKAVACH_DASHBOARD_URL | Base URL for dashboard links in alert templates (defaults to https://app.agentkavach.dev) |
⚠️ Never commit secrets
.env file. Never hardcode them in your source code or commit them to version control.