24Observe ships with an analyst that investigates incidents for you — but it is also a toolkit for building your own. Because every capability is a plain, safe, scoped REST API, you can stand up a custom triage bot, a remediation assistant, or a chatops integration in an afternoon. This cookbook shows you the patterns: how to give an agent least-privilege access, how to let it call the platform without an SDK, how to run a safe observe-reason-act loop, and recipes for the things agents actually do.
Most monitoring tools were built for a human clicking a dashboard, and an API was added later — which is why driving them with code so often feels like fighting the product.
24Observe is the other way around. The human dashboard and the programmatic interface are the same surface, so anything a person can do, an agent can do too — and the things an autonomous caller needs to be safe are built in rather than bolted on. Authentication is a bearer token. Requests and responses are JSON. Every mutating call is safe to retry. Every token is scoped and capped. And there are pre-built tool definitions so a model can call the platform directly without you writing and maintaining a wrapper. You are not bending a human tool into an automation; you are using an interface designed for exactly this.
That design choice is what makes building an agent here a short project instead of a long one. The rest of this page is the practical path: mint the token, wire the tools, run the loop, and the recipes for the common jobs.
Never hand an agent broad access. Create a token scoped to exactly the actions it needs, with a daily cap so a mistake is bounded. The token is shown once — store it in your secret manager, not your code.
From the dashboard, create a personal access token, choose the narrow set of scopes the agent requires — for a read-only triage bot, that might be just reading incidents, logs, and the context graph — and attach a daily mutation and data limit. For a bot that will also acknowledge incidents, add that one write scope and nothing else. The guiding rule: grant the smallest set that lets the agent do its job, because that set is the worst case if the token ever leaks.
# Every call is bearer-authenticated against the public API. export OBSERVE_TOKEN="obs_…" # from your secret manager, never source export OBSERVE_API="https://api.24observe.com" curl -s "$OBSERVE_API/api/v1/incidents?status=open" \ -H "authorization: Bearer $OBSERVE_TOKEN"
Two things to notice. The response includes headers telling you how much of the token’s daily budget remains and when it resets, so your agent can pace itself without spending a request to check. And the token can only ever do what its scopes allow — the dangerous actions (disabling detections, deleting data, reading stored secrets) are off-limits to any token, so least-privilege is a floor the platform enforces, not just a habit you maintain.
If your agent framework speaks tool definitions, you do not write an integration at all. Fetch the pre-converted definitions and pass them straight to the model.
# Pre-converted tool definitions, ready to drop into your agent framework. curl -s "$OBSERVE_API/openapi/openai-tools.json" # OpenAI-style curl -s "$OBSERVE_API/openapi/anthropic-tools.json" # Anthropic-style curl -s "$OBSERVE_API/openapi/langchain-tools.json" # LangChain-style
The model now knows every endpoint it is allowed to call, with the right shapes, and you have nothing to keep in sync — when the API changes, the definitions change with it. If you would rather connect a ready-made assistant in your editor or chat tool than build your own loop, a curated, fail-closed tool server exposes a vetted subset of the platform’s capabilities for exactly that; see the connect-an-agent docs. For a custom agent, read on.
The shape of almost every ops agent: observe the current state, reason about it with your model and the tools, and act — carefully, idempotently, within budget.
import os, uuid, requests
API = os.environ["OBSERVE_API"]; TOK = os.environ["OBSERVE_TOKEN"]
H = {"authorization": f"Bearer {TOK}"}
# OBSERVE — pull what needs attention
incidents = requests.get(f"{API}/api/v1/incidents?status=open", headers=H).json()
for inc in incidents:
# REASON — gather evidence with read-only tools, let your model decide
logs = requests.get(f"{API}/api/v1/logs/search",
params={"q": inc["title"], "hours": 1}, headers=H).json()
decision = my_model.decide(inc, logs) # your reasoning step
# ACT — idempotent + scoped; a retry with the same key never double-acts
if decision.acknowledge:
requests.post(f"{API}/api/v1/incidents/{inc['id']}/ack",
headers={**H, "Idempotency-Key": f"ack-{inc['id']}"}) The two habits that make this production-grade are visible in those few lines. Idempotency: every mutating call carries an Idempotency-Key, so when the agent retries after a timeout — and it will — it cannot create duplicates; retrying the same key is a no-op, and reusing a key with a different body returns a clear conflict so a buggy loop surfaces immediately. Budget awareness: the agent reads its remaining daily allowance off the response headers and slows down as it approaches the cap, rather than slamming into it. Together with a scoped token, that is the whole safety story: bounded permissions, bounded volume, safe retries, full attribution.
Common patterns, each a few calls. Mix them into whatever workflow you are automating.
Read the incident, search the related logs and read the relevant metrics, walk the context graph for what it touches, and post a summary back as an incident update — a custom first-responder in your own chatops voice.
Run a query over a window, pull the matching events, and let your model summarise what changed — a natural-language layer over your logs for a chat command or a daily digest.
With a narrow write scope, let the agent acknowledge incidents it has triaged to stop the paging, post updates as it learns more, and resolve them when a downstream signal confirms recovery — always idempotently.
Create, update, and bulk-adjust monitors from code — provision checks for a new service as part of your deploy pipeline, or sweep settings across many monitors at once.
Resolve an entity, read its neighbourhood, and pull an incident’s blast radius — the same topology the built-in analyst reasons over, available to your agent to narrow a search before it starts digging.
Instead of polling, subscribe to signed event webhooks and have your agent wake only when something happens — an incident opens, a detection fires — then run its loop. Less noise, faster reaction.
The fastest way to lose faith in an ops agent is to give it write access on day one and watch it do something surprising. The fastest way to build trust is to earn it in stages. Here is the path most teams take.
Start read-only. Give your first agent a token that can only read — incidents, logs, metrics, the context graph — and let it do nothing but observe and explain. Have it summarise new incidents into your chat, post a digest of overnight activity, or answer questions about what happened. There is no blast radius to worry about because it cannot change anything, so you can let it run wide and watch how it reasons. This stage alone is genuinely useful, and it tells you whether the agent’s judgement is sound before that judgement can touch anything.
Add the safe writes. Once you trust its reading, grant the low-risk actions: acknowledging an incident it has triaged, posting an update, adding a note to a case. These are reversible, low-stakes, and high-value — they take the busywork off your humans without putting anything at risk. Keep the scope to exactly these actions and the daily cap conservative; you can always widen it. Watch the audit log for a week and you will quickly see whether the agent acts the way you would.
Approach the consequential actions carefully — or not at all. Provisioning monitors from a deploy pipeline is a natural next grant because it is bounded and predictable. The genuinely consequential moves — anything that changes production behaviour — are deliberately kept as human-approved proposals rather than autonomous actions, and most teams are happy to leave them there. The lesson from everyone who has built ops automation is the same: the value is overwhelmingly in the investigation and the safe, reversible actions, and the risk is overwhelmingly in the irreversible ones. You can capture almost all of the value while taking almost none of the risk, simply by being deliberate about where on that line each token sits.
Throughout, the platform is on your side. Scopes cap what each agent can reach, daily limits cap how much it can do, the hard guardrails make the truly dangerous actions impossible for any token, idempotency keeps retries safe, and the audit log records every move for review. You are never trusting the agent blindly — you are trusting it within a box whose walls you set and the platform enforces. That is what makes building on 24Observe feel less like handing over the keys and more like hiring a junior who can only reach what you have shown them.
Mint a scoped token, grab the tool definitions, and your agent can investigate, summarise, and act — safely, idempotently, and fully audited.