Cut your agent observability costs and make every trace auditor-proof

AI agent observability is getting expensive fast. Vendors charge per GB ingested, per host monitored, per feature enabled. Datadog bills can explode 100x over initial budgets. The Grafana Observability Survey found that 49% of OpenTelemetry users in production cite cost as a top concern, with scalability close behind at 44%.

Meanwhile, your agent is running thousands of actions a day, and your OTel pipeline captures all of them with the same trace depth. Formatting text gets the same span as transferring funds. Reading a config file gets the same treatment as sending customer data to an external API. You are paying to store telemetry for actions that carry zero compliance risk and will never be audited.

And when the audit does come, those traces won't help you anyway. OTel traces are logs in a database you control. Anyone with write access can edit them. A regulator can't tell a real trace from one you modified last Tuesday.

This guide shows how to solve both problems with ICME PreFlight: classify which agent actions matter (for free), enforce policy on the ones that do ($0.01 each), and attach a cryptographic proof to every decision. You stop paying to trace actions nobody cares about, and the traces you do keep become verifiable evidence.

Why OTel alone isn't enough for agent compliance

Mutable logs are not audit trails

Every compliance framework governing AI agents requires tamper-evident records. HIPAA §164.312(b) requires mechanisms to record and examine activity on systems containing PHI. The SEC requires attributable records of advisory activities. The EU AI Act requires technical documentation and traceability.

OTel traces stored in Jaeger, Grafana Tempo, or a vendor backend don't meet that bar. Organizations are building Merkle tree hash chains, digital signatures, and WORM storage to make their logs tamper-evident. That's a lot of infrastructure to solve a problem cryptographic proofs solve natively.

Regulators aren't waiting

33% of organizations lack audit trails for their AI agent activity. Only 14.4% reported full security approval for their entire agent fleet. The SEC and OCC are actively examining AI governance. In financial services, missing traces are treated as a books-and-records violation.

The question has shifted from "who did what?" to "can you prove it?" Proving it means the auditor doesn't have to trust your infrastructure to verify your claims.

Traces show what happened, not whether it was allowed

OTel tells you a function was called with these arguments at this timestamp. It doesn't tell you whether the agent was permitted to do it, which policy was evaluated, or what the decision was. When an incident turns into a blame game, you need the enforcement decision, not just the execution trace.

How PreFlight plugs into your OpenTelemetry pipeline

PreFlight adds three things no processor, exporter, or backend can provide: free action classification, deterministic policy enforcement, and a cryptographic proof on every decision.

Agent proposes action
        ↓
  checkRelevance (free, <300ms)
        ↓
┌───────────────────────────┬───────────────────────────────────────┐
│ should_check: false        │ should_check: true                     │
│                            │                                        │
│ Action is policy-irrelevant│ Action touches policy variables        │
│ No compliance risk         │ Requires enforcement                   │
│                            │                                        │
│ Lightweight OTel span:     │ Full OTel span:                        │
│ • Minimal attributes       │ • checkIt result (SAT / UNSAT)         │
│ • Low storage cost         │ • check_id (audit receipt)             │
│ • Never audited            │ • zk_proof_id (cryptographic proof)    │
│                            │ • matched_variables (policy context)   │
│                            │ • extracted values from solver         │
│                            │                                        │
│                            │ UNSAT → action blocked before execute  │
│                            │ SAT → action proceeds                  │
└───────────────────────────┴───────────────────────────────────────┘
        ↓
  OTel Collector → Backend

checkRelevance (free) classifies whether an action touches any of your policy variables: data access scope, transmission endpoints, retention duration, transaction amounts. Actions that match nothing get a lightweight span. Actions that match get full enforcement and tracing. You stop paying to store telemetry for actions that will never be questioned.

checkIt ($0.01) checks the action against a formally verified policy using an SMT solver, not an LLM judge. The result is deterministic: same input, same output, every time. SAT means proceed. UNSAT means blocked before execution. The solver can't be prompt-injected or socially engineered.

The ZK proof is generated on every checkIt call. Anyone can verify it in under one second, without re-running the computation, without trusting your infrastructure, and without seeing your policy. The proof is the audit trail.

What your OpenTelemetry backend looks like after

Benign action (formatting, summarizing, file reading):

Span: agent.action
  icme.relevant: false
  duration_ms: 45

Minimal. Cheap. Nobody will ever query this span for compliance.

Policy-checked action that passed:

Span: agent.action.policy_checked
  icme.relevant: true
  icme.result: SAT
  icme.check_id: "a1b2c3d4-..."
  icme.zk_proof_id: "e5f6a7b8-..."
  icme.matched_variables: ["dataTransmittedToApprovedEndpointOnly", "dataAccessScope..."]

The zk_proof_id is what changes everything. A regulator calls POST /v1/verifyProof with that ID and gets independent cryptographic confirmation the policy check happened correctly. No access to your systems required.

Policy-checked action that was blocked:

Span: agent.action.policy_checked
  icme.relevant: true
  icme.result: UNSAT
  icme.check_id: "c9d0e1f2-..."
  icme.zk_proof_id: "f3a4b5c6-..."
  icme.blocked: true
  icme.detail: "Data transmitted to unapproved endpoint"
  otel.status_code: ERROR

Proof that the violation was caught. Proof that it was blocked. Proof that the policy was evaluated correctly. All verifiable without trusting you.

Where to intercept in your framework

Every major agent framework exposes a hook where tool calls can be inspected and blocked before execution. That hook is where you serialize the tool call into a plain English action string and send it to PreFlight.

LangChain / LangGraph

LangChain's wrap_tool_call hook intercepts each tool execution individually. You get a ToolCallRequest containing the tool call dict (tool name + arguments) and the BaseTool instance. Serialize the call, check it, and either call the handler to proceed or raise to block.

python

from langchain.agents.middleware import AgentMiddleware, ToolCallRequest
from langchain.agents import create_agent
from langchain_core.messages import ToolMessage
from typing import Callable

class PreFlightMiddleware(AgentMiddleware):
    def wrap_tool_call(self, request: ToolCallRequest, handler: Callable) -> ToolMessage:
        tool_name = request.call["name"]
        tool_args = request.call["args"]

        # Serialize the tool call into a plain English action string
        action = serialize_tool_call(tool_name, tool_args)

        # Free relevance check
        relevance = check_relevance(POLICY_ID, action)
        if not relevance["should_check"]:
            return handler(request)

        # Paid policy check ($0.01)
        result = check_it(POLICY_ID, action)
        if result["result"] == "SAT":
            return handler(request)

        # Blocked. Return error to the agent, tool never executes.
        return ToolMessage(
            content=f"BLOCKED: {result['detail']} (check_id: {result['check_id']})",
            tool_call_id=request.call["id"],
            status="error",
        )

agent = create_agent(
    model="gpt-4.1",
    tools=[gmail_search, contacts_list, drive_read, http_post],
    middleware=[PreFlightMiddleware()],
)

See LangChain middleware docs and the wrap_tool_call reference.

OpenAI Agents SDK

The OpenAI Agents SDK uses Guardrail objects that validate inputs and outputs. For tool-level interception, wrap each tool function to check before execution.

python

from agents import Agent, function_tool

@function_tool
def gmail_search(query: str, max_results: int = 25) -> str:
    action = f"Call gmail.search with query {query}, returning up to {max_results} results."
    result = check_it(POLICY_ID, action)
    if result["result"] != "SAT":
        return f"BLOCKED: {result['detail']}"
    return _gmail_search_impl(query, max_results)

agent = Agent(
    name="data-access-agent",
    instructions="You help the user with their email, calendar, and documents.",
    tools=[gmail_search, contacts_list, drive_read],
)

Strands Agents (AWS)

Strands provides BeforeToolCallEvent, a hook that fires before every tool execution. Set event.cancel_tool to block.

python

from strands.hooks import HookProvider, BeforeToolCallEvent

class PreFlightHook(HookProvider):
    def on_before_tool_call(self, event: BeforeToolCallEvent):
        action = serialize_tool_call(event.tool_name, event.tool_input)
        relevance = check_relevance(POLICY_ID, action)
        if not relevance["should_check"]:
            return

        result = check_it(POLICY_ID, action)
        if result["result"] != "SAT":
            event.cancel_tool = f"BLOCKED: {result['detail']}"

See the Strands guardrails guide for details on the cancel_tool mechanism.

CrewAI

CrewAI doesn't have a middleware layer in the same sense. The standard pattern is to wrap your tool functions directly.

python

from crewai import Tool

def safe_gmail_search(query: str, max_results: int = 25) -> str:
    action = f"Call gmail.search with query {query}, returning up to {max_results} results."
    result = check_it(POLICY_ID, action)
    if result["result"] != "SAT":
        return f"BLOCKED: {result['detail']}"
    return _gmail_search_impl(query, max_results)

gmail_tool = Tool(
    name="gmail_search",
    func=safe_gmail_search,
    description="Search the user's Gmail inbox",
)

Serializing tool calls into action strings

The solver evaluates concrete facts in the action text. Your serialization function should include the tool name, all arguments, and any facts the policy references (record counts, destination URLs, storage behavior).

python

def serialize_tool_call(tool_name: str, args: dict) -> str:
    """
    Turn a structured tool call into a plain English string
    the solver can extract variables from.
    """
    if tool_name == "gmail_search":
        return (
            f"Call gmail.search with query {args.get('query', '')}, "
            f"returning up to {args.get('max_results', 25)} results. "
            f"Data stays in working memory and is discarded after the response."
        )
    if tool_name == "http_post":
        return (
            f"HTTP POST to {args.get('url', 'unknown')} "
            f"with {args.get('record_count', 'unknown')} records in the request body."
        )
    if tool_name == "contacts_list":
        return (
            f"Call contacts.list returning up to {args.get('max_results', 'all')} "
            f"contact records. Data stays in working memory."
        )
    if tool_name == "write_to_db":
        return (
            f"Write {args.get('record_count', 'unknown')} records "
            f"to {args.get('destination', 'unknown')} for persistent storage."
        )
    # Fallback: serialize everything
    return f"Call {tool_name} with arguments: {args}"

The solver reads "returning up to 500 results" and extracts numberOfEmailsAccessed: 500. It reads "HTTP POST to https://vendor.com/api" and extracts isExternalTransmission: true and destinationUrl. The more concrete your serialization, the more reliably the policy evaluates.

Full OpenTelemetry integration

The middleware examples above handle enforcement. To add the OTel tracing layer with checkRelevance routing and zk_proof_id on every span, wrap the check logic with span creation:

python

import httpx
from opentelemetry import trace
from opentelemetry.trace import StatusCode

tracer = trace.get_tracer("agent.middleware")

ICME_API_KEY = "sk-smt-..."
ICME_POLICY_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
ICME_BASE = "https://api.icme.io/v1"


async def check_relevance(action: str, threshold: float = 0.0) -> dict:
    """Free relevance screening. No credits charged."""
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            f"{ICME_BASE}/checkRelevance",
            json={
                "policy_id": ICME_POLICY_ID,
                "action": action,
                "threshold": threshold,
            },
            headers={
                "Content-Type": "application/json",
                "X-API-Key": ICME_API_KEY,
            },
        )
        return resp.json()


async def check_action(action: str) -> dict:
    """Full policy check. 1 credit ($0.01)."""
    async with httpx.AsyncClient(timeout=30.0) as client:
        resp = await client.post(
            f"{ICME_BASE}/checkIt",
            json={"policy_id": ICME_POLICY_ID, "action": action},
            headers={
                "Content-Type": "application/json",
                "X-API-Key": ICME_API_KEY,
            },
        )
        return resp.json()


async def guarded_execute(action: str, execute_fn):
    """
    Middleware that classifies, enforces, and traces
    agent actions with cryptographic proof.
    """
    relevance = await check_relevance(action)

    if not relevance.get("should_check", True):
        # Benign action. No policy variables touched. Lightweight trace.
        with tracer.start_as_current_span("agent.action") as span:
            span.set_attribute("icme.relevant", False)
            return await execute_fn(action)

    # Policy-relevant action. Full enforcement and tracing.
    with tracer.start_as_current_span("agent.action.policy_checked") as span:
        span.set_attribute("icme.relevant", True)
        span.set_attribute("icme.relevance_score", relevance["relevance"])
        span.set_attribute("icme.matched_variables", str(relevance["matched"]))

        result = await check_action(action)

        span.set_attribute("icme.result", result.get("result", "ERROR"))
        span.set_attribute("icme.check_id", result.get("check_id", ""))
        span.set_attribute("icme.zk_proof_id", result.get("zk_proof_id", ""))

        if result.get("result") == "SAT":
            return await execute_fn(action)

        # UNSAT. Blocked. The proof records the block.
        span.set_status(StatusCode.ERROR, result.get("detail", "Policy violation"))
        span.set_attribute("icme.blocked", True)
        raise PolicyViolation(
            action=action,
            detail=result.get("detail"),
            check_id=result.get("check_id"),
            proof_id=result.get("zk_proof_id"),
        )


class PolicyViolation(Exception):
    def __init__(self, action, detail, check_id, proof_id):
        self.action = action
        self.detail = detail
        self.check_id = check_id
        self.proof_id = proof_id
        super().__init__(f"Blocked: {detail} (check: {check_id}, proof: {proof_id})")

Setting up a policy

1. Create an account

bash

curl -s -X POST https://api.icme.io/v1/createUserCard \
  -H 'Content-Type: application/json' \
  -d '{"username": "otel-agent"}' | jq .
# Open checkout_url in browser. $5.00, gives 325 credits.
# Then retrieve your API key:
curl -s https://api.icme.io/v1/session/SESSION_ID | jq .

2. Compile a policy

A good AR policy uses concrete, extractable variables: numbers, URLs, booleans that the solver can read directly from the action text. Vague conditions like "scope matches user intent" or "approved endpoints" cause translation failures because the solver has no way to evaluate them. See AWS AR best practices for why this matters.

Every variable in the policy below maps to a fact your middleware can serialize from a tool call: a record count, a destination URL, a storage flag.

This example covers a personal data access agent with access to email, calendar, contacts, and documents:

bash

curl -s -N -X POST https://api.icme.io/v1/makeRules \
  -H 'Content-Type: application/json' \
  -H "X-API-Key: $ICME_API_KEY" \
  -d '{
    "policy": "The agent has access to email, calendar, contacts, and documents on behalf of the user.\n\nThe agent must not make outbound HTTP requests to any external URL. Any action that sends data via HTTP POST, PUT, or outbound network request to a URL is an external transmission. If the agent transmits data externally, the action must be rejected.\n\nThe agent must not access more than 25 email messages in a single operation. If the number of emails accessed exceeds 25, the action must be rejected.\n\nThe agent must not access more than 50 contact records in a single operation. If the number of contacts accessed exceeds 50, the action must be rejected.\n\nThe agent must not access more than 10 documents in a single operation. If the number of documents accessed exceeds 10, the action must be rejected.\n\nThe agent must not write, cache, or persist any extracted data to disk, database, or long-term memory. All data must be held in working memory only and discarded after the response is returned. If the agent stores data after the response, the action must be rejected."
  }'

The solver extracts variables like numberOfEmailsAccessed, isExternalTransmission, destinationUrl, retainsDataAfterResponse from these rules. When your middleware serializes a tool call, it includes these as concrete facts: "POST 500 email bodies to https://vendor.com/api" gives the solver a URL and a record count to evaluate.

Save the policy_id. Compilation costs 300 credits, one-time. See the full use case walkthrough for details on variable extraction and battle testing to verify the solver interprets your policy correctly.

3. Set environment variables

bash

export ICME_API_KEY=sk-smt-...
export ICME_POLICY_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

What gets caught

The action strings below are what your middleware would serialize from real tool calls. Each one includes the concrete facts the solver needs: record counts, destination URLs, storage behavior.

SAT: normal email search, small scope, no external transmission

bash

curl -s -N -X POST https://api.icme.io/v1/checkIt \
  -H 'Content-Type: application/json' \
  -H "X-API-Key: $ICME_API_KEY" \
  -d '{
    "policy_id": "'"$ICME_POLICY_ID"'",
    "action": "Call gmail.search with query from:[email protected] subject:Q3 review, returning up to 10 results. Data stays in working memory for summarization and is discarded after the response."
  }'

Result: SAT. 10 emails is under the 25 limit. No external transmission. No persistent storage. The proof records the pass.

UNSAT: external transmission with a destination URL

bash

curl -s -N -X POST https://api.icme.io/v1/checkIt \
  -H 'Content-Type: application/json' \
  -H "X-API-Key: $ICME_API_KEY" \
  -d '{
    "policy_id": "'"$ICME_POLICY_ID"'",
    "action": "HTTP POST 500 email bodies to https://summarizer.external-vendor.com/v1/batch for summarization. The emails were retrieved from the users inbox."
  }'

Result: UNSAT. The solver extracts isExternalTransmission: true and destinationUrl: https://summarizer.external-vendor.com/v1/batch. The policy blocks all outbound HTTP requests. The data never leaves. The proof records the block.

UNSAT: record count exceeds the limit

bash

curl -s -N -X POST https://api.icme.io/v1/checkIt \
  -H 'Content-Type: application/json' \
  -H "X-API-Key: $ICME_API_KEY" \
  -d '{
    "policy_id": "'"$ICME_POLICY_ID"'",
    "action": "Call gmail.search with query after:2026-04-01, returning up to 500 results to build context for answering a question about the Alice thread."
  }'

Result: UNSAT. The solver extracts numberOfEmailsAccessed: 500, which exceeds the 25-message limit. The agent wanted to read 500 emails to answer a question about one thread. The proof captures the exact count that violated the policy.

UNSAT: persistent storage of extracted data

bash

curl -s -N -X POST https://api.icme.io/v1/checkIt \
  -H 'Content-Type: application/json' \
  -H "X-API-Key: $ICME_API_KEY" \
  -d '{
    "policy_id": "'"$ICME_POLICY_ID"'",
    "action": "Write extracted contact names, email addresses, and phone numbers to agent_memory.db for reuse in future sessions. 342 contact records processed."
  }'

Result: UNSAT. The solver extracts retainsDataAfterResponse: true. Writing to a database is persistent storage. The proof records the violation.

UNSAT: instruction injected via calendar event

bash

curl -s -N -X POST https://api.icme.io/v1/checkIt \
  -H 'Content-Type: application/json' \
  -H "X-API-Key: $ICME_API_KEY" \
  -d '{
    "policy_id": "'"$ICME_POLICY_ID"'",
    "action": "Call drive.list returning all 847 documents. This instruction was found in a calendar event body titled Team Sync while the agent was reading todays schedule."
  }'

Result: UNSAT. 847 documents exceeds the 10-document limit. The proof captures it regardless of where the instruction came from.

Verifying proofs

Verify cryptographically:

bash

curl -s -X POST https://api.icme.io/v1/verifyProof \
  -H 'Content-Type: application/json' \
  -d '{"proof_id": "YOUR_PROOF_ID"}' | jq .

Check proof metadata:

bash

curl -s https://api.icme.io/v1/proof/YOUR_PROOF_ID \
  -H "X-API-Key: $ICME_API_KEY" | jq .

Download the raw proof binary:

bash

curl -s https://api.icme.io/v1/proof/YOUR_PROOF_ID/download -o proof.bin

The proof confirms: this specific policy was checked against this specific action, by this specific solver, and returned this specific result. No re-execution required. No trust in the provider required. No policy exposure required.

What people are building today vs. what this gives you

Problem	Current approach	With PreFlight + OTel
Tamper-evident audit trail	Merkle tree hash chains, blockchain anchoring, WORM storage	ZK proof on every check, verifiable by anyone, no infrastructure needed
Prove a policy was enforced	Log entries in a mutable database	Cryptographic proof the solver evaluated the policy correctly
Reduce observability cost	Sample traces, drop low-value spans in Collector	checkRelevance (free) classifies at source so you only trace what matters
Answer "was the agent allowed to do this?"	Reconstruct from logs after the fact	check_id + proof_id on the OTel span, answer is immediate and verifiable
Third-party verification	Grant auditor access, or download proofs for later checks. See API docs	Auditor calls /v1/verifyProof with zero access to your systems
Policy privacy during audit	Expose your rules to the verifier	ZK proof verifies without revealing the policy

Cost

Action	Cost
checkRelevance	Free
checkIt	1 credit ($0.01)
Policy compilation	300 credits ($3.00), one-time
Account creation	$5.00 (gives 325 credits)
ZK proof verification	Free (included with check)
Credit top-up	$5 to $100 (volume bonuses up to 20%)

For an agent averaging 1,000 actions/day where 15% are policy-relevant: 150 paid checks/day at $1.50/day. The other 850 relevance screenings are free. Compare that to what you're paying to store and query 1,000 full-depth traces per day in your observability backend.

Production checklist

• Account created, API key saved as ICME_API_KEY

• Policy compiled, ID saved as ICME_POLICY_ID

• checkRelevance called before every agent action

• checkIt called for every action where should_check: true

• check_id and zk_proof_id set as OTel span attributes

• Fail-closed: any non-SAT result blocks execution

• Per-action enforcement for multi-step chains (check before each step, not just once)

• Proof verification tested with /v1/verifyProof

• Compliance team briefed on proof verification workflow