Marketing Accelerant is an AI marketing analytics platform I worked on for a client. It runs 15+ specialized LLM agents covering Brand Voice, Creative Content, CMO Strategy, SEO, Email Campaigns, Google Ads, Meta Ads, Video Studio and more. All of them serve enterprise clients through one FastAPI backend.

The agents were not the hard part. The hard part was everything around them: model selection, context management, cost control, error recovery and human approval. This post walks through the middleware that makes that work in production.

The middleware stack

Every agent runs through a composable middleware chain, built per request from a set of flags:

def build_agent_middleware(
    configurable: dict | None = None,
    *,
    agent_slug: str | None = None,
    include_summarization: bool = True,
    include_todo: bool = True,
    include_tool_selector: bool = False,
    include_approval: bool = True,
    include_retry: bool = True,
    include_error_handler: bool = True,
    include_loop_guard: bool = False,
    # ...
) -> list:

Order matters. This is the full chain, top to bottom:

  1. Runtime model selection: picks the LLM provider from the request config.
  2. Workflow middleware: manages agent-specific workflow state.
  3. Auto-summarization: compresses context when it gets too long.
  4. Todo list: tracks progress on multi-step tasks.
  5. Tool selector: narrows 100+ tools down to the 24 most relevant.
  6. Model retry: retries transient failures such as rate limits and timeouts.
  7. Model call limit: caps a run at 8 LLM calls.
  8. Prompt caching: one middleware per provider, for Anthropic, OpenAI, Google and Bedrock.
  9. Tool loop guard: detects and breaks tool call loops.
  10. Research tool limits: per-tool call caps. KB search gets 4, web search 4, deep research 2.
  11. Human-in-the-loop: approval gates for destructive tools.
  12. Error handler: catches tool failures and recovers.
  13. Tool contract enforcement: checks that tool inputs and outputs match their contracts.

Each middleware handles one concern. An agent opts in or out with flags on its class:

class FrameworkMarketingAgent(BaseAgent[T]):
    include_todo_middleware = True
    include_tool_selector = False
    include_brand_voice = True
    include_approval_middleware = True
    include_loop_guard = False
    tool_selector_always_include: list[str] = []

The CMO Strategy agent turns the tool selector on because it orchestrates other agents and needs access to many tools. The Brand Voice agent turns it off; it only needs KB search and content generation. Each agent gets the middleware it needs and nothing else.

Request-scoped model selection

The platform supports OpenAI, Anthropic, Google Gemini and AWS Bedrock. The model is chosen at request time, so one agent can run on Claude for one client and GPT-4 for another depending on their configuration.

The first middleware in the chain, runtime_model_selection_middleware, reads the request config and injects the right LLM. Every middleware after it sees the model selected for that specific request. There is no global state and there are no singletons.

This was painful to build. Each provider has its own API shape, token counting and streaming behavior. In return, we can:

  • Route simple tasks to cheaper models. Summarization uses a utility model at temperature 0.3.
  • Let enterprise clients bring their own API keys.
  • Fall back to another provider when one is down.

Auto-summarization at 120K tokens

Long conversations eat context windows. These agents can run for dozens of turns full of tool calls, research results and user feedback. Left alone, the conversation hits the context limit and the agent crashes.

The summarization middleware fires on its own:

TrackingSummarizationMiddleware(
    model=utility_model,  # cheap model, temperature 0.3
    trigger=[("tokens", 120_000), ("messages", 100)],
    keep=("messages", 20),
    trim_tokens_to_summarize=32_000,
)

When the conversation reaches 120K tokens or 100 messages, whichever comes first, it:

  1. Keeps the 20 most recent messages intact.
  2. Takes up to 32K tokens of older messages.
  3. Summarizes them with the utility model.
  4. Replaces the old messages with the summary.

The 120K threshold is about 70% of the smallest context window we support, which is Haiku's 200K. That leaves room for the system prompt, the tools and the next response without risking an overflow.

The tracking part of TrackingSummarizationMiddleware records when summarization fired, how many tokens it compressed and how much context it kept. We use that to debug quality problems when an agent forgets something from earlier in the conversation.

Tool selector: 100+ tools, 24 per request

The platform has over 100 tools: knowledge base search, web search, URL fetching, analytics queries, email sending, ad management, content generation, calendar scheduling and more. Handing every tool to every agent is a mistake. The LLM spends tokens reading descriptions of tools it will never call, and with a menu that large it sometimes picks the wrong one.

RequestScopedToolSelectorMiddleware uses a lightweight classifier LLM to pick the 24 most relevant tools for each request:

RequestScopedToolSelectorMiddleware(
    agent_slug="cmo",
    model=create_classifier_llm_for_selection(selection),
    max_tools=max(24, len(always_include) + 8),
    always_include=["knowledge_base_search", "web_search"],
)

Some tools, like KB search, are always included. The rest are chosen from the agent type and the user's message. Ask the CMO agent about campaign performance and it gets analytics and reporting tools. Ask the same agent about brand strategy and it gets content and research tools.

This cut irrelevant tool calls by roughly 40% and reduced tokens spent on tool descriptions by about 60%.

Human-in-the-loop with spend warnings

Some tools are destructive: sending emails, publishing ads, modifying campaigns. They need human approval before they run:

HumanInTheLoopMiddleware(
    interrupt_on={
        tool_name: {
            "allowed_decisions": ["approve", "edit", "reject"],
            "description": _approval_description,
        }
        for tool_name in TOOLS_REQUIRING_APPROVAL
    },
)

When a tool involves money, such as ad spend or email sends, the approval prompt includes a spend warning. The user can approve the call as is, edit its arguments or reject it. This is LangGraph's interrupt() pattern. The graph pauses, sends the tool call to the frontend and resumes when the user responds.

Error recovery that does not retry blindly

The usual approach to tool errors is to retry three times and hope. That works for network glitches. It is terrible for business logic errors; you do not want to retry sending a malformed email.

We use contract-aware retries instead. The enforce_tool_contracts middleware validates tool inputs against their Pydantic schemas before execution and sorts errors into retriable ones, like network failures and rate limits, and non-retriable ones, like validation and auth failures. Only retriable errors are retried, with exponential backoff starting at 750ms.

DEFAULT_MODEL_RETRY = ModelRetryMiddleware(
    max_retries=2,
    retry_on=_should_retry_model_error,
    on_failure="continue",  # don't crash the agent
    initial_delay=0.75,
    max_delay=8.0,
)

on_failure="continue" matters. If every retry fails, the agent receives an error message and decides what to do next: try another approach, ask the user or report the failure. The conversation does not crash.

Why 15 agents

The first version had a single general-purpose agent, and it was bad. It wrote ad copy when asked for analytics. It started a research workflow when the user wanted a quick answer. Its system prompt was 4,000 tokens of instructions trying to cover every case.

Splitting it into specialized agents fixed that:

  • Each agent has a focused system prompt of 200-500 tokens, where a single agent doing everything needed 4,000.
  • Tool selection is scoped per agent.
  • Persona and formatting rules belong to each agent. The SEO agent outputs structured audits; the Creative agent writes prose.
  • Failures stay isolated. A bug in the Email agent does not break Brand Voice.

Routing happens at the API layer. The frontend knows which agent to call from the conversation type. We do not use an orchestrator agent that routes to sub-agents. Direct routing is faster and easier to debug.

What I would do differently

Starting over, I would:

  1. Build the middleware stack first. We bolted middleware onto existing agents over months. Treating it as a first-class abstraction from day one would have saved a lot of refactoring.
  2. Add structured logging earlier. Debugging a 15-agent system with print() statements does not scale. We added structured JSON logging with request correlation IDs only after too many production debugging sessions that took hours.
  3. Skip the orchestrator agent. The temptation is strong. Resist it. Direct routing with a good middleware stack is simpler and more predictable than an LLM deciding which LLM to call.