The Delta guide

Delta Agent Harness#

Delta is a lean runtime for building production agents that do durable knowledge work. It gives one model call a workspace, tools, memory, delegation, human review hooks, tracked-usage budgets, durable execution, and production visibility without turning the harness into a framework.

This is the canonical Delta guide. It is self-contained and written for both humans and language models. Start with the quickstart, then use the later sections as the exact operating reference.

Choose a path:

• Create and test a local agent
• Build an agentic assistant or AI feature
• Give a coding agent the right Delta context
• Integrate through the CLI or HTTP API
• Understand tools, MCP, memory, and review
• Deploy and operate in production
• Look up every setting

This guide describes the source checkout that contains it. A running daemon reports its harness version at /healthz. Delta reads DELTA_BUILD at runtime. The Docker build argument persists it into the image environment; a direct binary deployment must export it when starting the daemon.

A Delta agent is three things:

1. Engine: the compiled delta binary. It owns the model loop, tool execution, persistence, budgets, events, and HTTP API.
2. Bundle: five small files that define one agent. They hold its identity, policy, dynamic context, product vocabulary, and local configuration.
3. State: the workspace plus a SQLite database. They hold files, conversations, checkpoints, memory, events, and self-file revisions.

Delta is designed for work that crosses files, tools, systems, and people. It is not limited to a source repository. When a task genuinely needs advanced coding, Delta can delegate that part to an installed Codex, Claude Code, or other command-line coding agent.

Start here#

Prerequisites#

• Bun 1.3 or newer
• A model credential, such as an OpenRouter, Anthropic, or OpenAI API key
• Optional: an Exa key for built-in web search
• Optional: an installed coding CLI for the code tool
• Optional: the host grep command for the grep tool
• Optional: the host unzip command for lightweight DOCX and XLSX extraction

Delta installs three ways: curl -fsSL https://deltaharness.dev/install.sh | sh, bunx @carrara-labs/delta-harness, or a prebuilt binary from GitHub Releases. To build from source, clone the repository. Commands in this guide use ./dist/delta ... from a source checkout; if you installed the binary, substitute delta .... All commands run from the repository root unless stated otherwise.

Build the binary#

bun install --frozen-lockfile
bun run build

The build creates dist/delta, a Bun-independent executable for the current operating system and architecture. Build on the target platform, or use the Docker build for Linux. Optional built-ins can still call external grep, unzip, and coding CLI executables. During repository development, run the test suite with:

bun test

Create an agent#

./dist/delta init ./my-agent

delta init attempts to create these five scaffold files with exclusive writes:

my-agent/
├── delta.env
├── vocab.json
├── DELTA.md
├── POLICY.md
└── PROMPT_CONTEXT.md

It never overwrites an existing path. In a partially scaffolded directory it creates only the missing files. A permissions, disk, or path failure can leave a partial scaffold and returns a nonzero exit status; rerun after fixing the error to fill the remaining files.

Set a model credential and a stable agent ID in my-agent/delta.env:

OPENROUTER_API_KEY=your-key
DELTA_MODEL_PRIMARY=anthropic/claude-sonnet-5
DELTA_AGENT_ID=my-research-agent

The agent ID must remain stable for the life of the agent. Local memory is scoped by it. delta.env is a local-development convenience inside the agent workspace, and file tools can read workspace files. Use a restricted development key, never commit this file, and keep production secrets in the process environment or a secret manager. Add at least these paths to the repository ignore rules:

my-agent/delta.env
my-agent/.delta/

Give the agent a concrete identity in my-agent/DELTA.md:

# Persona

You are Meridian, a research operator for the product team.

# Mission

Turn ambiguous market questions into sourced, decision-ready briefs.

# Success

The reader can make a decision without repeating the research.

# Learned

Keep recommendations explicit and separate evidence from inference.

POLICY.md can remain as generated for the first run. Its comment-only starter causes Delta to use the embedded review policy when a reviewed-write MCP tool is present.

Launch the agent and Cockpit#

./dist/delta dev ./my-agent --port 8080

With the generated configuration, the launcher:

• reads my-agent/delta.env
• uses my-agent as the workspace, unless delta.env sets DELTA_WORKSPACE
• stores local state in my-agent/.delta/delta.db, unless delta.env sets DELTA_DB
• defaults the daemon bind to 127.0.0.1
• defaults local Cockpit editing and normalized model-call capture to on
• opens http://localhost:8080/dev

delta.env overlays the launcher's inherited environment. DELTA_BIND, DELTA_INSPECT_WRITE, and DELTA_CAPTURE_CALLS from that merged environment override their development defaults. --port, or the automatically selected free port, always replaces PORT. For workspace and database paths, only values in delta.env override the launcher defaults; inherited DELTA_WORKSPACE and DELTA_DB values are ignored. Relative overrides resolve from the bundle directory.

Do not set a public DELTA_BIND for local development unless DELTA_CONTROL_TOKEN and a deliberate inspection policy are also configured. Otherwise the run API can become reachable beyond the local machine. The bundled browser UI does not attach those tokens, so a token-protected local Cockpit also needs an authorization-injecting proxy.

Omit --port 8080 to let Delta select a free port. Keep the browser closed when needed:

./dist/delta dev ./my-agent --port 8080 --no-open

The terminal prints the selected port. delta dev runs the ordinary production daemon as a child, so the execution behavior is the same as a directly launched Delta.

Send the first task#

In another terminal:

./dist/delta send --port 8080 "Using only DELTA.md and POLICY.md, create brief.md with the agent mission, success criteria, risks, and next steps."

The default delta send path creates an asynchronous task, then prints the run's dev event stream as JSON Lines until it finishes. It needs the control credential to create the task and the inspection credential to read that stream when those gates are enabled. For only the final text:

./dist/delta send --port 8080 --json "Read brief.md and summarize it in five bullets."

The --json name is historical. This mode makes a synchronous request and prints only output_text; it needs only the control credential. delta send and delta watch read credentials from their own process environment and do not load the bundle's delta.env.

At this point the agent is real: it can reason across multiple model turns, use files and tools, persist every checkpoint, continue a conversation, and be inspected in the Cockpit.

Develop and acceptance-test the agent#

Treat the five-file bundle as code and keep one fixed acceptance task for the agent:

1. Start Delta on a pinned port and send the task without previous_response_id, which creates a fresh session.
2. In Cockpit, inspect the run timeline, model calls, selected tools, tool arguments, results, usage, and final answer.
3. Verify the expected workspace artifact, budget behavior, policy behavior, and a deliberate failure case such as a missing input file.
4. Edit the smallest relevant bundle file. DELTA.md takes effect on the next run; policy, vocabulary, all prompt context, MCP, and provider changes require a daemon restart.
5. Run the identical task in another fresh session and compare the artifact, tool choices, cost, and call trace.
6. Before release, repeat the task through the authenticated production API and inspect the production run with the separate inspection credential.

bun test verifies the Delta engine. This repeatable task verifies the particular agent bundle and its connected systems.

Use Delta with coding agents#

The documentation exposes three layers so Claude Code, Cursor, Codex, and similar tools can receive only the context they need:

1. Immediate handoff: use Copy agent context from the documentation header menu, then paste it into the coding agent with your task. The copied brief explains Delta's integration shapes, current source-checkout workflow, bundle contract, safety rules, and authoritative URLs.
2. Focused retrieval: point an agent to /llms.txt for a concise index, /guide.md for the canonical guide, or /llms-full.txt when a tool explicitly needs the complete corpus.
3. Persistent project context: place a short, relevant Delta section in the target repository's root AGENTS.md, or install the official Delta Skill once it ships. A hosted AGENTS.md is a source to copy from; coding agents only load it automatically after it is present in the repository they are working in.

The header's Copy .md action copies the complete current guide. Use that for a full page handoff. Prefer the smaller agent context for routine implementation work so the coding agent can retrieve only the relevant guide sections instead of carrying the entire manual in every turn.

Delta ships as the npm package @carrara-labs/delta-harness, installable via install.sh, bunx, or a prebuilt release binary; the brief points agents at these alongside the source-checkout commands in this guide. Still planned: a portable SKILL.md and a small installer that can add or update a bounded Delta section in the target repository's AGENTS.md. MCP is only necessary later if agents need live registry search or operational actions; static product knowledge does not require it.

Mental model#

What one run does#

Every request follows the same durable loop:

request
  -> durable queue row
  -> load identity, policy, context, memory, and relevant capabilities
  -> model call
  -> zero or more parallel tool calls
  -> checkpoint messages, usage, tool intents, and active tools
  -> repeat until final text, cancellation, budget exhaustion, or failure
  -> on a handled outcome, durable response and run.finished event
  -> optional background reflection

A run may last for seconds or hours. Delta has no overall wall-clock limit. Its loop stops on successful-main-model-call, recorded fresh-token, and recorded model-cost budgets. Individual model and ordinary tool calls have separate timeouts, with the exceptions documented under built-in capabilities.

The prompt spine#

Delta rebuilds a small, ordered system spine for every model turn:

1. Delta's base behavior and operating norms
2. boot-stable rendered context
3. DELTA.md, read once at the start of the current execution attempt
4. POLICY.md, rendered as the last instruction layer
5. an index of currently resident tool names and descriptions

The full JSON tool schemas are sent separately in the provider request's tools field. Conversation history follows the system spine. Per-turn context, request instructions, and retrieved capabilities are appended as user-role blocks. This matters because request-scoped text can specialize a task, but it does not become system policy.

The stable prefix is deliberately cache-friendly. Volatile values such as time and request metadata stay outside it. {{model}} is the configured primary on the first turn and the model that served the previous successful turn after that, so it cannot predict a fallback that will serve the call currently being assembled.

Sessions and runs#

One API response ID identifies one run. Passing that ID as previous_response_id appends another run to the same session and preserves the full current conversation. It is a session lookup, not a branch pointer: passing an older response ID does not fork or rewind the conversation, so pass the latest response ID. Runs in one session execute serially. The shipped daemon executes up to DELTA_MAX_CONCURRENCY different sessions concurrently (default 8, clamped 1–256; see Run and call the agent).

Do not invent or reuse an unknown response ID. Delta returns 400 instead of silently starting a new session.

Two ways to build#

Delta is one runtime, but products put it to work in two shapes. Both are the same agent with the same primitives - threads for context, tools for reach, and review-driven reflection for learning. They differ only in who opens the thread and where the human gives feedback.

• An agentic assistant. A person chats with the agent. Each topic or task is its own thread; the agent uses tools and permissioned app access to do real work, and gets better from how its proposals are received.
• An agentic feature. A product turns a one-shot LLM call - "generate a job description," "draft the outreach," "summarize the deal" - into an agent that iterates to a reviewable deliverable and improves across generations. The feature is a pre-seeded thread: the same agent, opened by the product with task context instead of by a person with a chat message.

The rest of this guide is the reference for every primitive named here. This section shows how they compose into each shape.

A thread is the unit of context#

A thread is a Delta session (see Sessions and runs). Open one with a first POST /v1/responses; continue or switch by passing that thread's latest response ID as previous_response_id. Everything that makes a long task survivable - compaction, recall, the working plan - is scoped to the thread, so two threads never bleed into each other. A person managing several threads, or a product running one thread per job, gets clean isolation without extra work. Your product owns the list of threads and their titles; Delta owns each thread's execution and memory.

Learning spans threads; context does not#

The isolation above is deliberate, and it is only half the design. The learning rail is the other half, and it runs the opposite way: a lesson distilled in one thread becomes available to every later thread of the same kind. Tag related work with a caller-declared task_type (see Request fields) - jd-generation, weekly-report - and reflection tiers what it learns to that use case. Generation fifty opens a fresh, isolated context but recalls what generations one through forty-nine taught the agent about the job. Isolation where you want a clean slate; memory where you want compounding.

Use case 1: an agentic assistant#

Give a person a chat surface backed by one agent.

• Threads for context. One thread per topic or task; the user switches between them and each keeps its own working context.
• Tools and permissioned app access. Connect the agent to your systems over MCP - retrieve contacts, read a CRM, draft in a document store - under whatever scopes your gateway enforces. Delta calls the tools; your product owns identity and permission.
• Self-improvement from review. When the agent proposes work and a human accepts, edits, or rejects it, that verdict becomes a learning signal (see Learn from human review). The agent's DELTA.md and scoped memory improve without a prompt change.

Use case 2: an agentic feature#

Turn a static feature into an agent that does the whole job and learns from every correction. The mechanism is the same review loop, opened by your product instead of by a person. Job-description generation, end to end:

1. Open a thread per job. POST /v1/responses with no previous_response_id, seeded with the source material and tagged with the feature's task_type:

   jsonCopy

   {
     "input": "Draft a JD and specs for the Staff PM role. Hiring-manager transcript: ... Recruiter transcript: ...",
     "metadata": { "task_type": "jd-generation", "user_id": "acme" }
   }
   

2. Let it work and iterate. The agent pulls context, drafts, and proposes; continue the thread with previous_response_id as the hiring manager reacts. This job's transcripts and drafts stay in this thread.

3. Propose, do not just answer. Delta's write rail bundles the deliverable into one proposal a human reviews (see POLICY.md and Learn from human review), stamped back to the run.

4. Feed the verdict back. When the reviewer accepts with edits, your control plane sends the disposition into the same thread as a new run:

   jsonCopy

   {
     "input": "Review outcome: accepted with edits. Proposed: ... Accepted: ... Reviewer note: lead with scope, cut adjectives.",
     "metadata": { "review_kind": "submission_disposition", "submission_id": "sub_123", "reflect": true }
   }
   

5. It learns from the diff, and the next job starts smarter. Reflection grounds the lesson in the gap between what was proposed and what was accepted, tiers it to jd-generation, and the next job's thread recalls it at open.

What Delta owns and what you own#

Delta owns the loop, threads, context management, reflection, and the memory tiers. Your product - or a control plane - owns opening and titling threads, the review surface, and, critically, authenticating the disposition and constructing the real proposed-versus-accepted digest. Delta trusts that digest; it does not verify the reviewer, so the trusted plane must be the one asserting that a human reviewed the work.

The learning loop ships wired to a submission-and-review model: an agent that proposes through the review inbox and is reviewed there gets the propose-review-learn cycle today. A feature on a different product surface reuses every harness primitive but supplies its own equivalent of that disposition digest and sender.

Configure the five-file bundle#

delta.env: local launch configuration#

delta dev parses this file as trimmed KEY=value pairs. Blank lines and full lines beginning with # are ignored. One matching pair of outer single or double quotes is removed. It is not a shell parser: export, variable expansion, escape processing, and inline-comment removal are not supported, and malformed lines are silently skipped.

The direct daemon does not automatically load delta.env. In production, pass the same settings as real environment variables. The container entrypoint can seed the four non-secret bundle files, but never delta.env.

Keep credentials out of version control. A useful local file starts small:

OPENROUTER_API_KEY=your-key
DELTA_MODEL_PRIMARY=anthropic/claude-sonnet-5
DELTA_AGENT_ID=my-research-agent

# Optional built-in web search
EXA_API_KEY=your-exa-key

# Optional post-run learning
DELTA_REFLECT=1

DELTA.md: identity and writable self-memory#

DELTA.md is the agent's compact, durable self-file. The generated structure is:

# Persona

Who the agent is and who it serves.

# Mission

The standing goal it exists to advance.

# Success

The concrete outcome that defines good work.

# Learned

Durable lessons worth carrying into every future run.

The file is human-editable and agent-editable. The model can replace it through the remember tool. A content-changing overwrite is atomic and size-capped; since 0.2.15 a refusal for size tells the agent exactly how many bytes over it is and hands back the current on-disk file as a merge base, so an over-cap save resolves in one informed retry instead of blind compression attempts that end in the repeat-failure breaker. When prior content exists, Delta records it in SQLite before retaining the newest 20 previous revisions. The first write and a same-content write do not create a revision. The Cockpit can compare and restore recorded revisions. The filesystem rename happens just before the database snapshot, so a process crash in that narrow interval can leave the new file in place without a revision for the immediately prior content.

Important behavior:

• Delta reads one snapshot of DELTA.md for each execution attempt. An uninterrupted run keeps that snapshot. A run resumed after a daemon restart loads the current file again.
• A change made during uninterrupted execution becomes visible to the next fresh or restarted execution attempt, not the current in-memory attempt.
• The default budget is about 800 tokens, implemented as a 3,200-byte write ceiling.
• A file above the prompt budget but no larger than 1 MB is head-and-tail elided as a recovery behavior. A file above 1 MB loads as empty, and an oversized self-write is rejected by its byte cap.
• The agent must write the complete replacement file, not only a new lesson.

Keep this file lean. Every future model call includes it.

POLICY.md: fixed operator rules#

POLICY.md is the fixed, highest-priority prompt guidance for a daemon process. It is loaded once when the daemon starts, placed after writable identity as the last instruction layer before the tool index, and cannot be changed by the normal root-file tools or remember.

Use it for rules such as:

# Policy

- Never send an external message without human approval.
- Cite the source for every material factual claim.
- Stop and ask when an action could spend more than $500.

Important behavior:

• A missing, unreadable, heading-only, or comment-only file uses Delta's embedded reviewed-write policy when a compatible write rail exists.
• A real custom policy replaces that embedded policy.
• The default policy budget is about 800 tokens.
• Delta refuses to boot when a custom policy exceeds the budget. Fixed rules are never silently truncated.
• An edit on disk requires a daemon restart.

Human review is policy plus tools, not a built-in approval server. Delta's embedded policy recognizes an MCP tool whose name ends with the configured reviewed-write suffix, such as propose_change. It tells the agent to bundle work into one human-reviewable proposal. The actual queue, approval UI, and side effect belong to the connected product.

Prompt guidance is not an authorization boundary. Delta does not evaluate POLICY.md before each tool call. MCP services, the trusted gateway, and any approval product must enforce permissions and block unapproved side effects even when the model ignores or misinterprets the prompt.

PROMPT_CONTEXT.md: stable and live context#

This optional file has two sections:

## Stable
Engine {{engine.version}} | agent {{agent.id}} | profile {{profile}}

## Turn
Model {{model}} | date {{now.date}} | timezone {{now.tz}}
Requester city: {{request.city}}

## Stable is rendered once at boot and joins the cached prefix. It can use:

• {{engine.version}}
• {{agent.id}}
• {{profile}}

## Turn is rendered for every model turn. It can use:

• {{model}}
• {{now.iso}}
• {{now.date}}
• {{now.tz}}
• {{request.<key>}} from metadata.context

Both section templates are loaded at daemon boot. The turn block is re-rendered from that boot-loaded template, so editing either section requires restart.

Example request:

{
  "input": "Prepare the regional launch note.",
  "metadata": {
    "context": {
      "city": "Paris",
      "country": "FR"
    }
  }
}

Request context is treated as bounded data. Delta accepts at most 24 safe keys, reduces each value to one line of at most 200 characters, and leaves unknown placeholders visible. The source file is ignored if it exceeds 100 KB. Rendered stable and turn blocks are capped at 2,000 and 4,000 characters respectively.

vocab.json: product bindings#

The vocabulary file lets the same engine fit a product without compiling product names into the runtime. It controls:

• which MCP verb suffixes are pinned in the lean core
• which tool suffix is the reviewed-write rail
• the nouns used by the embedded review policy
• which request metadata keys identify a hydration subject
• how a learned artifact maps into the product's write-tool arguments
• the default local memory namespace

The generated neutral vocabulary is enough for a plain agent. A product-specific example is:

{
  "coreVerbs": ["get_dashboard", "list_open_work", "propose_change"],
  "writeVerbSuffix": "propose_change",
  "writeNoun": "workspace",
  "runRefKey": "run_ref",
  "learningTargetKind": "note",
  "taskNoun": "task",
  "itemNoun": "review item",
  "subjectKeys": ["account"],
  "writeShape": {
    "summary": "{{summary}}",
    "details": "{{brief}}",
    "provenance": {
      "key": "{{run_ref_key}}",
      "value": "{{run_id}}"
    },
    "items": [
      {
        "kind": "{{target_kind}}",
        "source_kind": "{{kind}}",
        "content": "{{content}}",
        "confidence": "{{confidence}}"
      }
    ]
  }
}

MCP tools are named <server>__<tool>. A core verb matches the suffix after __. For example, crm__get_dashboard matches get_dashboard.

writeVerbSuffix is resolved with the first insertion-order tool whose full name ends with that text. It must identify exactly one connected tool. If two servers expose propose_change, use a more specific suffix such as crm__propose_change; otherwise policy rendering and learned-write promotion can bind to the wrong server.

writeShape is the exact argument object sent to the tool whose name ends in writeVerbSuffix when Delta promotes a fact, preference, or pitfall. It replaces the built-in envelope completely; it is not merged with it. Delta recursively resolves these placeholders:

A string leaf that is exactly one placeholder keeps the value's type. If that value is unavailable, Delta removes the object key or array item. Placeholders inside a longer string are converted to text, and unavailable values become an empty string.

Without writeShape, Delta sends this logical envelope:

{
  "summary": "Reflection (<kind>) from run <run-id>",
  "details": "Post-task or post-review reflection details",
  "<runRefKey>": "<run-id>",
  "items": [
    {
      "kind": "<learningTargetKind>",
      "content": "<learning text>",
      "confidence": 0.8
    }
  ]
}

The confidence key is omitted when no numeric value exists. The custom shape must be a JSON object at its root.

DELTA_VOCAB can supply the same JSON as an environment variable and takes precedence over vocab.json. Malformed vocabulary falls back to neutral defaults. If the wording of writeNoun changes, the derived memory namespace also changes. Set DELTA_MEMORY_NAMESPACE explicitly when changing only the wording.

Choose a model provider#

Delta has no model SDK dependency. It supports three wire formats:

• OpenAI-compatible Chat Completions, used by OpenRouter and many other gateways
• native Anthropic Messages
• OpenAI Responses

OpenRouter#

OpenRouter is the default endpoint and requires the least configuration:

OPENROUTER_API_KEY=your-key
DELTA_MODEL_PRIMARY=anthropic/claude-sonnet-5

MODEL_API_KEY can be used instead of OPENROUTER_API_KEY. Set DELTA_MODEL_FALLBACKS to a comma-separated model list to try additional models on the same provider:

DELTA_MODEL_FALLBACKS=openai/gpt-5.5,google/gemini-3.5-flash

Primary credential precedence is MODEL_API_KEY, then OPENROUTER_API_KEY. A defined but empty MODEL_API_KEY suppresses the fallback key. The fallback key name is not endpoint-aware, so set MODEL_API_KEY explicitly whenever MODEL_BASE_URL is not OpenRouter. Delta does not validate a usable credential at boot or in /healthz; the first model call exposes a bad route or key.

Anthropic directly#

Use the base URL without /messages because Delta appends that path:

MODEL_BASE_URL=https://api.anthropic.com/v1
MODEL_API=anthropic
MODEL_API_KEY=your-anthropic-key
DELTA_MODEL_PRIMARY=claude-sonnet-5

Delta sends native Messages requests and uses Anthropic prompt-cache breakpoints on the system prefix and recent user or tool blocks. Since 0.2.14 the rolling breakpoints are chained into contiguous 20-block lookback windows rather than clustered at the tail, so a turn issuing many parallel tool calls can still reach the previous turn's cache write. A turn of N parallel calls is roughly 2N blocks; below about 10 calls a turn the placement makes no difference, and beyond about 19 the cache is lost regardless, because that burst is one message wider than the whole lookback window.

OpenAI directly#

For the Responses API:

MODEL_BASE_URL=https://api.openai.com/v1
MODEL_API=responses
MODEL_API_KEY=your-openai-key
DELTA_MODEL_PRIMARY=gpt-5.6-sol

The GPT-5.6 family (gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna) is fully supported and priced in the built-in table. Since 0.2.16, an api.openai.com Responses lane is a first-class integration, not a ported one:

• Reasoning carry. The model's encrypted reasoning items are captured, persisted with the message, and replayed verbatim on the next request - OpenAI's own guidance for consecutive tool calls, and required for stateless (store:false) use of reasoning models. The assistant phase field (GPT-5.5+) round-trips with them, so the model never re-reads an intermediate update as a final answer. Both are stripped before any other wire and from compaction-retained rows, so nothing outside this lane ever sees the sealed payload.
• Explicit prompt-cache breakpoints (GPT-5.6+). The same placement engine that drives Anthropic caching marks one stable point (caching instructions + tools + the first user message) plus two rolling points, alongside the provider's implicit breakpoint. Cache writes bill 1.25× on 5.6 and are metered from the nested cache_write_tokens usage field.
• DELTA_TEXT_VERBOSITY (low/medium/high → text.verbosity) and DELTA_REASONING_SUMMARY=auto (→ reasoning.summary, streams reasoning summaries to the reasoning-delta consumer).

These fields are allowlisted to openai.com hosts. Any other Responses-compatible endpoint - a proxy, the ChatGPT subscription backend - receives the same request surface as 0.2.15, because an unknown field is a legitimate 400 on a strict endpoint. A configured knob the lane cannot render is named at boot and in the /v1/status controls block rather than silently ignored.

For an OpenAI-compatible Chat Completions endpoint, omit MODEL_API. Delta appends /chat/completions to MODEL_BASE_URL.

Model IDs pass through unchanged. DELTA_MODEL_FALLBACKS shares the primary base URL, wire, headers, and credential, so every ID in that list must be valid on that endpoint.

Provider and model failover#

There are two levels of failover:

1. DELTA_MODEL_FALLBACKS tries more model IDs on the primary endpoint.
2. DELTA_PROVIDERS adds endpoints after the primary provider.

Example:

BACKUP_OPENAI_KEY=your-key
DELTA_PROVIDERS=[{"label":"openai","baseUrl":"https://api.openai.com/v1","api":"responses","models":["gpt-5"],"apiKeyEnv":"BACKUP_OPENAI_KEY"}]

Inside one provider, a retriable failure gets two retries by default, for up to three wire attempts per model, then advances to the next model. A non-retriable client 4xx skips retries but still advances to the next model on that provider. Credential-wide 401, 403, broker 409, and shared-subscription 429 errors skip the remaining models. After the provider's model list is exhausted, network errors, 401, 403, 409, 429, and 5xx errors can advance to the next provider; a normal client 4xx and an exhausted HTTP 408 do not. Failed attempts do not increment the run's step count and usually have no usage to charge. The timeout applies per wire attempt, so retries and fallbacks can make one model turn last longer than DELTA_MODEL_TIMEOUT_MS.

Once answer text has started streaming, Delta does not retry or switch providers and concatenate a second answer.

Each DELTA_PROVIDERS entry supports baseUrl, models, api, label, apiKey, apiKeyEnv, brokerMintUrl, and brokerAuthEnv. baseUrl is required. models may be one string or an array and otherwise reuses the primary model list. Only anthropic and responses are recognized as api; every other value selects compatible Chat Completions. An inline apiKey, including an empty string, wins over apiKeyEnv. Fallback entries cannot configure static headers and do not inherit MODEL_HEADERS. Invalid JSON is logged and the whole fallback list is ignored. Entries must be JSON objects; a null entry currently aborts configuration loading.

MODEL_HEADERS accepts a JSON object of extra static headers for the primary provider only. Authentication, account, protocol, cookie, and hop-by-hop headers are reserved and rejected at boot.

Codex subscription access#

Delta has two distinct Codex paths:

1. Delegated coding: the built-in code tool runs an installed Codex CLI. The CLI uses its own credentials from the process user's home directory.
2. Primary model access: Delta can call an OpenAI Responses-compatible Codex backend with a short-lived bearer minted by a trusted broker.

The primary-model subscription path is an infrastructure integration, not a standalone login flow. A complete deployment commonly uses:

MODEL_API=responses
MODEL_BASE_URL=https://chatgpt.com/backend-api/codex
DELTA_MODEL_PRIMARY=<Codex model ID accepted by this backend>
DELTA_BROKER_TOKEN_URL=https://control.example/api/broker/openai-token
DELTA_BROKER_AUTH=machine-gateway-token
MODEL_HEADERS={"originator":"codex_cli_rs"}

The harness requires the Responses wire, an endpoint-native model ID, an allowed backend host, and a usable broker response. DELTA_BROKER_AUTH and MODEL_HEADERS are optional to Delta but may be required by the deployed broker or backend.

Delta sends a GET to the broker with the optional bearer and a 15-second timeout. A usable JSON response contains non-empty accessToken, non-empty accountId, and an ISO expiresAt more than five minutes in the future. Delta caches the token and coalesces concurrent mints. A 401 or 403 invalidates it and permits one re-mint retry; 409 immediately advances to another provider; a 429 temporarily cools the shared credential and advances to a keyed provider.

On this backend, max_output_tokens is never sent - the endpoint rejects it at any value, which would fail every sub-agent call with a billed 400 (fixed in 0.2.15; live-verified 3/3 children, with the pre-fix behavior reproducing 0/3 the same minute). One header note for a static-key Codex lane: chatgpt-account-id is a reserved header the subscription credential owns, so MODEL_HEADERS may not set it - the daemon refuses with may not set the reserved header 'chatgpt-account-id'. That refusal is correct and the backend does not require the header on the static-key path (verified both ways); if you hit it, remove the header rather than hunting for a config mistake that does not exist.

The 0.2.16 Responses additions (reasoning carry, phase, explicit cache breakpoints, verbosity, reasoning summaries) are not sent to this backend: its parameter surface is undocumented and rejects fields it does not recognize, so each field joins only after a wire probe proves it - until then this backend receives requests byte-identical to 0.2.15.

Subscription tokens are sent only to exact HTTPS hosts in DELTA_BROKER_ALLOWED_HOSTS, whose default is chatgpt.com. Add the real backend host explicitly when required. The broker URL must be HTTPS, except for loopback development. Configure a keyed provider in DELTA_PROVIDERS as a production fallback for broker exhaustion, authentication failure, or rate limiting. Subscription-route cost_usd is API-rate-equivalent consumption from Delta's price table, not the incremental subscription bill.

Utility model and reasoning#

Compaction, reflection, and eval_n judging use DELTA_UTILITY_MODEL, which defaults to anthropic/claude-haiku-4.5. Set it to an empty value to use the main provider cascade for those calls. The utility model is tried on each compatible provider. Native Anthropic translates dotted Claude version segments to dashes; Responses providers are skipped for a Claude utility model. Any utility-lane failure falls back to the main cascade, and reasoning effort is not applied to utility calls.

Set a daemon default for extended reasoning:

DELTA_REASONING_EFFORT=high

Or override it for one run with metadata.reasoning_effort. Delta lowercases and trims the value. OpenRouter and Responses receive reasoning.effort (on an openai.com Responses lane, DELTA_REASONING_SUMMARY=auto rides the same object); a directly compatible Chat Completions endpoint receives reasoning_effort. Native Anthropic picks the wire by model. Claude 4.6+ and every Claude 5 model use thinking:{type:"adaptive"} with output_config.effort - they reject the older thinking:{type:"enabled"} budget wire; on this path the OpenAI-only none and minimal map to low (an always-on reasoning model cannot disable thinking), and Delta raises the request's max_tokens by an effort-based headroom so adaptive thinking has room. Claude 4.5 and earlier keep the legacy budget wire, mapping none, minimal, low, medium, high, and xhigh to disabled, 1,024, 4,096, 8,192, 16,384, and 32,768 thinking tokens, with an unknown value using the 16,384-token mapping. Other providers validate their own accepted values and can return a clean 4xx.

Run and call the agent#

CLI reference#

delta run reads configuration from the process environment and bundle files from DELTA_WORKSPACE. It does not load delta.env, connect configured MCP servers, start HTTP or Cockpit, or provide crash recovery. Its conversation database is in memory, but file writes and DELTA.md changes persist in the workspace.

Synchronous HTTP request#

curl -sS http://127.0.0.1:8080/v1/responses \
  -H 'content-type: application/json' \
  -d '{
    "input": "Create a decision brief from the files in the workspace.",
    "metadata": {
      "user_id": "user_123",
      "task_id": "task_456",
      "task_type": "decision-brief"
    }
  }'

When DELTA_CONTROL_TOKEN is set, add the machine bearer to every run, task, queue, and upload request:

-H "authorization: Bearer $DELTA_CONTROL_TOKEN"

The terminal response has this shape:

{
  "id": "resp_...",
  "object": "response",
  "model": "anthropic/claude-sonnet-5",
  "status": "completed",
  "output_text": "...",
  "output": [
    {
      "type": "message",
      "role": "assistant",
      "content": [{ "type": "output_text", "text": "..." }]
    }
  ],
  "previous_response_id": null,
  "usage": {
    "input_tokens": 0,
    "output_tokens": 0,
    "total_tokens": 0,
    "cost_usd": 0
  }
}

Continue a conversation#

Use the prior response's id:

curl -sS http://127.0.0.1:8080/v1/responses \
  -H 'content-type: application/json' \
  -d '{
    "input": "Now turn that brief into an executive email.",
    "previous_response_id": "resp_previous"
  }'

The new run joins the same session. Its active conversation, compacted summaries, and session-scoped user identity remain available. A prior response ID is only a session lookup, not a history snapshot or branch point. Continuing from an older response still appends to the current session head and sees later turns. Multiple continuations into one session serialize rather than fork.

Stream final text#

curl -N http://127.0.0.1:8080/v1/responses \
  -H 'content-type: application/json' \
  -H 'accept: text/event-stream' \
  -d '{"input":"Write the brief.","stream":true}'

The stream emits response.output_text.delta frames, followed by one response.completed frame with the full response. Tool-calling turns can emit intermediate assistant text before the terminal answer, so concatenating every delta is not guaranteed to equal the final output_text; treat response.completed as authoritative. Reasoning deltas are available to the internal dev stream, but are never mixed into final answer text.

Start and monitor a long task#

Create the task:

curl -sS http://127.0.0.1:8080/v1/tasks \
  -H 'content-type: application/json' \
  -d '{"input":"Review every report in the workspace and synthesize the findings."}'

Delta acknowledges only after the queue row is durable:

{"id":"resp_...","object":"task","status":"queued"}

Inspect status:

curl -sS http://127.0.0.1:8080/v1/tasks/resp_...

The task view carries lifecycle metadata for a polling host: created_at (accepted and enqueued), started_at (dequeued and began executing), and finished_at (terminal), which separate queue wait from execution time. A failed or cancelled run also exposes a first-class error field with the run's last error - the provider or fatal error that ended it - so a plain poller learns why a run failed before its first token, without shelling in. Per-tool errors remain in the message stream.

Follow live task events:

curl -N http://127.0.0.1:8080/v1/tasks/resp_.../events

This task event endpoint is live-only. If the task is already terminal, it immediately emits the final done frame. Use /v1/dev/stream?since=<event-id> when replayable inspection is required.

The shipped queue runs up to DELTA_MAX_CONCURRENCY cross-session runs at once (default 8, clamped 1–256), with no admission or backlog cap. Each run is IO-bound async work on one event loop - this is "how many provider calls in flight", not threads - and sessions stay serial (one run per session). The mechanism itself is proven correct well past this; the practical ceiling is your provider's concurrent-request tolerance (a subscription key wants a low value; a high-limit API key can run 25+), then per-run context memory (~4–15 MB per active run). Raise it for a high-throughput agent; lower it to 1–2 to be gentle on a rate-limited subscription. Put hard capacity limits and backpressure in the gateway. For long work, prefer /v1/tasks over keeping a synchronous request open. Delta disables Bun's server idle timeout, but proxies and load balancers still need suitable request, read, and SSE timeouts. Task and response streams do not bound queued frames for slow consumers, so clients must consume promptly and gateways should cap concurrent or abandoned streams.

Cancel a queued or running task:

curl -sS -X DELETE http://127.0.0.1:8080/v1/tasks/resp_...

Cancellation is cooperative. Signal-aware model and tool work receives an abort. A detached, signal-ignoring tool may still finish, so verify external state before retrying its side effect.

Task status has five values:

GET /v1/tasks/:id includes result only for done, failed, or cancelled. Handled outcomes store a complete Responses-compatible object with status, output_text, output, model, previous response ID, and usage. Failed and cancelled output begins with a Delta status marker rather than pretending to be a successful answer. An unexpected exception outside Delta's handled run path can currently mark a task failed with an empty {} result and no normal run.finished event. A task or response SSE client waiting for that event can continue receiving heartbeats, so clients need their own terminal-status polling and timeout. Operators must alert on structured errors and malformed terminal results.

Progress events. GET /v1/tasks/:id/events follows the live run as Server-Sent Events; add ?coarse=1 to drop the per-token output_text.delta / reasoning.delta frames and keep only the structural heartbeat (run/turn/tool lifecycle, model.call, checkpoint, error). For a host that can't hold an SSE connection, GET /v1/tasks/:id/events?since=<id> returns a bounded JSON page { "events": [...], "cursor": N, "done": bool } - pass the returned cursor as the next since and stop when done is true (limit defaults to 200, max 500). model.call events carry cache_shortfall_tokens, the previous request's gross input minus this call's cache reads, so a host can watch cache health live. Prefer it to cache_hit_pct, which is also present but is a ratio whose denominator grows as history is appended, so it moves even when caching is perfect and a healthy turn can read anywhere from 59% to 100%. A model.retry event fires on every retry, re-auth, model switch, or provider failover, so a retrying or stalled provider is visible instead of silent dead air, and model.call also carries wall_ms and retries next to latency_ms so the pre-call gap (wall_ms minus latency_ms) is queryable. Both surfaces require the run's owner (see identity, below).

Handled non-streaming API errors use this shape:

{"error":{"message":"human-readable reason"}}

The request root must be a JSON object. A valid JSON scalar such as null currently escapes the normal validation path and can produce a generic 500 response instead of this envelope, so a public gateway should reject non-object bodies before forwarding them.

Request fields#

Useful metadata:

The trust model: the control token authenticates the gateway; the gateway asserts which of its users a request is for via the x-delta-user header, which is the sole tenancy authority - Delta never takes ownership from a request-body field. As of 0.2.4 Delta enforces ownership of task IDs: GET/DELETE /v1/tasks/:id and …/events are served only to the run's owner, and a cross-tenant ID and an unknown ID return the same 404 so existence isn't disclosed. The x-delta-user principal is canonicalized into the stored run at ingress, so recall, reflection, and event identity all key on that owner - a body user_id can no longer point them at another tenant - and the idempotency dedupe is scoped to the owner. Session ownership is still enforced too: continuing with previous_response_id is rejected across owners (a 404, indistinguishable from an unknown ID). A run with no asserted owner stays open, which is the single-tenant path. By default a task's idempotency key is freed once its run reaches a terminal state (a later dispatch with the same key runs fresh); a request that sets idempotency_terminal: true on a durable run instead re-attaches to its own terminal run, so a fire-and-forget gateway that lost the 202 can re-POST the same key and land the accepted run rather than a second (0.2.9, owner-scoped). Review-widening authorization (review_kind, widen_authorized) is stripped from every request body by default - a shared control token is not proof a human reviewer set them - so a caller can't self-authorize widening a user-scoped memory; DELTA_TRUST_REVIEW_METADATA=1 opts a single-tenant operator back in.

For a daemon that serves multiple users behind one control token, set DELTA_STRICT_TENANT=1: every run must then be owned (creating a task without x-delta-user is 401), unowned runs are inaccessible, a null-owned session can't be continued by another principal, and /v1/queue requires a principal and numbers positions only among the caller's own entries. A trusted gateway should still authenticate the caller and inject one canonical x-delta-user; end users must never reach Delta directly.

Queue visibility#

curl -sS http://127.0.0.1:8080/v1/queue \
  -H 'x-delta-user: user_123'

The named caller sees its own queued and running rows with IDs and session IDs. Other users' rows remain opaque and show only status, position, and age. The header is an identity hint expected from a trusted control plane, not authentication by itself.

Suspend safety#

curl -sS http://127.0.0.1:8080/v1/busy \
  -H 'authorization: Bearer <DELTA_CONTROL_TOKEN>'

Returns {"busy":true|false,"running":N,"queued":N}. busy is the durable queued-or-running truth read from the run table, so a host managing suspend and resume never suspends a Machine with work still owed, including a queued run that has not started. It sits behind the /v1/ control-token gate and is deliberately not folded into the open, data-free /healthz. A host polls it before suspend and after every terminal task; the hosting contract covers the full lifecycle.

Upload files#

curl -sS http://127.0.0.1:8080/v1/files \
  -F 'file=@./brief.docx' \
  -F 'file=@./chart.png'

Delta validates the full batch before writing any file. Limits are 50 files per request, 25 MB per file, and 100 MB total. The later filesystem writes are not transactional, so a storage failure on file N can leave earlier files from that batch in place. Files land under inbox/YYYY-MM-DD/, and the first upload creates FILES.md with organization conventions. Pass the returned paths to the agent in the next request.

Images are attached to a vision-capable model only after Delta itself has verified their provenance through upload, read_file, or an MCP image response. Up to four recent images smaller than about 3.4 MB each are attached. Provenance is process-local, so after a daemon restart re-read a workspace image, re-upload it, or retrieve it again through MCP before expecting an inline attachment. Old markers remain file references and can be re-read.

Built-in capabilities#

The trusted tier exposes the built-in tools below. Tool failures are returned to the model as values, so one failed dependency does not crash the daemon.

DELTA_TOOL_TIMEOUT_MS supplies the outer timeout for ordinary queued-run tools and defaults to 120 seconds. In normal queued execution, that outer signal replaces the search and fetch helpers' fallback 30-second signal; setting it to 0 therefore leaves those tools without a harness ceiling. MCP calls have an internal 60-second ceiling, and scheduling calls use 15 seconds. code, spawn_subagent, and eval_n deliberately use no outer timeout. Asynchronous work that ignores abort can continue detached, and synchronous blocking code cannot be preempted.

Workspace safety#

Normal file tools resolve every model-supplied path inside the workspace and reject traversal and escaping symlinks. Their protected-name rule applies only at the workspace root. They cannot write these root paths:

• POLICY.md
• vocab.json
• PROMPT_CONTEXT.md
• DELTA.md, except through remember
• root .env, .env.*, or delta.env
• root .delta and everything under it

The same names below a subdirectory are not protected, so notes/POLICY.md is writable. These are mutation guards, not read controls or a process sandbox. read_file can read workspace configuration, and the code tool runs a separate process with the daemon OS user's access. Never place secrets anywhere in the agent workspace. Use a dedicated OS user, VM, or purpose-built sandbox for stronger isolation.

Large tool results#

All tool results pass through one central cap, 20,000 characters by default. If a result is larger, Delta keeps its beginning and end in the conversation and saves the complete result under:

.delta/spill/<run-id>.<call-id>.txt

The inline message tells the model where to re-read the full output. Change the inline cap with DELTA_TOOL_RESULT_MAX_BYTES.

Coding CLI#

The default command is:

codex exec --sandbox workspace-write --skip-git-repo-check

Override it as a whitespace-separated command:

DELTA_CODE_CLI=claude --print

The command runs with the workspace as its current directory and has no harness timeout. Since 0.2.15 the built-in default appends --disable apps --disable plugins: a delegated CLI signs in with its own account, and account connectors live server-side on that login - a session with no send-capable tool was demonstrated listing the operator's real inbox through a Gmail connector nothing on the host granted. The default requires codex-cli 0.146.0 or newer; an older binary resolves at boot and then rejects the flags on every call. A deployment that deliberately relies on a connector sets DELTA_CODE_CLI explicitly - the operator's value is used verbatim, never modified. Delta splits DELTA_CODE_CLI on spaces without shell or quote parsing, so quoted arguments and executable paths containing spaces are unsupported. The default container does not install a coding CLI. Add one to the image and provision its credentials if the code tool is required. Delta's default asks Codex for its own workspace-write sandbox; a custom CLI receives no equivalent sandbox from Delta.

Subagents and eval_n#

spawn_subagent starts the same Delta binary in one-shot run mode with:

• a new in-memory database and conversation
• the same workspace
• the same bundle files, loaded again under child defaults
• the remaining parent fresh-token and cost budget
• at most one level of nesting
• a default-deny child environment
• a narrow allowlist of provider and run variables plus one static primary credential

The child forwards the safe process environment plus the primary base URL, wire type, legacy model name, model fallbacks, utility model, profile, model and stream timeouts, result cap, remaining budget values, and one static primary key. It does not inherit the parent's MCP configuration, Exa key, telemetry, control-plane token, subscription broker token, provider cascade objects or secrets, model headers, custom prices, reasoning, vision settings, coding CLI, stable agent ID, learning and hydration settings, or custom self and policy caps. A broker-only parent therefore cannot currently power a spawned subagent unless a static primary model key is also available. The same bundle does not guarantee the same resolved behavior: the child can reset to the default coding CLI, default identity, and 800-token file caps.

The child allowlist forwards both DELTA_MODEL_PRIMARY and the legacy DELTA_MODEL alias, so a subagent runs on the parent's configured model. The child also does not receive MODEL_HEADERS, DELTA_MODEL_PRICES, or reasoning settings, so a custom-header route, custom metering, or parent reasoning configuration is not reproduced in the child.

Subagents share the workspace. A subagent can also receive the remember tool under the trusted tier, so it can replace the shared DELTA.md; its revision database is temporary. Use delegation only for tasks whose file and self-memory effects are intentional.

eval_n runs variants concurrently against the same workspace. Use it for drafting, analysis, or other non-mutating work. Do not use it for concurrent edits to the same files.

Each child prints a DELTA_USAGE marker. Valid parsed usage is added to the parent and constrained by the remaining fresh-token and dollar budget. Missing or malformed child usage fails open as zero. External spend by a coding CLI or subscription route is not discovered through this marker. The current eval_n judge call is not charged to the parent usage total.

All tool calls returned by one model turn execute concurrently, including mutations, and up to DELTA_MAX_CONCURRENCY different sessions can run concurrently (default 8). The DELTA.md self-file is guarded by optimistic concurrency - a write that raced a diverged base is refused and the run re-reads, re-merges, and retries - but there is no compare-and-swap layer for other files or external resources. Same-session serialization does not serialize sibling tool calls within one turn. Design mutating tools to be idempotent and conflict-aware, and avoid parallel tasks that can touch the same state.

Scheduling#

Delta does not keep a local alarm clock because a production VM may be suspended. When both settings are present, it exposes scheduling tools backed by the control plane:

DELTA_CONTROL_URL=https://control.example
DELTA_CONTROL_TOKEN=machine-token

For a chat-fronted agent, Delta Connect 0.3.0 implements this control-plane surface itself: a loopback scheduler that stores schedules durably and fires each due one as a turn delivered back to the originating chat. Point DELTA_CONTROL_URL at the local Connect gateway and scheduling turns on with no external control plane.

Schedules support:

• one-time ISO timestamps
• intervals of at least 60 seconds
• five-field cron expressions with an optional IANA timezone

The control plane owns storage, time, and wake-up behavior. Delta only requests and manages schedules through its authenticated API.

Connect MCP tools#

Delta implements MCP Streamable HTTP and newline-framed stdio without an SDK dependency. It initializes every configured server at daemon boot, discovers its tools, and registers them as <server>__<tool>.

HTTP server#

DELTA_MCP_SERVERS should be a JSON array. Give each entry an explicit transport; if you omit it, Delta infers it from the entry shape (url implies http, command implies stdio) and logs that it did so:

DELTA_MCP_SERVERS=[{"name":"crm","transport":"http","url":"https://crm.example/mcp","headers":{"authorization":"Bearer service-token"}}]

The expanded JSON is:

[
  {
    "name": "crm",
    "transport": "http",
    "url": "https://crm.example/mcp",
    "headers": {
      "authorization": "Bearer service-token"
    }
  }
]

An MCP tool named get_account becomes crm__get_account. Names that cannot fit the model API's tool-name rules are dropped.

Malformed JSON is treated as no configured servers. Delta validates each entry's shape at boot and skips any malformed entry (with a warning), but it does not verify a server is reachable. Inspect boot logs plus /v1/dev/config rather than assuming a non-empty variable produced tools.

Stdio server#

DELTA_MCP_SERVERS=[{"name":"local","transport":"stdio","command":["bun","run","server.ts"],"env":{"MODE":"production"}}]

The stdio child inherits the daemon environment plus the explicit env object. Treat a stdio MCP process as trusted code inside the same machine boundary.

Authentication#

Static HTTP headers apply to discovery and tool calls. For a user-scoped run, pass a short-lived token:

{
  "input": "Update my account brief.",
  "metadata": {
    "user_id": "user_123",
    "authToken": "short-lived-user-token"
  }
}

For that run, Delta replaces the HTTP MCP Authorization header with Bearer <authToken>. One run has one shared authToken, and Delta applies it to every HTTP MCP server called by that run. Use tokens with the correct audience across all connected servers, or place differently scoped connectors on separate daemons. Each MCP service must enforce the user's permissions.

Delta also supports one rotating MCP credential attached to a named HTTP server:

DELTA_MCP_REFRESH_URL=https://auth.example/token
DELTA_MCP_REFRESH_FILE=/data/mcp-refresh-token
DELTA_MCP_REFRESH_SERVER=crm
DELTA_MCP_REFRESH_CLIENT_ID=delta-agent
DELTA_MCP_REFRESH_TOKEN=first-boot-refresh-token

The required fields are URL, file, and server. The seed token and client ID are optional. On an MCP 401, Delta rotates once and retries unless a per-run user token supplied the authorization.

The refresh file is authoritative persistent state. Delta seeds it only when absent, does not create its parent directory, and does not repair an existing empty or corrupt file. Pre-create a private writable parent outside the agent workspace, use one refresh-token file per daemon or machine, and include its current value in the recovery design. Never restore a stale backup of a refresh token that may already have been exchanged. Refresh coalescing is process-local.

Do not also set a static Authorization header on the selected rotating server. Static headers are applied after the rotating credential and can overwrite it, including on the 401 retry. A partial or unmatched refresh configuration only warns and may leave the server unauthenticated, so assert the expected connection at startup.

Failure and reload behavior#

One unavailable MCP server does not prevent the daemon from starting. Its tools are absent and the failure is logged. MCP configuration is boot-time in the shipped daemon, so restart after changing DELTA_MCP_SERVERS.

Delta implements a deliberately small MCP subset around protocol version 2025-06-18. It calls tools/list once at boot, does not paginate or consume list-changed notifications, and requires a restart for rediscovery. HTTP accepts plain JSON or one complete JSON object on a single SSE data: line; multiline SSE events are unsupported. Stdio is newline-delimited JSON, not Content-Length framing. Tool results retain text and image content only; structured content, resources, audio, and embedded resources are not passed to the model.

Every MCP result is framed as untrusted data before it reaches the model. The server still owns the side effects and authorization of its tools.

Crash idempotence is inferred from the MCP tool name. Names containing an underscore- or dot-delimited segment such as get, list, search, read, version, versions, or file are treated as read-only and may be re-executed after interruption. All other MCP tools are treated as non-idempotent. Name mutations so they do not contain those read segments. In particular, a mutating name such as get_or_create is unsafe because Delta will classify it as idempotent.

Progressive tool loading#

A large connector surface should not put hundreds of schemas in every prompt. The trusted tier:

1. pins all built-in tools
2. pins MCP tools whose suffix appears in vocab.coreVerbs
3. leaves other allowed tools searchable through search_tools
4. activates up to five matches at a time
5. persists those activations with the run

Delta applies a 60-name threshold to the initially requested pinned set. If that set is larger, it warns and falls back to the built-ins plus vocab.coreVerbs. The fallback is not capped a second time, so an oversized coreVerbs list can still exceed 60. Keep the declared core small; this threshold is a guardrail, not a strict invariant or a restriction on the total MCP registry.

Memory and self-improvement#

Delta has three separate learning surfaces. They solve different problems and should not be conflated.

1. The living self-file#

DELTA.md carries a small lesson into every future request. Use it for a compact operating preference or identity change that is important enough to pay for on every model call.

The remember tool is explicit, reversible, and visible in the workspace. It is not the same as automatic post-run reflection. Concurrent updates are atomic but last-writer-wins, so two runs can overwrite each other's lesson without corrupting the file.

2. Governed local memory#

Optional reflection runs after a successful response and distills at most one reusable artifact:

• fact
• preference
• pitfall
• procedure

Enable it for every successful run:

DELTA_REFLECT=1

Or enable it for one request:

{
  "input": "Prepare the weekly revenue review.",
  "metadata": {
    "reflect": true,
    "task_type": "weekly-revenue-review"
  }
}

Reflection runs best-effort in the background after the response has been released. It has no durable work queue or retry, and shutdown or a crash can drop it. A failure never changes the completed user response. A per-run reflect: false does not disable a daemon that has DELTA_REFLECT=1. Reflection usage is added later to the run row, but it is not added to the already stored terminal response payload or the earlier run.finished event and does not emit a normal model.call usage event. Cockpit can therefore later show more run usage than the API result or exported terminal telemetry.

Local memory is structured by four independent axes:

It is also keyed by product namespace, agent ID, user ID when private, and caller-declared task type when shared by use case. The local recall query currently returns agent, current-user, and matching-task-type rows. org rows can participate in shared promotion state but are not recalled by the local rail.

Governance rules:

• self-rated confidence below 0.6 is rejected
• a human-review correction receives at least 0.8 confidence
• content is capped at 500 characters
• duplicate content confirms one row and records the distinct producing run
• one identity is normally capped at 200 evictable rows; rows protected by staged or claimed promotion can temporarily take it over that limit
• memory unused for 90 days stops being recalled
• recall scores lexical relevance, aliases, confidence, recency, and prior use, but applies no minimum relevance threshold, so a zero-overlap row can still rank into the bounded result
• recall is inserted only at the first run of a session

Intended privacy routing, after a trusted gateway has supplied canonical metadata, is:

• a run tied to a user writes user-scoped memory
• a model cannot widen that private memory to a shared audience
• widening user-derived learning requires a human review turn with explicit widen_authorized
• user memory is never promoted to an external shared store
• a userless run with task_type can create and recall memory shared only for that caller-declared use case

These rules are not an authentication layer. Caller-controlled identity and review fields can change scopes unless a gateway normalizes them as described under request fields. Agent recall also includes legacy rows whose agent_id is empty. Separate databases or migrate and delete legacy rows before relying on agent-level isolation.

The local memory rail is deterministic SQLite retrieval, not vector search. It gives the harness a useful no-backend fallback while leaving semantic enterprise knowledge to an optional product store.

3. External context and reusable skills#

Task-start hydration can call product MCP reads before the first model turn of a new session:

DELTA_HYDRATE_TOOLS=crm__get_dashboard,crm__list_recent_notes
DELTA_HYDRATE_SEARCH_TOOL=crm__search_knowledge

Subject-scoped reads run only when request metadata contains a key declared by vocab.subjectKeys. If the vocabulary contains "account", Delta checks account, account_id, and accountId.

Task-keyed search requires metadata.authToken, even when no subject is present. With a trusted gateway, this avoids falling back to a shared daemon credential for user knowledge. Hydration is best-effort, waits at most 20 seconds, and injects at most about 16,000 characters across its blocks. The wait race does not abort the underlying calls, which can continue detached. Hydration calls configured tools directly, without the normal tool journal, idempotence classifier, or central result cap. Configure read-only, least-privilege tools only.

Each configured hydration read receives the resolved subject fields plus limit: 20. The search tool receives both q and query, the same subject fields, and limit: 8; the task text is truncated to 500 characters and must contain at least two non-space characters. Delta sends that whole object unchanged. A strict MCP schema must declare every supplied property or tolerate extras.

Separately, the capability adapter searches once per execution attempt for a reusable procedure, so it can repeat after crash recovery. Its 20-second wait likewise does not abort a late underlying call. Delta loads the first reference returned by the adapter and exposes other matches as references. A custom adapter may rank by task relevance; the built-in skill-registry adapter currently uses its returned index order. The block is ephemeral, bounded, and framed as untrusted directory data.

The built-in adapter recognizes skill-registry-style MCP tools with this concrete contract:

• it is considered bound only when a matching skill create, update, or propose tool exists; skill_search and skill_get alone do not activate retrieval
• directory search uses a tool whose name ends in skill_search, normally <server>__skill_search, and calls it with { "limit": 25 }
• loading a procedure uses a tool whose name ends in skill_get and calls it with { "name": "<exact-name>" }
• creation or update discovers a tool whose name matches skill followed by create, update, or propose
• a create sends name, body, and a bounded description; an update sends name, body, base_version, and baseVersion; reflected promotions also send both change_summary and changeSummary
• an update retries once after a reported version conflict by loading and rebuilding from the fresh version
• the current built-in search does not send the task query and keeps the returned index order; custom capability adapters can rank by relevance
• only one matching capability-tool family is bound, selected by insertion order
• skill_search extracts JSON-like objects with a string name and optional string description
• skill_get accepts only a string body and integer version of at least 1; other response shapes are treated as absent

Strict skill-registry tool schemas must accept the paired snake-case and camel-case fields above.

The engine boundary remains generic:

• a capability store searches, reads, and proposes versioned procedures
• a curated store accepts proposed facts, preferences, and pitfalls

Facts, preferences, and pitfalls go to the MCP tool selected by vocab.writeVerbSuffix; vocab.writeShape maps them into that product's exact arguments. Procedures use the capability adapter instead.

These external stores are optional. A plain Delta works with local memory only.

Choosing a skills backend (DELTA_SKILLS, added 0.2.7). The capability backend is selectable, and the default preserves prior behavior:

DELTA_SKILLS=mcp    # default - the skill-registry MCP tools above (unchanged)
DELTA_SKILLS=local  # read skill folders from the workspace, no service required
DELTA_SKILLS=off    # no skills at all - no retrieval, no skill tools, no mention

With local, drop a folder tree in the workspace and Delta discovers it at boot:

skills/
  cite-check/
    SKILL.md        # front matter with name + description; the body is the procedure
    rubric.md       # reference files, data, or scripts - read on demand

Only each skill's name and description enter the prompt (progressive disclosure). The agent reads SKILL.md and any bundled files on demand with read_file, and runs bundled scripts with the code tool. Discovery is direct-child-only and rejects symlinks. Since 0.2.15 the index refreshes without a restart: each search re-checks every skill's SKILL.md (modification time and size) and re-scans when anything moved, so skills added, renamed, or re-described after boot are findable. Frontmatter descriptions may use YAML block scalars (description: > or |) - previously such a skill registered but could never be surfaced by search - and a description under ten characters logs a loud warning, because an unsearchable skill is a defect defined by its silence. Local skills are read-only in this release: reflection proposes improvements only to a backend that accepts writes (the MCP registry). off is fully invisible. Default mcp needs no change.

Promotion to shared knowledge#

Non-user learning enters a durable SQLite outbox before any external proposal:

• procedures target the capability adapter
• facts, preferences, and pitfalls target the curated adapter
• a review-grounded correction may promote immediately
• the same lowercased, whitespace-normalized reflection content must recur in two distinct runs by default; a semantic paraphrase counts as different content
• only trusted memory can promote
• the local outbox carries an idempotency key, but the bundled skill-registry and curated MCP adapters do not transmit it to their remote tools; a crash after remote success can therefore duplicate a proposal
• a failed proposal can be retried when a later successful reflection drains the outbox and becomes terminal after five attempts; there is no independent retry timer

Change the recurrence threshold with DELTA_PROMOTE_MIN_RUNS. Agent, user, and task-type rows remain locally useful when no external adapter is bound. An org row is not part of local recall.

Promotion rows are scoped by the normalized content, full identity and artifact dimensions, namespace, and selected adapter. Changing the memory namespace, write noun, suffix, or adapter can strand staged rows. Drain or deliberately migrate the outbox before changing those bindings.

Learn from human review#

Delta is ready to participate in a review loop, but it does not ship a universal approval service. The connected product must provide the proposal tool, review state, user interface, and eventual side effect.

The recommended loop is:

agent work
  -> one proposal through <server>__<writeVerbSuffix>
  -> human accepts, rejects, or edits it in the product
  -> control plane sends the disposition back as a new Delta run
  -> review reflection learns from proposed versus accepted work

The review run must be explicit:

{
  "input": "Review outcome: item 1 accepted with edits. Proposed: ... Accepted: ... Reviewer note: ...",
  "metadata": {
    "review_kind": "submission_disposition",
    "submission_id": "submission_123",
    "reflect": true,
    "widen_authorized": false
  }
}

Only the exact review_kind value submission_disposition switches reflection to the correction-focused rubric. Delta does not validate reviewer identity, fetch the submission, or structurally compare proposed and accepted work. It asks the reflection model to ground learning in the supplied digest. The trusted control plane must authenticate the disposition and construct the real proposed-versus-accepted content rather than forwarding an end-user claim that review happened.

Durable execution and budgets#

Tiers and the tool envelope#

Two capability tiers ship. The tier sets which tools an agent may use and its budget ceilings; a request may narrow the tier but never escalate it.

Set the daemon ceiling:

DELTA_PROFILE=trusted

A request can choose metadata.profile: "safe" on a trusted daemon. It cannot choose trusted on a safe daemon. Unknown or more-permissive request values stay at the placement ceiling.

Renamed in 0.2.7 - no action needed for a live agent. These two tiers were previously named work and chat; they are now trusted and safe, naming the capability level rather than an activity. The old names still resolve as aliases, so an existing DELTA_PROFILE=work or DELTA_PROFILE=chat keeps working unchanged. The only observable difference is the canonical name reported in telemetry and GET /v1/status, which is now trusted / safe. No tool or budget behavior changes.

Custom tool envelope. Instead of a named tier, set the exact tool surface directly:

DELTA_ALLOWED_TOOLS=read_file,web_search,write_file,grep
DELTA_PINNED_TOOLS=read_file,web_search

The list is intersected with the tier's own allowed set, so it can only narrow a tier, never escalate it - a safe daemon cannot be widened this way. To build a broad custom envelope, start from trusted (which allows every tool) and list exactly what you want. A set-but-empty or malformed list falls back to the safe floor rather than allowing everything.

Lower the token and cost ceilings:

DELTA_MAX_TOKENS=250000
DELTA_MAX_COST_USD=1

Since 0.2.15, DELTA_MAX_STEPS completes the third axis and, like the other two, can raise or lower the profile's value. Unlike them it is floored at 1 - a zero-step budget would fail every run before its first model call with an error the operator cannot diagnose. Fractions floor. Honest scoping: across the measured fleet the binding constraint is tokens (the maximum observed step count is 62 of 100), so this is a knob for deep-research shapes at high reasoning effort, not something a typical deployment needs.

DELTA_MAX_STEPS=200

What the budget measures#

One step is one successfully completed main-loop model call. Failed provider attempts, compaction, reflection, hydration, capability retrieval, and the eval_n judge are not steps.

The fresh-token guard uses recorded main and child usage:

fresh input tokens + output tokens

Prompt-cache reads do not repeatedly consume the fresh-token budget. Dollar usage comes from the provider when available or from Delta's price table. A compatible provider that omits or misreports usage weakens both token and dollar enforcement, so verify metering with production routes before relying on caps.

For a model that is not priced, add dollars per million tokens:

DELTA_MODEL_PRICES={"my-model":{"in":2,"out":10,"cacheRead":0.2}}

Without a known price, Delta warns and records zero model cost for that path. A dollar cap cannot protect an unpriced model. Table lookup follows the served model ID and prefix matching; verify the selected row. A price override without cacheWrite values cache writes at 1.25 times input. Current parsing also accepts negative prices, so validate overrides externally because a negative value can weaken the cost guard.

Budgets are hard loop guards against usage Delta records, checked between model calls. The final call, its parallel tools, or background reflection can move recorded usage past the nominal threshold before the next guard. They do not cap failed provider attempts without usage, a child with a missing or malformed usage marker, provider under-reporting, coding CLI or subscription-account spend, MCP or search API fees, cloud cost, or the current eval_n judge. Subscription calls record API-rate-equivalent cost rather than the subscription's actual marginal bill. Treat these values as one-call-granularity model-loop guards, not prepaid authorization or a complete spend limit.

Environment overrides use JavaScript number conversion. An exported empty DELTA_MAX_TOKENS or DELTA_MAX_COST_USD becomes zero rather than unset. Validate all budget variables before starting the daemon.

What an exhausted run returns#

Since 0.2.15, a run that hits a budget ceiling hands back what it already has instead of a sentence of counters. output_text carries the current plan, the paths of every spill file and research artifact this run produced (enumerated from disk under the run's own id, so nothing stale or forged can enter), a note if a durable DELTA.md write already committed, and advice to narrow the question rather than repeat it. The whole handoff is bounded at 10 KiB and 20 paths per artifact family, and truncation is named rather than silent. An ephemeral (store: false) run gets no paths, because its artifacts are wiped the moment the run settles. The operator diagnostic - budget exhausted: N/M steps, … - lives in runs.error and the error telemetry event, as it always did; a consumer that previously string-matched the counters out of output_text must read runs.error instead. The run's status remains failed.

Compaction#

Delta compacts when the previous model request exceeded DELTA_COMPACT_AT_TOKENS, default 120,000, and there is more than a minimal tail of active messages to shed. It preserves a recent tail sized by a token budget (default 24,000 tokens) snapped to a turn boundary, so an assistant tool call always travels with its result, rather than a fixed message count. It asks the utility model for a summary under 350 words from at most 60,000 characters of older history. A failed compaction is a no-op. The compaction call is charged to run usage but not the step count.

If a provider reports a context-window overflow earlier, Delta forces one compaction and retries the same model turn once. Forced overflow handling may also elide an oversized tool message. If compaction cannot reduce the context, the run fails cleanly.

Checkpoints and crash recovery#

SQLite uses WAL mode, foreign keys, transactional schema migrations, and a five-second busy timeout. Durable state includes:

• sessions and runs
• active and compacted messages
• model step count and usage
• active tool names, whose schemas are reconstructed from the boot registry
• tool intents and results
• events
• local memory and promotion state
• writer lease
• exact model-call captures when enabled
• DELTA.md revisions

The queue writes a run before acknowledging it. Each assistant message and all tool intents it creates commit atomically. Each tool result and its conversation message also commit atomically.

On restart:

• rows left as running resume before queued rows drain
• an unfinished model call can run again because no response was checkpointed; the provider may already have billed the interrupted attempt, so recovery can duplicate model charges
• a completed tool journal result is replayed without re-executing the tool
• an interrupted idempotent tool may execute again
• an interrupted non-idempotent tool is not blindly re-fired
• the model receives an [interrupted] result and must verify whether the side effect happened
• step count, usage, compaction, and activated tools survive

A resumed execution is recovery, not deterministic replay. It reloads current boot configuration, POLICY.md, prompt context, vocabulary, MCP registry, provider routes, and a fresh DELTA.md snapshot. Keep bundle and infrastructure changes coordinated with in-flight recovery.

Events and all run-state transitions are not one atomic log. A crash at a boundary can leave a durable state change without its expected event or can produce a repeated lifecycle event. Treat the event stream as operational observability, not an exactly-once audit ledger, and reconcile against task and run state.

Single writer#

Delta uses a renewable writer lease to reject a different machine holder. The lease defaults to 30 seconds and never goes below 5 seconds. On a renewal miss the heartbeat renews-or-reacquires: it reclaims the lease when it is free or already self-held - so a Fly suspend/resume across a wall-clock jump no longer tears the daemon down without releasing - and the process exits only when a genuinely different live holder owns it.

The holder ID defaults to FLY_MACHINE_ID, then the host name. A process with the same holder can immediately reacquire the lease so a crash restart does not wait for expiry. That also means the lease does not distinguish two live processes on the same machine. Do not set one manual DELTA_LEASE_HOLDER across different machines, because that defeats cross-machine exclusion too.

Binding the same port catches an ordinary duplicate start before work resumes. It does not protect two processes using different ports. In particular, two delta dev launches choose different free ports and can open the same bundle database under the same host identity. Never run two daemons for one database or bundle. Enforce one replica in the process manager or control plane rather than treating the current lease as a complete same-machine mutex.

The lease is coarse, best-effort exclusion, not per-write fencing. Clock skew, an event-loop pause longer than the TTL, or identical holder identities can permit overlapping work. Infrastructure must enforce one live owner for each database and volume.

Local retention#

Defaults:

The sweep runs once at boot and then hourly. Set DELTA_RETENTION_SWEEP_MS=0 to disable only the periodic sweep. When telemetry export is active, the ordinary event sweep is skipped entirely and exporter overflow owns event deletion, while the sweep still prunes the journal. Set journal retention longer than the maximum run duration: deleting an old completed intent or result from an active multi-day run can remove restart replay protection.

This is not whole-database retention. Sessions, runs, messages, memory, promotions, captured calls, .delta/spill, and .delta/media are not age-pruned. Workspace trash is swept only at startup. SQLite reuses deleted pages but Delta does not run VACUUM. Plan database and file maintenance according to workload.

Shutdown is interruption-oriented, not drain-aware. On a termination signal Delta clears timers, force-stops the server, and closes the database without waiting for queued work, background reflection, or telemetry flush. Before planned maintenance, gate new traffic and use external queue and activity checks to wait for an acceptable state, or rely on documented crash recovery. The four-worker queue cap does not bound background reflection, because completed runs can start multiple untracked reflections concurrently.

Long-running tasks and context management#

Delta is built to run for hours across dozens of tool calls without the active context window filling up and without dropping earlier findings. The mechanism is a single principle: restorable context - nothing load-bearing is ever unrecoverable. Every compression the engine performs leaves a pointer back to the full thing on disk or in the local database, so a fact found at step 3 is still available at step 50.

The active window stays bounded#

Before every model call, the engine estimates the fully assembled request - system spine, tool schemas, ephemeral blocks, message history, and reserved output - using serialized byte size rather than a naive character count, and treating attached images as a fixed token reserve rather than their base64 length. If the estimate exceeds DELTA_COMPACT_AT_TOKENS, it compacts before sending, so a resumed or continued session cannot overflow on its first call. It also compacts when the previous call's real gross prompt size crossed the threshold, since cached tokens still occupy the model's window.

Compaction is archive-safe. Older turns are summarized into a structured note and their message rows are marked inactive - never deleted or overwritten. The recent tail is kept verbatim, sized by a token budget rather than a fixed message count, and snapped to a turn boundary so an assistant tool call always travels with its result. The request being served is pinned verbatim - since 0.2.15 this is the compacting run's own request, read fresh from the run record and bound to the session, never the session's first request (on threaded sessions the first-run pin re-instructed the agent to do old work in 42 of 42 measured cases). The summary is wrapped in a trusted-request / untrusted-history envelope with an explicit end marker so a weak model cannot read the historical note as fresh instructions. Load-bearing identifiers (IDs, counts, paths, years) that the summarizer drops despite an audit-and-retry pass ride a machine-built appendix inside the summary's size cap, so compression can no longer silently lose the values a task turns on.

The context window is one dial:

DELTA_COMPACT_AT_TOKENS=120000

Bounding what the model writes (0.2.12, opt-in)#

Tool results have always been capped on arrival and capped again at compaction. A tool call's arguments never were: they live on the assistant message and are replayed on every later turn, so an agent that files large results page by page pays, forever, to re-read text it already banked.

DELTA_TOOL_ARG_MAX_BYTES bounds them. It applies as both the per-value threshold and the total ceiling, so the invariant is simply "a stored tool call's arguments never exceed N bytes".

# 0 (the default) disables it entirely - this is also the rollback lever.
# 4096 is the recommended value if you enable it.
DELTA_TOOL_ARG_MAX_BYTES=4096

Only a call that succeeded is touched, and only its largest values, largest first, until the object fits. The keys and every small field survive, so send_email keeps to and subject and loses only body - the model can still see what it did, which is what stops duplicate side effects. The full arguments stay in the execution journal under that table's existing retention, and recall can list them, search inside them, and read one back.

It ships opt-in for its first cycle. The rail is measured and sound, but the guard that stops a model copying the placeholder back into a later call was written in response to a live data-loss bug and cannot be proven complete against arbitrary MCP argument schemas. Turn it on deliberately, on one lane, and watch compaction count and post-compaction input rather than steady-state cache hit.

recall gains two modes alongside its keyword search: calling it with no arguments lists what has been dropped, and passing artifact: {run_seq, call_id, field} reads one back a page at a time.

Tight (60,000–90,000) is cheaper and lower latency and compacts more often. The default 120,000 is safe on any model with a 200,000-token window. Large (160,000+) keeps more continuity and performance and is safe to run because compaction is restorable - raise it only up to the model's real window minus output headroom.

At a tight ceiling the retained tail stays a flat 24,000 tokens on purpose: measured at 60,000, a proportional 12,000-token tail could not hold two capped tool results and a heavy run compacted 55 times in 92 turns, where the flat tail plus the ledgers below finished the same prompt in 18 turns.

The summary preserves facts across many generations#

When a prior summary is already present, the engine merges forward - it instructs the summarizer to preserve every prior fact and only add new ones, rather than lossily re-summarizing a summary each generation. After each summary it audits the result against an anchor index harvested by class with a budget each: URLs, email addresses, slugs, recurring proper names from the agent's own text, years, and numbers, plus everything the previous summary carried. What the summarizer dropped rides a machine-built, defanged appendix inside the summary's byte budget, and a summary the model could not finish (a length stop) is retried once with the better candidate kept. Measured on a 23-run battery: the summarizer dropped 30% of the audited anchors and the appendix carried 97% of those back, against 30 of 30 lost per compaction on the previous engine.

Two deterministic ledgers ride every summary as well, because they do not depend on the summarizer at all: the spill paths of every large result the compaction covered, and the calls it summarized (builtin calls with their arguments, deduped and newest first; MCP tools by name and count). The second one exists because the most expensive failure on a long run is not a lost fact but a repeated search: after a cut the agent re-runs what it already ran. One fixed line at the end names recall as the way to read an earlier result back instead.

What a summary cannot do is hold a whole region of work. On real production compactions, a first-generation summary answers about one grounded question in four about the turns it replaced and a second-generation one almost none, whatever writes it: a 350-word note over 300 KB of transcript is a capacity limit, not a prompt problem. The design bet is therefore structure in the summary and a recovery path that works, not a longer summary.

Recovering what scrolled out#

Two tools let an agent reach past the live window:

• recall searches this conversation's earlier turns, including ones compacted out of the active window. It runs on a full-text index over the message table (case and Latin diacritics folded, prefix matching), any word of the query qualifies a row (function words are dropped), rare words outrank common ones, and a whole-phrase match ranks first, so "Maria Delgado at Acme" finds the row that says "Maria Delgado, Acme Robotics". It returns the matching finding - the whole message when it is reasonably sized, not a fragment - along with the on-disk path of any large result that was spilled. On identical questions drawn from production compactions, this search answered 53% where the old whole-phrase match answered 21%, and 72% on the facts the agent itself had written down. Searches are scoped to the newest 5,000 rows of the current session (an index probe, not a scan); a caller cannot search another session's transcript.
• Tool results larger than the inline cap are spilled to a file under .delta/spill, and compaction records every spill path in an artifact ledger inside the summary, so a large result is always one read_file away.

An agent can also keep its own working plan with the todo tool. The plan is re-injected every turn as an ephemeral block that survives compaction, because it is rebuilt from a per-session store rather than living in the message history. It is the cheapest way to hold a running list of findings across a long run, and the tool tells the agent when the plan outgrows its budget so it can offload longer notes to a workspace file instead of silently losing them.

Delegating heavy exploration#

For work that would bloat the primary window, the research tool runs one to three sub-agents in parallel, in-process. Each explores in its own bounded context and returns only a short summary plus the path to a findings file the parent writes, so the primary conversation absorbs the signal, not the exploration. A research child's callable tools are the parent's read-only tools and nothing else: admission is positive and fail-closed on a tool's own read-only flag, so a builtin, MCP, or future tool that is not explicitly marked read-only never reaches a child. A child therefore cannot write_file, remember, move or delete, run code, or delegate further, which is also what keeps nesting exactly one level deep. Give a child a question to answer rather than a change to make. It is forged from the same system spine as the parent, so it inherits the same norms and policy, and it starts on the parent's pinned tool set with search_tools for the rest.

Boundaries#

The token estimate is a conservative heuristic, not exact provider tokenization, so the post-provider overflow path remains a backstop that sheds aggressively and retries once. The summary audit harvests identifiers by class (URLs, emails, slugs, recurring proper names from the agent's own text, years, numbers) with a budget per class and 60 in total; anything outside those classes, or beyond the budget, the summarizer preserves but the engine does not verify. recall is an index probe over the newest 5,000 rows of the session, not a scan of an unbounded transcript, and it is lexical: a paraphrase that shares no word with the original does not match. The re-injected plan is bounded and meant for a running list, not a large document - large findings belong in a workspace file. These are the seams to watch on genuinely long, high-volume runs.

Debug and inspect with Cockpit#

The Cockpit is an active JavaScript application embedded in every compiled binary at /dev. Its asset contains no embedded runtime data or credentials; it reads the separate /v1/dev/* inspection API after loading.

delta dev is the fastest local path because it defaults editing and normalized successful-call capture to on unless delta.env explicitly overrides those settings.

Five tabs#

The header shows the stable agent ID plus cumulative tokens, cost, and the count of distinct runs that have at least one completed captured model call in the current live view.

Normalized model-call inspection#

With DELTA_CAPTURE_CALLS=1, each successfully served main-loop model turn stores Delta's normalized request and response representation:

• the assembled system spine
• active conversation messages
• ephemeral context, request instructions, and retrieved capability blocks
• active tool schemas
• reasoning effort and cache key
• returned assistant message and tool calls
• actual model and provider
• token usage, cost, and latency

Images remain file markers instead of copied base64. Capture occurs before provider-specific wire transformation. It is not raw HTTP and does not include the final URL, headers, SSE chunks, failed attempts, or retry traffic. The Cockpit's turn detail shows this normalized request and response after redaction on read.

Captured rows contain full prompts and outputs in SQLite before the read-time redaction. There is no automatic retention for the calls table. Leave capture off in production unless that storage and sensitivity are intentional.

Capture records the provider call that successfully served a model turn. Failed provider attempts, internal transport retries, and cascade candidates that fail before a successful result are not stored as call rows.

Hydration, capability lookup, reflection, promotion, compaction, and judging can call models or tools through internal paths that do not all pass through the ordinary tool journal, checkpoint, or call capture. Cockpit provides deep visibility into the main execution loop, but it is not a complete record of every internal invocation or decision.

Replayable event stream#

The Cockpit and delta watch use:

GET /v1/dev/stream?since=<event-id>&run=<run-id>&session=<session-id>&live=1

Persisted frames have monotonically increasing IDs and can be replayed. Live text and reasoning deltas are ephemeral and have no durable ID. The Cockpit stream terminates a client after its buffer grows beyond 2,048 frames. Bundled Cockpit and delta watch do not automatically resume, so a robust custom client must persist the last durable ID and reconnect with since.

Inspection API#

Apply edits#

Cockpit edits write bytes to disk. They do not hot-reload the daemon:

• an edited DELTA.md applies to the next run
• an edited POLICY.md, PROMPT_CONTEXT.md, or vocab.json requires restart
• changed provider, MCP, or environment configuration requires restart

This keeps the dev launcher behavior equal to production behavior.

The Files tree hides .delta, but that is presentation, not an access-control boundary. With root inspection, a direct /v1/dev/files request can currently address paths under the workspace .delta directory; when inspection writes are enabled, it can damage live state such as the development database. Treat the inspection token as root-equivalent, keep writes off in production, and block arbitrary inspection-file calls at a proxy unless the operator explicitly needs them.

Inspect production safely#

delta send and delta watch connect only to localhost. delta dev is a local launcher that opens a localhost Cockpit URL; it is not a remote-attachment command, although an explicit bind override can make its child daemon listen elsewhere.

Production inspection uses the same /dev and /v1/dev/* surfaces, with these gates:

• DELTA_INSPECT_TOKEN protects inspection data with a distinct bearer
• without that token, inspection data is loopback-only
• DELTA_INSPECT_WRITE=1 separately enables edits and self-file revert
• DELTA_INSPECT=off removes both /dev and the inspection API
• DELTA_CONTROL_TOKEN protects run-driving endpoints but does not grant inspection

The bundled browser UI does not store or attach bearer tokens. For remote use, put a trusted same-origin proxy in front of the daemon. It must authenticate the operator, inject the inspect bearer for /v1/dev/*, and inject the control bearer for Cockpit task and upload requests. Loopback is checked from the immediate socket peer, so a same-host reverse proxy makes remote traffic appear local. Never rely on tokenless loopback inspection behind a proxy; set a distinct inspect token or disable inspection, and block /dev plus /v1/dev/* unless the operator is authenticated.

Telemetry and events#

Restoring a volume blinds the exporter until the row counter catches up. Exported ids are <daemon uuid>:<events row id> and a collector dedupes on them. A database restored from an older snapshot re-issues ids the collector has already seen, so its events are dropped as duplicates, silently. After any restore, advance the events sequence past the collector's last id for that daemon, or start the lane from an empty database.

Delta emits one correlated event stream and gives you three ways to consume it. Two are free and built in; the third ships events off the box to a collector you run.

1. Local, always on - every durable event is written to the local SQLite events table.
2. Live tail - in-process subscribers drive task SSE and the Cockpit's GET /v1/dev/stream (replay from a cursor, then follow).
3. Ship it out (optional) - a background exporter pushes NDJSON envelopes to an HTTP endpoint you own.

The stream is a correlated event log, not distributed tracing: there are no spans or parent links, only a flat correlation spine - user → agent → session → run → task → entity → turn. Field names follow a subset of OpenTelemetry GenAI conventions (gen_ai.request.model, gen_ai.usage.*, latency_ms), but Delta embeds no OpenTelemetry SDK and emits no OTLP; the wire format is line-delimited JSON.

The fastest path: no collector at all#

Before wiring anything, note that (1) and (2) already exist with zero configuration. delta dev opens the Cockpit; GET /v1/dev/stream?since=<event-id>&run=<run-id>&live=1 replays the persisted events table from a cursor and then follows live (gated by DELTA_INSPECT_TOKEN). For a single agent, that is your dashboard. The exporter below is for when you run many agents and want their events in one queryable place.

Enable export#

Set two variables on the daemon and the exporter starts:

TELEMETRY_URL=https://your-collector.example/telemetry/ingest
TELEMETRY_TOKEN=a-dedicated-collector-token

TELEMETRY_URL is the only thing that turns the exporter on. If TELEMETRY_TOKEN is absent, Delta falls back to DELTA_CONTROL_TOKEN as the bearer - convenient, but it sends a control credential off the box, so prefer a dedicated token. The exporter does not enforce HTTPS or an allowlist, so validate TELEMETRY_URL yourself and never post a control token to a plaintext or untrusted endpoint.

What exports by default, and what needs consent#

For the three payload-bearing event types - model.call, tool.call, tool.result - the exporter applies a safe-attribute allowlist. By default (no DELTA_CAPTURE_PAYLOADS) a closed set of operational attributes survives: model and provider, token usage and cost, latency, cache-hit percent, reasoning effort, served speed, the fallback flag, is_error, error.class, and executed tool names and ids. That is enough to run cost, latency, fallback, and error dashboards on a default deployment. What is stripped without consent: prompt text, tool arguments, tool results, the 200-char error.message snippet, and the model's raw requested-name list (tool_calls) - anything an attacker-influenced string could ride out on.

To also export that sensitive payload - full prompts, tool arguments, tool results, error.message, and tool_calls - turn on capture:

DELTA_CAPTURE_PAYLOADS=1

Enable it wherever you already trust the collector; the default subset already gives you the operational picture without shipping any prompt, argument, or result off the box.

A separate flag, DELTA_CAPTURE_CALLS, captures true-to-life full request/response bodies into a local calls table for the Cockpit. It is local-only, storage-heavy, and never exported - right for deep local debugging, wrong for the telemetry stream. Don't confuse the two.

A reference collector you can copy#

A complete, single-agent collector lives in the repo - a Postgres table and a ~60-line Bun handler, distilled from the multi-tenant one Carrara Labs runs in production:

• docs/telemetry-collector/schema.sql - the agent_events landing table
• docs/telemetry-collector/server.ts - the NDJSON ingest handler
• docs/telemetry-collector/README.md - run it, point an agent at it, and the dashboard queries

Apply the schema, run the server, set the three variables above, and rows appear in agent_events within a couple of seconds of the agent doing work.

Delivery contract#

The exporter:

• sends application/x-ndjson
• wakes every two seconds
• sends up to 200 records per batch
• gives each request a 15-second timeout
• marks rows exported only after a 2xx
• retries network and non-2xx failures
• provides at-least-once delivery only while an unexported row remains in the bounded outbox
• gives every record a restart-stable <daemon-id>:<row-id> deduplication key
• bounds its outbox at 50,000 rows by default
• deletes exported rows first on overflow
• drops the oldest unexported records only when the unexported backlog alone exceeds the cap

Retries are serial on a fixed two-second cadence with no exponential backoff or jitter. When telemetry is active, normal event retention is disabled: the exporter owns events deletion up to the outbox cap, and visibility reports backlog count rather than oldest age. Your collector must deduplicate on event.id (the exporter is at-least-once), and you should monitor collector health and backlog externally - if the collector stays down, the oldest unexported rows are dropped once the outbox fills.

Record shape#

With DELTA_CAPTURE_PAYLOADS=1, an exported model.call record has this shape:

{
  "event.id": "daemon-uuid:123",
  "event.name": "model.call",
  "event.time_unix_ms": 1783950000000,
  "user.id": "user_123",
  "agent.id": "agent_123",
  "session.id": "sess_123",
  "run.id": "resp_123",
  "task.id": "task_123",
  "entity.id": "entity_123",
  "turn": 2,
  "attributes": {
    "gen_ai.request.model": "anthropic/claude-sonnet-5",
    "gen_ai.provider": "openrouter",
    "gen_ai.response.finish_reasons": ["tool_calls"],
    "gen_ai.usage.input_tokens": 1200,
    "gen_ai.usage.output_tokens": 300,
    "gen_ai.usage.cached_tokens": 900,
    "gen_ai.usage.cost_usd": 0.004,
    "latency_ms": 1800,
    "gen_ai.request.effort": "medium",
    "tool_calls": ["read_file", "run_code"]
  }
}

Every attribute above except tool_calls is in the default safe subset, so it exports without DELTA_CAPTURE_PAYLOADS. A fallback: true and a speed field appear when those conditions hold.

Events and their attributes#

The stream carries run lifecycle, turn boundaries, model and tool calls, context management, the learning loop, and errors. The attributes worth querying:

Where the model comes from: the executing model is only on model.call (capture on). run.finished carries usage but not the model id, so for per-run model read it from model.call or from your own agent registry.

The dashboard is SQL#

Once events land, the dashboard is just queries - no UI required to start. A few starters (full set in the collector README):

-- Spend and tokens per agent (dollar figures are token×price; notional under a flat subscription)
SELECT agent_id, count(*) AS runs,
       round(sum((attributes->>'gen_ai.usage.cost_usd')::numeric), 4) AS cost_usd
FROM agent_events WHERE event_name = 'run.finished' GROUP BY 1 ORDER BY cost_usd DESC;

-- Which model / provider is actually executing (needs DELTA_CAPTURE_PAYLOADS=1)
SELECT attributes->>'gen_ai.request.model' AS model,
       attributes->>'gen_ai.provider'      AS provider, count(*) AS calls
FROM agent_events WHERE event_name = 'model.call' GROUP BY 1, 2 ORDER BY calls DESC;

-- Errors, clustered by type
SELECT attributes->>'error.type' AS error_type, count(*) AS n
FROM agent_events WHERE event_name = 'error' GROUP BY 1 ORDER BY n DESC;

Mistakes to avoid#

1. Leaving DELTA_CAPTURE_PAYLOADS off. You get event counts but no model, tool names, latency, or per-call cost. The most common gap.
2. Assuming that flag leaks prompts. It doesn't - operational attributes only. Don't stay blind out of a misplaced privacy fear.
3. Returning 5xx on a malformed line. The daemon then retries the whole batch forever. Skip bad lines and answer 2xx for accepted work; reserve 5xx for a genuine storage failure - which should make the daemon retry.
4. Not deduping on event.id. Delivery is at-least-once; without ON CONFLICT DO NOTHING you double-count everything.
5. No body cap. Bound the batch (the reference collector caps at 8 MB → 413) so an authed client can't OOM the collector.
6. Trusting identity from the payload. Derive tenant/owner from the authed token, never from ids on the wire, or one caller can forge another's rows.
7. Posting a control token to a plaintext or untrusted collector. Use a dedicated TELEMETRY_TOKEN over TLS.
8. Not monitoring backlog. When telemetry is on, the daemon stops its own event retention; if the collector is down, unexported rows climb to the 50,000 cap and then the oldest are dropped. Watch it externally.
9. Expecting the model on run.finished. It's on model.call (capture on) or your own registry.
10. Treating "capture off" as content-free. Non-payload events (recall, error) can carry bounded excerpts or diagnostic strings. Give the collector a data-classification and retention policy regardless.

Privacy#

Payload filtering does not mean content-free telemetry. Other events retain attributes: recall can include a bounded memory excerpt, and errors or promotion events can include diagnostic strings. Credential-like patterns are scrubbed, but personal or business data can remain. Send telemetry only to a trusted collector with an explicit data-classification and retention policy.

OpenTelemetry#

The stream is NDJSON with OTel-GenAI-inspired names, but it is not OTLP and not semantic-convention complete: fields such as gen_ai.provider, gen_ai.usage.cached_tokens, gen_ai.usage.cost_usd, and latency_ms are Delta-specific or legacy-shaped. Send NDJSON to a collector that understands this contract (like the reference one above), or add a translation layer before an OTLP-only collector.

Deploy Delta#

Production shape#

The intended unit is one daemon per agent:

trusted gateway or control plane
  -> one Delta process or VM
     -> one persistent SQLite database
     -> one persistent workspace
     -> model providers and MCP services
     -> optional telemetry collector

Do not run multiple writers against one database. Do not share one workspace between unrelated agents. A single persistent mount may contain both, but they remain separate logical state.

Run the compiled binary directly#

Create a production workspace containing only the four non-secret bundle files. Do not reuse a development workspace that still contains a populated delta.env:

mkdir -p runtime/workspace
cp my-agent/DELTA.md my-agent/POLICY.md my-agent/PROMPT_CONTEXT.md my-agent/vocab.json runtime/workspace/

Set real environment variables, then launch the daemon:

export PORT=8080
export DELTA_BIND=127.0.0.1
export DELTA_DB="$PWD/runtime/delta.db"
export DELTA_WORKSPACE="$PWD/runtime/workspace"
export DELTA_AGENT_ID=my-research-agent
export OPENROUTER_API_KEY=your-key
export DELTA_MODEL_PRIMARY=anthropic/claude-sonnet-5
export DELTA_CONTROL_TOKEN=replace-with-a-long-random-token
export DELTA_INSPECT_TOKEN=replace-with-a-different-long-random-token

./dist/delta

The direct defaults are PORT=8080, DELTA_DB=data/delta.db, and DELTA_WORKSPACE=workspace. The default bind is Bun's all-interface behavior, so set DELTA_BIND deliberately.

Check liveness and build identity:

curl -sS http://127.0.0.1:8080/healthz

{"ok":true,"version":"0.2.17","build":"optional-commit"}

build appears only when DELTA_BUILD is set. This endpoint does not test the model, MCP servers, telemetry, or other dependencies. It is liveness and version metadata, not readiness.

Run a production acceptance smoke#

Use the gateway URL and the two distinct credentials provisioned on that agent. Replace crm__get_dashboard with one required, read-only MCP tool and include any arguments its policy expects. Run this on a placement with reflection off if the smoke must not create local memory or external proposals; request metadata cannot disable a daemon-wide DELTA_REFLECT=1.

export DELTA_URL=https://agent.example
export DELTA_CONTROL_TOKEN=run-api-token
export DELTA_INSPECT_TOKEN=separate-root-inspection-token

curl -fsS "$DELTA_URL/healthz"

SMOKE_RESPONSE="$(curl -fsS "$DELTA_URL/v1/responses" \
  -H "authorization: Bearer $DELTA_CONTROL_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"input":"Call crm__get_dashboard once using read-only smoke-test arguments, then report its result and the exact tool name."}')"

SMOKE_RUN_ID="$(SMOKE_RESPONSE="$SMOKE_RESPONSE" bun -e '
const response = JSON.parse(process.env.SMOKE_RESPONSE ?? "{}");
if (!response.id || response.status !== "completed") {
  console.error(JSON.stringify(response));
  process.exit(1);
}
process.stdout.write(response.id);
')"

curl -fsS "$DELTA_URL/v1/dev/config" \
  -H "authorization: Bearer $DELTA_INSPECT_TOKEN"

curl -fsS "$DELTA_URL/v1/dev/runs/$SMOKE_RUN_ID" \
  -H "authorization: Bearer $DELTA_INSPECT_TOKEN"

The status assertion matters because a terminal Delta failure is a valid Responses object returned with HTTP 200; curl -f alone cannot detect it. In /v1/dev/config, verify the harness version and build, model route, mcp_servers, required names under tools.mcp, and credential-presence booleans. Inspect /v1/dev/runs/$SMOKE_RUN_ID/calls as an additional check when normalized successful-call capture is deliberately enabled.

Finally, verify that the collector received correlated run, model, and expected tool events for SMOKE_RUN_ID, with stable event.id values. Delta has no dependency-readiness endpoint, so provider, MCP, and telemetry verification must exercise those paths.

Build and run the container#

Build with optional commit provenance:

docker build \
  --build-arg DELTA_BUILD="$(git rev-parse HEAD)" \
  -t delta:local .

DELTA_BUILD is exact provenance only when the build context is a clean, immutable checkout of that commit. On a dirty tree it is merely a HEAD label and does not describe the changed bytes. The root .dockerignore allowlists only the Dockerfile, build metadata, source, Cockpit asset, and entrypoint, so credentials and unrelated workspace state are not sent to the builder.

The final image contains the compiled binary, CA certificates, unzip for lightweight document extraction, Litestream, and the entrypoint. It does not contain Bun or a coding CLI.

Seed the four non-secret bundle files on the first start:

DELTA_SELF_MD_B64="$(base64 < my-agent/DELTA.md | tr -d '\n')"
DELTA_POLICY_MD_B64="$(base64 < my-agent/POLICY.md | tr -d '\n')"
DELTA_CONTEXT_MD_B64="$(base64 < my-agent/PROMPT_CONTEXT.md | tr -d '\n')"
DELTA_VOCAB_JSON_B64="$(base64 < my-agent/vocab.json | tr -d '\n')"

export DELTA_SELF_MD_B64 DELTA_POLICY_MD_B64 DELTA_CONTEXT_MD_B64 DELTA_VOCAB_JSON_B64

Run with persistent /data and real API protection:

docker volume create delta-data

docker run -d \
  --name delta-agent \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  --env-file my-agent/delta.env \
  -e DELTA_DB=/data/delta.db \
  -e DELTA_WORKSPACE=/data/workspace \
  -e DELTA_CONTROL_TOKEN=replace-with-a-long-random-token \
  -e DELTA_INSPECT_TOKEN=replace-with-a-different-long-random-token \
  -e DELTA_SELF_MD_B64 \
  -e DELTA_POLICY_MD_B64 \
  -e DELTA_CONTEXT_MD_B64 \
  -e DELTA_VOCAB_JSON_B64 \
  -v delta-data:/data \
  delta:local

The entrypoint writes seed files only when the target does not exist. A later image or environment change does not overwrite an evolved DELTA.md or other workspace file. Malformed base64 is skipped with a warning, so startup can continue with a missing or neutral bundle; verify all four files after first boot. Explicit /data paths after --env-file prevent an edited local delta.env from relocating persistent state.

The container defaults are:

PORT=8080
DELTA_DB=/data/delta.db
DELTA_WORKSPACE=/data/workspace

Persist the entire /data mount. Persisting only the database loses uploaded files, working artifacts, and the current self-file. Persisting only the workspace loses conversations, checkpoints, memory, events, and revisions.

The checked-in Dockerfile has no USER instruction, so the standard image runs as root. Before a production deployment, derive an image or set a runtime user with only the permissions Delta needs, and make sure that identity owns the persistent /data paths. The restart policy in this example is only a local Docker baseline; a production orchestrator or service manager must also supervise the process and its persistent volume.

Add the coding tool to an image#

The code tool degrades to a normal tool error when its executable is missing. Build a derived image when advanced coding is required:

FROM delta:local

# Install the chosen coding CLI here, then run Delta as a dedicated user.
# Provision CLI authentication at runtime rather than baking it into the image.

Exact installation depends on the chosen CLI. Confirm that DELTA_CODE_CLI resolves inside the final image and that its home-directory credentials survive VM restarts without being baked into a public layer.

Deploy on Fly Machines#

fly.toml.sample is a topology example, not a one-command managed service. It describes:

• one agent per app or Machine
• one shared CPU
• 512 MB starting memory, to be load-tested against the real workload
• /data on a persistent volume
• health checks against /healthz
• no Fly Proxy service; a trusted private 6PN control plane or gateway owns access
• lifecycle owned by an external controller

Create an app-specific fly.agent.toml from this checked-in topology, replacing the app name and region:

app = "delta-<agent>"
primary_region = "iad"

[build]
  dockerfile = "Dockerfile"

[env]
  PORT = "8080"
  DELTA_BIND = "0.0.0.0"
  DELTA_DB = "/data/delta.db"
  DELTA_WORKSPACE = "/data/workspace"
  DELTA_AGENT_ID = "my-research-agent"
  DELTA_BUILD = "<source-commit>"
  DELTA_MODEL_PRIMARY = "anthropic/claude-sonnet-5"

[[mounts]]
  source = "delta_state"
  destination = "/data"

[checks]
  [checks.delta_health]
    type = "http"
    port = 8080
    method = "get"
    path = "/healthz"
    interval = "15s"
    timeout = "2s"
    grace_period = "5s"

[[vm]]
  memory = "512mb"
  cpus = 1
  cpu_kind = "shared"

Create the Fly app and a persistent volume named delta_state in the configured region. Provision the provider, control, inspection, MCP, and telemetry credentials as Fly secrets. On the first boot, also provide DELTA_SELF_MD_B64, DELTA_POLICY_MD_B64, DELTA_CONTEXT_MD_B64, and DELTA_VOCAB_JSON_B64 from the bundle-seeding step above, or populate /data/workspace through another trusted provisioning path. Without a stable DELTA_AGENT_ID and those bundle files, the Machine starts a neutral daemon rather than the intended agent.

Deploy with fly.agent.toml after adapting the provider, region, tools, telemetry, and exact source commit. This private topology intentionally declares neither [[services]] nor [http_service], so it does not create a direct Fly Proxy URL. A [[services]] block requires at least one [[services.ports]] entry; do not add an incomplete service merely to attach a health check. The top-level check reaches port 8080, and Delta must listen on 0.0.0.0 inside the Machine. Route access through the trusted private control plane, or add a deliberate public gateway using Fly's current service-port syntax. The repository sample does not automate app, volume, secret, gateway, or lifecycle provisioning. Never put secret values in the TOML file.

Because the sample has no Fly Proxy service, it has no proxy auto-start or auto-stop policy. Suspend and resume through an external lifecycle controller using the Machines API. That controller must gate new traffic and avoid stopping while /v1/busy reports work in flight unless it deliberately accepts interruption and recovery. /v1/busy is the purpose-built suspend-safety signal; the hosting contract documents the full wake-before-dispatch, busy-check-before-suspend, suspend-after-terminal lifecycle. Enforce exactly one Machine for the volume, for example by deploying without high availability and asserting the invariant in the controller. A standalone deployment without that controller should keep the Machine running. Delta itself does not provision, suspend, or resume cloud infrastructure.

The 100 MB upload batch limit is not a streaming memory guarantee: multipart parsing buffers and copies request data. A 256 MB microVM can exhaust memory near the cap, especially with document extraction, coding CLIs, or subagents. Set a lower gateway upload limit or size and load-test the Machine for worst-case concurrent work.

Back up SQLite with Litestream#

The entrypoint uses Litestream when either of these is configured:

• a mounted LITESTREAM_CONFIG, default path /etc/litestream.yml
• a direct LITESTREAM_REPLICA_URL

Use this S3-compatible configuration. It provides one-second replication, hourly snapshots, and seven-day retention:

dbs:
  - path: ${DELTA_DB}
    replicas:
      - type: s3
        bucket: ${LITESTREAM_S3_BUCKET}
        path: ${LITESTREAM_S3_PATH}
        endpoint: ${LITESTREAM_S3_ENDPOINT}
        region: ${AWS_REGION}
        access-key-id: ${AWS_ACCESS_KEY_ID}
        secret-access-key: ${AWS_SECRET_ACCESS_KEY}
        sync-interval: 1s
        snapshot-interval: 1h
        retention: 168h

Save it outside the image or use etc/litestream.yml. The image does not copy that file to /etc/litestream.yml, so mount it explicitly:

-e LITESTREAM_CONFIG=/etc/litestream.yml
-e LITESTREAM_S3_BUCKET=your-bucket
-e LITESTREAM_S3_PATH=agents/my-agent
-e LITESTREAM_S3_ENDPOINT=https://your-s3-endpoint
-e AWS_REGION=auto
-e AWS_ACCESS_KEY_ID=your-key
-e AWS_SECRET_ACCESS_KEY=your-secret
-v "$PWD/etc/litestream.yml:/etc/litestream.yml:ro"

Add those options to the earlier docker run command. Keep the model, bundle seed, authentication, port, and /data volume options from that command.

For one genuine first boot against an empty replica, also add:

-e DELTA_BOOTSTRAP=1

Recreate the container without that option as soon as the first database exists.

First-boot rules are intentionally strict:

1. If the local DB exists, Delta keeps it and Litestream starts replication.
2. If the DB is missing and a replica is configured, restore must succeed.
3. A genuine first boot against an empty replica must set DELTA_BOOTSTRAP=1.
4. Remove DELTA_BOOTSTRAP=1 after the database exists.
5. If the volume is later lost while bootstrap remains enabled, the container starts blank instead of restoring.

DELTA_BOOTSTRAP has an effect only when backup is configured and the local database is absent. An existing local database always wins, even when a replica contains newer state. A restored stale volume can therefore start and replicate from stale local data. The entrypoint has no replication-freshness watchdog, and /healthz remains green if backup export has stalled. Monitor Litestream's last successful sync externally and alert on lag.

Litestream backs up only DELTA_DB. It does not back up the workspace, including the current DELTA.md, uploads, working files, spill files, or trash. Use volume snapshots, object-storage synchronization, or another file backup for DELTA_WORKSPACE. Complete recovery also needs the current rotating MCP refresh-token file when that feature is enabled; never restore a stale, already-spent token. Database, workspace, and refresh state have no coordinated point-in-time snapshot. For a consistent backup, quiesce new and active work before capturing them together, or define and test an ordering plus reconciliation procedure for database rows that refer to missing files.

Upgrade and roll back#

Delta runs SQLite migrations transactionally when opening the database. It records harness and schema versions and refuses to open a database whose schema is newer than the binary understands.

Before a production upgrade:

1. record /healthz version and DELTA_BUILD
2. confirm SQLite replication and workspace backup freshness
3. stop or suspend the single writer cleanly
4. take a database and workspace snapshot
5. deploy the pinned image digest
6. verify /healthz, a model request, required MCP tools, Cockpit access, and telemetry
7. monitor resumed runs and promotion failures

There is no automatic pre-migration snapshot. A rollback after a schema-changing release may require restoring the pre-upgrade database together with a compatible workspace snapshot.

Upgrading to 0.2.17 (schema v15 to v16). The recall index is a one-way migration, backfilled once at boot (about a second per 100 MB). Any v15 database (0.2.13 through 0.2.16) upgrades directly; a v16 database will not open under an older binary, so the snapshot in step 4 above is the rollback. Stop the old daemon before the new one opens the same volume: the write lease lives inside the database, and a still-running writer sees database schema is locked during the rebuild (a machine replacement on Fly already sequences this). No configuration change is required; DELTA_CACHE_DIAGNOSIS=1 is the only new wire-level option and is opt-in. After the upgrade, verify /healthz reports the new version, pragma user_version reads 16, the agent's DELTA.md hash is unchanged, and telemetry ids keep increasing past the pre-upgrade high-water (see Telemetry stops after a volume restore). The operator page with the Fly, systemd and Connect recipes and the acceptance checks is docs/upgrade-0.2.17.md in the repository.

Upgrading to 0.2.7 from an earlier version. This release is drop-in: with the new environment variables unset, a deployment behaves exactly as 0.2.6. Two things are worth knowing:

• The tiers were renamed work to trusted and chat to safe. Old names still resolve as aliases, so no configuration change is required; only the canonical name in telemetry and GET /v1/status changes. If a dashboard or alert keys on the literal string work or chat, update it to trusted / safe.
• New settings, all opt-in: DELTA_ALLOWED_TOOLS / DELTA_PINNED_TOOLS (a custom tool envelope, see Tiers and the tool envelope); DELTA_SKILLS = mcp | local | off (skills backend, default mcp unchanged, see reusable skills); DELTA_SAFE_MODE=1 (recovery boot, below). Leaving each unset changes nothing.

Safe mode (recovery). If an agent's self-file or bundle becomes corrupted - accidentally, or by a prompt-injection that reached the writable memory - boot it once with:

DELTA_SAFE_MODE=1

The daemon starts neutral and unwedgeable: the safe tool floor, no MCP servers, no self-write, no reflection, and it skips the workspace prompt layers (DELTA.md, POLICY.md, PROMPT_CONTEXT.md) so a poisoned self-file cannot influence it. You can always reach the agent to diagnose it, revert the self-file from a prior revision through the Cockpit, then restart normally. Safe mode ignores DELTA_ALLOWED_TOOLS and the self-write flags, so it cannot be widened.

Since 0.2.8, safe mode is observable and self-aware. GET /v1/status reports safe_mode: true, so an edge client (a channel gateway's /safemode or /status) can confirm it without reading the boot log. The agent is also told: in a safe-mode boot the system spine drops the configured agent name and states that the persona, policy, and learned self-file are not loaded, so the agent does not present as its configured identity or role-play one from earlier conversation history. To demonstrate this cleanly, pair a safe-mode restart with a fresh conversation so no prior identity carries over.

Read current model, provider, tier, and budget (GET /v1/status). A secret-free, seam-token-gated endpoint returns the running agent's version, tier, safe_mode, model, provider, reasoning effort, budget ceilings, and MCP server names - no keys. Since 0.2.15 it also reports tool usability in three states, because the operator's next action differs for each: tools.registered (in the live registry, read per request), tools.unusable (registered, but a live precondition says a call fails right now - a missing EXA_API_KEY, for example; this heals the moment a credential lands in the environment or the vault, no restart), and tools.omitted (never registered this boot, each with a reason - no vault, an unresolvable code CLI, sub-agent depth, missing control-plane wiring, or an allowlist name that matches nothing; fix the configuration and restart). Names and engine-authored reasons only, never values. The daemon also prints one stderr line at boot when anything is omitted or unusable. The allowlist remains a ceiling, not a guarantee - but the gap between the two is no longer silent. Since 0.2.8 the model view also names the provider in human terms (anthropic-native, openai-native, openrouter, codex-sign-in) with a provider_chain for the failover cascade, and reasoning_effort always resolves (an unset effort reports default rather than being omitted). It backs edge tooling such as a channel gateway's /model, /provider, and /status commands. Since 0.2.16 the model view also carries a controls block - speed, cache_ttl, text_verbosity, reasoning_summary, and unmapped, the list of configured knobs the primary lane's wire or host cannot render (the boot line and this field share one computation, so they can never disagree).

Security model#

Put a trusted gateway in front#

Set both credentials in production:

DELTA_CONTROL_TOKEN=run-api-token
DELTA_INSPECT_TOKEN=separate-root-inspection-token

When DELTA_CONTROL_TOKEN is set, every /v1/* route except the separately gated /v1/dev/* routes requires Authorization: Bearer <token>. /healthz remains public and data-light.

The run API is a daemon seam for a trusted control plane, not a complete public multi-tenant API:

• task lookup and cancellation are ID-based, without per-user ownership checks
• x-delta-user is trusted only for queue projection
• the request JSON endpoints have no built-in body-size limit
• the daemon has no rate limiter
• one control token can drive every run on that daemon

The gateway should authenticate users, authorize every task and previous_response_id, inject one canonical user identity and the machine token, strip caller-supplied privileged metadata, cap request bodies and queue depth, rate limit, log access, and expose only required routes. It must construct review and widening claims from verified control-plane state. POLICY.md is model guidance and cannot replace these checks.

Separate run access from root inspection#

The inspect token grants access to transcripts, prompts, memory, configuration shape, files, and selected database state. It is intentionally different from the control token. Keep Cockpit writes off unless an authenticated operator needs them:

DELTA_INSPECT_WRITE=0

For hardened agents with no inspection requirement:

DELTA_INSPECT=off

Without an inspect token, /v1/dev/* is available only to the immediate loopback socket peer. A same-host proxy defeats the intended remote distinction unless it injects a token. The /dev application contains no embedded runtime data, but direct public access to it is still unnecessary when inspection is disabled.

Treat the VM as the execution boundary#

The built-in file tools are workspace-confined, and model-directed code and subagent processes receive a narrowed service-token environment. That does not make Delta a general untrusted-code sandbox:

• read_file can read workspace files, including a workspace delta.env
• the delegated coding CLI has every filesystem and process capability available to the daemon's OS user; only the CLI's own sandbox can narrow it
• stdio MCP servers are trusted local child processes; since 0.2.10 they receive only process plumbing (PATH, HOME, SHELL, TMPDIR, LANG, LC_*, TERM) plus their own explicit env object, not the daemon's whole environment
• HTTP MCP tools can perform whatever side effects their servers authorize
• subagents share the workspace

Run one agent under a dedicated OS identity or isolated VM. Keep provider, control, telemetry, and MCP credentials in the process environment or secret manager, not workspace files. Add a purpose-built sandbox if arbitrary generated code must be treated as hostile.

The secret vault#

Third-party credentials an agent needs at runtime (a search key, a client API key, an MCP backend bearer) can live in an encrypted vault instead of the deployment environment. One rule governs it: a secret value never enters model-readable state.

Enable it with a key. Prefer the file form on a real deployment, because an environment variable is copied into the process environment block that any same-UID process can read:

DELTA_VAULT_KEY_FILE=/run/secrets/vault.key
# or, for local development:
DELTA_VAULT_KEY=<at least 16 characters>

Generate a key with openssl rand -base64 32. With neither set there is no vault at all: the routes answer 503, the list_secrets tool is not registered, and a {{vault:NAME}} reference fails closed. There is deliberately no plaintext fallback, and safe mode never carries a vault.

Provide a secret over the seam. Names match ^[A-Z][A-Z0-9_]{0,63}$, values are capped at 4 KB:

curl -X PUT "$BASE/v1/secrets/EXA_API_KEY" \
  -H "authorization: Bearer $DELTA_CONTROL_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"value":"<the credential>","purpose":"web search"}'

PUT is create-only and answers 409 if the name exists, so a compromised or prompt-injected gateway flow cannot silently swap an established credential. Rotation and deletion are operator acts on the inspect-gated surface (POST / DELETE /v1/dev/secrets/:name), which also requires DELTA_INSPECT_WRITE=1.

GET /v1/secrets returns names, purposes, and timestamps. There is no route that returns a value - not even to the authenticated gateway. A lost secret is re-provided, never recovered.

Reference a secret anywhere in your MCP configuration with {{vault:NAME}}:

DELTA_MCP_SERVERS='[{"name":"kb","transport":"http","url":"https://kb.example.com/mcp",
  "headers":{"authorization":"Bearer {{vault:KB_TOKEN}}"}}]'

The reference is resolved in engine code at the moment of egress - per call for HTTP headers, at spawn for a stdio server's env - so the configuration itself only ever holds the name. Because resolution happens at call time, a credential provided mid-conversation takes effect immediately; a backend that could not connect at boot for want of its credential is reconnected automatically when that credential arrives. The built-in web_search behaves the same way: with no EXA_API_KEY in the environment it falls back to the vault, so handing an agent a search key lights the tool up without a redeploy.

The model's entire view of the vault is one read-only tool, list_secrets, which returns names and purposes. There is deliberately no tool that returns a value; such a tool would turn every other tool into an exfiltration path. When a task needs a credential the agent does not have, it should ask its human for it by name.

As defense in depth, a value is registered for exact-value redaction the moment it is resolved for egress, in its raw, percent-encoded, and JSON-escaped forms. Any later reflection of it - an MCP backend echoing its own authorization header inside an error body, for instance - is replaced with [vault:NAME] before it reaches the model, the transcript, a spill file, a research artifact, or telemetry.

GET /v1/status reports vault.enabled, a live vault.count, and vault.declared: the names your configuration actually wires a destination for. An edge that offers a secret-entry interface should validate a request against declared, so an injected agent cannot invent a credential name and talk a human into providing it.

Known limits, stated plainly. The ciphertext lives in the daemon database, outside the model-writable workspace, so the confined file tools cannot reach it; but the delegated code tool runs arbitrary code as the daemon's OS user by design, and on Linux a same-UID process can read another's environment. The vault raises the cost of extraction from cat to an active attack on process state, and it keeps values out of transcripts, logs, and configuration - it does not replace the VM as the execution boundary. A secret injected into a configured MCP server goes wherever that server's operator-set URL points; operator configuration is trusted input and is never model-writable.

Protect stored state#

SQLite contains raw run requests, which can include metadata.authToken, plus messages, tool arguments and results, memory, and optional normalized call captures. Cockpit projections redact on read, but the database and its replica retain the original bytes.

Use short-lived user tokens, encrypted storage, restricted backup credentials, and an explicit retention policy. Do not enable call capture by default in production. Remember that Litestream copies the sensitive database to the replica.

Network protections#

web_fetch blocks private, loopback, link-local, and cloud-metadata destinations and revalidates redirects. Keep that protection on in production. Validation and the actual fetch perform separate DNS resolution, so a hostile hostname can still attempt a public-to-private DNS rebinding between them. Use network egress controls when SSRF is a hard security boundary. DELTA_FETCH_ALLOW_PRIVATE=1 is an explicit local-development escape hatch and lets model-driven fetches reach private services.

Subscription model tokens are sent only to exact HTTPS hosts in DELTA_BROKER_ALLOWED_HOSTS. Static MODEL_HEADERS cannot override authentication or protocol headers. MCP HTTP authorization can be replaced per run, so the MCP service must validate token audience and scope.

That broker route is the exception. Static model credentials sent to MODEL_BASE_URL, static and per-run MCP credentials, rotating MCP refresh credentials, scheduling control tokens, and telemetry credentials do not receive an equivalent harness-enforced HTTPS host allowlist. Treat every configured endpoint as trusted operator input, require validated HTTPS outside local development, and apply outbound allowlists where credentials are valuable. Be especially careful with TELEMETRY_URL because it can fall back to the production control token.

Configuration reference#

The values below are the current public operating surface. Unless noted otherwise, changing them requires a daemon restart.

Process, identity, and state#

Primary model#

OPENAI_API_KEY and ANTHROPIC_API_KEY are not read automatically as the primary key. Put the chosen value in MODEL_API_KEY, or reference another key with apiKeyEnv inside DELTA_PROVIDERS.

Codex subscription broker#

Tools, MCP, and context#

Identity, policy, and learning#

Profiles, context, and retention#

API, Cockpit, scheduling, and telemetry#

Container backup and bundle seeding#

The checked-in Litestream template also expects LITESTREAM_S3_BUCKET, LITESTREAM_S3_PATH, LITESTREAM_S3_ENDPOINT, AWS_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY.

Configuration validation is intentionally uneven in the current runtime. Invalid or reserved MODEL_HEADERS and an invalid vision regular expression fail startup. Malformed model-price JSON falls back, while negative prices are accepted. Many positive count settings are floored or clamped to at least one, the lease TTL is clamped to five seconds, and a zero retention sweep disables only periodic work. Other numeric values use raw JavaScript Number conversion with limited range validation. Empty exported budget values become zero. Validate the full environment against an operator-owned schema before launching production.

Removed bundle variables are ignored with a migration warning: DELTA_PLAYBOOK_FILE, DELTA_STEERING_FILE, DELTA_CHARTER_FILES, DELTA_CHARTER_TOOL, DELTA_PLAYBOOK_MAX_TOKENS, and DELTA_STEERING_MAX_TOKENS. Move their content or limits to the fixed files and current variables shown above rather than relying on those names.

Production checklist#

Before serving real work:

• Assign one stable DELTA_AGENT_ID.
• Keep one daemon, one database, and one workspace per agent.
• Prevent a second same-host process from opening that database, even on another port.
• Enforce one live Machine or volume owner outside the best-effort writer lease.
• Use a persistent volume for both database and workspace.
• Seed all four non-secret bundle files before the first container or Fly run.
• Set DELTA_CONTROL_TOKEN and put a trusted gateway in front.
• Have the gateway authorize task and prior-response ownership, inject user_id, and strip all caller-supplied privileged metadata.
• Set a different DELTA_INSPECT_TOKEN, or set DELTA_INSPECT=off.
• Keep DELTA_INSPECT_WRITE and DELTA_CAPTURE_CALLS off unless explicitly needed.
• Configure a real provider fallback for long-running production work.
• Add exact pricing for every non-OpenRouter model used by a dollar budget.
• Verify that every provider and child route reports usage; treat budgets as tracked-model loop guards, not total spend caps.
• Validate DELTA_PROFILE, budgets, prices, headers, regular expressions, and every numeric setting before launch.
• Give MCP calls the least privilege and use per-run user tokens where required.
• Validate MCP discovery, framing, result shapes, and rotating-token persistence against the actual server.
• Validate every model, MCP, refresh, control, and telemetry endpoint as trusted HTTPS.
• Name mutating MCP tools without read-like underscore or dot segments.
• Review POLICY.md and verify the product's human-review rail end to end.
• Verify that vocab.json matches real MCP names and schemas.
• Test workspace path, upload, image, and document behavior with production-like files.
• Size or cap upload memory and queue depth at the gateway.
• Install and authenticate the coding CLI only if the agent needs it.
• Run the container as a dedicated non-root identity with writable persistent paths.
• Test one subagent task if delegation is part of the agent design.
• Send telemetry only to a validated HTTPS endpoint and deduplicate event IDs.
• Classify recall and error attributes as potentially sensitive, and monitor exporter backlog and age externally.
• Back up SQLite and the workspace separately.
• Quiesce or reconcile unmatched database, workspace, and rotating refresh-token snapshots.
• Remove DELTA_BOOTSTRAP=1 after first boot.
• Put body limits, rate limits, user authorization, and task ownership at the gateway.
• Set journal retention longer than the longest possible run.
• Gate new work and check queue activity before planned stop or cloud suspension.
• Test a kill and restart during a tool-using task, then verify recovery behavior.
• Record the image digest, harness version, schema version, and build commit.

Troubleshooting#

delta send cannot connect#

delta dev chooses a free port unless --port is supplied. Read the launcher output and pass that port to delta send. The CLI always connects to localhost.

The default async delta send creates work through the control API and then tails the inspection stream. A gated daemon therefore needs both DELTA_CONTROL_TOKEN and DELTA_INSPECT_TOKEN in the CLI process. delta send --json needs only the control token. delta watch needs the inspect token. None of these clients loads the bundle's delta.env automatically.

The daemon has no usable model credential#

For the primary provider, use MODEL_API_KEY or OPENROUTER_API_KEY. OPENAI_API_KEY and ANTHROPIC_API_KEY are not primary-key aliases. Confirm that the direct daemon actually received the variables, because only delta dev loads delta.env.

Direct Anthropic or Responses calls hit the wrong path#

Set MODEL_BASE_URL to the versioned base, such as https://api.anthropic.com/v1 or https://api.openai.com/v1. Delta appends /messages, /responses, or /chat/completions.

An MCP server or tool is missing#

Check that DELTA_MCP_SERVERS is valid JSON. Malformed JSON, a non-array value, or an unusable entry (no name, an http entry with no url, or a stdio entry with no command) is dropped with a specific startup warning while the daemon boots with the remaining backends. A missing transport is inferred from the entry shape. Restart the daemon after changes. Read boot logs for the server-specific connection failure. Confirm that the combined <server>__<tool> name uses only letters, numbers, underscores, and hyphens and fits within 128 characters.

If the tool is connected but not initially visible, ask the model to use search_tools, or add its exact suffix to vocab.coreVerbs.

A bundle edit has no effect#

POLICY.md, vocab.json, all of PROMPT_CONTEXT.md, provider config, and MCP config require restart. DELTA.md changes apply to the next run. An existing session still carries prior conversation history, so start a new thread when testing a changed identity.

POLICY.md prevents startup#

A custom policy over DELTA_POLICY_MAX_TOKENS is a fatal configuration error. Trim it or deliberately increase the budget. Delta never truncates fixed rules.

The code tool returns an executable error#

The standard container does not include Codex or Claude Code. Install the selected CLI, confirm it is on PATH, set DELTA_CODE_CLI, and provision its home-directory authentication. Since 0.2.15 a missing CLI is loud rather than silent: the boot log and GET /v1/status name code as omitted with the reason. Delta does not parse shell quoting, so use an executable path and arguments without spaces.

A subagent cannot call the model or tools#

Children have a restricted environment and do not inherit MCP, Exa, broker, telemetry, control, or fallback-provider secrets. A broker-only parent cannot currently power a child through its broker credential. Give the parent a supported static primary key, or avoid subagent delegation for that deployment.

If the child reaches the provider but selects the wrong model, set DELTA_MODEL to the same value as DELTA_MODEL_PRIMARY. The current child environment forwards the legacy name only. Routes that require MODEL_HEADERS are not reproducible in a child, and child usage does not use a parent DELTA_MODEL_PRICES override.

A second local daemon opens the same bundle#

Stop it immediately. Two processes on the same host share the default lease-holder identity, and different ports bypass the port-bind collision. Never run two delta dev commands against one bundle. Keep exactly one process per database and workspace.

The production Cockpit loads but shows no data#

The application asset contains no embedded runtime data, but its API calls need the inspect bearer, which the page does not attach itself. Use a trusted same-origin proxy that authenticates the operator and injects the correct inspect and control tokens. A 403 without an inspect token usually means the client is not loopback. A 404 can mean DELTA_INSPECT=off.

Usage cost stays at zero#

OpenRouter can report cost directly. Other paths use Delta's price table. Add the served model ID to DELTA_MODEL_PRICES, including input, output, and cache-read dollars per million tokens. If tokens also remain zero, the compatible endpoint may be omitting usage; do not rely on token or dollar caps until that route reports it correctly.

A run reports an interrupted tool#

The daemon restarted after arming a non-idempotent call but before recording its result. The tool may or may not have committed its side effect. Inspect the external system and the tool journal before retrying.

A timed-out tool may still be running#

Delta races the configured timeout and tells signal-aware asynchronous tools to abort. A tool that ignores the signal may continue detached, while synchronous blocking work cannot be preempted at all. Verify its outcome before another mutation.

Telemetry does not arrive in an OTLP collector#

Delta sends a custom newline-delimited JSON schema with a subset of GenAI-inspired fields, not OTLP. Point it at a compatible ingestion route or translate the records at the collector boundary.

The container restores or starts incorrectly#

If a backup is configured and the local DB is absent, restore must succeed unless DELTA_BOOTSTRAP=1. Use bootstrap only for the initial empty replica, then remove it. A missing explicitly configured Litestream file is fatal. An existing local DB always wins even if stale. Workspace and rotating MCP refresh-token recovery are separate from Litestream.

Telemetry stops after a volume restore#

Exported event ids are <daemon uuid>:<events row id> and a collector dedupes on that id. Restoring a database from an older snapshot rewinds the row counter below what the collector already holds, so every event after the restore is dropped as a duplicate with no error on either side, while the daemon log looks healthy. Before booting a restored database, advance the sequence past the collector's last id for that daemon: UPDATE sqlite_sequence SET seq = <collector max + margin> WHERE name = 'events'. Roll-forward upgrades keep the sequence and need nothing.

An older binary refuses the database#

The database schema is newer than the binary's migrations. Do not force it open. Deploy a compatible binary or restore a database snapshot taken before the schema upgrade.

Current boundaries#

These are deliberate truths of the current codebase:

• Human-review rails are prewired through policy, vocabulary, MCP, and reflection. Delta does not include a universal approval backend or UI.
• delta send and delta watch are localhost-only clients; delta dev is a local launcher, not remote attachment.
• MCP configuration, policy, vocabulary, and all prompt context do not hot-reload.
• The writer lease blocks different machine identities but is not a same-machine mutex when processes use different ports.
• A Fly sample exists, but cloud provisioning and suspend or resume lifecycle remain external control-plane work.
• Litestream protects SQLite only. The workspace needs its own backup.
• The run API needs a trusted gateway for end-user authentication, authorization, body limits, and rate limits.
• Task, response, session, metadata identity, and review claims have no built-in tenant-ownership enforcement.
• /healthz is liveness and version metadata, not dependency readiness.
• Captured calls, conversations, and most database tables are not automatically age-pruned.
• Captured calls are normalized successful main-loop calls, not raw provider traffic or complete internal-call traces.
• Root inspection can directly address hidden workspace .delta paths; its token is root-equivalent and writes can corrupt live state.
• Token and dollar budgets cover recorded model usage only and can overshoot by one call plus background work.
• An invalid daemon DELTA_PROFILE currently falls open to trusted; deployment validation must reject it.
• Four workers bound queued sessions, not sibling tools or untracked background reflections.
• Shutdown does not drain tasks, reflection, or telemetry, and long-lived SSE endpoints do not impose production backpressure.
• Events are operational, at-least-once observability rather than a transactionally complete audit ledger.
• Subagents share the workspace and receive a restricted provider and tool environment.
• Subagent model forwarding uses the legacy DELTA_MODEL name and omits parent headers, custom prices, and reasoning configuration.
• eval_n is not safe for concurrent edits to shared files.
• Exported model and tool events omit attributes unless DELTA_CAPTURE_PAYLOADS=1.
• Bundled promotion adapters do not guarantee remote idempotency, and retry drain depends on a later reflected run.
• Task-start hydration and capability lookup can outlive their 20-second wait and bypass the ordinary tool journal.
• Reflection is best-effort background work with no durable reflection queue.
• The standard container runs as root unless the deployment overrides it.
• store is accepted for compatibility but does not turn off durable state.

Within those boundaries, Delta gives a small bundle a capable execution core: start with one request, inspect the durable main loop, add the tools and learning rails the work needs, and keep the result understandable all the way to production.