tranquil
Docs

ADR-0023: celld is the workflow-engine substrate

Status: Proposed · Date: 2026-08-06

Work in progress — this ADR may be modified and iterated on freely until implementation starts, and it is explicitly gated on the Phase 2 spike checklist in the design notes: a passed spike flips it Active; a failed spike supersedes it toward the Restate fallback (Option B).

References

Context

The “Engine” is a promissory note. The docs describe @tranquil/sdk workflows — workflow('name', w => w.trigger(webhook({ path }) | schedule('cron') | event('name') | manual())), w.emit(), tranquil run <name>, stable webhook URLs, “runs headless on the Engine” — and tranquil-examples ships “coming soon” placeholders. No SDK or Engine code exists anywhere. Building it bespoke means building a scheduler, durable state, crash recovery, and replication before the first workflow runs.

celld[^1] is Deno Land’s self-hosted, distributed Durable Objects + Workers daemon[^2]: a single binary where every node embeds V8 and executes Wrangler bundles. A cell is a Durable Object — named, single-threaded, with a private SQLite database — that serves HTTP, holds hibernatable inbound WebSockets, opens outbound ws:/wss: connections, sets alarms, and makes outbound fetches[^3]. Writes replicate to an S3-compatible bucket before they are acknowledged (RPO = 0), and the bucket is the only coordination layer: nodes discover owners and peers from bucket leases with compare-and-swap — no control plane, no consensus service. Peer HTTP is HMAC-authenticated and meant for private networks (Tailscale-class overlays)[^2]. Supported: module Workers, fetch, JS RPC (WorkerEntrypoint/RpcTarget, promise pipelining), service and DO bindings, static assets. Not supported and not planned: cron triggers, queues, KV, R2, the Cache API. Planned: D1 and “Workflows (durable execution)”[^3]. It is pre-1.0 with an evolving compatibility surface, Apache-2.0, and takes contributions as emailed patches only.

Prior art: the team already chose and ran Restate. For the healthcare-era integration backend, inbox draft ADR-0002 (active, 2026-04) adopted Restate — journaled steps replayed after crashes, keyed-workflow dedup, console introspection — with live Restate Cloud sandbox/dev/prod environments (deployment management). The durable-execution bar in this ADR comes from that hands-on use, and the cell taxonomy below deliberately mirrors Restate’s proven three-construct split.

The constraint that reframes the choice: Tranquil is individual-client, local-first. The engine must run on the user’s machine with zero accounts before any shared infrastructure exists.

Decision

The Engine is not built bespoke. Workflows deploy as Wrangler-bundled Workers plus Durable Object cells onto a celld node — initially a single client-managed node against a local MinIO bucket — behind a thin @tranquil/sdk facade that preserves the already-documented trigger surface and provides Restate-class durable-execution semantics from v1 (journaled steps, automatic retries, durable sleep, awakeables), so the substrate stays swappable. Cells never touch CDP: browser and user-context steps are capability callbacks to a connected Tranquil client over the client’s WebSocket, terminating in the same host-renderer capability layer ADR-0022 defines.

  1. What a workflow deploys as. The SDK CLI compiles workflow() files into one Worker entry (routes — Hono inside, per inbox draft ADR-0007) plus generic DO classes, emits wrangler.jsonc, and runs celld deploy . --bucket … (esbuild on PATH)[^2]. Where workflow code executes, stated plainly to preempt misreads: in the celld daemon’s embedded V8 (the Workers runtime) — a separate OS process, never inside the Tranquil client and never in a webview, headless or otherwise. The client’s webviews execute only the browser portion of capability steps; ADR-0022 scripts execute in Deno subprocesses. Three substrates, one TypeScript toolchain.

  2. Trigger mapping. The documented SDK surface maps onto celld primitives:

    SDK surface (already documented)celld primitive
    webhook({ path, validate })Worker fetch route /hooks/<wf>; validate runs in the Worker, then forwards to the workflow’s Coordinator cell. Stable URL = node listen address + route.
    schedule('cron')No native cron[^3] → the Coordinator stores the expression in SQLite, computes the next fire time, and storage.setAlarm(next)[^4]; the alarm handler re-arms, then starts a run. Missed-while-down semantics are a spike item.
    event('name') / w.emit()DO-binding JS RPC to subscribers’ Coordinators; reliability is emulated (no queues) with an SQLite inbox table plus alarm-driven drain/retry.
    manual() / tranquil run <wf> --inputCLI POST to the local node’s Worker route /runs/<wf>.

  3. Cell taxonomy — deliberately mirroring the Restate construct split that already worked (restate.service → Worker routes, restate.object → Coordinator, restate.workflow → Run):

    Cell classInstance nameOwns
    UserRootuser:<id> (one, in individual mode)workflow registry + versions, CLI/app auth material
    WorkflowCoordinatorwf:<name>config, schedule alarm, event inbox, run spawning, recent-run index
    WorkflowRunrun:<wf>:<runId>the durable-execution journal + executor (specific 4); every run its own cell so per-run alarms never contend
    ClientHubclient:<deviceId>terminates the app’s outbound WebSocket; presence; routes capability requests per their affinity; parks awakeables durably while no eligible client is connected

  4. Durable execution model — Restate-class semantics, minimal v1. This is how durable functions are written, and how anyone knows one hit a snag. Restate is the semantic north star[^5]; v1 implements its core subset on the WorkflowRun cell.

    Programming surface (in @tranquil/sdk — deliberately the common subset of Cloudflare Workflows’ step.do / step.sleep / step.waitForEvent[^6] and Restate’s ctx.run / ctx.sleep / awakeables[^7]):

    • step.do(name, opts?, fn) — a journaled side effect: the result is persisted before the run advances; on resume it is replayed, not re-executed.
    • step.sleep(name, duration) — a durable timer that survives crash and restart, backed by the run cell’s alarm.
    • step.waitForEvent(name, { timeout? }) — a durable promise (awakeable): journaled pending, resolved externally, result journaled.
    • TerminalError — non-retryable; fails the run immediately. Everything else thrown is transient by default.
    • Journaled ctx.now() / ctx.random() for nondeterminism the orchestrator itself needs.
    • The determinism rule, documented loudly: code between steps replays on resume, so it must be deterministic — all I/O and nondeterminism goes inside step.do. v1 runs steps sequentially (parallel steps deferred; this also keeps one pending timer per run cell).

    Failure semantics — how you know it snagged. A transient error (network timeout, any non-terminal throw) triggers automatic retry with exponential backoff under per-step { retries, backoff, timeout } options; retry scheduling and the per-attempt timeout both ride the run cell’s alarm (the alarm doubles as watchdog). A daemon or node crash mid-step means the alarm fires on restart, the run re-invokes the workflow function, and it replays from the journal: finished steps return their recorded results; the interrupted step re-executes. Step bodies are therefore at-least-once while step results are effectively-once — side effects inside steps should be idempotent, and the docs must say so. Exhausted retries move the run to failed, retained for manual retry or cancel.

    Journal schema (SQLite in the run cell): (run_id, seq, step_name, status, attempt, result_json, error, started_at, ended_at, next_retry_at). Engine-internal — workflow code may never depend on it.

    Observability. Run cells answer status queries — current step, attempt count, last error, next retry time — over JS RPC/HTTP. They surface in the client’s Runs panel (the same panel as ADR-0022 local runs: one Runs surface, two run kinds) and in the CLI: tranquil runs list | describe | retry | cancel. Terminal failures push a notification through the ClientHub to connected clients.

    Alarm multiplexing. A Durable Object has exactly one alarm — setAlarm overwrites[^4] — so each cell multiplexes it across its pending timers: due-times live in SQLite, the alarm is always armed to min(next due), and the handler re-arms before doing work. The Coordinator does the same across cron fires and inbox drains.

    The upgrade path is the design constraint. The step surface is the contract; engines swap beneath it, in order: (a) v1 — this journal in the run cell; (b) celld’s planned native “Workflows (durable execution)”[^3] when it ships — likely Cloudflare-Workflows-shaped, since celld tracks Cloudflare APIs, making the swap near-mechanical; (c) Restate (or Cloudflare Workflows proper) as the engine behind the same SDK if celld is outgrown — a retarget the team has hands-on experience with (inbox draft ADR-0002). Nothing in workflow authoring may depend on journal internals.

  5. Capability callbacks — how a workflow reaches the browser. Cells never speak CDP:

    webhook ─▶ Worker /hooks/<wf> ─▶ WorkflowCoordinator ─▶ WorkflowRun (journals each step)
                                       ▲ cron alarm            │ "needs browser / user context"
CLI ────▶ Worker /runs/<wf> ───────────┘                       ▼
                                                      ClientHub (client:<device>)
                                                         ┃ hibernatable WS, runId-correlated frames
                              ┌──────────────────────────┻──────────────────────────┐
                              │ Tranquil app — Deno WS client → host-renderer       │
                              │ capability layer (ADR-0022): tabs/ui/CDP run local  │
                              └─────────────────────────────────────────────────────┘
   all cell writes ── replicate-before-ack ──▶ S3 bucket (MinIO local → R2/Tigris shared)

    Mechanically these callbacks are step.waitForEvent awakeables: the run journals a pending request, the ClientHub delivers it, and the client’s completion resolves the awakeable with the result journaled. The same shape serves webhook-waits and human-approval steps.

    Each capability request carries an affinity: any (served by the first hub with a connected client) or device:<id> (pinned — required for steps that need that device’s login sessions or active window, since per-window browser sessions never roam between devices; an authenticated scrape step is de facto pinned). With no eligible client connected, the awakeable parks durably and the run suspends — parked, not failed — waking whenever one connects.

    The hub↔client protocol is plain runId-correlated frames initially, not capnweb: hibernation drops a cell’s in-memory JS state while keeping the socket, which conflicts with capnweb session state. Running capnweb inside the hub is spike item 9, not the default.

  6. Individual-client mode. Spike: user-run celld + MinIO started by a deno task (developers only). Phase 3 product default is Mode A: the client hosts the engine — celld and MinIO spawned and supervised as child processes by the app’s main process. The engine is up whenever a Tranquil client is open on that machine and stops on quit: the Tranquil client is also the engine. Consequence: schedules that came due while nothing was open fire via alarm catch-up on next launch — spike item 3 is load-bearing for exactly this. Mode B (deferred upgrade): celld + MinIO as app-installed user-level background services (macOS LaunchAgent) so schedules and webhooks fire with the editor closed — adopt only when proven needed. In both modes, browser-dependent steps park at the ClientHub until a client connects. Service supervision is a main-process concern — classic IPC per ADR-0015; this ADR needs no RPC-scope change (the app’s hub connection is an outbound client socket, not the guest↔host boundary).

  7. Cross-client promotion. Same bundles, shared bucket (R2[^8] or Tigris[^9]), each device a node advertising a Tailscale address[^10]. UserRoot, Coordinators, and Runs become shared-namespace cells; ClientHub stays per-device. The headline capability: any device can pick up a failed run. Every journal write replicates to the bucket before it is acknowledged and ownership is only a lease — so a run orphaned by a dead or offline device is acquired by any other device’s node: lease expiry → CAS takeover → SQLite materialized from the bucket → pending alarm fires on the new owner → replay from the journal. This is a key differentiator over a single local restate-server, and it depends on alarms surviving ownership migration (verify; spike item 12 tests it locally with two nodes on one machine). Resume rules by step kind: pure/compute steps resume on any node with no client app involved; browser steps resume their orchestration anywhere, but the awaited capability is served per its affinity — any-client requests go to whichever hub has a connected client, device-pinned requests stay durably parked until that device returns. Single-writer implications, stated plainly: a coordinator lives on exactly one node at a time; lease handoff on laptop-close means failover latency — so no interactive or latency-sensitive state belongs in cells.

  8. Deployment topologies — same substrate, no new machinery. (a) Solo: one machine, the degenerate hub. (b) Hub-and-spoke — the recommended first cross-client shape: one always-on machine with a Tranquil client left open is the de facto hub. Leases gravitate to whatever is up, so its node accumulates cell ownership, fires schedules reliably, serves as the stable webhook-ingress target, and its open client doubles as the default executor for any-affinity browser steps; laptops are spokes — capability providers and optional nodes. The hub role is operational, not structural: if the hub dies, spokes’ nodes take over its cells exactly per specific 7. (c) Full peer fleet: the general case, no designated hub. Promotion between topologies is pointing more devices at the bucket — never a re-architecture. Verify: whether a non-owner node accepts and proxies requests to a cell’s owner (webhook-via-any-node vs owner-only shapes the hub’s stable-URL story).

Options considered

The field was largely weighed once already, for the healthcare-era integration backend, in inbox draft ADR-0002 — its findings are carried forward here rather than re-litigated.

  • Option A: celld (chosen). Durable Objects + per-cell SQLite + alarms + JS RPC + replication for free[^1]; local-first (no accounts, runs offline); cross-client sharing is native — one bucket; Apache-2.0 and forkable; the same vendor as the Deno runtime bet. “Adopt, don’t invent,” echoing ADR-0003’s reasoning for capnweb.
  • Option B: Restate — the incumbent with in-house experience; the designated fallback. Inbox draft ADR-0002 chose Restate and it ran for real: journaled steps replayed from the last committed entry after a crash; keyed workflows so re-invoking the same workflow ID is a no-op (dedup); a console showing state, journal, and pending invocations of every workflow; each integration as three constructs — stateless restate.service (poller), restate.object (per-tenant cursor/state), restate.workflow (per-record run) — which this ADR’s cell taxonomy deliberately mirrors. Live Restate Cloud sandbox/dev/prod environments exist, with per-env ingress/admin keys and versioned-deployment lifecycle (deployment management). Why it is not the substrate here: that stack is cloud-first — Restate Cloud + AWS Lambda + per-environment accounts and keys — while individual-client mode needs an engine on the user’s machine with zero accounts; and Restate’s cross-machine story is a distributed restate-server cluster or Restate Cloud (data leaves the machine), versus celld’s share-through-a-bucket. It is the explicit fallback if the spike fails: local restate-server (single binary; the local-dev pattern that stack already proved) for individual mode, Restate Cloud for shared mode — behind the unchanged SDK step surface. Its semantics are the north star regardless of substrate[^5].
  • Option C: bespoke Deno engine per the original Engine docs — full control, one runtime dialect; but it re-implements scheduling, durability, crash recovery, and replication. Exactly the “custom cron + retry tables” status quo inbox 0002 already rejected, one level up.
  • Option D: Cloudflare-hosted Workers/DO/Workflows — mature, with real cron and queues; but data leaves the machine, accounts are required, and local-first is forfeited.
  • Option E: already rejected in inbox draft ADR-0002 — carried forward, not re-litigated. AWS SQS + Lambda (manual idempotency keys, no replay after partial execution, opaque mid-flight state). Step Functions (verbose JSON/YAML state machines, logic outside the definition, AWS lock-in). XState (in-process; state lost on crash; retries need a second library — the benchmark onboarding workflow took ~135 lines vs ~70 in Restate). Vercel Workflow DevKit (Vercel-coupled; the HTTP-hook step model is less composable than ctx.run).
  • Option F: run the engine inside the client (hidden window / webview). Superficially attractive — “the client is already running; it’s also the server” — and it matches the intuition that automation code lives in a background webview. Rejected as the execution substrate: a renderer dies with the app (no durability, no journal or replay, no leases); Chromium throttles hidden-renderer timers, so schedules drift or stall; workflow code would inherit the client’s ambient privileges, violating the capability model; and there is no cross-machine story. The healthy kernel survives elsewhere: the client hosts the engine as a supervised child (Mode A, specific 6), and one always-open client is the recommended hub topology (specific 8).
  • Option G: do nothing — the Engine docs remain fiction, and the docs debt compounds.

Consequences

RiskMitigation
celld is pre-1.0 with an evolving surface, patch-by-email contributionsPin the binary version; re-run the spike checklist on every upgrade; the thin SDK facade keeps the substrate swappable — the documented SDK surface is the contract, celld an implementation detail. Budget for vendoring/local patches.
No cron, no queues[^3]Alarm-based cron emulation and SQLite-inbox events are correctness we own; watch upstream (queues “if demand appears”).
Two daemons per user (celld + MinIO)One supervised pair under ~/.tranquil/engine/ (Mode A). MinIO is AGPL-3.0 and its upstream repo is now archived[^11] — actively track alternatives and any future celld filesystem-bucket backend; settle this before Mode B ships.
Workers API ≠ Deno API for workflow codeAuthors write against @tranquil/sdk; Deno tooling still formats/lints/checks the TypeScript; esbuild bundles it.
Upstream native “Workflows” may obsolete the run journalA good outcome — swap beneath the SDK per the upgrade path in specific 4.
Run journals replicate to the bucket and can hold sensitive step results (scraped page content, tokens)Carry forward the journal-encryption concern inbox draft ADR-0002 flags (its ADR-0006): encryption at rest and retention/GC settled before any shared-bucket promotion.

Relationships: inbox draft ADR-0007 carries forward — Hono runs on Workers, so the API surface lives inside the celld Worker. ADR-0022 is an independent decision: the local runner and the ClientHub are peer channels into the same host capability layer, and either ADR stands without the other. ADR-0003/ADR-0015 are untouched by this ADR — the hub connection is an outbound client socket, not the guest↔host boundary.

Re-evaluate if: the spike fails any go/no-go item (→ Option B, Restate, behind the unchanged SDK surface); celld stalls or relicenses; or Cloudflare-hosted operation becomes acceptable to the product.

[^1]: celld documentation — cells as Durable Objects (named, single-threaded, private SQLite), replication to an S3-compatible bucket before acknowledgment (RPO = 0), bucket-lease coordination with no control plane. https://celld.dev/docs/ [^2]: denoland/celld — “self-hosted, distributed Durable Objects”; single binary, V8 executing Wrangler bundles, celld deploy . --bucket … (esbuild on PATH), HMAC-authenticated peer HTTP for private networks. Announced by Ryan Dahl: “celld = V8 + S3 + SQLite + LTX + Tokio … exactly the Cloudflare Workers/DO JavaScript APIs” (https://x.com/rough__sea/status/2085001943693549887). https://github.com/denoland/celld [^3]: celld Cloudflare compatibility — supported: module Workers, fetch, JS RPC (WorkerEntrypoint/RpcTarget, pipelining), service/DO bindings, DO alarms, hibernatable inbound WebSockets + outbound ws:/wss:, static assets. Not planned: cron triggers, queues, KV, R2, Cache API. Planned: D1, “Workflows (durable execution)“. https://celld.dev/docs/cloudflare-compat/ [^4]: Durable Object alarms — one alarm per object (setAlarm overwrites), getAlarm/deleteAlarm, at-least-once alarm() execution with automatic retries (exponential backoff, up to 6). https://developers.cloudflare.com/durable-objects/api/alarms/ [^5]: Restate durable execution — every step journaled; on failure “Restate replays the journal, skipping completed steps and resuming from exactly where it left off.” https://docs.restate.dev/concepts/durable_execution [^6]: Cloudflare Workflows — durable multi-step applications with step.do, step.sleep/step.sleepUntil, step.waitForEvent, automatic retries and state persistence; the API shape the v1 step surface stays compatible with. https://developers.cloudflare.com/workflows/ [^7]: Restate SDK actions — ctx.run (“safely wrap any non-deterministic operation … and have Restate persist its result”), durable timers, awakeables (“a generated unique ID that an external system can resolve or reject”). https://docs.restate.dev/foundations/actions [^8]: Cloudflare R2 — S3-compatible object storage; a candidate shared bucket for cross-client mode. https://developers.cloudflare.com/r2/ [^9]: Tigris — S3-compatible object storage; the other shared-bucket candidate. https://www.tigrisdata.com/ [^10]: Tailscale — the private-overlay class celld’s peer HTTP expects between nodes. https://tailscale.com/ [^11]: MinIO — the local S3-compatible bucket for the spike; AGPL-3.0, and the upstream minio/minio repository is archived on GitHub (verified 2026-08-06), which makes the alternatives watch a real obligation, not hygiene. https://github.com/minio/minio