tranquil
Docs

Smoke Tests

The tranquil-test-suite repo holds deterministic GUI smoke tests that run without an LLM in the loop. They launch a fully isolated Tranquil instance, drive it over the Chromium DevTools Protocol (CDP), assert real UI behaviour, and tear the app down — so regressions in shipped features are caught mechanically.

They are the automated counterpart to Agent Debugging (interactive/LLM-driven CDP inspection) and to tranquil-client’s yarn verify (interactive isolated launch). Same isolation contract, run as pass/fail assertions.

What they cover

Ports of checks first verified by hand against a live build. Each suite runs as a step and reports ok:

  • ADR-0012 — the vertical tab panel: lives in the right dock, mirrors the center pane’s tabs, file octicons (icon-markdown / icon-code), click-to-activate, live update as tabs open, toggle removes/restores + reveals the dock, close buttons.
  • ADR-0013 — business-theme ui-variables resolve to dark values (dark background, light text — not the light fallback), and no button{…!important} rule is loaded.
  • Settings default tab — Settings opens on the Tranquil tab, pinned first in the menu.
  • ADR-0019 — the browser User-Agent: the settings combobox lists the platform-variant presets and the applied UA reflects the current setting.
  • .url bookmark save — saving a browser tab writes a .url file that reopens the page.
  • Focus routing — three suites: Ctrl+Tab cycles the center pane from a focused dock; the tree-view keeps its selection while a pathless tab is active; opening a background tab doesn’t steal focus.
  • ADR-0022 — tranquil-rpc injects trusted webviews only — a file:// page under a registered trusted root gets the guest runtime (window.tranquilHost.ping() → "pong"); an untrusted file:// page gets no runtime at all. Guards the audience-tagging change to the capability registry against regressing webview RPC.
  • Main-thread performance — tree-view folder-count badges are positioned by a measure-then-apply pass rather than a read/write loop, landing where the theme’s own centring transform (translateY) expects; the Runs panel coalesces renders to one per animation frame instead of rebuilding per stdout chunk, and grows its output by appending the delta rather than rewriting the whole buffer, falling back to a full replace once the ring buffer’s front truncates.
  • Markdown preview toggle — markdown-preview:toggle swaps the pane item in place instead of opening a second tab, in both directions, driven through the real tab-bar button rather than a raw command dispatch — so a broken button matcher fails the same way a broken swap would.
  • Tree-view tab-bar buttons — the New File / New Folder / Refresh / Collapse All cluster hides completely with no project open (none of the four mean anything against an empty project) and reappears the moment a folder is added back.
  • Browser toggle with a browser tab already open — guards a real bug: with a browser tab already open, typing Cmd+N, Cmd+W, Cmd+T fast could leave a bogus extra tab behind. tranquil-browser’s URL opener has a same-URL-within-50ms debounce guard that declined by returning undefined instead of false, so atom.workspace.open() treated the URL as unclaimed and fell through to Pulsar’s default “unrecognized URI → open as a file path” behaviour, creating a plain TextEditor named after the URL’s last path segment (“blank”). The suite confirms the debounce window is actually hit by reading window.prevUrl/window.prevId directly, not just inferring it from the outcome shape.
  • ADR-0025 — automation debugger — breakpoints bind to the correct TypeScript line through the inline source map rather than the naively-wrong generated line, round-trip through the store and render as real, visibly-painted gutter dots (not just correct model state), and a breakpoint with no source-map mapping anywhere near it stays genuinely unbound. Runs last on purpose: it pauses the app at a breakpoint for up to 45s, so anything else that needs to report should already have.

Running

cd tranquil-test-suite
deno task test

Requirements:

  • Deno (installed at ~/.deno/bin/deno).
  • A sibling ../tranquil-client checkout with node_modules installed. Electron is spawned directly — no Node is invoked.
  • A display: the suite opens a real window (fine locally on macOS). Headless CI would need a virtual framebuffer (e.g. xvfb) — not set up yet; the run model today is on-demand, whenever you touch UI or owned-package code.

A green run reports every suite as ok and exits 0; any failed assertion fails its step and exits non-zero. Set SMOKE_DEBUG=1 to see the launched app’s stdout/stderr.

The suite drives its instance on its own CDP port (9223), not the 9222 a normal dev instance uses — so you can leave a dev instance (yarn start) running and run the tests at the same time. Only something already on the test port blocks a run; override it with TRANQUIL_TEST_CDP_PORT.

How it’s built

tranquil-test-suite/
  deno.json                       # tasks.test, @std/assert + @std/path
  smoke/
    main_test.ts                  # Deno.test: launch → run suites as steps → teardown
    lib/launch.ts                 # isolated Electron launch (Deno.Command) + teardown
    lib/cdp.ts                    # CDP over Deno's native WebSocket (test port 9223)
    atom-home/config.cson         # FIXED config (theme + disabled packages) for determinism
    suites/adr-0012-vertical-tabs.ts
    suites/adr-0013-ui-variables.ts
    suites/settings-default-tab.ts
    suites/browser-user-agent.ts
    suites/url-bookmark-save.ts
    suites/focus-ctrl-tab-center.ts
    suites/focus-treeview-keep-selection.ts
    suites/focus-background-tab.ts
    suites/rpc-webview-injection.ts
    suites/main-thread-perf.ts
    suites/markdown-preview-toggle.ts
    suites/tree-view-empty-project-controls.ts
    suites/browser-toggle-empty-pane.ts
    suites/debugger-breakpoints.ts
  fixtures/                       # project content the suites open
    index.html                    # opened as a background tab by the focus-routing suite
    data/sample.json              # gives the vertical tab panel a non-markdown icon to check
    notes/welcome.md              # opened by the markdown preview toggle suite
    debug/breakpoint-drift.ts     # deliberately shaped so naive 1:1 TS→JS line mapping fails
    automations/chatty.ts         # streams output for the Runs-panel coalescing check
    rpc/                          # file:// page for the RPC injection suite
  • lib/launch.ts mirrors tranquil-client/scripts/verify.js. It finds the Electron binary via node_modules/electron/path.txt (no Node), seeds a throwaway ATOM_HOME from smoke/atom-home/config.cson, and spawns Electron with a throwaway --user-data-dir, ATOM_HOME/ATOM_RESOURCE_PATH/NODE_PATH set, ELECTRON_RUN_AS_NODE deleted, and TRANQUIL_VERIFY=1. That env var activates the gated filter in src/main-process/atom-application.js (openPaths) which drops the tranquil-client resource-path root, so only tranquil-test-suite opens as the project — no flash of the app’s own source. It also sets TRANQUIL_CDP_PORT=9223 on the spawned app (start.js honours it), so the instance exposes CDP on the test port, not 9222 — the suite runs alongside a normal dev instance. teardown() sends SIGTERM to the Electron main (SIGKILL fallback) and removes the temp dirs.
  • lib/cdp.ts — a tiny CDP client that connects on TRANQUIL_TEST_CDP_PORT (default 9223, the port launch.ts hands the app). connectAtom() finds the editor window (the page whose atom global has a loaded project), evalInAtom<T>(expr) runs JS and returns its value (awaiting promises), waitFor(expr) polls until truthy (used instead of fixed sleeps). The pre-flight guard checks the test port, so a dev instance on 9222 never blocks a run.
  • atom-home/config.cson pins core.themes to tranquil-business-dark so theme/colour assertions are stable across machines (the real ~/.tranquil doesn’t set it), and disables the Pulsar-only packages (pulsar-updater, about, git-diff, styleguide, …) at load time. pulsar-updater in particular fires an on-launch update check whose “update available” notification would otherwise pop mid-run; disabling these at load time (rather than tranquil-config’s runtime disablePackage) makes the package set and UI deterministic.

Determinism

  • Fixed config for the theme; on a dev machine the harness also symlinks ~/.tranquil/dev so owned packages load exactly as in interactive use (bundled fallback otherwise).
  • Poll-based waits, never fixed sleeps.
  • Semantic colour assertions (dark bg / light text) rather than exact RGB — robust to minor theme tweaks, still catches the light-fallback regression.
  • Subset assertions on rows/tabs — the app auto-opens some panes, so suites assert their fixtures are present rather than an exact list.

Adding a smoke test

  1. Add smoke/suites/<name>.ts exporting export async function <name>(cdp: Cdp): Promise<void>; use cdp.evalInAtom / cdp.waitFor and @std/assert. Drive the app into a known state first.
  2. Register a step in smoke/main_test.ts: 
    await t.step("<name>", () => yourSuite(cdp));
  3. If it needs new content, add tracked files under fixtures/. For a test that needs a trusted file:// page (e.g. RPC injection), register a trusted root in the suite before opening it — see rpc-webview-injection.ts and fixtures/rpc/.
  4. deno task test (and deno fmt / deno lint / deno check) should stay green.