mcp-coordinator

Troubleshooting

First-line self-help for the most common mcp-coordinator failure modes. Each entry is symptom -> likely cause -> fix. If none of these match, open an issue on GitHub with the relevant log lines (~/.mcp-coordinator/logs/server.log in daemon mode).

References:

Quick facts you will need below:

1. Port already in use (3100 or 1883)

Symptom. server start (or the daemon log) shows Error: listen EADDRINUSE: address already in use 127.0.0.1:3100 (or :1883), and the process exits. Since issue #273 the daemon-mode CLI waits for the child to accept a connection before saying anything, so it reports Coordinator failed to start with the tail of the log and exits non-zero. Older releases printed Coordinator started in background and exited 0 – on those, the crash is only visible in ~/.mcp-coordinator/logs/server.log.

A bind can also be refused rather than busy: a command launched from an agent running under a sandboxed Bash may be denied the listen outright. It lands in the same place, with the daemon’s own error at the end of server.log.

Since issue #280, only 3100 is fatal. A busy MQTT port no longer stops the coordinator: it logs MQTT broker failed to start - continuing without MQTT at warn and carries on. The three MQTT tools (wait_for_message, get_queued_messages, mqtt_publish) then report themselves unavailable; the other 23 tools, the REST API, the dashboard SSE feed and every stdio-mode session are unaffected. If you would rather the boot fail loudly, set COORDINATOR_MQTT_REQUIRED=true.

So a coordinator that starts but whose MQTT tools all refuse is usually this, not a broken install – grep server.log for that warning before looking further.

Likely cause. Both listeners fail closed on bind: the HTTP server (httpServer.listen(port, bindHost)) and the embedded MQTT TCP broker (tcpServer.listen(tcpPort, "127.0.0.1")) each reject on the error event, so a busy port aborts boot. Usually a previous coordinator is still running, or another service holds 3100/1883.

Fix. Point at free ports, or free the busy one:

# Move the HTTP port
mcp-coordinator server start --port 3200
# Move the MQTT TCP port
COORDINATOR_MQTT_TCP_PORT=1884 mcp-coordinator server start

To find and stop the current holder, use mcp-coordinator server stop (see section 3) before re-launching, or inspect the port with lsof -i :3100 (macOS/Linux) / netstat -ano | findstr :3100 (Windows).

2. Server won’t start – boot fails closed (OAuth / Phase 2)

These guards fire only when COORDINATOR_OAUTH_ENABLED=true. Phase 1 (the default) does not require them.

Symptom: secret entropy: .... Boot aborts with e.g. secret entropy: contains dictionary word "changeme", secret entropy: all bytes identical, or secret entropy: 74.2 bits estimated, minimum 128.

Symptom: BootValidationError: ... must be set. Boot aborts naming a missing variable.

Symptom: OAuth (Phase 2) is not supported on the Bun runtime.

3. server stop says “no PID” or the server seems stuck

Symptom: No server PID file found. Is the server running? (exit 1).

Symptom: Server (PID N) is not running. Cleaning up PID file.

Symptom: Server did not stop gracefully. Sending SIGKILL.

mcp-coordinator server stop --timeout 30   # allow a slow audit flush
mcp-coordinator server stop --force        # SIGKILL now, no graceful shutdown

If all else fails, delete ~/.mcp-coordinator/server.pid by hand and stop the OS process directly.

4. MQTT client can’t connect

The coordinator embeds an MQTT broker (Aedes). See the connection contract in docs/ for the full topic list. Common connection failures:

Symptom: connection refused / times out from another machine or a LAN IP.

Symptom: WebSocket handshake fails (browser / strict WS client).

Symptom: CONNECT rejected (MQTT auth rejected in the log).

Symptom: connected, subscribed, but no messages arrive.

5. /readyz returns not_ready (503)

Symptom. GET /readyz returns HTTP 503 with {"status":"not_ready","checks":{...}}. (/livez stays 200 – it never checks downstream deps, so orchestrators do not restart the pod over a transient dependency blip.)

Likely cause. Only db and mqtt gate readiness (tree_sitter and git_cochange are reported but optional). Look at the failing check:

Fix. Inspect the failing check and its error string, then resolve the underlying dependency – free the MQTT port, or make the data directory writable. /readyz flips back to 200 ready once both are green.

git_cochange reports unavailable (does not cause the 503)

Layer 4 never gates readiness, so this check explains a degraded conflict detector, not a failing pod:

"git_cochange": {
  "available": false,
  "status": "false",
  "last_error": "SQLITE_READONLY: attempt to write a readonly database",
  "scheduler_circuit_open": true,
  "optional": true
}

Git-side failures never reach the breaker: build() handles those itself and simply retries at the normal refresh interval. A tripped breaker always means SQLite.

Metrics: mcp_coordinator_git_cochange_scheduler_failures_total, mcp_coordinator_git_cochange_scheduler_circuit_open, and mcp_coordinator_git_cochange_builds_total{outcome="scheduler_error"}.

6. 401 Unauthorized on HTTP / MCP / SSE

Symptom. REST or MCP requests return 401 with a WWW-Authenticate: Bearer realm="coordinator", error="invalid_token" header.

Likely cause. COORDINATOR_AUTH_ENABLED=true requires a valid Bearer JWT on protected routes; the token is missing, malformed, or expired. A common gotcha: EventSource/SSE cannot set headers, so a browser /api/events connection 401s unless the token is passed as a query param.

Fix.

7. Docker: pnpm: command not found / Corepack

Symptom. Building your own image (or running pnpm locally) fails with pnpm: command not found or a Corepack error, even though packageManager is pinned in package.json.

Likely cause. Node 25+ no longer bundles Corepack. The provided Dockerfile (Node 26 alpine) installs it explicitly before enabling it.

Fix. Install Corepack, then enable it so the pinned pnpm version is used:

RUN npm install -g corepack@latest && corepack enable

Locally, run corepack enable once and let it resolve the right pnpm.

8. macOS: standalone binary won’t open

Symptom. Running the downloaded binary shows “cannot be opened because the developer cannot be verified”, or it is silently killed on first launch.

Likely cause. The release binaries are Bun-compiled and ad-hoc signed (codesign --sign -), not notarized by Apple. Gatekeeper quarantines unnotarized downloads and blocks the first run.

Fix. Clear the quarantine attribute, then run it:

xattr -d com.apple.quarantine ./mcp-coordinator
./mcp-coordinator --version

(Or right-click the binary in Finder and choose Open once to approve it.) Two related macOS notes: OAuth is unsupported on this Bun binary (section 2), and the Anthropic quota reader is macOS-only (it reads the Keychain; on Linux/Windows the quota endpoint returns 503 fail-open).

8b. An agent shows as online after it died — or disappears while still working

list_agents(online_only: true), coordinator_status and wait_for_peers all report agents that are flagged online and have been seen recently.

“Seen” is refreshed by real work, not only by the explicit heartbeat tool: announce_work and post_to_thread both count. An agent that has done none of those for longer than the TTL drops out of the online list.

The TTL defaults to 900 seconds (15 minutes) and is tunable:

COORDINATOR_AGENT_ONLINE_TTL_SECONDS=1800   # more patient

Two things this deliberately does not do:

If an agent vanishes mid-task, it is idle longer than the TTL between coordination calls: raise the value, or have it call heartbeat periodically.

8c. register_agent fails with “already registered in org …”

Symptom. Registering an agent id that works fine in one org is rejected in another:

agent id 'builder' is already registered in org 'acme'. Agent ids are globally
unique in this release, not per-org — choose a different id.

Cause. You are on a pre-v11 schema. The agents primary key was already (org_id, id), but a global UNIQUE index on agents(id) survived because five foreign keys referenced agents(id) rather than the composite key — so ids were effectively global. (Older still: the same situation surfaced as a raw UNIQUE constraint failed: agents.id, naming neither the id nor the owning org.)

Fix. Upgrade. Schema v11 (#231) rewrites those foreign keys as (org_id, <col>) REFERENCES agents(org_id, id) and drops the global index: agent ids are now unique within an org, and two orgs may both use builder. The migration runs automatically on first open. To stay on the old schema, keep prefixing ids with the org (acme-builder).

Two things the upgrade can report:

8d. claim-task returns success: false with a conflict field

Symptom. A claim is refused even though the thread itself is unclaimed — claimed_by is null, assigned_to is null, status is "open":

{ "success": false, "claimed_by": null, "assigned_to": null, "status": "open",
  "conflict": { "thread_id": "…", "files": ["src/shared.ts"] } }

Cause. Another agent already holds a different thread whose target_files overlap yours. The claim CAS used to predicate on thread id alone, so two agents could claim two different threads touching the same file and both start editing it. The overlap guard closes that (#258).

conflict names the thread holding the files and which files collided — the other fields all look normal precisely because nothing is wrong with your thread.

Fix. Nothing to repair: claim a different thread and come back. Clients that already handle success: false by refetching need no change. One agent holding two overlapping threads is still allowed — a single worker serializes itself.

8e. Boot aborts on foreign key violations

Symptom. Startup fails with a message naming one or more constraints and row counts, e.g.:

The database has 3 foreign key violation(s) this migration did not create and
cannot repair — thread_messages.thread_id -> threads (3 rows). These rows
already violate the current schema, so this is a data problem rather than a
coordinator bug. Back up the data directory, list them with
`sqlite3 <data-dir>/coordinator.db 'PRAGMA foreign_key_check;'`, then delete or
re-point them and restart. Aborting; the database is left unchanged.

Cause. The v0.9 schema migration ends with a PRAGMA foreign_key_check over the whole database, and the boot sequence re-enters that migration on every start — so in practice it is the coordinator’s startup integrity gate, not a one-off migration self-check. Any violation anywhere in the file stops boot, fail-closed.

A violation usually means rows were written while foreign keys were off, or a parent row was deleted directly in SQLite. The message distinguishes the two possible culprits:

Before this was split out (#285) every violation was reported as “this indicates a migration bug” with no table, constraint or count — which sent operators looking for a coordinator bug that was not there.

Fix. Back up <data-dir> first, then list the offending rows:

sqlite3 ~/.mcp-coordinator/coordinator.db 'PRAGMA foreign_key_check;'

Each result row is table|rowid|parent|fkid. Delete or re-point those rows (DELETE FROM <table> WHERE rowid = <rowid>;, or set the column to a row that exists), then restart. The coordinator never deletes them for you — losing coordination history to make a schema check pass is not its call. —

8f. Cannot find module '@babel/runtime/...' — a broken dependency tree

Symptom. The daemon stops starting, with a stack trace naming a package you never installed:

Error: Cannot find module '@babel/runtime/helpers/defineProperty'
Require stack:
- node_modules/.pnpm/broker-factory@3.1.15/.../bundle.js
- node_modules/.pnpm/worker-timers@8.0.33/.../bundle.js
- node_modules/.pnpm/mqtt@5.15.2/node_modules/mqtt/build/lib/client.js

Since #282 the CLI recognises this and prints the fix instead of the stack. If you are on an older build, the stack above is what you get.

Fix.

pnpm install --frozen-lockfile

It restores the missing links without touching package.json or pnpm-lock.yaml (“Lockfile is up to date, resolution step is skipped”). mcp-coordinator doctor reports the same thing without waiting for a failed start — its dependency probe runs first and stays usable when the tree is broken.

Cause. Not your code, and rarely this repo. Two mechanisms are known:

Remove-Item "<worktree>
ode_modules" -Force   # deletes the link, not the target

A test that proves nothing. Running node -e "require.resolve('@babel/runtime/helpers/defineProperty')" from the repo root fails even when everything works: under pnpm’s strict layout a transitive package is not hoisted to the root. It costs twenty minutes to chase. The only meaningful checks are mcp-coordinator doctor, an actual start, or node -e "import('./dist/src/serve-http.js')". —

8g. path is outside repoRoot, or an agent that coordinates with nobody

Both come from the same place: the coordinator turns every declared path into a repo-relative key, and everything downstream — hot_files, check_file_conflict, the Layer 1 score — joins those keys by exact string equality. A path it cannot map to the right key either gets refused or, worse, gets filed under a key nothing else shares.

Since #379 the roots come from git worktree list, run by the coordinator on its own COORDINATOR_REPO_ROOT, so an absolute path from any worktree of the repository maps to the same key as the main checkout — whether the worktree sits outside the root (git worktree add ../feature-x) or inside it, as Claude Code’s own do at .claude/worktrees/<name>/.

The in-root case is the one worth knowing about, because it never announced itself. Before #379 an agent there was accepted and silently stored under .claude/worktrees/<name>/src/foo.ts while everyone else stored src/foo.ts. No error, no log line. If you ran agents in native worktrees on an older build, their file_activity and working_files rows carry those long keys and will not match anything; they age out with their normal retention.

If you still get the rejection:

Note this is per repository, not per project: one daemon still has one COORDINATOR_REPO_ROOT, so two unrelated repos on one coordinator still merge their hot_files rows — #279, still open. One daemon per project remains the supported answer.

9. Windows notes

</content> </invoke>

This site is open source. Improve this page.