{
  "batch": "deep-54",
  "setup": "Hermes source clone at 5fac5ed7b5918185006761d85b28050e580c4ef8; uv sync --locked --python 3.14 (CPython 3.14.10), CLI install/validate/doctor verified. Used supplied tryit.py and Nous FakeLLMServer, isolated homes under test/scratch, no model credentials. Initial bare-Hermes HTTP 502 was fixed with NO_PROXY/no_proxy for loopback. Original empty-plan runs include catalog --install-deps and enable. Supplemental runner uses the same AIAgent/FakeLLMServer and direct local-directory configuration to avoid repeated PM environment publication; Neural Mesh additionally selects memory.provider as its README requires. Added test-only pytest, pytest-asyncio, pytest-timeout, PyYAML and pinned nemo-switchyard 0.4.0a1 in scratch. Runtime/test claims below are local observations; pinned source URLs and exact excerpts are in sources.json. No live X account, external model pair, SSH/Slurm/EC2 host or GUI/device was used; these limits are not represented as successful end-to-end runs.",
  "plugins": [
    {
      "pr": 152256,
      "name": "nvidia-switchyard",
      "validate": "Passed original --install-deps catalog admission and runtime doctor (5 hooks); no native agent tool is registered. Original enable exceeded the harness 120-second limit; a later genuine CLI enable succeeded. Manual rule-11 concern remains: providers._runtime obtains resolved provider api_key and credential_headers constructs raw subscription/Codex bearer headers, rather than credentialed_provider_request with requires_auth.",
      "inside": "Pinned 114c3e12c1931d86cf75affc6c6ec5754865d107, repository NVIDIA/switchyard-local-first-routing, switchyard/ folder. Loaded real Hermes hooks, llm_request middleware, provider profile, switchyard CLI and Desktop/dashboard code. In a fresh Hermes conversation after removing the test-only bundled copy, three terminal calls reached the pinned plugin: status gave a clean missing SWITCHYARD_CLOUD_API_KEY error, list returned the VGR preset, and pin invalid-tier returned an argparse choice error. No actual local/cloud model pair or subscription account was used; routing behavior was assessed through source and the plugin test suite. Its corrected provided suite also passed the real sidecar/mock-tier routing and single-tier model paths.",
      "tests": "174/174 pytest tests passed, 0 skipped (1115.07 seconds), in the documented layout with unchanged pinned source copied into Hermes/plugins/switchyard and its provider profile staged as documented. Real nemo-switchyard 0.4.0a1 and loopback MockTier servers exercised routing, fallbacks, access controls, lifecycle, setup, removal and Hermes parity. External HTTP endpoints were failed locally with a test-only transport guard. Earlier external-layout failures were excluded after correcting the test layout; an unguarded run also hit the sandbox network gate on efficient.example. No live subscription or production model account was used.",
      "useful": 4,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 24,
      "rank": 4,
      "best": "Verification-gated routing, model presets and a Desktop monitor give Hermes a substantial local-first inference capability through public extension surfaces.",
      "worst": "Subscription routing obtains the raw resolved api_key and builds Codex bearer headers instead of using the catalog-required credentialed_provider_request path.",
      "blocked": null
    },
    {
      "pr": 153861,
      "name": "summon",
      "validate": "Passed all 15 catalog checks with security verdict safe, no warnings; original --install-deps/enable and runtime doctor passed.",
      "inside": "Pinned ccfc8c4163fa736785b4f0cd7322742f39c6d435, repository gtmcknight/hermes-summon, root folder. Loaded and registered the X platform; no agent tools are added. Three calls through the real Hermes terminal tool inspected its real registered adapter: requirements/empty-config validation returned false, connect with no bot credentials returned false with gateway-setup guidance, and a private gateway notice returned success without a public post. The account-dependent polling, learning and posting flow was inspected and exercised only against test doubles in its unit suite; no real X request or post was made.",
      "tests": "32/32 unittest tests passed, including adapter polling/reply, allowlists, daily budget, privacy guards, setup defaults, signing, splitting and videos. Relay E2E was attempted after installing Wrangler, but its local runtime aborted with uv_interface_addresses returned Unknown system error 1; the relay checks consequently could not connect. This is a sandbox/runtime limitation, not a verified relay-code defect.",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 2,
      "best": "A real X gateway can summon the user’s own Hermes under a thread and carry approved learning back into its skills.",
      "worst": "Its audience and ongoing value depend on X accounts, paid API access or the author’s limited relay, and the live external service path remains unverified here.",
      "blocked": null
    },
    {
      "pr": 153900,
      "name": "annotary",
      "validate": "Passed catalog admission, original --install-deps/enable and runtime doctor (10 tools). Automatic admission passing does not resolve the manual rule-11 concern: accounts.token_for reads profile tokens from files and server._secrets reads owner credentials from secrets.json, rather than using Hermes secret scope.",
      "inside": "Pinned 253abf811b4be90f57834f8233730fbc688e7ced, repository josh-andrews-dev/annotary-hermes, root folder. Loaded real Hermes tools (deferred), Annotary CLI/skill and Desktop/backend surfaces. All 5 planned calls ran against the downloaded checksum-pinned v0.5.0 private server: list empty documents, share Markdown, read its text/version, create a span-anchored comment, and reject an option-like invalid document name cleanly. No real user name was supplied; default owner identity was left unchanged. The phone/Desktop rendering and wake-up through a running gateway were not exercised interactively.",
      "tests": "331 pytest tests passed, 3 skipped (240.10 seconds), using the real pinned Annotary server and isolated profiles; 29/29 Node Desktop tests passed. No model-driven gateway wake loop or physical phone test was run.",
      "useful": 4,
      "reliable": 2,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 25,
      "rank": 3,
      "best": "Versioned documents, exact-line threads and editor/reviewer wake rules turn agent drafts into a practical review workflow across Hermes surfaces.",
      "worst": "Despite passing the automated catalog gate, its owner and profile bearer tokens are read directly from plugin-owned files rather than Hermes’s secret store, conflicting with rule 11.",
      "blocked": null
    },
    {
      "pr": 154010,
      "name": "own-compute",
      "validate": "Passed catalog admission and original --install-deps/enable; security safe, no warnings. Doctor passed with 15 tools and 2 hooks. Internal manifest id is cloud-compute, as disclosed in README.",
      "inside": "Pinned 97522db41e7f6352089bd350df45bc64267a71e6, repository Siioh/hermes-own-compute, root folder. Loaded in real Hermes with deferred tools, /compute, guide skill, context and approval hooks. Main calls returned no-host setup guidance, supported host types, and an empty host list. Initial malformed env_exec was rejected by Hermes schema validation before invocation; a corrected-schema follow-up reached the plugin and returned structured not_found for the missing instance. Inspected secret-scope reads, fail-closed approvals, save-before-delete, host adapters, EC2 and Desktop code. No Docker/SSH/Slurm/EC2 workload or cloud-agent branch round trip was run live.",
      "tests": "Full pytest run: 3174 passed, 13 failed, 30 skipped (352.70 seconds). Targeted rerun of all 13 with a Git discovery ceiling: 2 passed, 11 failed. Two initial failures were enclosing-checkout discovery artifacts; remaining failures include 9 cloud-agent Stop/collect expectations returning not_fetched, a timer-shell fixture and a path-sensitive /work assertion. Those failures are not presented as a green suite; live host tests remain skipped.",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 1,
      "best": "It gives Hermes a substantial, approval-aware control plane for the user’s own containers, servers, cluster jobs and agent runs.",
      "worst": "The cloud-agent Stop/collect regression tests remain red, and its large remote-compute surface was not verified against a real host here.",
      "blocked": null
    },
    {
      "pr": 149750,
      "name": "neural-mesh",
      "validate": "Passed catalog admission (15 checks; caution only for destructive commands in the historical demo transcript). Runtime doctor FAILED on unknown on_pre_compress/on_memory_write hooks and unregistered declared hooks; manifest metadata does not match current Hermes hook contracts.",
      "inside": "Pinned 26abb610f3cbdd4215fbb898d4a2478317dfc5e5, repository D0xedDevi0/hermes-neural-mesh, root folder. Loaded as the selected memory.provider; four mesh tools were offered. All 5 planned calls ran inside AIAgent: empty stats, remember a dated deployment-cadence fact, recall it, dated recall, malformed date rejected with a clear message. Supplemental direct-provider probe confirmed unrelated dated query returns that deployment fact whereas ordinary recall correctly says no match.",
      "tests": "23/23 unittest tests passed offline. Supplemental relevance probe reproduced false-positive dated recall; no external service required.",
      "useful": 4,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 24,
      "rank": 5,
      "best": "A fully local memory provider gives Hermes persistent recall and dated, superseding facts without a service or key.",
      "worst": "The manifest fails runtime doctor on obsolete hook names, and dated recall returns unrelated valid memories because it has no relevance floor.",
      "blocked": null
    },
    {
      "pr": 151448,
      "name": "muster",
      "validate": "Passed catalog admission (16 checks); security caution is the README tip-jar link, not a failing gate. Doctor passed with 2 tools and 14 hooks. Original CLI enable FAILED and a warm retry failed again: PM/uv could not find the editable muster distribution under workspace/plugin-sources; automatic validation itself remained green.",
      "inside": "Pinned d02a2d4775ff6cee77b686101f0778d8e13e15eb, repository jollyroger1480/muster, root folder. Loaded through real Hermes; tools are deferred from the initial offer. Four AIAgent calls ran: muster_status returned an empty fleet, empty-goal muster_fanout returned a clear validation error, a one-job fanout returned a real lifecycle handle, and status showed its RUNNING row with has_handle=true. Completion, Stop/Message and visual Desktop interactions were not exercised live; inspected ledger, health derivation, lifecycle controls and Desktop/dashboard code. The successful functional run used manual local-directory configuration, so it is not evidence that the normal catalog install/enable path succeeds.",
      "tests": "34/34 pytest tests passed (final isolated run, 5.28 seconds), covering ledger, hooks, health, lifecycle actions, digest and UI helpers.",
      "useful": 4,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 24,
      "rank": 6,
      "best": "A durable, privacy-conscious fleet ledger makes real Hermes workers visible and controllable across chat and Desktop.",
      "worst": "The standard Hermes enable path failed twice on a missing editable package source, so a working manual load does not make its normal installation reliable.",
      "blocked": null
    }
  ]
}