{
  "batch": "deep-28",
  "setup": "Installed Hermes once from https://github.com/NousResearch/hermes-agent at fdde27e7dae174cb4ba842589e92508852b7dd9f using uv sync --locked --python 3.14; install/validate/doctor CLI verified. Fetched all six exact commits from field.json. Used the supplied tryit.py with empty plans and Nous tests/fakes/fake_llm_provider.py, then scripted tool calls. Deferred tools need not appear in the initial offered list: direct calls demonstrated registration. All reported executions are local observations, not entrant claims. Desktop-only entries were additionally exercised through real Hermes discovery and its API router mounting function on an in-process FastAPI test client; Rich Paste also through Hermes invoke_middleware. No paid model, account or device was used. Time Machine received a separate sample workspace; supplemental runs aligned conversation cwd with it. Plugin source was not patched. Tests ran without Hermes on PYTHONPATH; Hermes remained importable from its editable installation, affecting tests that assume Hermes is absent. Browser rendering could not be verified: the available Chromium aborted in this runtime. No physical 3DS or Desktop GUI was available. Rank prioritizes demonstrated behavior and likely continued installation; equal totals have distinct ranks.",
  "plugins": [
    {
      "pr": 153524,
      "name": "shadow-army",
      "validate": "Passed: tryit.py catalog validation --install-deps, enable and doctor all exited 0. Runtime doctor registered 2 tools and 6 hooks; no built-in override was granted. Reviewed the pinned manifest, bridge, policy/service, setup, Desktop/dashboard code and tests.",
      "inside": "Pinned source: https://github.com/SilentKnight87/shadow-army/tree/aa340f06ffe10dce5c63cfabc0e0c6de3e9d1e41/shadow-army . Empty-plan chat completed; the initial tool list deferred the plugin tools. All three planned direct calls ran inside AIAgent: army_gate(nonexistent-card) returned a card-not-found error; army_verdict(PASS, empty findings) refused an unbound caller; army_gate(empty card) returned a nonempty-id requirement. The plugin also registers a delegation backend, verification skill and review pane. No actual independent model review was claimed from this fake-model run; successful binding, stale-receipt and evidence cases were inspected in code and exercised in unit tests.",
      "tests": "Local full pytest: 151 passed, 6 failed, 236 subtests passed (157 tests). Three failures came from non-repository fixtures finding the enclosing Git repository; one real-Hermes host-probe subprocess timed out at 120 seconds; two setup-count/context-name assertions failed. A rerun with Git discovery stopped at the scratch boundary and the timed-out host case explicitly deselected gave 154 passed, 2 failed, 1 deselected, 236 passing subtests. The two remaining failures are the additional Hermes-version warning caused by a missing expected runtime context name; no clean full-suite pass is claimed.",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "Code-bound, expiring independent-review receipts and observed check evidence give Hermes a meaningful completion gate that a review prompt cannot enforce.",
      "worst": "Its session and board context dependencies produce compatibility failures in the current Hermes tests, and the full host dispatch probe did not finish within its own timeout.",
      "blocked": null
    },
    {
      "pr": 151448,
      "name": "muster",
      "validate": "Catalog admission passed (exit 0), including matching 2 tools/14 hooks and Desktop SDK surface. Scanner caution md_link_exfil points to the README's local-install command rather than an exfiltration implementation. Doctor passed. Native enable failed twice in tryit.py; a normal pinned Git install with --yes-deps --enable reproduced the same missing editable plugin-sources distribution path in Hermes PM. This is an observed integration/staging failure, not evidence of malicious plugin code.",
      "inside": "Pinned source: https://github.com/jollyroger1480/muster/tree/d02a2d4775ff6cee77b686101f0778d8e13e15eb . Empty-plan chat completed but the refused enable left muster unavailable; status and empty fanout returned tool-not-found and later planned calls were not reached. In a clearly separate developer-config run with muster explicitly enabled before registration, status returned an empty fleet, empty goals returned a helpful error, and valid fanout returned a real public-lifecycle child handle. The fourth status result was not captured, so child completion and Stop/Message were not established here. Read the ledger, health analysis, lifecycle operations, registrations, pane/dashboard and tests; no automatic cancellation or approval bypass was found.",
      "tests": "Local pytest: 34 passed, 0 failed. Covers registration, durable ledger updates, crash reconciliation, stall/spin classification, lifecycle launch/control mocks and budget/digest behavior; this is not proof that the native enable path works.",
      "useful": 4,
      "reliable": 2,
      "polish": 5,
      "creative": 3,
      "beyond": 5,
      "total": 24,
      "rank": 6,
      "best": "A durable fleet ledger with health states, user-priced spend, and public lifecycle controls makes delegated Hermes workers much easier to supervise.",
      "worst": "The pinned package could not be enabled through the normal CLI in this Hermes build, so its strong unit tests and developer-mode launch cannot remove the reliability cap.",
      "blocked": null
    },
    {
      "pr": 149202,
      "name": "3ds-platform",
      "validate": "Passed: catalog validation --install-deps, enable and doctor all exited 0; security scan safe, no core override, no model tools or hooks declared/registered. Read plugin manifest/README, main app README, adapter, CLI, cryptographic protocol/pairing, approval and media paths, compatibility helpers and tests.",
      "inside": "Pinned source: https://github.com/rndaom/Hermes3DS/tree/3182c0b1cc97072e6aa047dd463474e3b12b84d7/plugin . Loaded as a native 3ds gateway platform, device-management CLI and optional streaming speech provider; empty-plan AIAgent chat completed and correctly added no model tool. Three real Hermes terminal calls exercised its installed CLI: 3ds devices returned no paired devices (exit 0); unpair missing-console returned No such device (exit 1); away invalid-option returned an argparse choice error (exit 2). These are handled negative inputs, not crashes. The Python suite exercised loopback encrypted pairing/reconnect, revoked devices, replay, streaming and request failures. A physical console, full gateway-to-agent chat on that console and Tailscale access were not available; no hardware success is claimed.",
      "tests": "Local Python pytest: 226 passed, 12 failed (238 tests). Eleven failures were ffmpeg/ADPCM duration or speech chunk assertions (for example a requested 1500 ms clip reported 1573 ms); one adversarial rich-text timing case exceeded 3 seconds. Host C runner: Noise IK vectors/tamper/replay/codecs, cache, autoplay, camera YUV and tones passed; JPEG round trip skipped without libjpeg-turbo. The ADPCM decoder comparison against ffmpeg failed on 22 blocks/44902 samples, so the script stopped before its encoder comparison. No 3DS-toolchain app build or physical-device run was performed.",
      "useful": 3,
      "reliable": 3,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 26,
      "rank": 5,
      "best": "A source-readable encrypted handheld platform with native approvals, camera, voice and session controls is an unusually complete way to put a user's own Hermes on their own hardware.",
      "worst": "Its audience requires a homebrew-capable 3DS, and both Python audio timing tests and the C decoder comparison against ffmpeg failed while physical-device behavior remains unverified.",
      "blocked": null
    },
    {
      "pr": 149553,
      "name": "hermes-time-machine",
      "validate": "Passed: catalog validation --install-deps, enable and doctor exited 0; five tools/five hooks matched the manifest, security scan safe, no core overrides. Read README, manifest, configuration, handlers/snapshot/storage, effect parsing/compensation, approval gate, memory hooks and test modules.",
      "inside": "Pinned source: https://github.com/ddaikodaiko/hermes-time-machine/tree/1ac9c78e539dd518d6c8d8ffd64395b72bab30d4 . Empty-plan chat completed; tools are deferred in the first model request, but direct atm_* calls worked. The default harness cwd overlaps plugin data, and checkpoint correctly refused that unsafe workspace; list/effects returned empty results and invalid rewind failed cleanly. With a separate configured workspace, checkpoint and automatic mutation checkpoint succeeded and reused unchanged content; rewind preview and invalid-id handling worked. One run exposed a real configuration edge: a write inside the configured protected workspace was classified as outside because effect preflight uses session cwd instead. Aligning the conversation cwd removed that false approval gate. All final five calls ran successfully as intended: checkpoint captured one file; read_file read its old contents; write_file changed it and received a time_machine restore-point annotation; rewind preview identified exactly note.txt as modified; invalid rewind returned No checkpoint found. No irreversible remote action or human approval was fabricated.",
      "tests": "The documented unittest runner passed all 122 tests after placing temporary fixtures deeper than the effect parser's ancestor-search limit (with short paths). An initial 121/122 pass found the enclosing repository in the test for a missing .git; an overlong isolation path then hit the expected target-string truncation assertion. Both were fixture artifacts, resolved without changing plugin code. Initial pytest collection also failed because the hyphenated checkout was treated as an unparented __init__; unittest is the documented runner. Passing cases cover snapshots/restores, compensation, approvals, effect-memory replay and malformed inputs.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 29,
      "rank": 2,
      "best": "Keeping irreversible-effect memory after chat rollback or compaction solves a real agent failure that ordinary file snapshots and prompts cannot solve reliably.",
      "worst": "Effect classification uses session cwd while checkpoints honor the configured workspace, producing a false outside-workspace approval gate when those directories differ.",
      "blocked": null
    },
    {
      "pr": 152408,
      "name": "rich-paste",
      "validate": "Passed: catalog validation --install-deps, enable and doctor exited 0; declared llm_request middleware matched registration and the Desktop bundle passed SDK-surface/no-core-override admission checks. Read README/manifest, interleave.py, cache/download/security backend, page API, Desktop conversion/composer code and Python/TypeScript tests.",
      "inside": "Pinned source: https://github.com/ZhenningLang/rich-paste/tree/1e3ec4bb45ce016aee1ce3e4d07d1d19fcc46e16 . Empty-plan Hermes chat completed, correctly adding no model tool: its agent contribution is request middleware. Through actual Hermes discovery and API mounting, four calls returned: version 200, private-loopback image URL 200 with per-image blocked_address, malformed urls string 400 with urls must be a list, save-markdown 200 with a cache file. Hermes invoke_middleware on a two-image model request produced text/image/text/image in the intended order; malformed request data passed through without a callback result. This verified request rewriting and backend errors, not an actual Desktop clipboard or live vision-model benchmark. macOS clipboard integration, cloud/local transcription and HEIC system support were not exercised.",
      "tests": "Python: 248 passed, 0 failed after bypassing the network proxy for loopback test requests; the initial 246 pass/2 fail run routed fake test hostnames to the proxy and returned 502. TypeScript explicit-glob runner: 167 passed, 4 skipped, 0 failed out of 171. A separate SDK-surface run pointed at the real Hermes source: 4 passed, 1 skipped (no packaged Desktop build). The shipped npm test command itself failed on Node 24 because it passes a directory rather than explicit test files; no source was changed to run them.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 29,
      "rank": 1,
      "best": "It fixes an everyday Desktop and multimodal-request problem at the composer and middleware layers, with tested image placement, sensible failures and substantial safety coverage.",
      "worst": "Native clipboard behavior is macOS-focused, screenshot matching has a disclosed order-fallback gap, and the npm test entry point needs an explicit file glob on Node 24.",
      "blocked": null
    },
    {
      "pr": 153978,
      "name": "imprint",
      "validate": "Passed: catalog validation --install-deps, enable and doctor exited 0; zero tools/hooks matches its standalone Desktop design and imprint_write auxiliary-task registration. Read README/manifest, runtime/model bridge, source ingestion, page routes, cited-answer and export checking paths, browser/HTML rendering security code, Desktop sources and broad tests. Its declared pypdf>=6.21.4,<7 was explicitly installed for the final API probe; an initial generic test dependency install had selected pypdf 7.",
      "inside": "Pinned source: https://github.com/Dgardenhire/hermes-imprint/tree/a8ce37bc2f62e0c1d287a36b836a227eef243747 . Empty-plan Hermes chat completed and correctly added no agent tool. Real Hermes discovery registered its writing task; Hermes's own API mounting function mounted the page backend. Five representative page calls: state 200; add Volunteer report text source 200; read its passages 200; empty question 400 (Type a question first); valid question 200 creating a background job. With Nous's fake provider configured to return JSON on its auxiliary route, that job completed with 42 meals and a citation chip resolving to the actual selected source's para. 1. The earlier default fake auxiliary prose produced a clean cut_off job error, not a crash. This tests actual PluginLlm routing and citation plumbing, not real-model factual accuracy or generated slide aesthetics. Browser-based drawing and native Desktop were not verified.",
      "tests": "Local pytest: 692 passed, 5 failed, 32 skipped (729 tests), plus 4857 passing subtests. Documented unittest run: 729 run, 3 failures, 2 errors, 32 skipped. The same five cases account for both: two tests assume the Hermes-safe web client is unavailable, but the editable Hermes install makes it available; two HTML-loop tests need an undiscovered browser; one expects a robots error but receives the earlier missing-browser error. Chromium was found separately but aborted on a prohibited Unix socket, so rendering could not be established. Many source, citation, number-check/export, import-security, revision and offline-font cases passed; no full clean suite is claimed.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "A coherent source notebook and branded publishing studio inside Desktop combines the user's Hermes model, verifiable citation links and code-enforced number checks into a substantial product.",
      "worst": "Its most ambitious browser-rendered design workflow remains unverified here, and number/quote matching cannot establish whether a generated headline faithfully represents its sources.",
      "blocked": null
    }
  ]
}