{
  "batch": "deep-53",
  "setup": "Installed Hermes from https://github.com/NousResearch/hermes-agent at fdde27e7dae174cb4ba842589e92508852b7dd9f with uv sync --locked --python 3.14 in test/scratch/hermes; used Nous tryit.py and its FakeLLMServer with AIAgent.run_conversation. All six plugin repositories were fetched at field.json SHAs. Set HOME/HERMES_HOME to scratch fixtures and bypassed the proxy for loopback. Empty-plan discovery and catalog/doctor checks ran; follow-up harness selects breadcrumbs memory and seeds its documented demo; Hermes routes deferred plugin tools through tool_call, so an empty added_by_plugin list is not treated as a load failure. Takeover uses a simulated ready screen, not host desktop installation. No real provider key, Google or GitHub account was used. Follow-up admission was also run without repeating --install-deps; fast-call runs reused completed admission/doctor evidence and enabled the same pinned plugin in an isolated config. Runtime/test observations below are local evidence; linked pinned sources identify the code and tests inspected. Repeated enable calls during concurrent dependency preparation sometimes timed out; completed empty-plan enables passed, and follow-up conversations with explicit fixture enablement succeeded.",
  "plugins": [
    {
      "pr": 149978,
      "name": "clearvoice",
      "validate": "Passed: validate --install-deps, enable and doctor exited 0; safe scan, matching 3 tools and SDK-only Desktop surface. Pinned source: https://github.com/thebangarang/clearvoice-hermes-plugin/tree/d4bc366ec72d4f2af1a1772b3a94f48bd0e0b156.",
      "inside": "Loaded inside real Hermes; doctor registered clearvoice_speak, clearvoice_transcribe and clearvoice_say_back, plus slash command, CLI and Desktop backend. All 4 planned calls ran through AIAgent: speak queued the appointment sentence with spoken=false/awaiting_user=true; blank text returned No text to speak; deliver=file was denied while disabled; relative transcription path returned an absolute-path error. Speech engines/model weights were absent from the source-installed interpreter, so no real transcription, synthesis or microphone/UI interaction is claimed. Hermes routed deferred tools through tool_call.",
      "tests": "Initial suite collected 163: 158 passed, 4 failed, 1 skipped. Three reproducible failures expose missing clearvoice_engine/data/piper_voices.json in the pinned tree; the fourth CLI relative-out-path test first encounters absent Phonon dependency. Tests use mocks for speech; no real speech quality benchmark was run.",
      "useful": 3,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "rank": 6,
      "best": "A local accessibility bridge with single-use human approval prevents an agent from putting words in the user's mouth.",
      "worst": "The pinned tree omits its bundled Piper voice index, breaking advertised non-English voice lookup and three tests.",
      "total": 23,
      "blocked": null
    },
    {
      "pr": 151964,
      "name": "takeover",
      "validate": "Passed: validate --install-deps, enable and doctor exited 0; 1 tool and 3 hooks matched; safe scan and SDK-only Desktop surface. Admission warns about private tools.bot_desktop and tools.interrupt imports; websocket backend requires in-process operation. Pinned source: https://github.com/shoumikgoswami/Hermes-takeover/tree/e35bd71c00e085304e3767126dbfe0e24ea07894.",
      "inside": "Loaded in real Hermes with request_takeover, session hook, browser-tool fencing hook, result hints, /takeover, CLI and Desktop/dashboard surfaces. All 3 planned calls ran: wait_seconds=0 posted a request, blank reason returned a clear error, and a 1-second request timed out cleanly. For these calls only, readiness and screen status were simulated and bootstrap disabled; no real browser control, desktop package installation or sign-in was attempted. The core suite targets Hermes leases and WebSocket transport with a fake screen, with transport limits recorded below.",
      "tests": "Core unittest discovery: 71 run, 62 passed, 1 assertion failure, 8 errors. The WebSocket failures follow the sandbox refusing AF_UNIX socket creation in the fake-screen fixture; these do not demonstrate a production plugin defect. The remaining setup test expects TightVNC wording but gets generic installer fallback text; it leaves shutil.which unmocked, so this is not treated as proof of a broken install. Separately, all 10 Desktop mocked-SDK tests and all 36 WebUI/dashboard guidance tests passed. Real screen end-to-end remains unverified.",
      "useful": 5,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 3,
      "best": "A reusable human handoff with browser fencing solves authentication and consent interruptions across Hermes interfaces.",
      "worst": "Private Bot Desktop dependencies and automatic privileged first-use setup make this less portable and more fragile than its one-step install suggests.",
      "total": 27,
      "blocked": null
    },
    {
      "pr": 153020,
      "name": "breadcrumbs",
      "validate": "Passed: validate --install-deps, enable and doctor exited 0; safe scan, matching 3 hooks and SDK-only Desktop surface. Warning: private imports in island.py, logview.py, seats.py and slot.py may break across releases. Pinned source: https://github.com/breadcrumbs-agency/hermes-breadcrumb-plugin/tree/b8975645e5174edbcc00d5dc86a78e536bce938b.",
      "inside": "Loaded in real Hermes; selected memory.provider=breadcrumbs and seeded the documented neutral demo inside the isolated profile. All 4 planned calls ran: mcp__crumb_find found green-tea source turns, mcp__crumb(d98e0bf6,0.5) returned keyword zoom, level 1 returned the exact 80 C/three-minute/two-minute text, and an invalid id returned unknown id. Registers memory provider, optional context engine, /island hooks, CLI and explorer. ACE compression, IPFS, a local summarizer and Desktop interaction were read rather than end-to-end exercised.",
      "tests": "unittest discovery: 694 run, 668 passed, 25 skipped, 1 timing-sensitive hive failure (busy instead of aged). All 27 hive tests then passed in isolation with loopback routed directly. Core recall/zoom/error calls passed inside real Hermes. No benchmark or ACE throughput claims treated as independently verified.",
      "useful": 5,
      "reliable": 4,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "rank": 1,
      "best": "Exact-source memory with deterministic zoom and a native context engine improves the agent itself without requiring another model service.",
      "worst": "Private Hermes seams and a timing-sensitive hive test temper confidence in the much broader sharing/context-engine feature set.",
      "total": 28,
      "blocked": null
    },
    {
      "pr": 153294,
      "name": "ci-red-to-green",
      "validate": "Passed: validate --install-deps, enable and doctor exited 0; safe scan, 0 tools/3 hooks match, Desktop stays inside SDK. In-process notice is informational (gateway-bound post_message), not an admission failure. Pinned source: https://github.com/kinnuworks/ci-red-to-green/tree/83009f03e864264524b8c18bed4ec5423680ab4b.",
      "inside": "Loaded in real Hermes; empty-plan conversation completed, adding /ci, 3 hooks, the fix-red-ci skill, morning-ci automation blueprint and Desktop proof surfaces, rather than a new model tool. Five real plugin-manager command calls: /ci status showed no watches, /ci watch malformed gave usage, /ci demo started the bundled incident, /ci status listed its four failing cases, /ci discard cleared it. Its real-Hermes E2E test additionally reproduced failing pytest output, applied the source fix, observed passing runs, verified first-code-execution approval and push blocking, and exported a source-only patch after proof. No live GitHub PR or polling account was exercised.",
      "tests": "96 passed with HERMES_AGENT_DIR set, including test_demo_through_the_real_plugin_manager (not skipped): 3 failing-before and 4 passing-after test invocations, proof ready, source-only patch. Unit tests cover log parsing, proof ledger, source/test guards, runners, API token/redirect handling and engine state.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 4,
      "rank": 5,
      "best": "It proves a source-only fix against unchanged tests and asks the person before opening a PR, rather than merely suggesting a patch.",
      "worst": "The supported path is limited to pytest/go test on GitHub Actions, and it still relies on the model and a compatible local environment to produce the fix.",
      "blocked": null,
      "total": 26
    },
    {
      "pr": 153724,
      "name": "roomsmith",
      "validate": "Passed: validate --install-deps, enable and doctor exited 0; safe security scan, all 11 tools/1 hook match and Desktop SDK check passed without warnings. A separate direct validate_plugin_dir check also passed all 16 checks. Pinned source: https://github.com/devSebb/roomsmith/tree/421e6f30fca43066f75ed7258281315c7eeb2fde.",
      "inside": "Loaded inside real Hermes with 11 tools, /room and an awareness hook. All 5 planned calls ran: created a calibrated 4 x 5 x 2.5 m study at revision 1; atomically placed a 1.2 x 0.75 x 0.6 m desk and USD 1,000 budget at revision 2; returned 3 geometric fit candidates; rejected width=-1 with out_of_range; inspect confirmed the original room and revision 2 remained intact. Source includes shared SQLite scene state, atomic revisions/undo, product dimensions and budget logic, GLB export and Desktop/dashboard editor. No actual Desktop session, vision photo reconstruction or live merchant extraction was exercised.",
      "tests": "255 Python tests passed; TypeScript tsc --noEmit passed; all 13 editor Node tests passed. Coverage includes geometry, placement, atomic operations, budgets, shortlist, product parsing/fetch security, hardening and backend routes.",
      "useful": 3,
      "reliable": 5,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "rank": 2,
      "best": "A shared editable 3D scene makes spatial planning measurable, persistent and reversible across the person and their agent.",
      "worst": "Its strong implementation serves an occasional room-design niche, and photo dimensions still require real measurements while merchant pages can lack usable product data.",
      "blocked": null,
      "total": 28
    },
    {
      "pr": 150110,
      "name": "hermes-colab",
      "validate": "Passed: validate --install-deps, enable and doctor exited 0; safe scan, 8 tools/1 hook match, OAuth schema and SDK-only Desktop surface pass. Pinned source: https://github.com/PietrangeloManco/hermes-colab/tree/f05cf934fff3b446f2e00cc32e19432e9e41e87b.",
      "inside": "Loaded in real Hermes with 8 Colab tools, background supervisor, completion hook and Desktop run page. All 5 planned calls ran: status listed no runs; doctor returned ready=false with exact missing OAuth-client setup; run from the isolated profile working directory was refused by private-path policy; unknown-run logs returned no such run; runtimes returned a clean sign-in instruction. A second 5-call conversation used an absolute missing script outside the profile and returned a clear does-not-exist error; all calls again completed. No Google account/runtime was available, so GPU allocation, real checkpoint transfer, preemption recovery and notification delivery were assessed from code and tests, not represented as live successes.",
      "tests": "Initial suite: 61 passed, 5 local-Jupyter tests skipped because the server dependency was absent. After installing jupyter_server/ipykernel, the full attempt showed 3 failures before a 300-second test-runner timeout. Isolated test_kernel_roundtrip failed in 32 seconds with RuntimeUnreachable: kernel websocket handshake timeout; the fixture kernel repeatedly exited under sandbox restrictions, so this does not establish a plugin protocol defect. Bundle/path guards, OAuth transport, supervisor state/recovery, progress/logs, output sync, tool errors and dashboard unit tests passed in the initial suite; actual runtime execution remains unverified.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 4,
      "best": "Hermes gains persistent supervised GPU jobs with checkpoint sync and restart handling while leaving Google OAuth credentials in Hermes core.",
      "worst": "Google client setup is substantial, and authenticated GPU/recovery operation remains unverified because no account was available and the local Jupyter fixture could not keep a kernel running.",
      "blocked": null,
      "total": 27
    }
  ]
}