{
  "batch": "deep-09",
  "setup": "Hermes source cloned once; uv sync --locked --python 3.14 succeeded (Hermes commit fdde27e7dae174cb4ba842589e92508852b7dd9f). CLI provides install/validate/doctor. Used supplied tryit.py and Nous FakeLLMServer with isolated scratch homes, no external model/account. Plugin tools are deferred: discovered schemas with tool_describe and invoked via tool_call. Runtime/test statements are local observations, not entrant claims. Some tests cannot use Unix sockets, multiprocessing semaphores or bubblewrap here; these limits are distinguished from code failures. Tests use Hermes venv with pytest/PyYAML/NumPy/PyNaCl/BLAKE3/HuggingFace Hub added, without Hermes source on PYTHONPATH. Initial parallel empty-plan baseline raced; discarded added_by_plugin deltas and used isolated sequential baselines for scored calls. Supplemental replay reused an enabled scratch home and changed only its fake model endpoint. Plugin README/manifest/code/tests were reviewed at each field.json pin; sources.json contains pinned primary-source excerpts, while execution results and test counts are this seat's observations. No judge-steering instruction was found; adversarial browser test fixtures were treated as test data. Ranking favors broad retained usefulness after safety and activation limits, not advertised feature counts.",
  "plugins": [
    {
      "pr": 150644,
      "name": "hermes-canvas",
      "validate": "Passed: catalog validate --install-deps exit 0; doctor exit 0 (4 tools, 7 hooks). First parallel dependency preparation warned inputs changed; the catalog checks still passed and enable succeeded. Pin: https://github.com/0xbyt4/hermes-canvas/tree/d6eeff0d6f7ec653262537bdd7bb001cc6d3c7f8 (pinned README, implementation and tests in sources.json).",
      "inside": "Loaded and enabled. Adds 4 deferred canvas tools, 7 work/steering/verification hooks, skill, subagent and Desktop pane. Local tryit: omitted root returned a clear absolute-path error; supplemental real-Hermes replay with an explicit scratch project root created/read a two-box plan, returned only the dependency-ready data box with a tagged delegate goal, and refused a cyclic update with dependency cycle: data -> report -> data. No call crashed. Desktop rendering and mid-build human gestures were read/tested, not interactively exercised. Additional five-call replay proved hook binding: a real write_file wrote a scratch JSON file, its terminal check exited 0, and canvas_get showed writes=1, status=done, check_exit=0; the cyclic update still failed cleanly.",
      "tests": "Passed: 306 pytest tests. Initial 305 pass/1 fail was the non-repository test discovering the enclosing assignment Git repo because TMPDIR is inside scratch; a Git discovery boundary inside scratch fixed the fixture without changing plugin code.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "Makes coding-agent plans inspectable and steerable while actual hooks track writes, checks, drift and subagent ownership.",
      "worst": "Its canvas veto is a best-effort parser rather than a security boundary, and a plain AIAgent session needed an explicit project root despite that argument being optional.",
      "blocked": null
    },
    {
      "pr": 153225,
      "name": "agent-pool",
      "validate": "Passed catalog validate --install-deps and doctor (3 tools, 10 commands). Initial empty-plan enable succeeded; a later fresh-home enable was refused by Hermes PM: editable agent-pool package metadata pointed at a missing staged distribution. Conservatively caps Reliable at 2 for this observed activation failure. Pin: https://github.com/maxcurrent420/agent-pool/tree/5b973a5c3dfd8633840bb438208211da1c78f3cf (pinned README, implementation and tests in sources.json).",
      "inside": "Loaded in initial real-Hermes home; later failed activation made its tools unavailable in that new conversation, so those bridge errors are not counted as plugin execution. Supplemental replay in the successfully enabled home: pool_status returned ledger/identity ready and an actionable missing Tor-sidecar warning; pool_pane_report persisted a local operation; invalid-claim pool_report_usage failed cleanly at the Tor readiness gate. No real Discord peer, Tor transfer, ingestion or merge was exercised.",
      "tests": "Full pytest: 413 passed, 47 failed, 11 skipped, 5 setup errors. Many failures/errors are denied AF_UNIX socket creation in ClamAV checks and denied bubblewrap namespaces; others include a stale install-deps expected-text assertion and sandbox-mode assertions. This is not a passing suite.",
      "useful": 3,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 23,
      "rank": 6,
      "best": "Adds signed encrypted peer-to-peer repository sharing and gateway-owned task coordination with explicit authorization and fail-closed Tor readiness.",
      "worst": "The Tor/sidecar/ClamAV setup is demanding, activation failed on a repeated fresh install, and its static audit falls back to an unsandboxed subprocess when bubblewrap fails.",
      "blocked": null
    },
    {
      "pr": 152918,
      "name": "look",
      "validate": "Passed: catalog validate --install-deps, enable and doctor all exit 0; 14 tools, 7 hooks, request/tool middleware and CLI/chat commands. Catalog warns about private core imports in benchmark/probe_profile.py and runtime/native_input.py. Pin: https://github.com/SilentHacks/look/tree/7c01a94aa651adedaf27e5216598c6deef293050 (pinned README, implementation and tests in sources.json).",
      "inside": "Loaded. Real-Hermes tool_describe found its deferred tools. gaze_benchmark completed 10 synthetic trials per noise level and returned baseline/fusion metrics; these are simulated reference-resolution results, not measured webcam accuracy. gaze_read and bad file-URL gaze_navigate both safely returned executed=false / No active browser task; the URL-validation branch was not reached. Real Chrome extension, pointing, speech, webcam and human confirmation could not be exercised on this Linux headless seat.",
      "tests": "Full pytest after installing NumPy/HuggingFace prerequisites and normalizing the test umask: 366 passed, 15 failed, 393 subtests passed. Fourteen failures directly raise PermissionError on forbidden AF_UNIX socket creation; the extension-setup assertion also fails in that constrained setup. A previous private-directory assertion was only the restrictive test umask and passed on rerun. Actual Mac Chrome interaction remains untested.",
      "useful": 4,
      "reliable": 3,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "Brings pointed-at browser evidence, private answer cards and guarded browser actions into Hermes through a substantial local multimodal stack.",
      "worst": "The supported beta is Apple Silicon Mac, so this seat verified only synthetic resolution and safe refusal rather than the actual browser interaction loop.",
      "blocked": null
    },
    {
      "pr": 153955,
      "name": "shadow",
      "validate": "Passed: catalog validate --install-deps, enable and doctor all exit 0. Declares model/provider override capabilities; registers 6 tools, 12 hooks, 3 middleware layers, Desktop/RPC and CLI/chat command surfaces. No provider/model override was granted for this test. Pin: https://github.com/akinduroifedayo/shadow/tree/38173bc9c696df3b1f87b5b9b60ccecd36a260ab (pinned README, implementation and tests in sources.json).",
      "inside": "Loaded in real Hermes. tool_describe exposed six deferred controller/replay tools. shadow_setup_check reported no verified isolation and saved nothing; shadow_trial_step refused an unstarted controller step with continue=false; bad-path shadow_read was blocked as not run in a test. These are guarded failures, not a completed comparison. No paid provider replay, outside-maker judge, cost meter or model switch was exercised; those paths are assessed from code and the plugin tests. Supplemental real CLI: doctor returned explicit missing-model, capability and live-isolation fixes without changes; status said nothing running; results with a nonexistent ID said no tests yet.",
      "tests": "Pytest completed: 1458 passed, 436 skipped, 7 warnings in 466 seconds. Warnings are unhandled watcher-thread assertions from a fixture event wait timing out; therefore not described as a clean all-pass. The test import issue was fixed by preimporting the inner shadow package, without Hermes source on PYTHONPATH. Separate scripted acceptance: 8 PASS, 4 SKIP(needs-mac), 0 FAIL; the skipped cases include a real 20-turn provider comparison, live gradual-switch evidence, live switch/undo and Desktop screenshots. These are fixture checks, not a live model result.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 29,
      "rank": 1,
      "best": "Improves the user's own agent by comparing models on saved real turns with blind order-swapped judging, spending limits and a reversible model switch.",
      "worst": "A live multi-provider comparison and the rendered Desktop/switch path remain unverified here, and the otherwise passing unit suite emitted seven watcher-thread timeout assertion warnings.",
      "blocked": null
    },
    {
      "pr": 150807,
      "name": "manyhands",
      "validate": "Passed: catalog validate --install-deps, enable and doctor all exit 0 (6 tools, 3 hooks, llm_request middleware, owner approval transport and /many command). Security-scan caution is a destructive-command example in README; not an executed command. Runtime warns that provides_middleware is an unknown manifest field but still registers the middleware. Pin: https://github.com/PhiBao/manyhands/tree/bc5ec1cb57bb26a133f7967bd06dee6441409544 (pinned README, implementation and tests in sources.json).",
      "inside": "Loaded in real Hermes. whoami identified the local operator with owner permissions. remember with valid text reported no person attached; recall returned no-person/empty results; blank remember returned text is required. An additional replay using the local CLI subject had the same roster limitation. Included real-PluginManager live integration passed role blocking, one-use approvals, memory scoping and audit scenarios. A separate fault probe through the real get_pre_tool_call_directive gate normally blocked an unattributed terminal call, but returned (None, None), i.e. no permission directive, when Runtime.audit raised a simulated SQLite disk-full error. No terminal command was actually dispatched in the fault probe.",
      "tests": "Selftest: 68 tests, 0 failures/errors, but logs an unhandled owner-denial-thread SQLite cannot commit - no transaction is active error. Live Hermes integration script: PASS. Supplemental fault probes reproduced the fail-open permission error both directly and through the real Hermes gate.",
      "useful": 4,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 24,
      "rank": 5,
      "best": "Gives a shared Hermes gateway per-human identities, private fact namespaces, owner approvals and an attributable local audit trail.",
      "worst": "The permission-hook wrapper catches unexpected exceptions and returns None, so an audit-store failure can silently remove this plugin's block on a forbidden tool.",
      "blocked": null
    },
    {
      "pr": 153240,
      "name": "flowcanvas",
      "validate": "Passed: catalog validate --install-deps, enable and doctor all exit 0; 7 tools and no hooks. Registers a Desktop page, dashboard API, /flow command and skill; catalog scan and Desktop SDK checks passed. Pin: https://github.com/EmotionalThrifty/arachne-flow/tree/46db25db1a7ef13e2adbf12777b7c06941df0895 (pinned README, implementation and tests in sources.json).",
      "inside": "Loaded. Real-Hermes tool_describe found its 7 deferred tools. flow_list listed starter templates and unavailable tool dependencies; flow_apply saved the five-node local-list-cleanup template; flow_run completed all nodes in 98 ms, removing empty values and duplicates while retaining Unicode, false and zero; invalid flow_get ../missing returned a structured not_found error. Paid media, external MCP, messaging delivery and the rendered Desktop UI were not exercised.",
      "tests": "Python pytest: 586 passed, 4 failed, 3 skipped, 330 subtests passed. All four failures are multiprocessing messaging-lock tests denied creation of OS semaphore resources in this sandbox, with secondary cannot join an unstarted process errors. Desktop JavaScript helper suite: 269 passed, 0 failed; this exercises UI logic with SDK/React stubs, not a rendered Desktop session.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 3,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "Turns Hermes tools and models into persistent visual graphs with parallel execution, typed dataflow, caching, templates and visible per-node results.",
      "worst": "It needs external providers for its headline media workflows, and background dispatch plus configurable MCP/tool allowances leave safety dependent on the chosen nodes and operator configuration.",
      "blocked": null
    }
  ]
}