{
 "batch": "deep-11",
 "setup": "Hermes cloned from github.com/NousResearch/hermes-agent (depth 1) into test/scratch and built with uv sync --locked --python 3.14 (uv installed per RUN.md; Python kept inside scratch). Each plugin was fetched at its pinned sha from field.json and run with RUN.md's tryit.py against Hermes's scripted fake model: an empty plan, then tool_search/tool_call plans, since plugin tools are deferred behind tool_search. Plugin tests were run with the Hermes venv's pytest.",
 "plugins": [
  {
   "pr": 151341,
   "name": "permission-canvas",
   "validate": "passed: hermes plugins validate --install-deps --json exited 0 (all admission checks ok); manifest, requires_hermes, loadable, capability probe, declared tools/hooks all ok",
   "inside": "Loaded and enabled; doctor OK with 0 tools and 0 hooks, because it registers a computer_use provider plus /pc-approve and /pc-canvas commands rather than tools. This Hermes build offered no computer_use tool (no cua-driver or desktop here), so computer_use list_apps and a bad raw-coordinate click both came back 'Tool computer_use does not exist'. The main path could not be exercised.",
   "tests": "passed: 314 passed, 9 skipped (unit suite, run against the Hermes source; the Windows/Edge integration runners were not run)",
   "useful": 3,
   "reliable": 4,
   "polish": 4,
   "creative": 5,
   "beyond": 5,
   "total": 26,
   "rank": 3,
   "best": "Colour-coded element-level permissions (allow, read-only, one-time approval, hidden with pixel masking) wrapped around Hermes computer_use as a real provider: a safety layer only a plugin can add.",
   "worst": "Only validated on Windows with cua-driver and a desktop, so its core path could not run here, and the README is dense.",
   "blocked": null
  },
  {
   "pr": 153900,
   "name": "annotary",
   "validate": "passed: hermes plugins validate --install-deps --json exited 0 (all admission checks ok)",
   "inside": "Loaded and enabled; doctor OK with 10 tools (annotary_*, deferred behind tool_search, which found annotary_share). On first use every tool crashed with 'OSError: [Errno 18] Invalid cross-device link': install.py downloads the pinned annotary binary into a TemporaryDirectory and os.replace()s it into the Hermes home, which fails whenever TMPDIR is on another filesystem. With TMPDIR on the same filesystem, annotary_share created doc v1, annotary_list listed it, and the bad inputs (doc '--rm', path /etc/passwd) were refused cleanly.",
   "tests": "passed: 331 passed, 3 skipped (pytest; downloads the real pinned binary)",
   "useful": 4,
   "reliable": 2,
   "polish": 5,
   "creative": 4,
   "beyond": 4,
   "total": 23,
   "rank": 6,
   "best": "Agents share Markdown plans that a person comments on line by line from Desktop, web or phone, with send-back/approve waking the agent: a real human-in-the-loop review channel with a Desktop page.",
   "worst": "First-run binary install crashes on any machine whose temp dir is on a separate filesystem (os.replace across devices). Arguments are only checked after that install, its tokens sit in plain 0600 files rather than Hermes's secret store, and every tool depends on a downloaded third-party binary.",
   "blocked": null
  },
  {
   "pr": 153316,
   "name": "hermes-lens",
   "validate": "passed: hermes plugins validate --install-deps --json exited 0 (all admission checks ok)",
   "inside": "Loaded and enabled; doctor OK. It adds a hermes_lens gateway platform (an aiohttp server for the iPhone app), a confirm-tools pre_tool_call hook, and 'hermes lens setup/try' CLI commands, but no model tools: tool_search found nothing for it. Its platform needs Ray-Ban Meta glasses, an iPhone build and a Gemini key, so the main path could not run here.",
   "tests": "passed: 67 passed (pytest with the Hermes source on the path; the 37 Swift tests were not run)",
   "useful": 3,
   "reliable": 4,
   "polish": 5,
   "creative": 5,
   "beyond": 5,
   "total": 27,
   "rank": 1,
   "best": "Takes Hermes onto your own hardware: Ray-Ban Meta glasses through an iOS app, as a new Hermes platform with tap-to-approve cards for gated actions. Its secrets go through Hermes's secret store.",
   "worst": "Needs smart glasses, an Xcode-built iPhone app and a paid Gemini Live key, so few Hermes users can run it, and none of it could be exercised offline.",
   "blocked": null
  },
  {
   "pr": 152651,
   "name": "workroom",
   "validate": "passed: hermes plugins validate --install-deps --json exited 0 (all admission checks ok) (7 Python requirements installed)",
   "inside": "Loaded and enabled; doctor OK with 31 tools and 14 hooks, plus a Desktop page and pane. Calls via tool_call: workroom_selftest passed its synthetic battery; workroom_add_project registered a folder; workroom_status listed it. Bad inputs: explain_figure with a string link was rejected by schema validation, and add_project '/' was refused with a clear error.",
   "tests": "passed: 730 passed, 16 skipped (tests/ui Playwright tests excluded)",
   "useful": 4,
   "reliable": 5,
   "polish": 5,
   "creative": 4,
   "beyond": 4,
   "total": 26,
   "rank": 2,
   "best": "A finished, local-only Office workroom that ties every figure in Word and PowerPoint to its Excel cell, lets only the owner approve releases, and checks the agent's claims through hooks, backed by 730 passing tests.",
   "worst": "Huge surface (31 tools, 14 hooks, about 70 routes) for a niche of Office-file users, and most of its benchmark claims are Windows/Office-only and unverifiable here.",
   "blocked": null
  },
  {
   "pr": 153302,
   "name": "lethe",
   "validate": "passed: hermes plugins validate --install-deps --json exited 0 (all admission checks ok)",
   "inside": "Loaded and enabled; doctor OK with 0 tools and 5 hooks (pre_tool_call, pre_llm_call, post_llm_call, on_session_finalize, on_session_start), plus /lethe commands and a Desktop panel. Off the record is off until /lethe or the Desktop panel arms it, and tryit cannot send slash commands, so the memory add went through as normal and read_file passed. The blocking path was checked only by its tests.",
   "tests": "passed: 67 tests, all passing dots (the summary line was cut from the captured output)",
   "useful": 3,
   "reliable": 4,
   "polish": 5,
   "creative": 4,
   "beyond": 4,
   "total": 24,
   "rank": 5,
   "best": "One-switch off-the-record chats: it blocks memory and skill writes and deletes the transcript at session end, with a polished Desktop river view and a disclosed demo.",
   "worst": "It deletes transcripts with no confirmation on /lethe-forget or the /forget route, its prompt note nudges the model to keep attempting saves, and its manifest carries a paste-in prompt telling the agent to edit the plugin.",
   "blocked": null
  },
  {
   "pr": 152914,
   "name": "culprit",
   "validate": "passed: hermes plugins validate --install-deps --json exited 0 (all admission checks ok)",
   "inside": "Loaded and enabled; doctor OK with 3 tools and 3 hooks. After two write_file edits, culprit_timeline listed the baseline plus 1 step (the second edit was not shown yet). culprit_bisect and culprit_revert (step 9999) were both stopped by its own approval hook ('requires approval', with no user present), as designed.",
   "tests": "passed: 29 passed, 1 skipped",
   "useful": 4,
   "reliable": 4,
   "polish": 4,
   "creative": 4,
   "beyond": 4,
   "total": 24,
   "rank": 4,
   "best": "git bisect over the agent's own tool calls: it snapshots every edit privately, finds the call that broke the tests and undoes just that one, with approval gates on anything that runs or changes files.",
   "worst": "The test command runs with shell=True, and in the run the timeline had not yet recorded the second edit, so step tracking may lag.",
   "blocked": null
  }
 ]
}