{
  "batch": "deep-52",
  "setup": "Installed NousResearch/hermes-agent source at fdde27e7dae174cb4ba842589e92508852b7dd9f with uv sync --locked --python 3.14 (CPython 3.14.10); verified CLI install/validate/doctor commands. Used RUN.md and supplied tryit.py with Nous FakeLLMServer and real AIAgent, no external model, account or key. Empty discovery runs preceded planned calls. Original --install-deps validation and enable succeeded for agent-si and triage, but redundant PM runtime preparation was slow; further probes used the same Hermes admission checks without dependency installation, declared dependencies preinstalled in the Hermes venv, and configuration-based enabling. Selected memory.provider=agent-si and disabled external session indexing; indexed Triage bundled demo skills. Expanded schemas with tools.tool_search.enabled=off for planned reruns because current Hermes defers plugin schemas behind tool_call by default. All runtime/test results below are local observations at the field.json pins; external functionality not exercised is stated explicitly. Plugin source was not repaired. Tied totals are ranked by expected retained installation breadth: agent-si over argus; no-cap over triage and barehand, with triage above barehand.",
  "plugins": [
    {
      "pr": 149242,
      "name": "agent-si",
      "validate": "Passed: supplied tryit.py Hermes plugins validate --install-deps exited 0, enable exited 0, doctor exited 0. Warning only: private imports from plugins.memory.config_schema; no failed admission check. Planned rerun with dependencies already present also passed.",
      "inside": "Loaded joelbrilliant/hermes-agent-si@adf7abc1d89b1e1ea686ac2fd29e2b95b454f4fe at repository root. Enabling alone registers an exclusive memory provider; selecting memory.provider=agent-si exposes brain_search, brain_get and brain_remember and per-turn recall. With session indexing disabled for isolation, all 4 planned calls ran in a completed real Hermes conversation: remembered a Tuesday meeting preference with the exact user quote, found it by search, retrieved Tuesday with source/date, and refused a fabricated Friday quote. Cross-agent session indexing and live Desktop configuration were read/tested through fixtures, not exercised on personal histories.",
      "tests": "Passed: 19 pytest tests, covering the provider and MemoryManager integration, sourced facts/disputes, recall fusion, settings and quote refusal; none failed or skipped.",
      "useful": 5,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 2,
      "best": "Gives Hermes durable local memory with cited user evidence and explicit disputes, without another model, network service or dependency install.",
      "worst": "Keyword-only retrieval misses synonyms, owner attribution protection requires setting a principal, and vendored top-level module names can collide with other plugins.",
      "blocked": null
    },
    {
      "pr": 154048,
      "name": "triage",
      "validate": "Passed: original Hermes plugins validate --install-deps exited 0; doctor exited 0 with 2 tools and 4 hooks. Warnings: private core imports in catalog_link.py, dashboard/plugin_api.py and hermes_hide.py; app-markup selectors in desktop/plugin.js require manual review (no failed admission checks).",
      "inside": "Loaded pinned Saderius/hermes-triage@913a61d62b0795926531ecdfbf8980a1e9c0b9c9, repository root. Adds skill_search/skill_load, request/tool middleware, four hooks, CLI/slash commands and Desktop composer chips. Empty run completed; initial schemas deferred behind Hermes's bridge, but all four direct planned calls dispatched: git commit search found bundled git-commit at 100%, skill_load returned its full body, empty search returned query-is-required, and nonexistent skill returned an unknown-name explanation. Real conversation completed without a crash. Desktop code was reviewed, not rendered interactively.",
      "tests": "Upstream suite passed: unittest discover ran 141 tests, 139 passed and 2 skipped. An additional isolated helper check against pinned hermes_hide.sync_hidden/restore_all reproduced a bug: first sync adopted a pre-existing disabled git-commit, and restore removed that disable while preserving an unrelated private-skill.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 5,
      "best": "Replaces an always-present skills index with local retrieval and explicit loading, integrated into Hermes's prompt pipeline and Desktop composer.",
      "worst": "Fresh-install restore can re-enable a skill the user had already disabled, and the documented shared new-chat attachment state can also cross windows.",
      "blocked": null
    },
    {
      "pr": 149272,
      "name": "no-cap",
      "validate": "Passed: Hermes catalog validation exited 0, all checks passed, no warnings; doctor exited 0, registering 4 hooks and no model tools. Dependency-free Python runtime.",
      "inside": "Loaded jedbrnth-eng/no-cap@56576ea3caf1c3ef55547b6897cde0b12c63e294, repository root. Adds post-tool evidence capture, final-reply receipt transformation, strict pre_verify and pre-LLM hooks, slash/CLI commands, Desktop receipt chips and Cap Board. Empty real Hermes conversation completed. Three planned terminal calls ran inside Hermes: nocap doctor reported no setup problems, nocap battery matched gold labels for all 125 bundled synthetic cases, and approving an unknown coach proposal returned a clear no-proposal message. Supplemental real AIAgent conversation ran a deliberately failing scratch pytest test; scripted final 'All tests pass.' was transformed into a CAP receipt citing 1 failed and exit 1. A separate real Hermes CLI doctor probe with a scratch competing transform correctly reported the conflict and exited 1. This verifies actual hook/diagnostic paths, not general detection accuracy; live Desktop rendering and external blind-eval figures were not independently reproduced.",
      "tests": "Full pytest suite completed with 30-second per-test bounds: 5,375 passed, 3 failed, 2 skipped, 17 xfailed and 1 deprecation warning; no per-test timeout fired. Failures were test_not_a_repo, the FIFO-.git artifact assertion (both under the enclosing task repository), and the mocked CLI doctor conflict assertion with installed Hermes; the separate real Hermes conflict-doctor probe correctly exited 1. Of the 17 expected failures, 14 are intentional build/file contradiction abstentions and 3 are known redaction defects. The latter 3 were explicitly rerun with --runxfail: all 3 failed, exposing synthetic credential fragments in persisted JSONL. Reliability 2 reflects this reproduced privacy defect, not a failed catalog gate or a crashing primary call.",
      "useful": 5,
      "reliable": 2,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "Turns Hermes's own tool results into inexpensive, inspectable receipts and can challenge unsupported completion claims without another model.",
      "worst": "Windowing long tool outputs can persist credential fragments in receipts, reproduced in all three shipped expected-failure redaction cases.",
      "blocked": null
    },
    {
      "pr": 152481,
      "name": "argus",
      "validate": "Passed: Hermes catalog validation exited 0, no failed checks; doctor exited 0 with 5 hooks and no model tools. Warning: private core imports for skill_ledger, skill_manager_tool and hermes_bootstrap in apply/apply_child; public registration itself passed.",
      "inside": "Loaded albertodvp/argus@bf0b6a3b7ecf978590605ad166956229c39e238d, subfolder argus. Adds five capture/lifecycle hooks, an auxiliary model slot, CLI/slash commands and an intent/skill-review dashboard. Empty conversation completed. Four real Hermes terminal calls inspected status, backfilled a two-day window with estimated model-call costs, ran derive, and refused a missing proposal in dry-run (exit 1 with a clear refusal, not a crash). Stock scripted auxiliary text could not assign the turn: its saved report recorded one failed model call and one unassigned turn. A supplemental Hermes CLI derive with schema-shaped scripted auxiliary responses created one intent, assigned its turn and judged it, with 4 model calls and 0 failures; this proves plumbing, not real-model judgement quality. No live skill change or rendered dashboard was exercised; apply safety was covered by tests and code review.",
      "tests": "Passed: all 115 pytest tests after installing declared mypy dependency (first run had 4 missing-mypy failures). Covers assignment/consolidation, model-error handling, scoped history, dashboard API, edited/locked skills, rollback and proposal/new-skill approval checks.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "Makes Hermes's self-learning inspectable across sessions and requires a person's decision before applying evidence-backed skill changes.",
      "worst": "Derivation can make hundreds of auxiliary calls, its history window only widens, and private core APIs plus model-dependent labels add maintenance and operating costs.",
      "blocked": null
    },
    {
      "pr": 153871,
      "name": "barehand",
      "validate": "Passed: catalog validation exited 0, no warnings; doctor exited 0 with 1 tool and 1 hook. No Python dependencies declared. The computer-use provider is a second role in the same plugin directory.",
      "inside": "Loaded mrv777/barehand@7e62bc9419afbe70f175de0515abca3ede66dd38, subfolder barehand. Adds a QMP/VNC computer_use backend, barehand_machine control tool, destructive-action approval hook, CLI/slash commands, skill and flight recorder. Empty run completed. Without QEMU or a configured endpoint, requirements gating hid barehand_machine: the first planned conversation got two unknown-tool refusals and stopped before the remaining calls, rather than a plugin-handler crash. Supplemental real Hermes conversation attached the plugin's synthetic QMP protocol fixture and selected computer_use.backend=barehand: status reported a running 128x96 display, computer_use captured it, snapshot_list returned none, missing snapshot name and unknown action returned clear errors; all 5 calls completed. No real QEMU guest, VNC hardware, OS installation or boot repair was run.",
      "tests": "Full pytest run: 724 passed, 11 failed, 7 skipped. Failures included restricted Unix sockets/process inspection, two Hermes capture_after tests attempting an unconfigured auxiliary vision model, four screen-text assertions, an RFB timeout and stopped-fake-VNC doctor assertion. Focused rerun of the 4 screen-text cases and RFB timeout passed all 5; stopped-fake-VNC doctor case still failed (1 failure of 6 focused tests). The complete suite is therefore not clean on this host; 7 integration/environment tests were skipped.",
      "useful": 3,
      "reliable": 3,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 26,
      "rank": 6,
      "best": "Extends Hermes down to firmware and bootloader screens over QMP/VNC, with snapshots, readable screen text, human approval gates and a replayable recorder.",
      "worst": "It is a specialist rescue/install tool with no real guest verified here, and the non-clean suite, including screen-text checks that passed only when rerun separately, leaves more runtime uncertainty than the stronger entries.",
      "blocked": null
    },
    {
      "pr": 152458,
      "name": "housemate",
      "validate": "Passed: Hermes catalog validation exited 0, all 17 checks passed, no warnings; its 5 declared dependency requirements were present in the test environment. Doctor exited 0, registering 7 tools and 1 hook. Platform runs in-process, as disclosed by the isolation audit.",
      "inside": "Loaded RaYYeR220/housemate@09c1a1d0a88bfa8762cee6b3ccf94271b0f172c5, repository root. Adds a gateway voice platform, seven room/timer/display tools, spoken-answer hook, CLI/slash commands, blueprints, streaming Piper provider and Rooms Desktop page. Empty conversation completed. Supplemental real AIAgent conversation used the plugin's actual loopback Hub, ControlServer, RoomRegistry, TimerService and SQLite store (no microphone/TTS device): listed Kitchen/Office, created a 600-second tea timer, listed it, rejected -1 seconds with a bounded-duration error, and cancelled tea; all five calls returned normally and the conversation completed. Full satellite microphone, wake-word, STT/TTS playback and live Desktop rendering were not exercised; shipped screenshots and UI code were reviewed.",
      "tests": "Passed on corrected invocation: 565 passed, 1 skipped, 4 slow/model-dependent tests deselected (-m 'not slow'). Initial invocation from the plugin root had 25 failures: local tools.py shadowed Hermes tools in approval tests, and loopback model-download fixtures hit the proxy; running from the parent with loopback excluded from proxy resolved them without source edits. Includes real WebSocket pairing, approvals, restart recovery, room arbitration, timers and Desktop contracts.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "Turns spare browser devices into a coherent Hermes room platform with durable timers, speaker arbitration and human approvals rather than merely wrapping a voice prompt.",
      "worst": "The complete hands-free experience needs a running gateway, paired browsers, downloaded wake/VAD models and working STT/TTS, and those live audio paths remain unverified here.",
      "blocked": null
    }
  ]
}