{
  "batch": "deep-10",
  "setup": "Installed real NousResearch/hermes-agent from source commit fdde27e7dae174cb4ba842589e92508852b7dd9f with uv sync --locked --python 3.14 (CPython 3.14.10); plugins help exposed install, validate and doctor. Used the supplied unmodified tryit.py with Nous FakeLLMServer and AIAgent.run_conversation, a fresh HERMES_HOME per plugin, first an empty plan then five scripted calls including bad input. The initial loopback requests went through the network proxy; setting NO_PROXY/no_proxy for localhost resolved those harness failures. Hermes defers plugin tools behind tool_call, so an empty added_by_plugin list did not mean absence: doctor registrations and successful tool_call dispatch verified them. Command-only plugins were exercised by terminal calls into their real Hermes CLI/PluginManager; their repository integration tests supplemented the conversations. Installed test dependencies in the scratch Hermes environment; kept Hermes off PYTHONPATH for plugin suites. All observations below are local runs of the field.json pins, not claims of live cloud, watch, camera, GPU-training or Desktop-GUI verification. RUN source: https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/RUN.md ; pin source: https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/field.json",
  "plugins": [
    {
      "pr": 152504,
      "name": "benchforge",
      "validate": "Passed: catalog validate --install-deps exit 0; security scan safe, declared capabilities matched, no core override; enable and doctor exit 0 (2 tools, 1 hook).",
      "inside": "Loaded bench_guide, bench_test, the pre_tool_call flash gate, /bench and hermes bench. Five scripted calls through real Hermes: prepared the pinned compiler fixture, got bench_guide, compiled/simulated the greenhouse draft (20/23, expected rollover and unplugged-sensor failures), tested the fixed sketch (22/22, passing receipt, approved=false), and rejected a bench missing steps with a clear error. Real PluginManager pre-tool checks subsequently blocked uploading both the failed firmware and the passing-but-unapproved firmware; no upload was executed. Arduino CLI 1.4.1/AVR core 1.8.6/Servo 1.2.2 came through the plugin installer, with Arduino's explicit proxy setting added only to its scratch configuration because its normal downloader could not resolve hosts in this sandbox. The actual plugin tools used this installed compiler. Code: https://raw.githubusercontent.com/ohjay-official/hermes-benchforge/62e633037f9ba253066d17818b533a8ee5549c32/__init__.py ; gate: https://raw.githubusercontent.com/ohjay-official/hermes-benchforge/62e633037f9ba253066d17818b533a8ee5549c32/benchforge/gate.py",
      "tests": "56/56 passed with the repository's unittest runner and its supported bare-AVR compiler adapter; 0 skipped after installing the pinned compiler (initial no-compiler run: 38 passed, 18 skipped). All 10 golden sketches matched their expected verdicts and failure sets using the real arduino-cli toolchain. The initial pytest invocation had a checkout-name import collision; using the repository's unittest runner resolved it without changing plugin source. Suites cover malformed benches, compile errors, simulator confinement, approval/cancel paths, receipt tampering and the repair loop. Suite: https://github.com/ohjay-official/hermes-benchforge/tree/62e633037f9ba253066d17818b533a8ee5549c32/tests",
      "useful": 3,
      "reliable": 5,
      "polish": 5,
      "creative": 5,
      "beyond": 4,
      "total": 26,
      "rank": 3,
      "best": "It actually compiles and fault-tests firmware on a simulated chip, produces reproducible receipts and checks approval before agent flashing; the observed draft-to-fixed results matched its claims (https://raw.githubusercontent.com/ohjay-official/hermes-benchforge/62e633037f9ba253066d17818b533a8ee5549c32/README.md).",
      "worst": "Its well-finished workflow serves the narrower Arduino Uno/ATmega328P niche and idealized listed parts, so it cannot validate other boards, real wiring or analog hardware behavior (https://raw.githubusercontent.com/ohjay-official/hermes-benchforge/62e633037f9ba253066d17818b533a8ee5549c32/README.md).",
      "blocked": null
    },
    {
      "pr": 153294,
      "name": "ci-red-to-green",
      "validate": "Passed: catalog validate --install-deps exit 0, security scan safe, declarations/no-core-override/Desktop checks passed; enable and doctor exit 0 (0 tools, 3 hooks).",
      "inside": "Loaded /ci, pre_llm_call/pre_tool_call/post_tool_call hooks, its skill, automation blueprint and Desktop backend; no tool schemas. Five scripted terminal calls invoked the registered /ci handler through the real Hermes PluginManager: malformed watch returned usage; demo created the bundled humanize incident; status showed four failing tests; discard succeeded; watch without a GitHub token returned a clear error. The repository's real-Hermes integration test reproduced the failure twice, observed approval and push-block directives, repaired the source, recorded three passing runs and full-suite runs, and emitted a source-only local patch. No real GitHub PR was sent. An extra real-Hermes probe then demonstrated a false proof: after fixing the original bug but adding two new regressions, three runs with PYTEST_ADDOPTS=-ktest_args1 were accepted as full-suite evidence and emitted a patch; the genuine full suite still had 2 failed/15 passed. Code: https://raw.githubusercontent.com/kinnuworks/ci-red-to-green/83009f03e864264524b8c18bed4ec5423680ab4b/runners.py ; https://raw.githubusercontent.com/kinnuworks/ci-red-to-green/83009f03e864264524b8c18bed4ec5423680ab4b/ledger.py",
      "tests": "96 passed, 0 failed in the supplied pytest suite, including test_hermes_e2e with HERMES_AGENT_DIR pointing to the real checkout. Additional regression probe exposed a product defect: status ready and proof accepted despite two actual full-suite failures; this is separate from the 96 passing repository tests. Integration test: https://github.com/kinnuworks/ci-red-to-green/blob/83009f03e864264524b8c18bed4ec5423680ab4b/tests/test_hermes_e2e.py",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 4,
      "total": 24,
      "rank": 5,
      "best": "It builds a useful approval-aware CI repair workflow around observed terminal results, immutable source-state evidence, protected tests and a reviewable patch/PR (https://raw.githubusercontent.com/kinnuworks/ci-red-to-green/83009f03e864264524b8c18bed4ec5423680ab4b/README.md).",
      "worst": "Its full-suite gate can certify a regressing fix because environment-based pytest filtering passes command validation and tests missing from the later run are ignored (https://raw.githubusercontent.com/kinnuworks/ci-red-to-green/83009f03e864264524b8c18bed4ec5423680ab4b/runners.py ; https://raw.githubusercontent.com/kinnuworks/ci-red-to-green/83009f03e864264524b8c18bed4ec5423680ab4b/ledger.py).",
      "blocked": null
    },
    {
      "pr": 151281,
      "name": "tironian",
      "validate": "Passed: catalog validate --install-deps exit 0, security scan safe, capability declarations matched, no core override and Desktop surface checks passed; enable and doctor exit 0. Doctor registered 11 hooks; the loader warned that provides_middleware is an unknown manifest field but still loaded the llm_execution middleware.",
      "inside": "Loaded the pinned plugin/ folder. Added /tironian, hermes tironian, 11 capture/lifecycle/output hooks, llm_execution routing middleware and the auxiliary judge slot; no tools. Five scripted terminal calls inside real Hermes: status and report saw the captured conversation, demo ran its actual pipeline over scripted history/student responses, promote nonexistent-family returned a clear refusal, and trainer check returned a clear not-set-up message (exit 1, no crash). The demo refused the weak student, passed the better student, accumulated shadow agreements, served locally, and fell back to the teacher exactly once for malformed output. GPU fine-tuning, a real trained student and the Desktop page were not exercised. Wiring: https://raw.githubusercontent.com/angelraph/tironian/9b590cc094f4b65cd70e979f632e1783424526a8/plugin/__init__.py ; router: https://raw.githubusercontent.com/angelraph/tironian/9b590cc094f4b65cd70e979f632e1783424526a8/plugin/core/engine.py",
      "tests": "104 passed, 0 failed after correcting loopback proxy routing; the initial proxy-induced failures were harness failures. Includes routing validation, fallback, session/place holdouts, privacy, demotion, trainer environment and provenance tests. Training in the demo was explicitly dry-run, not a trained model result. Test tree: https://github.com/angelraph/tironian/tree/9b590cc094f4b65cd70e979f632e1783424526a8/tests",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "It turns a user's own successful Hermes work into gated local-model routes, with held-out evaluation, shadow comparison and automatic fallback implemented in the model execution path (https://raw.githubusercontent.com/angelraph/tironian/9b590cc094f4b65cd70e979f632e1783424526a8/plugin/core/engine.py).",
      "worst": "The central promise still requires enough recurring history and a working training/serving stack; this seat verified the scripted pipeline and guardrails but could not establish real personalized-model quality (https://raw.githubusercontent.com/angelraph/tironian/9b590cc094f4b65cd70e979f632e1783424526a8/plugin/README.md).",
      "blocked": null
    },
    {
      "pr": 153113,
      "name": "standing-orders",
      "validate": "Passed: catalog validate --install-deps exit 0, five declared dependencies resolved, security scan safe, capabilities/no-core-override/Desktop checks passed; enable and doctor exit 0 (8 tools). Validator identified the standing_orders platform as trusted_inbound.",
      "inside": "Loaded eight so_* tools, /orders, hermes orders, the standing_orders gateway platform and Desktop backend. Five tool_call invocations in real Hermes: status reported camera not started; listing returned no orders; creating a mug reminder cleanly explained the missing reference frame; arming a nonexistent order returned no such order; adding a list item succeeded. No live camera, gateway alert delivery or real vision model was available. Separately downloaded the plugin's checksum-pinned CLIP ONNX models into scratch via its explicit model_dir argument and replayed all ten committed clips through the real engine with the committed cached vision answers: 9/10 correct, precision 1.0, recall 0.75, 81 vision calls versus 407 polling calls, 39 cache misses; the first mug-left-behind scenario did not fire. This is an offline replay, not live vision accuracy. Tools: https://raw.githubusercontent.com/jaxsonsprinkles/standing-orders/1959097a915e53624262c235e9eb102babece01f/standing_orders/hermes/tools.py ; replay: https://raw.githubusercontent.com/jaxsonsprinkles/standing-orders/1959097a915e53624262c235e9eb102babece01f/standing_orders/benchmark.py",
      "tests": "170 passed, 2 model-dependent tests skipped in the supplied pytest suite. Additional full recorded benchmark with real downloaded CLIP weights failed its all-scenarios-correct/zero-cache-misses acceptance: 9/10 correct and 39 misses; scenario 9 preserved the original action despite the camera prompt-injection fixture. The benchmark returned results normally; the external correctness assertion failed. Live-camera tests were not claimed. Suite: https://github.com/jaxsonsprinkles/standing-orders/tree/1959097a915e53624262c235e9eb102babece01f/tests",
      "useful": 3,
      "reliable": 3,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "total": 25,
      "rank": 4,
      "best": "It gives Hermes persistent camera-triggered agency through a real platform adapter, budgeted local/vision cascade, temporal rules, replay and correction machinery (https://raw.githubusercontent.com/jaxsonsprinkles/standing-orders/1959097a915e53624262c235e9eb102babece01f/README.md).",
      "worst": "Its claimed reproducible offline benchmark did not reproduce with the declared dependency range here, and the live camera implementation is limited to macOS AVFoundation (https://raw.githubusercontent.com/jaxsonsprinkles/standing-orders/1959097a915e53624262c235e9eb102babece01f/standing_orders/benchmark.py ; https://raw.githubusercontent.com/jaxsonsprinkles/standing-orders/1959097a915e53624262c235e9eb102babece01f/README.md).",
      "blocked": null
    },
    {
      "pr": 151543,
      "name": "seedguard",
      "validate": "Passed: catalog validate --install-deps exit 0, security scan safe, declarations and no-core-override checks passed; enable and doctor exit 0 (0 tools, 5 hooks). A benign unknown provides_middleware manifest warning did not prevent its three middleware registrations or actual redaction.",
      "inside": "Loaded /seedguard, five hooks and llm_request/llm_execution/tool_request middleware; no tools. Five calls ran in real Hermes: read_file returned a placeholder for a public BIP39 vector; terminal output masked a synthetic verification code; an unknown protected-secret placeholder was blocked before terminal execution; /seedguard doctor reported three auxiliary-path warnings; an invalid arming duration returned Minutes must be a number. Inspection of all six recorded main-model requests found neither the raw public mnemonic nor the synthetic OTP. The supplied real-Hermes integration test also exercised locked/armed rehydration through the actual middleware and hooks. Code: https://raw.githubusercontent.com/Investorquab/Seedguard/4809791d06471235fe3243fd1888c66eefb129e1/hooks.py",
      "tests": "157 passed, 1 opt-in CLI end-to-end test skipped (158 collected), including the real-Hermes PluginManager integration. Additionally attempted the repository CLI e2e script in both default and hardened modes: the scratch Hermes gateway failed to start before any provider request, so neither mode establishes plugin behavior here; its default PASS label was vacuous and was not counted, and hardened reported FAIL. The supplied tryit in-process conversation did work. Suite: https://github.com/Investorquab/Seedguard/tree/4809791d06471235fe3243fd1888c66eefb129e1/tests",
      "useful": 5,
      "reliable": 3,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 2,
      "best": "Its fail-closed main-provider middleware, output redaction and scoped timed release to allowed tools protect a boundary that a prompt cannot enforce, and the redaction/blocking worked inside Hermes (https://raw.githubusercontent.com/Investorquab/Seedguard/4809791d06471235fe3243fd1888c66eefb129e1/hooks.py).",
      "worst": "Its own doctor warns that title, compression and smart-approval side-model paths need extra mitigation, while the documented CLI path retains raw input locally; it is not complete secret protection at default settings (https://raw.githubusercontent.com/Investorquab/Seedguard/4809791d06471235fe3243fd1888c66eefb129e1/README.md).",
      "blocked": null
    },
    {
      "pr": 151546,
      "name": "wearos",
      "validate": "Passed: catalog validate --install-deps exit 0; capability, no-core-override and Desktop checks passed. Security scan was caution, not safe: sudo_usage in __init__.py:1407 and desktop/plugin.js:39-40. Enable and doctor exit 0 (3 tools, 1 hook). Manual review found a credential-store rule violation that the automated gate missed.",
      "inside": "Loaded watch_status, watch_pair, watch_send, /pair, hermes watch, on_session_end notifications, TTS registration and the Desktop backend. Five scripted tool_call invocations in real Hermes: status explained missing Tailscale/private lane and no devices; watch_pair opened a five-minute human-approval window without returning a code; watch_send cleanly reported no paired watch; watch_pair with an invalid code argument was refused; API status explained missing credentials. No device enrollment, notification delivery, revoke, live voice model or tailnet was exercised. Pairing and tool code: https://raw.githubusercontent.com/borealBytes/hermes-wearos-plugin/2991d28fa7aeeb86d52be77680f4be7c1d80a41a/__init__.py ; feature/setup scope: https://raw.githubusercontent.com/borealBytes/hermes-wearos-plugin/2991d28fa7aeeb86d52be77680f4be7c1d80a41a/README.md",
      "tests": "177 passed, 0 failed, plus 3 subtests passed, using the pinned source archived into a checkout named hermes-wearos-plugin. Initial socket failures disappeared with NO_PROXY for loopback; the remaining test failure in a checkout named wearos was pytest pre-importing that package name, and disappeared with the repository-style directory name. Tests use fake Tailscale/voice engines and local HTTP fixtures, so these passes are not evidence of real watch or cloud operation. Suite: https://github.com/borealBytes/hermes-wearos-plugin/tree/2991d28fa7aeeb86d52be77680f4be7c1d80a41a/tests",
      "useful": 3,
      "reliable": 2,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 24,
      "rank": 6,
      "best": "It extends Hermes onto a watch with a human-only pairing claim, device-bound private delivery, per-agent grants and a substantial Desktop control surface (https://raw.githubusercontent.com/borealBytes/hermes-wearos-plugin/2991d28fa7aeeb86d52be77680f4be7c1d80a41a/__init__.py ; https://raw.githubusercontent.com/borealBytes/hermes-wearos-plugin/2991d28fa7aeeb86d52be77680f4be7c1d80a41a/README.md).",
      "worst": "tts_air.store_token writes an HF token to plugin-data/hf_token and _hf_token reads that file even in the hosted plugin path, violating the catalog requirement that credentials come through Hermes rather than credential files (https://raw.githubusercontent.com/borealBytes/hermes-wearos-plugin/2991d28fa7aeeb86d52be77680f4be7c1d80a41a/tts_air.py).",
      "blocked": null
    }
  ]
}