test: add semantic regression corpus for Hermes rules #9

Merged

speedi57 merged 2 commits from hermes-semantic-fixture-tests into main

2026-09-04 10:07:21 +00:00

hermes commented

2026-09-04 09:55:09 +00:00

Owner

Goal

Add a deterministic semantic fixture corpus around the high-value Hermes rules introduced in the previous ruleset work, with an explicit bias toward false-positive prevention.

Corpus

Adds 48 cases covering 16 high-value semantic rules. Every covered rule has exactly three classes of scenario:

finding: concrete source-to-sink / invariant violation with enough evidence for high confidence;
no-finding: explicit safe implementation;
suppress: intentionally incomplete or ambiguous context where Hermes must not guess.

Covered areas include BOLA/IDOR, privileged authorization, mass assignment, path traversal, command injection, unsafe deserialization, credentialed CORS, HTTP timeouts, retry/idempotency, file uploads, uniqueness races, SQLAlchemy transaction/session state, frontend-only authorization, open redirects, React async races and multi-write atomicity.

CI invariants

Adds test/semantic-fixtures.test.mjs, automatically picked up by the existing npm test command and therefore by Forgejo Actions. It validates that:

fixture IDs are unique;
every fixture references an existing rule;
expectations are constrained to finding/no-finding/suppress;
concrete findings declare high confidence;
each covered rule has positive, safe and ambiguous cases;
non-finding coverage is at least 2x finding coverage;
ambiguous cases explicitly encode missing context.

Important scope

This PR establishes the deterministic regression corpus and its CI contract. It does not pretend that the Node test runner executes an LLM/Hermes review. The corpus is designed to become the stable input/expected-output contract for a subsequent Hermes evaluation runner, while already preventing accidental loss or weakening of the positive/negative/ambiguous test matrix.

## Goal Add a deterministic semantic fixture corpus around the high-value Hermes rules introduced in the previous ruleset work, with an explicit bias toward false-positive prevention. ## Corpus Adds 48 cases covering 16 high-value semantic rules. Every covered rule has exactly three classes of scenario: - `finding`: concrete source-to-sink / invariant violation with enough evidence for high confidence; - `no-finding`: explicit safe implementation; - `suppress`: intentionally incomplete or ambiguous context where Hermes must not guess. Covered areas include BOLA/IDOR, privileged authorization, mass assignment, path traversal, command injection, unsafe deserialization, credentialed CORS, HTTP timeouts, retry/idempotency, file uploads, uniqueness races, SQLAlchemy transaction/session state, frontend-only authorization, open redirects, React async races and multi-write atomicity. ## CI invariants Adds `test/semantic-fixtures.test.mjs`, automatically picked up by the existing `npm test` command and therefore by Forgejo Actions. It validates that: - fixture IDs are unique; - every fixture references an existing rule; - expectations are constrained to finding/no-finding/suppress; - concrete findings declare high confidence; - each covered rule has positive, safe and ambiguous cases; - non-finding coverage is at least 2x finding coverage; - ambiguous cases explicitly encode missing context. ## Important scope This PR establishes the deterministic regression corpus and its CI contract. It does **not** pretend that the Node test runner executes an LLM/Hermes review. The corpus is designed to become the stable input/expected-output contract for a subsequent Hermes evaluation runner, while already preventing accidental loss or weakening of the positive/negative/ambiguous test matrix.

hermes added 2 commits

2026-09-04 09:55:09 +00:00

test: add Hermes semantic review fixture corpus 23ffa35f4e

test: validate semantic fixture coverage and invariants

Validate rules / validate (pull_request) Successful in 7m40s

Details

6b5b1b360d