Make Kodus rules repository suitable as a source for AI code reviews

hermes commented

2026-08-31 08:32:10 +00:00

Owner

Context

This repository is intended to be the living source of rules used by an AI reviewer for repository and pull-request reviews. The current rules are useful, but the repository needs an explicit, machine-consumable contract so findings are precise, consistent, actionable, and safely mapped to GitHub review verdicts.

Requested work

1. Define and validate a rule schema

Add a documented schema (JSON Schema or equivalent) and CI validation for every rule file. Require, where applicable: a stable immutable rule ID, title, category, severity, scope (file, diff, or pull-request), language/framework applicability, path/include/exclude globs, detection guidance, exceptions and false-positive guidance, remediation guidance, and valid positive/negative examples. Reject malformed, duplicate, or incomplete rules in CI.

2. Normalize severity and verdict semantics

Document one severity model and its mapping to reviewer output:

critical: security breach, data loss/corruption, outage, or critical auth/payment failure; blocks merge;
high: important broken behavior, unsafe authorization, or severe reliability/performance issue; normally blocks merge;
medium: recoverable defect or meaningful risk; normally non-blocking unless project policy says otherwise;
low: minor quality or maintainability issue; informational.

Explicitly distinguish correctness issues from warnings and prevent style-only rules from becoming blocking findings. Define REQUEST_CHANGES, COMMENT, and APPROVE behavior.

3. Define a GitHub-compatible finding format

Specify a canonical finding containing rule ID/title, severity, file path, exact changed line or range when possible, concise technical problem, impact, correction, confidence, and blocking status. Document behavior for findings outside changed lines, missing locations, and uncertainty. Do not fabricate locations; security/auth uncertainty must fail closed.

4. Improve rule quality and consistency

Review existing rules for stable language conventions, consistent terminology/severity, narrow applicability paths, explicit exceptions, testable criteria, contradictory/duplicate coverage, and examples matching the rule text. Add checks for duplicate IDs/titles, invalid paths, missing examples, and severity/scope mismatches.

5. Add high-value missing checks

Consider explicit rules for SSRF, unsafe pagination/offset/limit/index validation (including negative values), insecure security fallbacks, timing attacks, case-normalization bypasses, TOCTOU/race conditions, cleanup on all lifecycle exits, transaction boundaries/idempotency/retries, tenant isolation, and sensitive data exposure in logs/errors/responses/telemetry/URLs.

6. Document project-context contracts

Document how consumers provide repository instructions, architecture constraints, linked repositories, API contracts, acceptance criteria, generated-code/migration exceptions, severity overrides, ignored paths, and language/framework selection. Make precedence explicit so policies cannot silently conflict.

7. Version and synchronize safely

Add a rule-set version/manifest and document compatibility, pinning by commit/tag, breaking changes, SHA verification, and failure behavior when the source is unavailable or invalid. Security-sensitive reviews must never silently use an old or partial rule set.

8. Test the rules

Add fixture-based tests for positive/negative examples, exceptions, zero/one/many cases, selection across multiple files, hostile inputs, concurrency-sensitive cases, expected severity/blocking status, and locations. Important rules should demonstrate failing behavior before the corresponding fix and passing behavior afterward.

9. Document contribution workflow

Add CONTRIBUTING guidance for adding/modifying rules, choosing severity, writing examples, running validation, and reviewing rule changes. Require contributors to identify sibling call sites/backends/exit paths and state what was not verified.

Acceptance criteria

All current rules validate against a documented schema.
Every rule has stable ID, category, severity, scope, applicability, exceptions, remediation, and valid examples.
CI rejects malformed or contradictory definitions.
Findings can be rendered as precise inline GitHub comments.
Blocking/non-blocking mapping to GitHub verdicts is explicit.
Consumers can verify the exact rule-set commit used.
Fixture tests cover current rules and representative edge cases.
README/CONTRIBUTING explain consumption and extension.

Non-goals

Implementing the AI reviewer itself.
Replacing project-specific architecture or security policies.
Treating every style recommendation as a merge blocker.

## Context This repository is intended to be the living source of rules used by an AI reviewer for repository and pull-request reviews. The current rules are useful, but the repository needs an explicit, machine-consumable contract so findings are precise, consistent, actionable, and safely mapped to GitHub review verdicts. ## Requested work ### 1. Define and validate a rule schema Add a documented schema (JSON Schema or equivalent) and CI validation for every rule file. Require, where applicable: a stable immutable rule ID, title, category, severity, scope (`file`, `diff`, or `pull-request`), language/framework applicability, path/include/exclude globs, detection guidance, exceptions and false-positive guidance, remediation guidance, and valid positive/negative examples. Reject malformed, duplicate, or incomplete rules in CI. ### 2. Normalize severity and verdict semantics Document one severity model and its mapping to reviewer output: - `critical`: security breach, data loss/corruption, outage, or critical auth/payment failure; blocks merge; - `high`: important broken behavior, unsafe authorization, or severe reliability/performance issue; normally blocks merge; - `medium`: recoverable defect or meaningful risk; normally non-blocking unless project policy says otherwise; - `low`: minor quality or maintainability issue; informational. Explicitly distinguish correctness issues from warnings and prevent style-only rules from becoming blocking findings. Define `REQUEST_CHANGES`, `COMMENT`, and `APPROVE` behavior. ### 3. Define a GitHub-compatible finding format Specify a canonical finding containing rule ID/title, severity, file path, exact changed line or range when possible, concise technical problem, impact, correction, confidence, and blocking status. Document behavior for findings outside changed lines, missing locations, and uncertainty. Do not fabricate locations; security/auth uncertainty must fail closed. ### 4. Improve rule quality and consistency Review existing rules for stable language conventions, consistent terminology/severity, narrow applicability paths, explicit exceptions, testable criteria, contradictory/duplicate coverage, and examples matching the rule text. Add checks for duplicate IDs/titles, invalid paths, missing examples, and severity/scope mismatches. ### 5. Add high-value missing checks Consider explicit rules for SSRF, unsafe pagination/offset/limit/index validation (including negative values), insecure security fallbacks, timing attacks, case-normalization bypasses, TOCTOU/race conditions, cleanup on all lifecycle exits, transaction boundaries/idempotency/retries, tenant isolation, and sensitive data exposure in logs/errors/responses/telemetry/URLs. ### 6. Document project-context contracts Document how consumers provide repository instructions, architecture constraints, linked repositories, API contracts, acceptance criteria, generated-code/migration exceptions, severity overrides, ignored paths, and language/framework selection. Make precedence explicit so policies cannot silently conflict. ### 7. Version and synchronize safely Add a rule-set version/manifest and document compatibility, pinning by commit/tag, breaking changes, SHA verification, and failure behavior when the source is unavailable or invalid. Security-sensitive reviews must never silently use an old or partial rule set. ### 8. Test the rules Add fixture-based tests for positive/negative examples, exceptions, zero/one/many cases, selection across multiple files, hostile inputs, concurrency-sensitive cases, expected severity/blocking status, and locations. Important rules should demonstrate failing behavior before the corresponding fix and passing behavior afterward. ### 9. Document contribution workflow Add CONTRIBUTING guidance for adding/modifying rules, choosing severity, writing examples, running validation, and reviewing rule changes. Require contributors to identify sibling call sites/backends/exit paths and state what was not verified. ## Acceptance criteria - All current rules validate against a documented schema. - Every rule has stable ID, category, severity, scope, applicability, exceptions, remediation, and valid examples. - CI rejects malformed or contradictory definitions. - Findings can be rendered as precise inline GitHub comments. - Blocking/non-blocking mapping to GitHub verdicts is explicit. - Consumers can verify the exact rule-set commit used. - Fixture tests cover current rules and representative edge cases. - README/CONTRIBUTING explain consumption and extension. ## Non-goals - Implementing the AI reviewer itself. - Replacing project-specific architecture or security policies. - Treating every style recommendation as a merge blocker.