How it works

One docs PR, from open to gate decision. Click for full size.

When an author opens a ready docs PR, a triage workflow labels it with a shell classifier. Trivial or frontmatter-only PRs get a quick Claude Haiku prose check and skip the full review. Everything else gets the full review. Vale runs on changed files and keeps findings only on added lines. Claims are found three ways, a regex floor plus two Claude Sonnet passes, and each one is verified by a bounded Claude Opus agent loop, routed to a lane by plain rules (some claims settle with no model call at all). A script drafts the review, Opus edits it without cloud credentials, and a validator rejects any edit that demotes or drops a finding. A separate publish job posts the results, but only if the reviewed commit is still the PR head. Finally the Sentinel, a pure function of GitHub state with no model, evaluates five gates and posts one commit status. markdownlint runs separately as part of the PR build.

Notable engineering details

• Models never decide a merge. Model output enters only as files that scripts validate, and findings may move up toward blocking, never down.
• Privilege split. The model job holds no cloud credentials, only the model key and a PR-scoped token, and runs only for authors with write access. The publish job runs only default-branch scripts and refuses to publish a review of a stale head.
• Redundant claim extraction. Regex plus two differently framed passes feed a claim floor, and each claim is routed by rules to source, URL, or web-search verification. A verdict whose only evidence is the page under review gets re-checked.
• The Sentinel fails closed. An evaluation error blocks the merge. The review:waived break-glass label is honored only from an active routing-team member, and the verdict summary names who applied it.
• Safe on untrusted PRs. The Sentinel never checks out PR code, and a test asserts it. A repo setting runs it report-only before it enforces.
• Models chosen by benchmark. Haiku for trivial PRs, Sonnet for extraction, Opus for verification and editing, with the benchmark results recorded next to the choices.

Code and write-up

• Review machinery and the Sentinel
• Review workflow
• Sentinel workflow
• The docs skills (open source)
• Treating Prompts Like Code

Code walkthrough

Step through the code, chapter by chapter

Built by code-walkthrough, my Claude Code skill, from pulumi/docs at a pinned commit. Every line links to GitHub, so you can check its work.

Back to the portfolio