Evidence-backed software factory pilot

Date: 2026-09-02
Status: Continuation active; trial 1/5 completed as PR #333, which remains open for a separate human merge decision
Branch: codex/software-factory-pilot-20260902 from origin/dev at 181445d948610ebd7b665d7333a4f6d0cbc7e7a3

Hypothesis

StratoFusion can make agent-assisted changes easier and safer to review by combining structured intake, task isolation, deterministic checks, and private runtime evidence, while retaining its existing architecture, CI, CASA gates, and human merge authority.

This is a falsifiable pilot. A do not adopt result is valid.

Inputs and interpretation

The external workflow is useful as a menu of ideas, not a repository operating contract. The creator's software factory video and skills repository propose isolated task branches, service boundaries, runtime evidence, before/after handoff, and iterative AI review. The video's description identifies Cursor as its sponsor. Its speed, throughput, and rebuild claims are therefore treated as creator/vendor self-report, not independent evidence.

Primary product documentation confirms that Cursor Cloud Agents use isolated VMs and separate branches and can produce screenshots, videos, and logs. Those capabilities are not being installed or evaluated in this pilot. Greptile markets an iterative 5/5 review loop and currently lists a free individual tier plus a paid Pro tier. Neither the score nor the service is part of this pilot. See Cursor Cloud Agents, , and .

PR	Base <- head	Files	Explicit acceptance criteria	Reported verification	Runtime or visual evidence	Privacy/deployment boundary	CI result	Review/revision signal	Automated reviewer
#332 Google OAuth verification response	`dev` <- `cdx-gqc007v/google-oauth-verification-response`	28	No	Exact commands/results, suite counts, build, Playwright, CASA	Evidence package described; no PR-linked before/after artifact	Explicit: no dashboard mutation, deployment, credential use, video upload, or Google reply	All reported checks passed	1 commit; no GitHub review objects	None observable
#330 reusable local-VM images	`dev` <- `cdx-c644i0q/local-vm-multi-machine-release`	6	No	Commands plus a GitHub Actions run and OCI revision labels	Runtime/image identity values reported; no visual artifact	Test-key and immutable-release boundary stated	All reported checks passed	1 commit; no GitHub review objects	None observable
#329 infrastructure management links	`dev` <- `cdx-c644i0q/infrastructure-management-tools`	25	No	Focused tests, lint, typecheck, Compose validation, live local E2E	Live local observations described; no linked screenshot/video	Read-only UI, private URLs, no host controls	All reported checks passed	2 commits; no GitHub review objects	None observable
#328 admin MFA freshness	`dev` <- `cdx-gq007v/casa-admin-mfa-freshness`	18	No	Exact pass counts across focused/full suites, build, Playwright, CASA	Redacted runtime proof explicitly left outstanding	Explicitly no certification/readiness claim or production proof	All reported checks passed	1 commit; no GitHub review objects	None observable
#327 two-admin MFA evidence	`dev` <- `cdx-gq007v/casa-mfa-two-admin-evidence`	10	No	Exact pass/skip/environment-limited results and reruns	Redacted evidence recorded in repository files; negative-boundary evidence outstanding	Detailed exclusions for identities, tokens, claims, IP/device data	All reported checks passed	2 commits; no GitHub review objects	CodeRabbit was requested; completed review not observable
#326 runner-platform framing	`main` <- `cdx-c644i0q/generalize-runner-platform-framing`	3	No	Social validation, focused content tests, check, diff check	Not applicable and not stated	Non-endorsement boundary stated	All reported checks passed	2 commits; no human review objects	CodeRabbit summary; a later run was rate-limited
#325 CI article punctuation	`main` <- `cdx-c644i0q/remove-em-dashes-and-publish-social`	5	No	Content tests, social validation, check	Not applicable and not stated	Not stated	All reported checks passed	1 commit; no human review objects	CodeRabbit summary
#324 tighten CI article	`main` <- `cdx-c644i0q/tighten-ci-article`	6	No	Content tests, social validation, check	Not applicable and not stated	Runner-time caveat stated	All reported checks passed	1 commit; no human review objects	CodeRabbit comment was rate-limited

Base <- head

Files

Explicit acceptance criteria

Reported verification

Runtime or visual evidence

Privacy/deployment boundary

CI result

Review/revision signal

Automated reviewer

#332 Google OAuth verification response

dev <- cdx-gqc007v/google-oauth-verification-response

Exact commands/results, suite counts, build, Playwright, CASA

Evidence package described; no PR-linked before/after artifact

Explicit: no dashboard mutation, deployment, credential use, video upload, or Google reply

All reported checks passed

1 commit; no GitHub review objects

None observable

#330 reusable local-VM images

dev <- cdx-c644i0q/local-vm-multi-machine-release

Commands plus a GitHub Actions run and OCI revision labels

Runtime/image identity values reported; no visual artifact

Test-key and immutable-release boundary stated

All reported checks passed

1 commit; no GitHub review objects

None observable

#329 infrastructure management links

dev <- cdx-c644i0q/infrastructure-management-tools

Focused tests, lint, typecheck, Compose validation, live local E2E

Live local observations described; no linked screenshot/video

Read-only UI, private URLs, no host controls

All reported checks passed

2 commits; no GitHub review objects

None observable

#328 admin MFA freshness

dev <- cdx-gq007v/casa-admin-mfa-freshness

Exact pass counts across focused/full suites, build, Playwright, CASA

Redacted runtime proof explicitly left outstanding

Explicitly no certification/readiness claim or production proof

All reported checks passed

1 commit; no GitHub review objects

None observable

#327 two-admin MFA evidence

dev <- cdx-gq007v/casa-mfa-two-admin-evidence

Exact pass/skip/environment-limited results and reruns

Redacted evidence recorded in repository files; negative-boundary evidence outstanding

Detailed exclusions for identities, tokens, claims, IP/device data

All reported checks passed

2 commits; no GitHub review objects

CodeRabbit was requested; completed review not observable

#326 runner-platform framing

main <- cdx-c644i0q/generalize-runner-platform-framing

Social validation, focused content tests, check, diff check

Not applicable and not stated

Non-endorsement boundary stated

All reported checks passed

2 commits; no human review objects

CodeRabbit summary; a later run was rate-limited

#325 CI article punctuation

main <- cdx-c644i0q/remove-em-dashes-and-publish-social

Content tests, social validation, check

Not applicable and not stated

Not stated

All reported checks passed

1 commit; no human review objects

CodeRabbit summary

#324 tighten CI article

main <- cdx-c644i0q/tighten-ci-article

Content tests, social validation, check

Not applicable and not stated

Runner-time caveat stated

All reported checks passed

1 commit; no human review objects

CodeRabbit comment was rate-limited

Gate	Result	Evidence
Isolated worktree and unique branch from latest `origin/dev`	Go	Worktree and branch listed above; shared checkout remains untouched
No overlap with relevant open PRs or another worktree	Go	Open PR #331 is a `main` release candidate touching unrelated release/CASA/product files; no open pilot branch exists
Existing stack can implement the helper without a dependency	Go	Node, TypeScript, PNPM, Playwright, and Chromium capture APIs already exist
Product/runtime behavior remains unchanged	Go	Scope is repository workflow files, local scripts/tests, docs, and drafts only
Secrets, provider state, auth, billing, rclone, database, and production remain out of scope	Go	Helper uses a fresh browser context or existing local PNGs, limits browser requests to read-only loopback traffic, and performs no upload
CI and human approval remain authoritative	Go	No workflow, branch protection, reviewer, merge, or deployment setting changes are proposed
Pilot can be removed cleanly	Go	New templates/skill/helper/docs/drafts are separable; package and index entries can be reverted
Baseline can prove time or review-effort improvement	No-go for that claim	No baseline review timings exist; the pilot will report completeness and evidence output, not time saved

Check	Result	Notes
`pnpm env:guard`	Pass	Windows-native PowerShell environment; no WSL
`pnpm evidence:pr -- --help`	Pass	Usage and local-only boundary rendered
`pnpm test scripts/pr-evidence/core.test.ts`	Pass after one useful failure	Final: 9 tests passed; initial generic token-redaction miss fixed and covered
`pnpm evidence:smoke`	Pass	Four-file local evidence package complete; images visually inspected
Focused ESLint and Prettier checks	Pass	`scripts/pr-evidence/*.ts` and touched article test formatted; no lint warnings
`pnpm social:validate`	Pass	6 repository social records validated, including the new draft
`pnpm test scripts/social/content-store.test.ts src/lib/articles.test.ts`	Pass	9 tests passed; new article draft explicitly remains outside published results
`pnpm check`	Pass	Repository lint and TypeScript checks passed
`pnpm audit --prod --json`	Pass after remediation	Zero production advisories reported after applying the two transitive overrides.
`pnpm casa:security`	Pass after remediation	Web and worker production audits, static scans, tracked-change secret scan, file-size check, deployment/supply-chain policy, and assessor handoff policy passed. Live database checks were skipped because `DATABASE_URL` was intentionally absent. Generated evidence changes were discarded.
GitGuardian Security Checks	Pass after owner disposition	GitGuardian classified the deliberately synthetic CLI-redaction fixture in commit `c3b856e3` as a generic CLI secret. The value was not a credential, was replaced in the current tree with a dynamically assembled fixture, and the sole maintainer dismissed incident `36837173` as a false positive. A fresh scan passed on closing head `0acd16c3`.
`pnpm build`	Pass after remediation	The production compile and generate phases completed successfully with the patched dependency graph.

Measure	Baseline	Pilot result	Confidence	Adopt?
Intake completeness	0/8 sampled PRs had explicit acceptance criteria	Issue and PR templates now request testable acceptance criteria and exclusions; actual author completion is unmeasured	Medium for structure, none for future compliance	Continue
Reviewer comprehension	Verification was strong, but handoff structure varied	The sole maintainer rated the evidence `partly` helpful: structured acceptance, safety boundaries, and exact checks helped verification, while fixture screenshots and duplicated result maintenance added little value. This is self-review, not independent evidence.	Low	Continue with revisions
Runtime evidence usefulness	No consistent linked evidence package in 8 PR bodies	Smoke package is deterministic, visually legible, named consistently, and honest about missing evidence	Medium for mechanics, low for real PR usefulness	Continue
First-pass verification	8/8 sampled PRs had passing CI	One initial helper-test failure exposed a redaction defect. A later CASA advisory failure was remediated with patched transitive versions. Closing CI run `33628767467` passed every repository job, including CASA, GitGuardian, and aggregate `build-and-test`, on head `0acd16c3`.	High for observed commands	Continue with the failures recorded
Added maintenance burden	No template/helper maintenance	889 lines across the helper, test and skill, plus a 43-line canonical testing section, concise templates and drafts; repeated-use burden is unknown	High for line count, low for ongoing effort	Continue narrowly
Privacy/security risk	Evidence practices were task-specific	Loopback/read-only request policy, fresh context, metadata redaction, ignored output and manual pixel review are explicit; deny-list sanitisation and screenshot pixels remain residual risks	Medium	Continue only with human inspection
Vendor/tooling cost	Existing GitHub, CodeRabbit, PNPM, and Playwright stack	No direct dependency or paid service added; two transitive packages were pinned to patched versions through existing PNPM overrides	High	Continue

Measure

Baseline

Pilot result

Confidence

Adopt?

Intake completeness

0/8 sampled PRs had explicit acceptance criteria

Issue and PR templates now request testable acceptance criteria and exclusions; actual author completion is unmeasured

Medium for structure, none for future compliance

Continue

Reviewer comprehension

Verification was strong, but handoff structure varied

The sole maintainer rated the evidence partly helpful: structured acceptance, safety boundaries, and exact checks helped verification, while fixture screenshots and duplicated result maintenance added little value. This is self-review, not independent evidence.

Low

Continue with revisions

Runtime evidence usefulness

No consistent linked evidence package in 8 PR bodies

Smoke package is deterministic, visually legible, named consistently, and honest about missing evidence

Medium for mechanics, low for real PR usefulness

Continue

First-pass verification

8/8 sampled PRs had passing CI

One initial helper-test failure exposed a redaction defect. A later CASA advisory failure was remediated with patched transitive versions. Closing CI run 33628767467 passed every repository job, including CASA, GitGuardian, and aggregate build-and-test, on head 0acd16c3.

High for observed commands

Continue with the failures recorded

Added maintenance burden

No template/helper maintenance

889 lines across the helper, test and skill, plus a 43-line canonical testing section, concise templates and drafts; repeated-use burden is unknown

High for line count, low for ongoing effort

Continue narrowly

Privacy/security risk

Evidence practices were task-specific

Loopback/read-only request policy, fresh context, metadata redaction, ignored output and manual pixel review are explicit; deny-list sanitisation and screenshot pixels remain residual risks

Medium

Continue only with human inspection

Vendor/tooling cost

Existing GitHub, CodeRabbit, PNPM, and Playwright stack

No direct dependency or paid service added; two transitive packages were pinned to patched versions through existing PNPM overrides

High

Continue

Outcome	Exact repository action
Adopt unchanged	Retain `.github/ISSUE_TEMPLATE/bug_report.yml`, `.github/ISSUE_TEMPLATE/feature_or_workflow.yml`, `.github/pull_request_template.md`, `.gitignore`, `package.json`, `docs/ai-skills/INDEX.md`, `docs/ai-skills/16-evidence-backed-delivery-pilot.md`, `scripts/pr-evidence/capture.ts`, `scripts/pr-evidence/cli.ts`, `scripts/pr-evidence/core.ts`, `scripts/pr-evidence/git.ts`, `scripts/pr-evidence/smoke.ts`, `scripts/pr-evidence/core.test.ts`, `public/docs/TESTING.md`, `src/lib/articles.test.ts`, and this report. Retain the article/social files as drafts until separate editorial approval.
Adopt with revisions	Retain the same files while changes are reviewed. Update the templates, skill, helper, test and `public/docs/TESTING.md` together; remove only a superseded file after its replacement and migration notes are accepted. Keep this report as the original measurement record.
Continue the pilot	Retain the same repository files and keep `content/articles/drafts/testing-a-small-ai-software-factory.md` and `content/social/drafts/testing-a-small-ai-software-factory.md` unpublished. Retain local `.tmp/pr-evidence/<task>/` packages only as long as their private review is active, then delete them locally.
Reject and remove	Retain only this report as the decision record. Remove the three `.github` template files, `docs/ai-skills/16-evidence-backed-delivery-pilot.md`, all six implementation/test files under `scripts/pr-evidence/`, both draft content files, and local `.tmp/pr-evidence/`. Revert only the pilot additions in `.gitignore`, `package.json`, `docs/ai-skills/INDEX.md`, `public/docs/TESTING.md`, and `src/lib/articles.test.ts`.

SOFTWARE FACTORY PILOT 2026 09 02

Evidence-backed software factory pilot

Hypothesis

Inputs and interpretation

Existing StratoFusion controls

Recent pull-request baseline

Baseline observations

Proposed pilot slice

Pre-implementation go/no-go checklist

Explicit exclusions

Pilot results

Verification results

Evaluation

Recommendation