# Local Automation Utility Layer v1 Status: draft Owner: Platform Operations Fairway task: `OPS-LOCAL-AUTOMATION-INVENTORY-001` Last updated: 2026-06-08 ## Purpose GPUaaS now has enough CI, deploy, UAT, branch-closeout, and evidence scripts that long-running agent loops should move into deterministic local utilities. The goal is not to replace Codex, Claude, Fairway, or human review. The goal is to move repeated polling, log collection, classification, evidence formatting, and closeout checks into scripts that produce stable artifacts. Authoritative results must come from deterministic utilities and environment state. Optional local LLM summarization can improve human-readable handoffs, but it must never decide pass/fail, deploy readiness, review approval, Fairway task status, branch deletion, or release readiness. ## Current Baseline As of this inventory: - `master` and `release/platform-control` are aligned across the configured remotes at `4b7edc37`. - Branch cleanup has reduced remote branch backlog; remote `codex/*` branches are cleared. - kind and dev-control `gpuaas-core` deployments are rolled out and current pods are Running/ready. - The dev registry ingress returns the expected Docker Registry `401` on `/v2/`. - Fairway active reconciliation is clean. This is a good point to formalize local automation because the current deploy baseline is known-good and the remaining work is workflow hardening, not feature delivery. ## Design Principles 1. Deterministic first. Utilities emit JSON and stable markdown/text artifacts from command output, API responses, file trees, or Kubernetes state. 2. Fairway aware, not Fairway hidden. Utilities should print or execute explicit Fairway evidence/checkpoint commands. They must not silently mark tasks done, approve reviews, delete branches, or release software. 3. Observe by default. Monitoring and summary utilities are read-only by default. Mutating modes need an explicit flag, documented scope, and reviewable evidence. 4. One utility, one responsibility. CI monitoring, deploy monitoring, UAT summarization, evidence packets, gate selection, closeout, ops smoke, and release verification are separate utilities with shared output conventions. 5. LLM summaries are optional. Ollama/Gemma/Qwen summaries may be generated from deterministic artifacts after validation. Missing or invalid LLM output falls back to deterministic markdown. 6. Sanitize before storage. Utility artifacts, local prompts, and generated packets must not store secrets, tenant-sensitive detail, raw credentials, or broad private URLs. Optional local LLM summaries consume deterministic sanitized packets, not raw logs or unbounded provider transcripts. ## Existing Automation Inventory | Lifecycle phase | Existing scripts and targets | Current inputs | Current outputs | Gap | |---|---|---|---|---| | CI pipeline trigger/watch | `scripts/ci/gitlab_pipeline_trigger.sh`, `scripts/ci/gitlab_pipeline_watch.sh`, `scripts/ops/local_ci_monitor.sh`, `scripts/ops/local_cicd_handoff_packet.sh`, Fairway `examples/session-adapters/ci-monitor.sh` | pipeline id, branch, SHA, GitLab env, handoff/handback fields | pipeline status, Fairway checkpoints/evidence through adapter, local CI monitor packet, CI/CD handoff or handback packet | GPUaaS wrappers now exist; next hardening is direct Fairway handback integration. | | CI failure classification | `scripts/ci/platform_control_ci_failure_classifier.sh`, `scripts/ci/platform_control_pipeline_history.sh`, `scripts/ci/platform_control_pipeline_duration_budget.sh` | pipeline/job JSON, recent pipeline history | classification JSON, duration/budget reports | Need one local utility that fetches job metadata/log snippets safely and produces `CI-FIX-*` suggestions. | | Deploy trigger/promotion | `scripts/ci/platform_control_promote_release_branch.sh`, `scripts/ci/dev_control_rke2_release_runtime_fast.sh`, `scripts/ci/platform_control_release_*`, `scripts/ci/platform_control_deploy_release_candidate.sh` | source ref, release profile, env profile | release candidate env, GitLab pipeline, deploy artifacts | Trigger path exists; need monitor utility to verify rollout state after pipeline pass. | | Deploy execution and rollout checks | `scripts/ci/platform_control_deploy.sh`, `scripts/ci/platform_control_remote_validation_local_dev.sh`, `scripts/ci/platform_control_remote_validation_*`, `scripts/ci/platform_control_remote_preflight.sh` | release candidate env, SSH host, kubeconfig, API URLs | deploy logs, remote validation output | Need deterministic deploy monitor that checks image freshness, rollout status, pod readiness, service health, ingress/proxy health, and produces a deploy evidence packet. | | QA and gate selection | `make verify-web`, `scripts/ci/frontend_e2e.sh`, `scripts/ci/sdk_codegen_smoke.sh`, `scripts/ci/ci_script_smoke.sh`, `scripts/ci/platform_control_change_aware_preflight.sh` | changed files, branch/range, env vars | test output, generated artifact checks, suggested release profile | Need gate selector that maps changed paths/task type to required local gates and explains rationale. | | UAT run wrappers | `scripts/ops/demo_uat_package.sh`, `scripts/ops/demo_uat_mutating.sh`, `scripts/ops/demo_scheduler_uat.sh`, app/browser smoke scripts, `scripts/ops/uat_harness_failure_injection.sh` | environment, UAT mode, artifact root, credentials | UAT run directories, summaries, logs | Need UAT summarizer that normalizes scenario status and classifies failures before broad reruns. | | UAT readiness gates | `scripts/ci/pre_uat_readiness_gate.sh`, `scripts/ops/edge_error_presentation_smoke.sh`, `scripts/ops/app_artifact_smoke.sh`, app runtime smoke scripts | readiness artifact directories, endpoints, app manifests | readiness JSON/markdown, smoke results | Need combined pre-UAT gate packet that says which missing prerequisite blocks UAT and which task owns it. | | Evidence payloads | `scripts/ci/platform_evidence_payload.sh`, `scripts/ci/platform_evidence_gate.sh`, `scripts/ci/platform_evidence_submit.sh`, `scripts/ci/ops_evidence_gate.sh` | CI, UAT, security, deploy, supply-chain artifacts | evidence bundle JSON, gate output, optional API submit | Need task-level evidence packet generator for Fairway evidence, reviews, deploy-runs, UAT runs, and closeout. | | Branch/worktree closeout | Git commands, Fairway reconcile, existing cleanup reports under `tmp-ux/` | branch refs, worktree list, Fairway task state, CI state | markdown/CSV reports, manual deletion decisions | Need dry-run closeout utility that classifies branches/worktrees and refuses unsafe deletion by default. | | Fairway coordination | `fairway session`, `fairway checkpoint`, `fairway record evidence`, `fairway reconcile active`, provider-event adapter, CI monitor adapter | task id, role, provider/session id, artifact paths | sessions, checkpoints, evidence, reconcile findings | Need helper that assembles closeout commands and verifies no stale active work remains. | | Ops endpoint smoke | `scripts/ops/observability_smoke.sh`, `scripts/ops/observability_stack_smoke.sh`, `scripts/ops/proxy_authz_api_bearer_load_smoke.sh`, `scripts/ops/smoke_app_proxy.sh`, `scripts/ops/database_operations_readiness.sh`, `scripts/ops/backup_restore_smoke.sh` | API URLs, tokens, kube context, endpoint list | smoke results, readiness reports | Need API-first ops smoke wrapper with endpoint classification and direct-DB usage warnings. | | Release readiness verify | promotion policy, platform evidence gate, remote validation scripts, CI status, deploy-run evidence, Fairway review state | source SHA, release branch, deploy/UAT/evidence refs | release candidate env, gate output | Need release verifier that produces a single blocked/pass/needs-human-triage packet without deploying or approving release. | ## Gap Classification | Gap | Type | Why it matters | First utility task | |---|---|---|---| | CI monitors require custom poll scripts and agent attention for handback interpretation. | Deterministic utility gap | Repeated CI waits still consume agent cycles and can leave stale Fairway state. | `OPS-LOCAL-CI-MONITOR-UTILITY-001` | | Failed CI job logs are not consistently collected, sanitized, classified, and turned into scoped follow-up tasks. | Fairway ingestion gap | Retryable runner noise and deterministic failures can be mixed in chat. | `OPS-LOCAL-CI-MONITOR-UTILITY-001` | | UAT outputs are spread across directories and summaries; there is no single scenario matrix. | Deterministic utility gap | Full UAT becomes the first defect detector and triage loop. | `OPS-LOCAL-UAT-SUMMARY-UTILITY-001` | | Deploy pass does not automatically mean live rollout readiness is verified. | Deterministic utility gap | A pipeline can pass while stale pods, image drift, or endpoint regressions remain. | `OPS-LOCAL-DEPLOY-MONITOR-UTILITY-001` | | Fairway evidence is often hand-written after the fact. | Fairway ingestion gap | Evidence quality varies and task closeout takes manual effort. | `OPS-LOCAL-EVIDENCE-PACKET-UTILITY-001` | | Required local gates depend on human memory of AGENTS.md rules. | Process documentation gap | UX/e2e and contract/codegen gates can be missed until CI. | `OPS-LOCAL-GATE-SELECTOR-UTILITY-001` | | Branch/worktree cleanup needs manual join of Git, CI, Fairway, and review state. | Deterministic utility gap | Branch cleanup can either lag or become unsafe. | `OPS-LOCAL-CLOSEOUT-UTILITY-001` | | Ops smoke checks are script-specific and not normalized by endpoint class. | Deterministic utility gap | Repeated direct checks produce inconsistent evidence. | `OPS-LOCAL-OPS-SMOKE-UTILITY-001` | | Release readiness is distributed across CI, deploy, UAT, evidence, reviews, and branch state. | Deterministic utility gap | Promotion decisions can rely on incomplete chat memory. | `OPS-LOCAL-RELEASE-VERIFY-UTILITY-001` | | Local LLM summaries are not yet schema-validated or bounded. | Optional local LLM summarization gap | LLM output could be mistaken for authority if not constrained. | Add as secondary mode after deterministic outputs exist. | ## Standard Utility Output Contract Every utility should write into a caller-provided artifact root, defaulting to `.fairway/artifacts//`. Minimum files: ```text summary.json summary.md fairway-evidence.sh raw/ ``` Required JSON fields: ```json { "utility": "name", "version": "v1", "result": "pass|fail|blocked|partial|timeout|needs_human_triage", "source_sha": "optional git sha", "environment": "kind|dev|demo|local|ci|unknown", "task_id": "optional Fairway task id", "detected_by": "utility name or gate", "expected_gate": "gate name", "findings": [], "next_actions": [], "artifacts": [] } ``` The generated `fairway-evidence.sh` should be reviewable shell text. It may print Fairway commands by default and execute them only with an explicit `--record` or equivalent flag. ## Local Retention And Redaction Contract This contract applies to `.fairway/transcripts/`, `tmp-ux/`, generated context packets, generated evidence packets, and any prompt file or stdin payload fed to a local LLM. Permitted content: - task ids, lane ids, owner roles, review domains, risk levels, source paths, target paths, changed files, command names, command exit status, and commit SHAs; - deterministic utility outputs such as `summary.json`, `summary.md`, `packet.json`, `packet.md`, `review-packet.md`, and `fairway-evidence.sh`; - sanitized log excerpts and failure snippets that remove credentials, tenant-sensitive values, payment references, raw bearer material, cookies, and private key material; - environment aliases such as `kind`, `dev-control`, `demo`, `staging`, or `private-url-redacted` when a full private URL is not required for approved evidence. Prohibited content: - secrets, access tokens, refresh tokens, bearer headers, passwords, API keys, SSH private keys, kubeconfigs, registry credentials, service-account credentials, session cookies, Stripe signing secrets, certificate private material, or raw credential exchange payloads; - tenant names, customer names, customer-specific hostnames, tenant ids, user PII, billing/payment references, private incident details, or raw data rows that identify a tenant or customer; - raw production logs, raw database exports, raw webhook payloads, raw request/response bodies, raw provider transcripts, or broad terminal captures when they contain or may contain prohibited content; - private URLs in prompts or generated packets unless the URL is required to reproduce an approved ops check and contains no credential, token, tenant, or customer-identifying path/query material. Storage expectations: - `.fairway/transcripts/` is for local provider-session continuity only. Store concise progress, final answers, and sanitized excerpts; do not preserve raw secret-bearing chat, shell, browser, or terminal output. - `tmp-ux/` is scratch working memory. Store only short-lived sanitized notes, reports, screenshots, and intermediate summaries needed for active work. - generated context packets are prompt/input artifacts. They may include goal, scope, owned files, commands, acceptance criteria, stop conditions, and sanitized current state. They must not include raw logs or secret catalogs. - generated evidence packets are durable review inputs only when sanitized. Prefer references to approved artifacts over copying raw source material. - prompt files for local LLMs must be generated from sanitized packet JSON or Markdown, not from raw logs, raw transcripts, direct database dumps, secret inventories, or unbounded command output. Retention expectations: - keep transcripts, prompt drafts, and `tmp-ux/` scratch artifacts local-only and out of commits unless intentionally promoted as sanitized evidence; - delete or refresh scratch memory when the task closes, is handed off, or the information is superseded; - use 30 days after task closeout as the default maximum local retention for non-evidence transcripts, local memory, local prompts, and packet drafts; - retain sanitized evidence packets according to the Fairway task/evidence lifecycle, because they are review artifacts rather than scratch memory; - immediately redact or delete any artifact that contains prohibited content, then create a security/governance follow-up if the source utility needs a sanitizer or contract fix. Local LLM input rule: ```text raw logs/provider output -> deterministic parser/sanitizer -> packet.json|packet.md -> optional local LLM summary ``` The local LLM output may improve a handoff or human-readable summary. It must not decide pass/fail, deploy readiness, review approval, Fairway task status, branch deletion, release readiness, or security exceptions. If packet generation cannot prove the input is sanitized, skip the LLM step and use the deterministic Markdown summary. ## Utility Sequence The existing Fairway tasks are broadly correct. The inventory changes only one detail: evidence packet generation should move before deploy monitor implementation if CI monitor needs reusable evidence formatting. The practical order is: 1. `OPS-LOCAL-CI-MONITOR-UTILITY-001` Build the shared GitLab/Fairway handback wrapper first because every later batch uses CI. 2. `OPS-LOCAL-EVIDENCE-PACKET-UTILITY-001` Create the reusable packet format and Fairway command generator early. CI, deploy, UAT, and closeout utilities can consume it. 3. `OPS-LOCAL-UAT-SUMMARY-UTILITY-001` Normalize UAT outputs before the next broad UAT run. 4. `OPS-LOCAL-DEPLOY-MONITOR-UTILITY-001` Convert the manual dev-control rollout checks from the recent recovery into a reusable observer. 5. `OPS-LOCAL-GATE-SELECTOR-UTILITY-001` Prevent repeated missed local gates for frontend, contracts, codegen, schema, and deploy profiles. This utility is now implemented as `scripts/ops/local_gate_selector.sh`. 6. `OPS-LOCAL-CLOSEOUT-UTILITY-001` Apply the branch cleanup lessons to a dry-run-first Git/Fairway closeout helper. This utility is now implemented as `scripts/ops/local_closeout_report.sh`. 7. `OPS-LOCAL-OPS-SMOKE-UTILITY-001` Normalize API-first ops endpoint smoke checks. This utility is now implemented as `scripts/ops/local_ops_smoke.sh`. 8. `OPS-LOCAL-RELEASE-VERIFY-UTILITY-001` Compose the previous utilities into a release readiness verifier. This utility is now implemented as `scripts/ops/local_release_verify.sh`. Optional local LLM summarization should wait until at least the CI monitor, evidence packet, UAT summary, and deploy monitor emit stable JSON. ## Proposed Utility Boundaries ### CI Monitor Utility Inputs: - GitLab project id and pipeline id. - Source SHA, branch/ref, expected completion window. - Optional Fairway task id or deploy-run id. Outputs: - pipeline `summary.json`. - failed job list with sanitized trace excerpts. - retryability classification. - `ci_fix_packet` with failed job wrapper/envelope replay fields when a deterministic CI follow-up is needed. - Fairway evidence/checkpoint commands. Initial implementation: ```bash scripts/ops/local_ci_monitor.sh --pipeline-id \ --ref master \ --source-sha \ --task-id \ --artifact-root .fairway/artifacts/ ``` The same utility supports fixture mode with `--pipeline-json` and `--jobs-json` so CI smoke can validate classification without live GitLab access. Failed-job trace capture must be reproducible. The standard command is: ```bash scripts/ops/local_ci_monitor.sh --pipeline-id \ --trace-lines \ --task-id \ --artifact-root .fairway/artifacts/ ``` When a CI fix is opened, attach the monitor `summary.json` and require the embedded `ci_fix_packet` to include: - `prefix`, `detected_by`, `expected_gate`, owner, and artifact path; - pipeline id and failed job ids; - failed command wrapper/envelope, for example `backend_build_and_tests -> ci_script_smoke -> security_dast_report`; - inherited environment deltas relevant to the failure; - exact local replay command, or a concrete reason replay is impossible; - proof downstream smoke steps affected by wrapper changes still see required tools such as `go`; - sanitized failed-job trace artifact paths. Private URLs and secret-shaped values in copied pipeline/job JSON, trace artifacts, wrapper/envelope text, env deltas, and replay commands must be redacted before persistence. ### CI/CD Handoff Packet Utility Inputs: - Packet type: `handoff` or `handback`. - Fairway task id. - For handoff: deploy-run task id, pipeline/deploy reference, source SHA, target environment/profile, expected window, decision dependency, safe rerun commands, stop conditions, and forbidden actions. - For handback: terminal status, failed stage/job, evidence path, failure prefix, `detected_by`, `expected_gate`, attempted bounded fix, recommended owner, next action, and optional `ci_fix_packet`. Outputs: - `packet.json`. - `packet.md`. - `fairway-evidence.sh`. Initial implementation: ```bash scripts/ops/local_cicd_handoff_packet.sh --packet-type handoff \ --task-id \ --deploy-run-id \ --pipeline-ref \ --source-sha \ --target-environment \ --expected-window \ --decision-dependency \ --safe-rerun-command \ --stop-condition \ --forbidden-action \ --artifact-root .fairway/artifacts/ ``` ```bash scripts/ops/local_cicd_handoff_packet.sh --packet-type handback \ --task-id \ --run-status \ --failed-stage \ --failed-job \ --evidence-path \ --failure-prefix CI-FIX \ --detected-by ci \ --expected-gate security_scan \ --recommended-owner \ --next-action \ --ci-fix-packet \ --artifact-root .fairway/artifacts/ ``` The helper is deterministic review/evidence input only. It must not approve reviews, close findings, rerun CI, deploy, merge, push, or authorize release. It redacts secret-shaped values and private URLs before writing artifacts so the packet can be attached as Fairway evidence. Authoritative classifications: - `pass` - `fail` - `timeout` - `retryable_runner_noise` - `deterministic_failure` - `blocked` - `needs_human_triage` LLM allowed: - draft a concise human summary from collected sanitized traces. Artifact safety contract: - Local automation utilities must treat generated artifacts as attachable evidence and redact secret-shaped values before writing or before final closeout. - Generated `summary.json`, `summary.md`, `fairway-evidence.sh`, copied raw fixture/catalog inputs, trace excerpts, deploy monitor packets, ops smoke packets, and evidence packets must not contain bearer tokens, `Authorization` header values, passwords, API keys, private-key material, raw access/refresh/id tokens, or `token=` URL parameters. - Optional local LLM summaries may only consume deterministic sanitized packets. Raw logs, raw endpoint catalogs, and raw trace files are not valid LLM inputs unless they have passed the same artifact redaction boundary. - cannot override deterministic classification. ### UAT Summary Utility Inputs: - UAT artifact directory. - Environment and run id. - Optional scenario catalog. Outputs: - scenario matrix JSON. - Product Quality flow coverage rows with `flow_id`, `matrix_id`, required subpath status, owner, follow-up task, expiry, and reason. - failure classification and suggested follow-up task metadata. - invariant coverage summary. - Fairway evidence commands. Initial implementation: ```bash scripts/ops/local_uat_summary.sh \ --run-dir dist/uat/kind/ \ --task-id \ --artifact-root .fairway/artifacts/ ``` The utility prefers structured `results.jsonl` emitted by `scripts/ops/demo_uat_package.sh`, and falls back to parsing `summary.md` tables when only Markdown evidence is available. It writes: - `summary.json` - `summary.md` - `fairway-evidence.sh` - `follow-up-suggestions.json` Structured full-UAT inputs should include `matrix_id`, `flow_id`, and `flow_subpaths` for the Product Quality row. Required subpaths are `happy`, `empty`, `blocked`, `recovery`, `negative`, `cleanup`, `fixture`, and `environment`. Missing subpaths produce blocked flow coverage rows. Skipped subpaths must include owner, Fairway follow-up task, and expiry so UAT closeout does not hide approved gaps. Status mapping: - `PASS` -> `pass` - `FAIL` -> `fail` - `WARN` / `BLOCKED` -> `blocked` - `SKIP` / `INFO` -> `skipped` - malformed or unknown rows -> `unknown` Authoritative classifications: - `product-bug` - `harness-fix` - `environment-gap` - `deploy-fix` - `ci-fix` - `ops-fix` - `flaky-retry-candidate` - `needs-human-triage` Suggested follow-up prefixes: - `UAT-BUG-*` for product/workflow invariant failures - `HARNESS-FIX-*` for harness, sequencing, parser, or flaky retry candidates - `OPS-FIX-*` for provider, capacity, credential, observability, secret, or environment gaps - `CD-FIX-*` for rollout, image freshness, ingress, proxy, or route failures - `CI-FIX-*` for contract, codegen, or CI gate failures ### Gate Selector Utility Inputs: - one or more changed files; - a file containing changed paths; or - a Git diff range. Outputs: - required local validation gates; - rationale and matched paths for each gate; - Fairway evidence command template. Initial implementation: ```bash scripts/ops/local_gate_selector.sh \ --base origin/master \ --head HEAD \ --task-id \ --artifact-root .fairway/artifacts/ ``` The same utility supports fixture mode with `--changed-file` and `--changed-files-file`, so `scripts/ci/ci_script_smoke.sh` validates the rule matrix without needing live Git state. Authoritative gate mappings: - `packages/web/**`, UX docs, portal docs -> `make verify-web` and `bash scripts/ci/frontend_e2e.sh`. - `doc/api/**`, generated clients, API route contracts, and frontend API client surfaces -> `CODEGEN_ENFORCE_CLEAN=1 bash scripts/ci/sdk_codegen_smoke.sh`. - `scripts/ci/**`, `scripts/ops/**`, and `.gitlab-ci.yml` -> `bash scripts/ci/ci_script_smoke.sh`. - schema, seed, and migration inputs -> `bash scripts/ci/platform_control_schema_drift_guard.sh`. - release/deploy/profile paths -> platform-control fast preflight, with post-push deploy monitor evidence after deployment. - Fairway operating-model or queue paths -> Fairway active reconciliation. - security, IAM, audit, secrets, PKI, or policy surfaces -> independent security review. The selector is advisory but deterministic. It does not run the gates, approve scope narrowing, mark tasks done, deploy, or decide release readiness. If a frontend e2e run is narrowed with `E2E_SPEC=`, task evidence still needs to record why the focused spec fully covers the touched journey. ### Closeout Report Utility Inputs: - local and remote branch inventory; - worktree inventory and dirty state; - optional Fairway task/review state; - optional CI status by branch; - optional push-intent evidence by branch. Outputs: - branch and worktree closeout disposition JSON; - markdown cleanup report; - Fairway evidence command template. Initial implementation: ```bash scripts/ops/local_closeout_report.sh \ --base-ref master \ --task-id \ --artifact-root .fairway/artifacts/ ``` The same utility supports fixture mode with `--branches-file`, `--worktrees-file`, `--fairway-tasks-file`, `--ci-status-file`, and `--push-intent-file`, so CI smoke can validate classification without mutating real branches or worktrees. Authoritative dispositions: - `safe_to_delete` when a branch is merged to the base, clean, CI/review state is acceptable, and remote push intent is present. - `preserve_for_review` when Fairway work is still active, blocked, or review-owned. - `preserve_unmerged` when a branch tip is not an ancestor of the integration base. - `dirty` when a checked-out worktree has uncommitted changes. - `missing_ci` when CI evidence is present but not passing. - `missing_review` when Fairway review state is not approved or explicitly not required. - `missing_push_intent` when a remote branch exists without closeout push-intent evidence. - `preserve_protected` for integration, release, recovery, and protected branches. - `needs-human-triage` when the utility cannot classify safely. The utility is dry-run only in the initial implementation. It does not delete remote branches, remove worktrees, prune remotes, push, merge, approve reviews, or change Fairway state. Cleanup execution remains a separate reviewed task. Reviewer merge-lane usage: - implementation lanes should keep work local until a coherent reviewed batch is ready; - if a task branch must be pushed, record push-intent evidence and expected closeout path; - reviewers should use named review branches or reviewed integration batches rather than detached `FETCH_HEAD` review; - after a batch is promoted to `master` and CI passes, run this report before any branch deletion pass; - remote branch deletion is allowed only after the report and reviewed cleanup task show merge ancestry, passing CI, acceptable Fairway review state, no dirty worktree, and explicit push-intent/cleanup approval. ### Ops Smoke Utility Inputs: - endpoint catalog JSON; - optional fixture result JSON; - optional per-endpoint headers supplied by the caller. Outputs: - endpoint classification JSON; - markdown summary; - Fairway evidence command template; - follow-up suggestions. Initial implementation: ```bash scripts/ops/local_ops_smoke.sh \ --endpoints-file doc/operations/.json \ --task-id \ --artifact-root .fairway/artifacts/ ``` Endpoint catalog shape: ```json { "endpoints": [ { "id": "api-health", "url": "https://aicloud-dev-api.core42.dev/healthz", "surface": "public", "expected_status": [200] } ] } ``` Surface classes: - `public`: externally reachable without operator identity. - `ops-only`: operator-authenticated control or observability surface. - `sre-only`: elevated operational surface for incident/debug access. - `internal-only`: cluster or service-internal surface; external exposure is a finding. Authoritative classifications: - `pass` - `route_missing` - `auth_policy_issue` - `service_down` - `upstream_unhealthy` - `tls_or_dns_issue` - `proxy_issue` - `needs-human-triage` The utility treats `401`/`403` as pass for protected `ops-only`, `sre-only`, and `internal-only` surfaces unless the endpoint catalog expects a different authenticated status. It uses HTTP/operator surfaces only. Repeated direct-DB inspection is explicitly out of scope; if an operational check needs SQL more than once, add an API/operator read model and track the missing surface. ### Deploy Monitor Utility Inputs: - environment, namespace, deployment set, source SHA, image host, release profile. - optional SSH/kube context and endpoint list. Outputs: - image freshness report. - rollout status report. - pod readiness report. - ingress/proxy/service health report. - Fairway deploy-run evidence commands. Initial implementation: ```bash scripts/ops/local_deploy_monitor.sh \ --environment dev-control-rke2 \ --namespace gpuaas-core \ --deployment gpuaas-api \ --expected-image aicloud-dev-registry.core42.dev/gpuaas-api@sha256: \ --source-sha \ --smoke-endpoint https://aicloud-dev-api.core42.dev/healthz \ --task-id \ --artifact-root .fairway/artifacts/ ``` The utility is observe-only. It may call `kubectl get` and `curl` in live mode, or read fixture JSON through `--deployments-json`, `--pods-json`, `--services-json`, `--ingress-json`, and `--endpoints-json`. It does not apply manifests, restart deployments, mutate secrets, promote releases, or change traffic. It writes: - `summary.json` - `summary.md` - `fairway-evidence.sh` - `raw/deployments.json` - `raw/pods.json` - `raw/services.json` - `raw/ingress.json` - `raw/endpoints.json` Authoritative findings: - `image_not_fresh` - `rollout_stuck` - `secret_missing` - `migration_failed` - `ingress_or_proxy_issue` - `service_crash` - `dependency_unhealthy` - `pass` - `timeout` - `needs_human_triage` ### Evidence Packet Utility Inputs: - task id, command result, artifacts, commit SHA, CI/deploy/UAT refs, reviewer notes. Outputs: - markdown packet. - JSON packet. - `fairway record evidence` command. - optional review packet and follow-up task suggestions. Initial implementation: ```bash scripts/ops/local_evidence_packet.sh \ --task-id \ --packet-type ci-failure \ --result fail \ --command-text "bash scripts/ci/sdk_codegen_smoke.sh" \ --artifact .fairway/artifacts//summary.json \ --commit-sha \ --ci-ref \ --detected-by local-ci-monitor \ --expected-gate sdk-codegen \ --artifact-root .fairway/artifacts/ ``` Supported packet types: - `ci-failure` - `deploy-run` - `uat-run` - `review` - `release-readiness` - `task-evidence` The utility writes: - `packet.json` - `packet.md` - `fairway-evidence.sh` - `review-packet.md` - `follow-up-suggestions.json` Example deploy packet: ```bash scripts/ops/local_evidence_packet.sh \ --task-id DEPLOY-CI-RUN-20260608-EXAMPLE \ --packet-type deploy-run \ --result pass \ --command-text "scripts/ci/platform_control_remote_validation_local_dev.sh --phases health,authz" \ --artifact .fairway/artifacts/deploy-run/summary.json \ --deploy-ref dev-control-rke2 ``` Example UAT packet: ```bash scripts/ops/local_evidence_packet.sh \ --task-id PSSM-UAT-EXAMPLE \ --packet-type uat-run \ --result blocked \ --command-text "scripts/ops/demo_uat_package.sh --env kind" \ --artifact .fairway/artifacts/uat/summary.md \ --follow-up HARNESS-FIX-EXAMPLE-001 \ --uat-ref kind-read-only ``` Example review packet: ```bash scripts/ops/local_evidence_packet.sh \ --task-id REVIEW-EXAMPLE \ --packet-type review \ --result partial \ --command-text "manual review of evidence packet" \ --reviewer-notes "needs ops review before closeout" ``` Example deterministic security review packet: ```bash scripts/ops/local_evidence_packet.sh \ --task-id SEC-REVIEW-EXAMPLE \ --packet-type review \ --result partial \ --command-text "review packet for high-risk auth/runtime change" \ --changed-file cmd/api/routes_platform_iam.go \ --changed-file packages/web/src/app/auth/session.ts \ --changed-file cmd/node-agent/main.go \ --task-tag surface:local-automation \ --risk-level high \ --reviewer-domain ops \ --reviewer-notes "security-rule selection is deterministic; LLM summaries are advisory only" ``` Security-rule selection inputs: - `--changed-file` paths. - `--task-tag` Fairway tags. - `--risk-level` as `low`, `medium`, `high`, or `critical`. Security-rule selection outputs: - `security_rule_selection` inside `packet.json`. - `security-rule-selection.json`. - a `Security Rule Selection` section in `packet.md`. - a `Security Rule Selection` section in `review-packet.md`. The selected rule families are deterministic. Optional local LLM summaries may summarize the packet but cannot approve, waive, route, close, or change any finding. Example path-to-rule outcomes: | Change type | Example input | Selected rule family | Expected review domains | |---|---|---|---| | API route | `--changed-file cmd/api/routes_platform_iam.go` | `api-web-services`, `authorization-access-control`, `input-validation-injection` | security, architecture, backend | | Frontend auth/session | `--changed-file packages/web/src/app/auth/session.ts` | `authentication-session-management`, `client-side-web-security` | security, backend, architecture, frontend | | Node-agent/runtime | `--changed-file cmd/node-agent/main.go` | `input-validation-injection`, `crypto-certificates`, `xml-serialization-file-handling` | security, backend, ops, architecture | | Kubernetes/edge | `--changed-file infra/k8s/overlays/dev-control/kustomization.yaml` | `cloud-kubernetes-cicd-containers-iac`, `hardcoded-credentials` | security, ops, architecture | | Supply-chain | `--changed-file pnpm-lock.yaml` | `supply-chain-security` | security, ops, governance | | Ops utility | `--changed-file scripts/ops/local_evidence_packet.sh` | `input-validation-injection`, `cloud-kubernetes-cicd-containers-iac`, `data-storage-privacy-logging`, `hardcoded-credentials`, `mcp-tooling-security`, `xml-serialization-file-handling` | security, ops, governance | For `high` or `critical` risk changes with any selected security rule family, the packet also requires `security`, `ops`, and `governance` review domains. Example release-readiness packet: ```bash scripts/ops/local_evidence_packet.sh \ --task-id RELEASE-READINESS-EXAMPLE \ --packet-type release-readiness \ --result pass \ --command-text "scripts/ci/platform_foundation_l2_promotion_gate.sh" \ --artifact .fairway/artifacts/release-readiness/summary.json \ --commit-sha ``` Must not: - approve reviews. - mark merge-ready. - close tasks. - mutate branches or release state. ### Gate Selector Utility Inputs: - changed-file list, branch/range, optional Fairway task id. Outputs: - required commands. - rationale mapped to AGENTS.md gates. - release profile suggestion when applicable. - evidence template. Required deterministic rules: - web UX/API consumption changes require frontend gates. - OpenAPI/AsyncAPI/generated artifacts require codegen clean gates. - deploy/profile/script changes require `ci_script_smoke.sh` and release profile validation. - schema changes require schema drift/migration gates. ### Closeout Utility Inputs: - local worktree list, branch refs, remotes, Fairway task/review state, CI state. Outputs: - branch/worktree closeout report. - deletion candidates and blockers. - Fairway closeout/evidence command set. Default mode: - dry-run only. Destructive mode: - must require explicit apply flag and satisfy merge, CI, review, task, and push-intent checks. ### Ops Smoke Utility Inputs: - environment profile, endpoint catalog, credentials/tokens by env var, optional kube context. Outputs: - endpoint status JSON and markdown. - classification by service and access class. - Fairway evidence command. Rules: - prefer APIs/operator surfaces. - flag repeated direct SQL as missing operator surface. ### Release Verify Utility Inputs: - source SHA, release branch, CI/deploy/UAT evidence refs, Fairway task/review ids. Outputs: - release readiness packet. - blocker list. - evidence refs. - pass/blocked/needs-human-triage result. Initial implementation: ```bash scripts/ops/local_release_verify.sh \ --source-sha \ --release-branch release/platform-control \ --ci-file .fairway/artifacts//summary.json \ --deploy-file .fairway/artifacts//summary.json \ --uat-file .fairway/artifacts//summary.json \ --evidence-file .fairway/artifacts//summary.json \ --reviews-file .fairway/artifacts//reviews.json \ --release-notes doc/operations/releases/.md \ --changelog CHANGELOG.md \ --artifact dist/ ``` The verifier performs live git checks by default for clean worktree state and remote containment of the source SHA. CI, deploy, UAT, evidence, review, release notes, changelog, and artifact inputs are explicit evidence files so operators can run the verifier before platform-control promotion, dev deploy, or production release gates without relying on chat memory. The utility supports fixture files for CI smoke and dry-run review: `--worktree-status-file`, `--refs-file`, `--ci-file`, `--deploy-file`, `--uat-file`, `--evidence-file`, and `--reviews-file`. Authoritative results: - `pass`: all deterministic release readiness checks passed. - `blocked`: required evidence is missing or explicitly non-pass. - `needs-human-triage`: input is malformed or optional attestation evidence is inconsistent. Rules: - signing or attestation evidence is required only when `--require-attestation` is provided; otherwise optional attestation paths are reported but do not block release readiness. - the verifier produces `release-run-packet.json` for Fairway evidence. - the verifier never tags, pushes, deploys, promotes release branches, approves reviews, or changes Fairway state. Must not: - tag. - push. - deploy. - approve release readiness. ## Follow-Up Task Mapping The active Fairway queue already contains the first implementation tasks: - `OPS-LOCAL-CI-MONITOR-UTILITY-001` - `OPS-LOCAL-UAT-SUMMARY-UTILITY-001` - `OPS-LOCAL-DEPLOY-MONITOR-UTILITY-001` - `OPS-LOCAL-EVIDENCE-PACKET-UTILITY-001` - `OPS-LOCAL-GATE-SELECTOR-UTILITY-001` - `OPS-LOCAL-CLOSEOUT-UTILITY-001` - `OPS-LOCAL-OPS-SMOKE-UTILITY-001` - `OPS-LOCAL-RELEASE-VERIFY-UTILITY-001` Recommended adjustment from this inventory: - Move `OPS-LOCAL-EVIDENCE-PACKET-UTILITY-001` immediately after `OPS-LOCAL-CI-MONITOR-UTILITY-001`, because CI, deploy, UAT, and closeout utilities all need a common evidence packet contract. No additional implementation tasks are required before starting `OPS-LOCAL-CI-MONITOR-UTILITY-001`.