Cloud Run · Publish Pipeline — Plumbing Complete, GitHub PAT Pending

DARE.CO.UK · INFRASTRUCTURE · 12 MAY 2026

Runbook + operational chronology of today’s containerisation of the dev-reports publish pipeline. Five phases shipped; first end-to-end execution gated on a 5-minute GitHub PAT mint. Generated immediately after the canary execution validated the rest of the chain.

TL;DR

• The dev-reports publish pipeline is now containerised end-to-end. Cloud Run Job devreports-publish exists in GCP project dare-devreports, with image us-central1-docker.pkg.dev/dare-devreports/devreports/publish:latest (490MB) built from xlab-co/toolkit/cloud-run/Dockerfile.
• The local launchd cron stays untouched. All five phases are additive — defaults in dare_dev_reports_publish.py reproduce the historical ~/Downloads layout exactly, so today’s macOS publish stack keeps working. Cloud Run is the parallel-run candidate, not a replacement (yet).
• Canary executed at 18:35Z and failed at the expected guard — GIT_TOKEN env var is required (Secret Manager binding) — proving image boots, secret binding resolves, entrypoint runs, and the failure mode is exactly the one telegraphed in entrypoint.sh. Everything from container provisioning through CDN, security layer, and DNS provider sitting in front of dare.co.uk.">CF-token secret resolution is working.
• Pending: 5-minute GitHub PAT mint for xlab-co/devreports-content read access, stored in Secret Manager as devreports-git-pat. Once that’s in, gcloud run jobs execute devreports-publish ships a real deploy. See action plan below.

What was plumbed in

Five phases, each independently shippable. Chronology of the session:

Phase 1 — Toolkit Linux portability

Two surgical ports in xlab-co/toolkit (commit f08566b):

• thumbnailer.py — new _resize_to_jpeg() helper. Prefers Pillow when importable (Linux containers always; macOS if installed); falls back to sips on macOS without Pillow. Linux without Pillow is an explicit error — container images install Pillow at build time. Closes the parked toolkit-README item; same code path on macOS and Linux once Pillow lands.
• dare_dev_reports_publish.py — three env-var-overridable paths:

python REPORTS_DIR = os.environ.get("REPORTS_DIR", ~/Downloads) STAGE_DIR = os.environ.get("STAGE_DIR", REPORTS_DIR/dev_reports_pages) THUMB_CACHE_DIR = os.environ.get("THUMB_CACHE_DIR", REPORTS_DIR/devreports_thumbs)

Defaults reproduce the historical layout exactly. Smoke-tested locally — 49 reports staged, 49 thumb cache hits, no breakage.

Phase 2 — Content repo

Created xlab-co/devreports-content (private). Migrated 56 markdown reports + .gitignore (excludes generated HTML / JPG / staging output) + README documenting filename conventions. This becomes the source-of-truth corpus the Cloud Run Job clones at runtime, decoupling content from the Mac filesystem.

Initial commit captures today’s catalog state. The local generator scripts (dare_audit.py, dare_session_report.py, dare_404_audit.py, narrator, etc.) still write to ~/Downloads/ today; a later session adds git add + commit + push steps to those scripts so generated reports flow through this repo automatically.

Phase 3 — GCP project + Secret Manager + Artifact Registry

Mirrors today’s audrey-experiments setup pattern:

Two gotchas surfaced and were resolved during setup:

1. Fresh-project IAM propagation delay — Artifact Registry create initially returned IAM_PERMISSION_DENIED even though dan@dare.co.uk has roles/owner. A 30s wait + retry succeeded. Standard GCP eventually-consistent behaviour; worth knowing.
2. Empty secret payload from a piped op read that silently failed auth — first op read | gcloud secrets create cf-pages-deploy --data-file=- created the secret but populated zero versions, because op had auth-timed-out and returned empty. Downstream Secret was not found errors at job creation time. Fix: explicit length check on the op-read output before piping to gcloud secrets versions add, plus a separate version-add step rather than create-with-version. Worth memorialising — silent empty pipes are exactly the kind of issue that’s easy to miss.

Phase 4 — Dockerfile + entrypoint + cloudbuild config

Three files in xlab-co/toolkit/cloud-run/ (commit 08e6141):

STASH44

The Dockerfile copies only the three toolkit modules the publish needs (seo_render_html.py, thumbnailer.py, dare_dev_reports_publish.py) rather than the full toolkit checkout — image stays small + the runtime surface is auditable.

The cloudbuild config has a small staging step that lays out the build context so the Dockerfile’s COPY toolkit/seo_render_html.py /app/ resolves cleanly. The toolkit repo’s flat root layout doesn’t match the Dockerfile’s toolkit/... path expectations; the staging step bridges that without restructuring the repo.

Phase 5 — First image build + Cloud Run Job + canary execution

STASH50

Build time: 3 min 28s. Image landed at us-central1-docker.pkg.dev/dare-devreports/devreports/publish:08e6141 + :latest (490MB). Cloud Run Job created:

STASH53

Canary executed at 18:35Z. Logs from Cloud Logging:

STASH54

This failure mode is the canary signal — every other layer of the chain works. The container provisioned, the image was pulled from Artifact Registry, the runtime SA resolved, the CLOUDFLARE_API_TOKEN secret bound from Secret Manager (otherwise we’d have failed before reaching entrypoint), the entrypoint ran, the config echoed correctly, and execution stopped at exactly the explicit guard for the one piece that’s still missing.

Architecture — end state

STASH56

Both paths converge on the same Cloudflare Pages project — once Cloud Run is validated, the cutover is just “disable the launchd plist” with no DNS or config changes.

Action plan — next session (≤ 1 hour for first successful run)

1. Mint a fine-grained GitHub PAT (5 min, interactive)

At https://github.com/settings/personal-access-tokens/new:

• Resource owner: xlab-co
• Repositories: Only select repositories → xlab-co/devreports-content
• Permissions: Repository permissions → Contents: Read-only
• Expiration: 1 year (or your preference)

Copy the PAT once issued. Per feedback_op_read_never_to_chat_stdout.md, never paste it into chat — straight from clipboard into the gcloud command below.

2. Store PAT in Secret Manager + grant access (3 min)

STASH60

3. Bind PAT to the Cloud Run Job (1 min)

STASH61

4. First end-to-end execution (2 min + ~3 min runtime)

STASH62

Expected output: git clone succeeds, dare_dev_reports_publish.py stages all reports, wrangler pages deploy ships to devreports.dare.co.uk. Verify via git log on the Cloudflare Pages project + a fresh load of the catalog page.

5. Compare output with the launchd cron (validation gate)

After the Cloud Run execution completes, compare the deployed catalog vs what launchd’s most-recent run produced. Equivalent content + identical thumbnail set = the migration is functionally complete.

Open items (Phase 6 and beyond)

• [ ] Cloud Scheduler trigger pointing at the Job (daily 06:00 UTC) — replaces the launchd plist’s schedule.
• [ ] Update local generator scripts to git add + commit + push to xlab-co/devreports-content after writing .md files, so reports flow into the cloud pipeline automatically.
• [ ] Parallel-run validation — run launchd + Cloud Run side-by-side for ~1 week, diff outputs daily.
• [ ] Decommission launchd — launchctl unload ~/Library/LaunchAgents/uk.co.dare.dev-reports.plist. Keep the plist file for ~30 days as rollback.
• [ ] Image size optimization (490MB → could probably hit ~300MB with multi-stage build pruning npm install artifacts; not urgent).
• [ ] Containerise dare_dashboard_refresh.sh next — same shape, different secrets. The dashboard cron has the same biometric drift problem.

Why this matters beyond today

Three categories of payoff:

Immediate operational wins:

• No more op run biometric prompts at cron-time — every Cloud Run execution reads secrets from Secret Manager bound at deploy. Eliminates the entire class of “biometric expired silently, cron no-op’d for hours” failure modes that have bitten the dashboard stack before.
• No more macOS TCC traps — Linux container has zero relationship to ~/Downloads, the macOS sandboxing model, or Touch ID at all.
• Identical behaviour across local dev, CI, production — the same dare_dev_reports_publish.py runs in three places now: Mac launchd (today), Cloud Run (today, pending PAT), eventually GitHub Actions if we want push-to-deploy.

Substrate compounding (per feedback_upstream_fix_downstream_compounding.md):

• The Pillow port closes the macOS-only dependency on thumbnailer.py — every future site using the toolkit can run on Linux now. Container is the forcing function; the port pays back wherever Linux is touched (GHA, Cloud Run, future containers).
• The env-var pattern (REPORTS_DIR, STAGE_DIR, THUMB_CACHE_DIR) generalises to every script that hardcodes ~/Downloads. Same one-line change shape; same backward-compat default.
• The Dockerfile + entrypoint + cloudbuild config is the template for every future cron job that should move to Cloud Run. Dashboard refresh, daily 404 audit, narrator generation, eventually audrey’s studio.* portal — all can copy this layout with surgical changes.
• The IAM gotchas resolved today (fresh-project propagation, empty-secret-from-failed-op-pipe, dual SA + service-agent grants) are now in the bloodstream. The next Cloud Run setup is materially faster.

Learning lab payoff (per user_portfolio_as_learning_lab.md):

• GCP fluency just leveled up. Today went from “we have a Vertex AI project” → “we operate full Cloud Run + Artifact Registry + Cloud Build + Secret Manager + IAM” in one session. Same vocabulary now applies for future client work.
• Container-native cron pattern memorised. The shape “git-tracked content → Cloud Run Job clones → runs publish → ships to CDN” is portable to every property’s content pipeline.

Portability

Same architecture applies anywhere the portfolio runs a content publish pipeline:

Linked memories

• feedback_upstream_fix_downstream_compounding.md — today’s containerisation is the canonical example of upstream-fix-downstream-benefit
• feedback_renderer_change_invalidates_artefacts.md — sibling lesson; containerisation is the upstream fix for the runtime-environment-changes class of the same trap
• feedback_op_read_never_to_chat_stdout.md — the silent-empty-pipe gotcha during Phase 3 is a close cousin; same instinct (verify what op returned before using it) applies
• feedback_just_in_time_permission_grants.md — applied to the GitHub PAT decision (fine-grained > broad gh CLI token, even for validation)
• project_gcp_audrey_foundation.md — the GCP foundation pattern that this work copies for the second project
• reference_gcp_cli_and_agency_iam.md — the agency-tenant IAM model that means dare-devreports slots cleanly under the dare.co.uk org alongside audrey-experiments
• feedback_thin_edge_canonical_source_uplift.md — same canonical-source-of-truth pattern, applied to operational pipelines (content repo IS the source of truth; Cloud Run is the thin proxy)
• feedback_operational_runbook_report_pattern.md — this report follows the shape

Cloud Run Job devreports-publish is provisioned, image built, secrets wired, canary validated. The 5-minute GitHub PAT mint is the only thing between here and a fully cloud-native publish pipeline.