# Self-hosting and operations

This chapter is for operators of an authorized CherrySub source checkout. It describes the implemented processes and operational contracts. The public staging installation is not a template you can treat as a production high-availability guarantee.

## Processes and boundaries

| Process/component | Role |
| --- | --- |
| Core server (`cmd/server`) | Receipt/store ingress, projections, runtime/admin APIs, webhook worker, reconciliation |
| Web control plane (`cmd/web-control-plane`) | Hosted accounts, projects, dashboard queries, credential management, export worker |
| Web application | Next.js dashboard, public site, and documentation |
| PostgreSQL | Immutable ledger, mutable projections, memberships, sessions, durable jobs/outbox |
| Email/report gateway | Fixed-template account email and private export object/download handling |
| Reverse proxy | HTTPS termination, private origin routing, protected observability |
| Optional sandbox broker | Development bridge for checked-in sandbox applications; not your production authentication system |
| CLI | Runs on the operator/developer's computer; uses hosted APIs and local store sign-in |

Core and web control plane share the intended database and compatible encryption-key configuration. Store/receipt verification happens before the transaction that appends and reduces evidence. Webhook network delivery does not hold a database transaction open.

## Local development

From the backend checkout:

```sh
cp .env.example .env
docker compose up --build
```

Review the example before changing it; it contains development defaults. Change `POSTGRES_PORT` and the host `DATABASE_URL` together if the host port conflicts, or `APP_PORT` for the application port. Container-to-container addresses are separate.

For host commands, load your intended development environment, then run `make migrate`, `make dev`, `make test`, and `make lint`. `go run ./cmd/server` exposes its serve/migrate/replay/reconcile commands. The web control plane is a separate process, typically on port 8082; the core defaults to 8080. With the intended development database/environment loaded, start the hosted process in another terminal:

```sh
HTTP_ADDR=:8082 WEB_COOKIE_SECURE=false go run ./cmd/web-control-plane
```

The insecure-cookie override is for local HTTP development only. Hosted verification/invitation/reset flows still need the configured email gateway; running locally does not disable email verification or store verification.

In the web checkout, use Node 24, `npm ci`, and `API_URL` pointing at the hosted control-plane API origin that exposes `/auth` and `/dashboard`. Start with `npm run dev`. The browser also needs the intended publicly reachable runtime API for its SDK integration; these roles must not be confused.

## Core configuration reference

| Variable | Purpose / default |
| --- | --- |
| `DATABASE_URL` | Required intended PostgreSQL connection |
| `HTTP_ADDR` | Listen address; core defaults to `:8080` |
| `LOG_LEVEL`, `LOG_FORMAT` | `info` and `json` defaults; text supported for development |
| `RECEIPT_RATE_LIMIT_RPS`, `RECEIPT_RATE_LIMIT_BURST` | Per-key, process-local receipt throttle; defaults 100/200 |
| `GOOGLE_PLAY_API_ENABLED` | Enables environment-backed Play reads independently of RTDN |
| `GOOGLE_APPLICATION_CREDENTIALS` | Application Default Credentials file path for environment-backed Google access |
| `GOOGLE_PUBSUB_AUDIENCE`, `GOOGLE_PUBSUB_SERVICE_ACCOUNT` | Exact authenticated push audience and identity; configured together |
| `APPLE_ISSUER_ID`, `APPLE_KEY_ID`, `APPLE_PRIVATE_KEY_B64` | Complete environment-backed Apple server-key tuple; all or none |
| `STORE_CREDENTIAL_MASTER_KEYS` | Comma-separated base64 32-byte encryption keys for managed app credentials |
| `ADMIN_API_KEYS` | Operator root bearer-key ring enabling `/admin/v1` |
| `WEBHOOK_MASTER_KEYS` | Base64 32-byte encryption-key ring for webhook signing secrets |
| `WEBHOOK_WORKERS` | Delivery concurrency; default four in the core configuration |
| `RECONCILIATION_INTERVAL` | `0` disables scheduled recovery; otherwise 1 minute–24 hours |
| `RECONCILIATION_LOOKBACK` | First-run lookback, default 24 hours; bounded 1 minute–30 days |
| `RECONCILIATION_APPLE_ENVIRONMENTS` | `production,sandbox` by default; choose the intended environments |

Managed store credentials can supply per-app reads without relying on one deployment-wide store key. The store encryption ring currently falls back to `WEBHOOK_MASTER_KEYS` when its dedicated variable is empty; configure dedicated compatible rings deliberately rather than assuming unrelated processes can decrypt each other's records.

## Hosted web and email/report configuration

The web control plane needs `DATABASE_URL`, its `HTTP_ADDR`, and the relevant store/webhook master keys. `WEB_SESSION_TTL` defaults to seven days and accepts one minute through 30 days. Keep `WEB_COOKIE_SECURE` enabled behind production-shaped HTTPS. `EXPORT_WORKERS` defaults to one and is bounded from one through eight. The hosted process accepts `SHUTDOWN_WINDOW` from one second to one minute (default ten seconds); the core uses its ten-second configured shutdown default.

Configure `EMAIL_GATEWAY_URL`, `EMAIL_GATEWAY_SECRET`, and `PUBLIC_WEB_URL` together. The gateway secret must match its deployment; the public web origin controls allowed account-action destinations. Without delivery configuration, email-dependent flows/exports are unavailable rather than pretending mail was sent.

The checked-in email/report gateway is a Cloudflare Worker with an email-sending binding and a private R2 `REPORTS` binding. Its public action/report origins and allowed sender are explicit configuration. Provision the appropriate sender and private bucket, store the shared secret in the platform secret store, and run that package's `npm ci` / `npm run check` before the approved environment's deployment. Its checked-in production environment is not configured.

After a gateway rollout, check health, unauthenticated request rejection, one authorized test-account email, and a bounded test report upload/download. Export links expire after 24 hours; private bucket lifecycle cleanup is a backstop, not the primary link authorization mechanism.

## Deploy the origin

Use the staging Compose/Caddy templates as a topology reference: expose HTTPS through the proxy, keep PostgreSQL and application ports private, run migrations deliberately, and wait for readiness before routing traffic. Immutable image tags, versioned configuration, and backups make rollback concrete.

Before a release, back up the database, record actual image references/configuration, retain required encryption keys, run formatting/tests/static analysis, validate Compose/proxy configuration, and check readiness on the new version. Production requires its own reviewed blue/green or equivalent rollout, compatible expand/contract migrations, origin protection, off-site recovery, and availability targets. Two processes on one host do not protect against host failure.

The production topology document records a preferred direction; it does not mean a multi-origin deployment is currently live. Never infer authorization to mutate another environment from the word staging in a command example.

## Reconciliation and missed observations

Run a bounded recovery pass with the core server command:

```sh
go run ./cmd/server reconcile --app-id APP_UUID
```

Scheduled recovery uses `RECONCILIATION_INTERVAL`. Successful streams retain cursors and intentionally overlap five minutes. Apple recovers signed notification history for selected environments. Google recovers void/refund evidence and bounded order-accounting enrichment. Each stream advances only after its full page set succeeds; one app/store failure does not stop every stream.

A PostgreSQL advisory lock prevents concurrent ownership of the same reconciliation run across replicas. Recovery does not bypass verification, invent another provider's history, or treat Google order-accounting data as an independent access decision.

## Replay and projection repair

Start with read-only verification:

```sh
go run ./cmd/server replay --verify --app-id APP_UUID
```

Replay rebuilds projections in an unlogged shadow schema, checks drift, and compares derived events without replacing live tables. It uses a global advisory lock and brief safe ledger-boundary locks; do not launch competing jobs into the fixed shadow schema.

Only after reviewing the report and establishing a recoverable backup should an operator deliberately use `replay --apply`. Apply performs the final compared projection replacement; it is not an event-history edit. Keep the original source ledger and mapping-version rules intact.

## Health, metrics, and response

Core `/livez` reports process liveness. `/readyz` and `/healthz` check database readiness. The web control plane has its own `/healthz`. `/metrics` is for protected internal scraping and is blocked by the staging public proxy.

Monitor ingest acceptance/deduplication/rejection/failure, receipt throttling, webhook attempts/duration/disabled endpoints, reconciliation runs/records/duration, export job failures, and database health. Labels are bounded; avoid placing customer tokens or private store payloads in logs/metrics.

For an incident, preserve evidence, identify the failing dependency, stop uncontrolled retry amplification, and restore the last known-good routing/configuration when needed. Test database and key restoration rather than relying only on a successful backup command. [Troubleshooting](/docs/troubleshooting) · [Security](/docs/security).