15 Empty Backups, 3 Hidden Bugs, 2 Rotated Passwords: One Afternoon in a GitOps Homelab

15 Empty Backups, 3 Hidden Bugs, 2 Rotated Passwords

It started with four words — “fix this merge conflict” — and ended four PRs later having rescued a corrupt Postgres database, uncovered three Helm bugs hiding behind one another, solved a Grafana CSRF mystery caused by a single missing character, and moved a secret into OpenBao where it actually belonged.

1 · A merge conflict

The ask was tiny. The repo had other ideas.

PR #9 added a first-party Helm chart and had drifted from main. Two files conflicted — ct.yaml and a Helm-lint workflow — both trivially resolved by taking main’s superset. The interesting part was the push: a pre-push gitleaks hook scans full history and found a plaintext Postgres password baked into an old commit.

That decision — “the leaked password must be rotated” — is the thread that pulled everything else loose.

Deleting a secret from the working tree does nothing to a full-history scan — the string lives in the commit graph forever.

2 · A leaked password that was never live

The Guacamole chart shipped an empty secret. That was the first clue.

The chart referenced a SealedSecret with encryptedData: {} — a placeholder. In the cluster, guacamole-db-creds had zero keys, and postgres-0 was stuck in CreateContainerConfigError: it couldn’t find POSTGRES_DB in an empty secret. The infamous leaked password wasn’t live anywhere — the database it belonged to had never even started with it.

So this wasn’t a rotation, it was an initialisation. We generated a fresh password, sealed it (bound to name + namespace), applied it, and the secret finally had three real keys. Then we restarted Postgres — and it got far enough to reveal a much bigger problem.

3 · The database that couldn’t be saved

Postgres started, tried recovery, and died on a file that wasn’t there.

FATAL: could not open file "global/pg_filenode.map": No such file or directory

The data directory was corrupt: pg_filenode.map — the map from system catalogs to on-disk files — was missing, and half of global/ was 0-byte files (one dated Mar 24 1979). pg_resetwal can’t fix that. This was the pre-existing corruption the chart’s own comments had warned about; our secret fix simply let Postgres get far enough to hit it.

With the data unrecoverable and the backups hollow, the only path back to a working Guacamole was to wipe the corrupt volume and let Postgres re-initialise. Nothing recoverable was lost — it was already gone. So we wiped it. And that’s when the real bugs showed up.

4 · Three bugs hiding behind each other

Each fix revealed the next. Classic onion.

Bug #1 — the liveness probe killed initdb

On slow NFS, first-time initdb takes >75 s. The liveness probe (30 s delay, kills after ~75 s) shot Postgres mid-initialisation, leaving a half-written data dir that the entrypoint then refused to re-init. This is almost certainly what corrupted the volume in the first place. Fix: a startupProbe with a 300 s budget that gates liveness until Postgres first accepts connections.

startupProbe:
  exec: {command: ["pg_isready"]}
  failureThreshold: 60
  periodSeconds: 5   # 300 s budget before liveness gates in

Bug #2 — a service-link env collision

Guacamole came up but its PostgreSQL auth extension refused to start: Property "postgresql-port" must be an integer. Because the Service was named postgres, Kubernetes injected the legacy Docker-link variable POSTGRES_PORT=tcp://10.x:5432, which overrode the intended POSTGRESQL_PORT=5432. Fix: enableServiceLinks: false.

spec:
  enableServiceLinks: false   # stop injecting POSTGRES_PORT-style link vars

Bug #3 — a Service with no endpoints

Even then, Guacamole timed out reaching postgres:5432 — while the pod IP worked fine. The Service’s classic Endpoints object was empty. The cause: the Service used targetPort: postgres (a named port), but the container’s port had no name. The named port never resolved, so the endpoint list stayed empty and the ClusterIP had no backend. Fix: name the container port postgres.

ports:
  - containerPort: 5432
    name: postgres   # must match the Service targetPort by name

Why the ClusterIP had no backend

We also rotated the Guacamole web admin, which was still the well-known guacadmin / guacadmin default — hashing the new password with Guacamole’s exact algorithm (verified against the known default hash before touching the live row) and storing the new credential in a SealedSecret record.

5 · Grafana said “Origin Not Allowed”

Dashboards wouldn’t load. The fix was one missing scheme… removed.

The browser console showed the OpenFeature provider failing fatally with OFREP request failed: forbidden, and the Grafana Live websocket refused. Grafana’s CSRF middleware was returning origin not allowed. A previous PR had already set csrf_trusted_origins — and it still failed.

Two facts collided. First, Grafana matches the browser’s Origin host against the trusted list — its own sample.ini shows example.com, a bare hostname. The config had https://apps.thecylon.org, with the scheme, which can never equal the host apps.thecylon.org. Second, the Cloudflare tunnel rewrites the Host header to thecylon.local (needed so Traefik routes it), so Grafana’s same-origin shortcut can’t apply either.

The Origin that never matched

The fix was to drop the scheme: csrf_trusted_origins: apps.thecylon.org thecylon.local — space-separated, host-only. Shipped through the central chart, reconciled by ArgoCD. No live patch.

# before — scheme included, can never equal the Origin host
csrf_trusted_origins: https://apps.thecylon.org
# after — host-only, space-separated
csrf_trusted_origins: apps.thecylon.org thecylon.local

6 · A token, and where secrets belong

“Can we put it in GitHub secrets?” led to a better answer: OpenBao, with no Kubernetes Secret at all.

An SRE tool needed to call Grafana’s API, so we minted a least-privilege Viewer service-account token. The question was where to keep it. GitHub secrets? Fine — but only a GitHub Actions workflow can read them; a running pod can’t. The right home was already installed: OpenBao.

We wrote the token to KV at secret/opensre/grafana and set up a read-only policy. But the real goal was to avoid materialising a Kubernetes Secret at all — so instead of the External Secrets Operator (which syncs into a Secret), we wired the Agent Injector: the token lands as a file in the pod, backed by tmpfs, and no Secret object ever exists.

Injector flow — a secret with no Secret

All of it went into the central charts as GitOps: the injector flag, plus a bootstrap Job that enables OpenBao’s Kubernetes auth and creates the role. Two honest caveats came with the PR — the OpenBao ArgoCD app is currently Degraded on an immutable-selector sync error, and OpenSRE isn’t deployed yet — so the change is verified by render and lint, not yet live.

7 · The unseal-key near-miss

Turning the injector on meant walking straight into OpenBao’s crown jewels — and a placeholder nobody had filled.

To make the injector actually work, its ArgoCD app had to sync — and it wouldn’t. The first blocker was a classic: the openbao-unsealer Deployment had a immutable selector that no longer matched the chart, so every sync failed. Deleting the Deployment let ArgoCD recreate it cleanly, and the Agent Injector finally rolled out. Then the next failure surfaced:

SealedSecret/openbao-keys: failed update:
Resource "openbao-keys" already exists and is not managed by SealedSecret

The chart shipped openbao-keys as a SealedSecret with an empty placeholder — encryptedData: {} — with a comment reminding whoever came next to “seal the real values before first sync.” Nobody ever did. Meanwhile the live openbao-keys was a plain Secret holding the root token and three unseal keys — the values that let OpenBao come back to life after a restart. Get this wrong and OpenBao locks sealed, permanently.

Get this wrong and OpenBao locks sealed, permanently.

The real fix was to seal the current keys — the exact live values, so the sealed blob is guaranteed to decrypt to what’s already running — fill the placeholder, and adopt the existing Secret in place with the sealedsecrets.bitnami.com/managed annotation. No delete, no risk. The plaintext copy of the keys sitting in the Secret’s last-applied-configuration annotation got stripped in the same pass.

Seal what’s real, adopt in place

8 · Takeaways

Signal	Lesson
Empty placeholder secret	An encryptedData: {} that “deployed fine” is a pod that never started. Check the workload, not just the sync status.
20-byte backups	A backup job that never fails is not the same as a backup that works. Assert on dump size.
Probe killed initdb	First-init on slow storage needs a startupProbe. An aggressive liveness probe is a data-corruption risk, not just a restart.
Named ports	A Service targetPort by name needs a matching named container port, or endpoints stay silently empty.
CSRF host vs URL	Read the tool’s own sample config. One scheme prefix was the whole bug.
Where secrets live	GitHub secrets reach workflows; pods need Vault/OpenBao or SealedSecrets. Match the store to the consumer.
Placeholder SealedSecrets	An encryptedData: {} “TODO: seal before first sync” is a live landmine — it can empty the very Secret it’s meant to protect. Seal the real values or don’t ship the resource.
Adopt, don’t delete	For crown-jewel Secrets, the sealedsecrets.bitnami.com/managed annotation adopts in place — no delete window where the keys are gone.
Pause before you seal	Changing a Secret live while GitOps still holds the old value invites a self-heal to overwrite it. Pause auto-sync until git and reality agree.

The four-word request was real: the merge conflict got fixed. Everything after it was the repo telling us, one failure at a time, what it had been hiding. Four words in, four PRs out — and a homelab that finally tells the truth about itself. That’s the job.