Add Sealed Secrets, Longhorn, Traefik config and scripts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
un-oeil 2026-10-11 20:58:44 +02:00
parent 827890b002
commit 8c418c3a3a
7 changed files with 377 additions and 22 deletions

184
README.md
View file

@ -10,34 +10,149 @@ bootstrap/root.yaml the only thing applied by hand: the "root" Application ->
cluster/ cluster-level Argo objects (app-of-apps) cluster/ cluster-level Argo objects (app-of-apps)
argocd.yaml Argo CD manages its own install from argocd-install/ argocd.yaml Argo CD manages its own install from argocd-install/
apps.yaml ApplicationSet: one Application per folder in apps/ apps.yaml ApplicationSet: one Application per folder in apps/
sealed-secrets.yaml Sealed Secrets controller (kube-system)
longhorn.yaml Longhorn storage (longhorn-system)
traefik-config.yaml tweaks to k3s's Traefik, from infra/traefik/
argocd-install/ Argo CD install (upstream install.yaml pinned + small patches) argocd-install/ Argo CD install (upstream install.yaml pinned + small patches)
infra/ plain manifests used by cluster/ Applications
apps/<name>/ one app per folder, deployed to namespace <name> apps/<name>/ one app per folder, deployed to namespace <name>
scripts/ prep-worker.sh (Longhorn prerequisites), seal-secrets.sh
``` ```
Argo reads the repo at `http://10.0.0.14:3000/un-oeil/argocd.git` (Forgejo's Argo reads the repo at `http://10.0.0.14:3000/un-oeil/argocd.git` (Forgejo's
internal address): pods can't reach `forgejo.un-oeil.fr` because the host has no internal address). The repo is public, so no credentials are needed.
hairpin NAT. The repo is public, so no credentials are needed.
## Bootstrap (once) ## Apps
Over WireGuard, with `KUBECONFIG=~/.kube/k3s-homelab.yaml`. Push this repo first: | App | URL | Data | Database |
Argo pulls from Forgejo, not from your laptop. |---|---|---|---|
| homepage | home.un-oeil.fr | config in git | none |
| ntfy | ntfy.un-oeil.fr | 2 Gi | SQLite |
| wikijs | wiki.un-oeil.fr | none | Postgres |
| paperless | paperless.un-oeil.fr | 2 + 50 + 5 Gi | Postgres + Valkey |
| linkwarden | links.un-oeil.fr | 20 Gi | Postgres |
| firefly | firefly.un-oeil.fr | 2 Gi | Postgres |
| hedgedoc | md.un-oeil.fr | 5 Gi | Postgres |
| freshrss | rss.un-oeil.fr | 2 Gi | SQLite |
| vikunja | tasks.un-oeil.fr | 5 Gi | SQLite |
| n8n | n8n.un-oeil.fr | 2 Gi | SQLite |
| searxng | search.un-oeil.fr | none | none |
| privatebin | paste.un-oeil.fr | 1 Gi | files |
| stirling-pdf | pdf.un-oeil.fr | none | none |
| it-tools | tools.un-oeil.fr | none | none |
| postgres | (internal) `postgres.postgres.svc` | 10 Gi | shared by the 5 above |
| whoami | (example, no DNS) | none | none |
Every volume is Longhorn, 2 replicas, on the workers' `sdb`. Total requests are
about 3 GiB RAM and 110 GiB of volumes (×2 replicas = 220 of 600 GiB).
## First rollout (in this order)
Over WireGuard, with `KUBECONFIG=~/.kube/k3s-homelab.yaml`.
**1. Prepare the workers for Longhorn.** On each of `10.0.0.141`, `.142`, `.143`:
```bash ```bash
kubectl apply --server-side -k argocd-install/ ssh un-oeil@10.0.0.141 'sudo bash -s' < scripts/prep-worker.sh
kubectl -n argocd rollout status deploy/argocd-server ```
kubectl apply -f bootstrap/root.yaml It formats `/dev/sdb`, which is the worker's own empty 200 GB disk (the zvol
`hdd-mirror/vm-disks/vm-14x-disk-0` on the host). It is **not**
`/hdd-mirror/vzdump-backups` or `hdd-mirror/restic-repo`. The script stops if
`sdb` isn't 200 GB or already holds anything. It also installs `open-iscsi` and
`nfs-common`.
**2. Push the cluster layer only:**
```bash
git add cluster/ infra/ scripts/ README.md
git commit -m "Add Sealed Secrets, Longhorn, Traefik config"
git push
```
Wait until both are healthy:
```bash
kubectl -n kube-system rollout status deploy/sealed-secrets-controller
kubectl -n longhorn-system get pods # all Running
kubectl get nodes.longhorn.io -n longhorn-system # 3 workers, Ready
``` ```
Admin password (user `admin`), then change it in the UI and delete the secret: **3. Back up the Sealed Secrets key.** Without it, nobody can decrypt the
secrets in this repo after a cluster rebuild. Keep it out of git, e.g. in
Vaultwarden:
```bash ```bash
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.yaml
``` ```
UI: either `kubectl -n argocd port-forward svc/argocd-server 8080:80` → **4. Generate and seal the app secrets**, then push the apps:
http://localhost:8080, or add `10.0.0.141 argocd.un-oeil.fr` to `/etc/hosts` → ```bash
http://argocd.un-oeil.fr. Argo CD is admin-only: **don't add it to Caddy**. scripts/seal-secrets.sh
git add apps/
git commit -m "Add apps"
git push
```
Postgres needs to start before the five apps that use it. Those apps restart
a few times until it's ready, which is normal.
**5. Caddy (LXC 100).** One block for every app. Caddy keeps the `Host` header,
and Traefik routes on it:
```
home.un-oeil.fr, ntfy.un-oeil.fr, wiki.un-oeil.fr, paperless.un-oeil.fr,
links.un-oeil.fr, firefly.un-oeil.fr, md.un-oeil.fr, rss.un-oeil.fr,
tasks.un-oeil.fr, n8n.un-oeil.fr, search.un-oeil.fr, paste.un-oeil.fr,
pdf.un-oeil.fr, tools.un-oeil.fr {
reverse_proxy 10.0.0.141:80 10.0.0.142:80 10.0.0.143:80 {
lb_policy round_robin
fail_duration 30s
# ntfy and HedgeDoc stream
flush_interval -1
}
}
```
Then `env $(cat /etc/caddy/infomaniak.env) caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy`.
Expect 14 `certificate obtained` lines in the journal.
**6. DNS.** Add the 14 A records → `10.0.0.10` at Infomaniak (zone file in
`../infomaniak-zone-un-oeil.fr.txt`).
## After the first deploy
- **Paperless:** user `admin`. Read the password with
`kubectl -n paperless get secret paperless-secrets -o jsonpath='{.data.PAPERLESS_ADMIN_PASSWORD}' | base64 -d`, then change it.
- **Wiki.js, FreshRSS, n8n, Firefly, Linkwarden, HedgeDoc, Vikunja:** the
first visit creates the admin or owner account. Do it right after the deploy.
- **Vikunja:** once your account exists, set `VIKUNJA_SERVICE_ENABLEREGISTRATION`
to `"false"` in `apps/vikunja/deployment.yaml`.
- **HedgeDoc:** same, with `CMD_ALLOW_EMAIL_REGISTER: "false"`.
- **ntfy:** nobody can use it until you create users:
```bash
kubectl -n ntfy exec deploy/ntfy -- ntfy user add --role=admin un-oeil
kubectl -n ntfy exec deploy/ntfy -- ntfy token add un-oeil # for Uptime Kuma / scripts
```
- **Uptime Kuma:** add the 14 URLs (`uptime-kuma-monitors.md` §3), plus a
Push monitor for the Postgres backup below.
## Postgres backups
`apps/postgres/backup-cronjob.yaml` streams `pg_dumpall` into restic on LXC 105
every night at 02:45, keeps 7 daily / 4 weekly / 3 monthly, then pings Uptime
Kuma. It's suspended until it has credentials:
1. On LXC 105, add a rest-server user `k3s-postgres`, named after its repo like
the existing ones (`--private-repos` limits each user to its own path):
`htpasswd -B /etc/restic-htpasswd k3s-postgres && systemctl restart rest-server`
2. Create a Push monitor "Backup – Postgres (k3s)" in Kuma, interval 90000 s.
3. Seal the credentials. The script adds them to the kustomization:
```bash
RESTIC_REPOSITORY='rest:http://k3s-postgres:<pw>@10.0.0.15:8000/k3s-postgres' \
RESTIC_PASSWORD='<new repo password, keep it in Vaultwarden>' \
KUMA_PUSH_URL='https://status.un-oeil.fr/api/push/<token>?status=up&msg=OK' \
scripts/seal-secrets.sh
```
4. Set `suspend: false` in the CronJob, push, then test it once:
`kubectl -n postgres create job --from=cronjob/postgres-backup test-backup`.
**Not covered:** the files on Longhorn volumes (Paperless documents, Linkwarden
archives, uploads). The workers' `sdb` disks are excluded from vzdump
(`backup=0`), and Longhorn has no backup target yet. Longhorn's backup target
can be an NFS export or S3 bucket, e.g. on the restic LXC. Until then, those
files survive a node failure (2 replicas) but not the loss of the pool.
## Launching an app ## Launching an app
@ -45,15 +160,40 @@ http://argocd.un-oeil.fr. Argo CD is admin-only: **don't add it to Caddy**.
- plain manifests, or - plain manifests, or
- a `kustomization.yaml` (Helm charts work too via `helmCharts:`), or - a `kustomization.yaml` (Helm charts work too via `helmCharts:`), or
- a `Chart.yaml` (umbrella chart with the real chart as a dependency + `values.yaml`). - a `Chart.yaml` (umbrella chart with the real chart as a dependency + `values.yaml`).
2. Commit and push. Argo picks it up within ~3 min (or hit *Refresh* in the UI) 2. Volumes: `storageClassName: longhorn`, and `strategy: Recreate` on the
and creates namespace `<name>`. Deployment (a Longhorn volume attaches to one node at a time).
3. To make it public: add an Ingress with host `<name>.un-oeil.fr`, then a Caddy 3. Secrets: add a `seal` line to `scripts/seal-secrets.sh`, or run `kubeseal`
vhost (LXC 100) pointing at Traefik on `10.0.0.141-143:80`. by hand. Never commit a plain Secret.
4. A Postgres database: create it by hand, because the init script only runs on
the first start:
`kubectl -n postgres exec -it postgres-0 -- psql -U postgres -c "CREATE ROLE x LOGIN PASSWORD '...'" -c "CREATE DATABASE x OWNER x"`
5. Commit and push. Argo picks it up within ~3 min and creates namespace `<name>`.
6. Ingress host `<name>.un-oeil.fr`, add the name to the Caddy block above, and
an A record → `10.0.0.10`.
Removing a folder deletes the app **and its resources, PVCs included**. Removing a folder deletes the app **and its resources, PVCs included**.
Longhorn, Sealed Secrets and Argo CD itself have `prune: false`, so a bad
commit can't delete them.
## Not there yet ## Bootstrap (once, already done 2026-10-09)
- Storage: only k3s `local-path` (data stuck on one node). Longhorn is the next app. ```bash
- Secrets: nothing yet. Don't commit plain Secrets; add Sealed Secrets or SOPS first. kubectl apply --server-side -k argocd-install/
- Forgejo webhook → Argo for instant syncs instead of 3 min polling. kubectl -n argocd rollout status deploy/argocd-server
kubectl apply -f bootstrap/root.yaml
```
Argo CD UI: `kubectl -n argocd port-forward svc/argocd-server 8080:80`, or
`10.0.0.141 argocd.un-oeil.fr` in `/etc/hosts`. It's admin-only: **don't add
it to Caddy**. Same for the Longhorn UI:
`kubectl -n longhorn-system port-forward svc/longhorn-frontend 8081:80`.
## Before relying on it
- Image versions were current on 2026-10-11. Several are new major versions
(Paperless 3, Vikunja 2, n8n 2, Homepage 2, Stirling-PDF 3), so check their
release notes for renamed settings if one misbehaves.
- Traefik trusts `X-Forwarded-For` from `10.0.0.0/24` and the pod network, so
apps see real client IPs through Caddy. Someone on WireGuard who hits Traefik
directly could fake that header. That's acceptable for a private network.
- Not there yet: Forgejo webhook → Argo for instant syncs, Longhorn backup
target, kube-vip on `.139`.

37
cluster/longhorn.yaml Normal file
View file

@ -0,0 +1,37 @@
# Longhorn: replicated block storage on the workers' 200 GB disk (sdb, mounted
# at /var/lib/longhorn by scripts/prep-worker.sh). Masters are tainted, so
# Longhorn only runs on the workers.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: longhorn
namespace: argocd
spec:
project: default
source:
repoURL: https://charts.longhorn.io
chart: longhorn
targetRevision: 1.13.0
helm:
valuesObject:
# The pre-upgrade hook job doesn't work under Argo CD.
preUpgradeChecker:
jobEnabled: false
persistence:
# local-path stays the k3s default; apps ask for longhorn by name.
defaultClass: false
defaultClassReplicaCount: 2
defaultSettings:
defaultDataPath: /var/lib/longhorn
defaultReplicaCount: 2
destination:
server: https://kubernetes.default.svc
namespace: longhorn-system
syncPolicy:
automated:
# No prune: a bad commit must not be able to delete the storage layer.
prune: false
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true

View file

@ -0,0 +1,26 @@
# Sealed Secrets controller: decrypts the SealedSecrets committed in apps/.
# Installed as kube-system/sealed-secrets-controller, the name and namespace
# kubeseal looks for by default.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sealed-secrets
namespace: argocd
spec:
project: default
source:
repoURL: https://bitnami.github.io/sealed-secrets
chart: sealed-secrets
targetRevision: 2.20.0
helm:
valuesObject:
fullnameOverride: sealed-secrets-controller
destination:
server: https://kubernetes.default.svc
namespace: kube-system
syncPolicy:
automated:
# No prune: deleting the controller loses its key, and every sealed
# secret in the repo becomes unreadable. Back the key up (README).
prune: false
selfHeal: true

View file

@ -0,0 +1,19 @@
# Tweaks to the Traefik that k3s bundles (infra/traefik/).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: traefik-config
namespace: argocd
spec:
project: default
source:
repoURL: http://10.0.0.14:3000/un-oeil/argocd.git
targetRevision: main
path: infra/traefik
destination:
server: https://kubernetes.default.svc
namespace: kube-system
syncPolicy:
automated:
prune: true
selfHeal: true

View file

@ -0,0 +1,16 @@
# Trust X-Forwarded-For from Caddy (10.0.0.10), so apps see the real client IP
# instead of Caddy's. Requests reach Traefik through ServiceLB, which rewrites
# the source to a node or pod address, hence the two ranges.
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
ports:
web:
forwardedHeaders:
trustedIPs:
- 10.0.0.0/24
- 10.42.0.0/16

35
scripts/prep-worker.sh Executable file
View file

@ -0,0 +1,35 @@
#!/usr/bin/env bash
# Longhorn prerequisites on one k3s worker. Run on each worker (141-143):
# ssh un-oeil@10.0.0.141 'sudo bash -s' < scripts/prep-worker.sh
#
# Formats /dev/sdb, the worker's own empty 200 GB disk. On the Proxmox host
# that disk is the zvol hdd-mirror/vm-disks/vm-14x-disk-0, separate from
# /hdd-mirror/vzdump-backups and hdd-mirror/restic-repo. The script stops if
# sdb already has a filesystem or partitions.
set -euo pipefail
DISK=/dev/sdb
MNT=/var/lib/longhorn
[ "$(lsblk -dno SIZE "$DISK" | tr -d " ")" = "200G" ] || { echo "$DISK is not the 200G Longhorn disk, stopping."; exit 1; }
if [ -n "$(blkid -o value -s TYPE "$DISK" 2>/dev/null)" ] || [ "$(lsblk -no NAME "$DISK" | wc -l)" -gt 1 ]; then
if findmnt -no SOURCE "$MNT" | grep -q "^$DISK$"; then
echo "$DISK already mounted on $MNT, skipping the disk part."
else
echo "$DISK already has data and isn't mounted on $MNT, stopping."; exit 1
fi
else
mkfs.ext4 -L longhorn "$DISK"
mkdir -p "$MNT"
echo "LABEL=longhorn $MNT ext4 defaults,noatime 0 2" >> /etc/fstab
mount "$MNT"
fi
apt-get update -qq
apt-get install -y -qq open-iscsi nfs-common
systemctl enable --now iscsid
modprobe iscsi_tcp
echo iscsi_tcp > /etc/modules-load.d/longhorn.conf
# multipathd grabs Longhorn's devices if it's running.
systemctl disable --now multipathd.service multipathd.socket 2>/dev/null || true
df -h "$MNT"; systemctl is-active iscsid

82
scripts/seal-secrets.sh Executable file
View file

@ -0,0 +1,82 @@
#!/usr/bin/env bash
# Generates random passwords for every app and writes them as SealedSecrets
# (apps/<app>/sealed-secret.yaml), which are safe to commit.
#
# Needs: kubectl + kubeseal, KUBECONFIG pointing at the cluster, and the
# sealed-secrets controller running (cluster/sealed-secrets.yaml synced).
#
# Existing files are never overwritten: Postgres only reads its passwords on
# the very first start, so regenerating them would lock the apps out.
# To rotate one app's secret, delete its file, rerun, and change the password
# in Postgres by hand.
#
# Optional, for the nightly Postgres backup (apps/postgres/backup-cronjob.yaml):
# RESTIC_REPOSITORY=rest:http://k3s-postgres:<pw>@10.0.0.15:8000/k3s-postgres \
# RESTIC_PASSWORD=... KUMA_PUSH_URL=https://status.un-oeil.fr/api/push/... \
# scripts/seal-secrets.sh
set -euo pipefail
cd "$(dirname "$0")/.."
rand() { openssl rand -hex "${1:-24}"; }
# seal <app> <secret-name> <file> KEY=VALUE...
seal() {
local app=$1 name=$2 out=apps/$1/$3; shift 3
if [ -e "$out" ]; then echo "skip $out (exists)"; return; fi
local args=()
for kv in "$@"; do args+=(--from-literal="$kv"); done
kubectl create secret generic "$name" -n "$app" "${args[@]}" --dry-run=client -o yaml \
| kubeseal --format yaml > "$out"
echo "wrote $out"
}
# Passwords shared between Postgres and the apps.
WIKIJS=$(rand); PAPERLESS=$(rand); LINKWARDEN=$(rand); FIREFLY=$(rand); HEDGEDOC=$(rand)
# If Postgres already has its passwords, a DB app missing its file would get a
# new password Postgres doesn't know.
if [ -e apps/postgres/sealed-secret.yaml ]; then
for a in wikijs paperless linkwarden firefly hedgedoc; do
if [ ! -e apps/$a/sealed-secret.yaml ]; then
echo "apps/$a/sealed-secret.yaml is missing but Postgres already has its password."
echo "Set it by hand: ALTER ROLE $a PASSWORD '...'; then seal the same value. Aborting."
exit 1
fi
done
fi
seal postgres postgres-secrets sealed-secret.yaml \
POSTGRES_PASSWORD="$(rand)" \
WIKIJS_DB_PASSWORD="$WIKIJS" PAPERLESS_DB_PASSWORD="$PAPERLESS" \
LINKWARDEN_DB_PASSWORD="$LINKWARDEN" FIREFLY_DB_PASSWORD="$FIREFLY" \
HEDGEDOC_DB_PASSWORD="$HEDGEDOC"
seal wikijs wikijs-secrets sealed-secret.yaml DB_PASS="$WIKIJS"
seal paperless paperless-secrets sealed-secret.yaml \
PAPERLESS_DBPASS="$PAPERLESS" PAPERLESS_SECRET_KEY="$(rand 32)" \
PAPERLESS_ADMIN_PASSWORD="$(rand 12)"
seal linkwarden linkwarden-secrets sealed-secret.yaml \
DATABASE_URL="postgresql://linkwarden:$LINKWARDEN@postgres.postgres.svc:5432/linkwarden" \
NEXTAUTH_SECRET="$(rand 32)"
# Firefly wants exactly 32 characters for both.
seal firefly firefly-secrets sealed-secret.yaml \
APP_KEY="$(rand 16)" DB_PASSWORD="$FIREFLY" STATIC_CRON_TOKEN="$(rand 16)"
seal hedgedoc hedgedoc-secrets sealed-secret.yaml \
CMD_DB_URL="postgres://hedgedoc:$HEDGEDOC@postgres.postgres.svc:5432/hedgedoc" \
CMD_SESSION_SECRET="$(rand 32)"
seal vikunja vikunja-secrets sealed-secret.yaml VIKUNJA_SERVICE_JWTSECRET="$(rand 32)"
seal searxng searxng-secrets sealed-secret.yaml SEARXNG_SECRET="$(rand 32)"
seal n8n n8n-secrets sealed-secret.yaml N8N_ENCRYPTION_KEY="$(rand 32)"
if [ -n "${RESTIC_REPOSITORY:-}" ]; then
seal postgres postgres-backup-secrets sealed-backup-secret.yaml \
RESTIC_REPOSITORY="$RESTIC_REPOSITORY" RESTIC_PASSWORD="${RESTIC_PASSWORD:?}" \
KUMA_PUSH_URL="${KUMA_PUSH_URL:?}"
grep -q sealed-backup-secret.yaml apps/postgres/kustomization.yaml \
|| sed -i 's/^ - sealed-secret.yaml$/&\n - sealed-backup-secret.yaml/' apps/postgres/kustomization.yaml
echo "Now set 'suspend: false' in apps/postgres/backup-cronjob.yaml."
fi
echo
echo "Done. Commit apps/*/sealed-secret.yaml and push."
echo "Paperless admin password, once synced:"
echo " kubectl -n paperless get secret paperless-secrets -o jsonpath='{.data.PAPERLESS_ADMIN_PASSWORD}' | base64 -d; echo"