diff --git a/README.md b/README.md index e02e10d..45f243b 100644 --- a/README.md +++ b/README.md @@ -10,34 +10,149 @@ bootstrap/root.yaml the only thing applied by hand: the "root" Application -> cluster/ cluster-level Argo objects (app-of-apps) argocd.yaml Argo CD manages its own install from argocd-install/ apps.yaml ApplicationSet: one Application per folder in apps/ + sealed-secrets.yaml Sealed Secrets controller (kube-system) + longhorn.yaml Longhorn storage (longhorn-system) + traefik-config.yaml tweaks to k3s's Traefik, from infra/traefik/ argocd-install/ Argo CD install (upstream install.yaml pinned + small patches) +infra/ plain manifests used by cluster/ Applications apps// one app per folder, deployed to namespace +scripts/ prep-worker.sh (Longhorn prerequisites), seal-secrets.sh ``` Argo reads the repo at `http://10.0.0.14:3000/un-oeil/argocd.git` (Forgejo's -internal address): pods can't reach `forgejo.un-oeil.fr` because the host has no -hairpin NAT. The repo is public, so no credentials are needed. +internal address). The repo is public, so no credentials are needed. -## Bootstrap (once) +## Apps -Over WireGuard, with `KUBECONFIG=~/.kube/k3s-homelab.yaml`. Push this repo first: -Argo pulls from Forgejo, not from your laptop. +| App | URL | Data | Database | +|---|---|---|---| +| homepage | home.un-oeil.fr | config in git | none | +| ntfy | ntfy.un-oeil.fr | 2 Gi | SQLite | +| wikijs | wiki.un-oeil.fr | none | Postgres | +| paperless | paperless.un-oeil.fr | 2 + 50 + 5 Gi | Postgres + Valkey | +| linkwarden | links.un-oeil.fr | 20 Gi | Postgres | +| firefly | firefly.un-oeil.fr | 2 Gi | Postgres | +| hedgedoc | md.un-oeil.fr | 5 Gi | Postgres | +| freshrss | rss.un-oeil.fr | 2 Gi | SQLite | +| vikunja | tasks.un-oeil.fr | 5 Gi | SQLite | +| n8n | n8n.un-oeil.fr | 2 Gi | SQLite | +| searxng | search.un-oeil.fr | none | none | +| privatebin | paste.un-oeil.fr | 1 Gi | files | +| stirling-pdf | pdf.un-oeil.fr | none | none | +| it-tools | tools.un-oeil.fr | none | none | +| postgres | (internal) `postgres.postgres.svc` | 10 Gi | shared by the 5 above | +| whoami | (example, no DNS) | none | none | +Every volume is Longhorn, 2 replicas, on the workers' `sdb`. Total requests are +about 3 GiB RAM and 110 GiB of volumes (×2 replicas = 220 of 600 GiB). + +## First rollout (in this order) + +Over WireGuard, with `KUBECONFIG=~/.kube/k3s-homelab.yaml`. + +**1. Prepare the workers for Longhorn.** On each of `10.0.0.141`, `.142`, `.143`: ```bash -kubectl apply --server-side -k argocd-install/ -kubectl -n argocd rollout status deploy/argocd-server -kubectl apply -f bootstrap/root.yaml +ssh un-oeil@10.0.0.141 'sudo bash -s' < scripts/prep-worker.sh +``` +It formats `/dev/sdb`, which is the worker's own empty 200 GB disk (the zvol +`hdd-mirror/vm-disks/vm-14x-disk-0` on the host). It is **not** +`/hdd-mirror/vzdump-backups` or `hdd-mirror/restic-repo`. The script stops if +`sdb` isn't 200 GB or already holds anything. It also installs `open-iscsi` and +`nfs-common`. + +**2. Push the cluster layer only:** +```bash +git add cluster/ infra/ scripts/ README.md +git commit -m "Add Sealed Secrets, Longhorn, Traefik config" +git push +``` +Wait until both are healthy: +```bash +kubectl -n kube-system rollout status deploy/sealed-secrets-controller +kubectl -n longhorn-system get pods # all Running +kubectl get nodes.longhorn.io -n longhorn-system # 3 workers, Ready ``` -Admin password (user `admin`), then change it in the UI and delete the secret: - +**3. Back up the Sealed Secrets key.** Without it, nobody can decrypt the +secrets in this repo after a cluster rebuild. Keep it out of git, e.g. in +Vaultwarden: ```bash -kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo +kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.yaml ``` -UI: either `kubectl -n argocd port-forward svc/argocd-server 8080:80` → -http://localhost:8080, or add `10.0.0.141 argocd.un-oeil.fr` to `/etc/hosts` → -http://argocd.un-oeil.fr. Argo CD is admin-only: **don't add it to Caddy**. +**4. Generate and seal the app secrets**, then push the apps: +```bash +scripts/seal-secrets.sh +git add apps/ +git commit -m "Add apps" +git push +``` +Postgres needs to start before the five apps that use it. Those apps restart +a few times until it's ready, which is normal. + +**5. Caddy (LXC 100).** One block for every app. Caddy keeps the `Host` header, +and Traefik routes on it: +``` +home.un-oeil.fr, ntfy.un-oeil.fr, wiki.un-oeil.fr, paperless.un-oeil.fr, +links.un-oeil.fr, firefly.un-oeil.fr, md.un-oeil.fr, rss.un-oeil.fr, +tasks.un-oeil.fr, n8n.un-oeil.fr, search.un-oeil.fr, paste.un-oeil.fr, +pdf.un-oeil.fr, tools.un-oeil.fr { + reverse_proxy 10.0.0.141:80 10.0.0.142:80 10.0.0.143:80 { + lb_policy round_robin + fail_duration 30s + # ntfy and HedgeDoc stream + flush_interval -1 + } +} +``` +Then `env $(cat /etc/caddy/infomaniak.env) caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy`. +Expect 14 `certificate obtained` lines in the journal. + +**6. DNS.** Add the 14 A records → `10.0.0.10` at Infomaniak (zone file in +`../infomaniak-zone-un-oeil.fr.txt`). + +## After the first deploy + +- **Paperless:** user `admin`. Read the password with + `kubectl -n paperless get secret paperless-secrets -o jsonpath='{.data.PAPERLESS_ADMIN_PASSWORD}' | base64 -d`, then change it. +- **Wiki.js, FreshRSS, n8n, Firefly, Linkwarden, HedgeDoc, Vikunja:** the + first visit creates the admin or owner account. Do it right after the deploy. +- **Vikunja:** once your account exists, set `VIKUNJA_SERVICE_ENABLEREGISTRATION` + to `"false"` in `apps/vikunja/deployment.yaml`. +- **HedgeDoc:** same, with `CMD_ALLOW_EMAIL_REGISTER: "false"`. +- **ntfy:** nobody can use it until you create users: + ```bash + kubectl -n ntfy exec deploy/ntfy -- ntfy user add --role=admin un-oeil + kubectl -n ntfy exec deploy/ntfy -- ntfy token add un-oeil # for Uptime Kuma / scripts + ``` +- **Uptime Kuma:** add the 14 URLs (`uptime-kuma-monitors.md` §3), plus a + Push monitor for the Postgres backup below. + +## Postgres backups + +`apps/postgres/backup-cronjob.yaml` streams `pg_dumpall` into restic on LXC 105 +every night at 02:45, keeps 7 daily / 4 weekly / 3 monthly, then pings Uptime +Kuma. It's suspended until it has credentials: + +1. On LXC 105, add a rest-server user `k3s-postgres`, named after its repo like + the existing ones (`--private-repos` limits each user to its own path): + `htpasswd -B /etc/restic-htpasswd k3s-postgres && systemctl restart rest-server` +2. Create a Push monitor "Backup – Postgres (k3s)" in Kuma, interval 90000 s. +3. Seal the credentials. The script adds them to the kustomization: + ```bash + RESTIC_REPOSITORY='rest:http://k3s-postgres:@10.0.0.15:8000/k3s-postgres' \ + RESTIC_PASSWORD='' \ + KUMA_PUSH_URL='https://status.un-oeil.fr/api/push/?status=up&msg=OK' \ + scripts/seal-secrets.sh + ``` +4. Set `suspend: false` in the CronJob, push, then test it once: + `kubectl -n postgres create job --from=cronjob/postgres-backup test-backup`. + +**Not covered:** the files on Longhorn volumes (Paperless documents, Linkwarden +archives, uploads). The workers' `sdb` disks are excluded from vzdump +(`backup=0`), and Longhorn has no backup target yet. Longhorn's backup target +can be an NFS export or S3 bucket, e.g. on the restic LXC. Until then, those +files survive a node failure (2 replicas) but not the loss of the pool. ## Launching an app @@ -45,15 +160,40 @@ http://argocd.un-oeil.fr. Argo CD is admin-only: **don't add it to Caddy**. - plain manifests, or - a `kustomization.yaml` (Helm charts work too via `helmCharts:`), or - a `Chart.yaml` (umbrella chart with the real chart as a dependency + `values.yaml`). -2. Commit and push. Argo picks it up within ~3 min (or hit *Refresh* in the UI) - and creates namespace ``. -3. To make it public: add an Ingress with host `.un-oeil.fr`, then a Caddy - vhost (LXC 100) pointing at Traefik on `10.0.0.141-143:80`. +2. Volumes: `storageClassName: longhorn`, and `strategy: Recreate` on the + Deployment (a Longhorn volume attaches to one node at a time). +3. Secrets: add a `seal` line to `scripts/seal-secrets.sh`, or run `kubeseal` + by hand. Never commit a plain Secret. +4. A Postgres database: create it by hand, because the init script only runs on + the first start: + `kubectl -n postgres exec -it postgres-0 -- psql -U postgres -c "CREATE ROLE x LOGIN PASSWORD '...'" -c "CREATE DATABASE x OWNER x"` +5. Commit and push. Argo picks it up within ~3 min and creates namespace ``. +6. Ingress host `.un-oeil.fr`, add the name to the Caddy block above, and + an A record → `10.0.0.10`. Removing a folder deletes the app **and its resources, PVCs included**. +Longhorn, Sealed Secrets and Argo CD itself have `prune: false`, so a bad +commit can't delete them. -## Not there yet +## Bootstrap (once, already done 2026-10-09) -- Storage: only k3s `local-path` (data stuck on one node). Longhorn is the next app. -- Secrets: nothing yet. Don't commit plain Secrets; add Sealed Secrets or SOPS first. -- Forgejo webhook → Argo for instant syncs instead of 3 min polling. +```bash +kubectl apply --server-side -k argocd-install/ +kubectl -n argocd rollout status deploy/argocd-server +kubectl apply -f bootstrap/root.yaml +``` +Argo CD UI: `kubectl -n argocd port-forward svc/argocd-server 8080:80`, or +`10.0.0.141 argocd.un-oeil.fr` in `/etc/hosts`. It's admin-only: **don't add +it to Caddy**. Same for the Longhorn UI: +`kubectl -n longhorn-system port-forward svc/longhorn-frontend 8081:80`. + +## Before relying on it + +- Image versions were current on 2026-10-11. Several are new major versions + (Paperless 3, Vikunja 2, n8n 2, Homepage 2, Stirling-PDF 3), so check their + release notes for renamed settings if one misbehaves. +- Traefik trusts `X-Forwarded-For` from `10.0.0.0/24` and the pod network, so + apps see real client IPs through Caddy. Someone on WireGuard who hits Traefik + directly could fake that header. That's acceptable for a private network. +- Not there yet: Forgejo webhook → Argo for instant syncs, Longhorn backup + target, kube-vip on `.139`. diff --git a/cluster/longhorn.yaml b/cluster/longhorn.yaml new file mode 100644 index 0000000..d2c6db4 --- /dev/null +++ b/cluster/longhorn.yaml @@ -0,0 +1,37 @@ +# Longhorn: replicated block storage on the workers' 200 GB disk (sdb, mounted +# at /var/lib/longhorn by scripts/prep-worker.sh). Masters are tainted, so +# Longhorn only runs on the workers. +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: longhorn + namespace: argocd +spec: + project: default + source: + repoURL: https://charts.longhorn.io + chart: longhorn + targetRevision: 1.13.0 + helm: + valuesObject: + # The pre-upgrade hook job doesn't work under Argo CD. + preUpgradeChecker: + jobEnabled: false + persistence: + # local-path stays the k3s default; apps ask for longhorn by name. + defaultClass: false + defaultClassReplicaCount: 2 + defaultSettings: + defaultDataPath: /var/lib/longhorn + defaultReplicaCount: 2 + destination: + server: https://kubernetes.default.svc + namespace: longhorn-system + syncPolicy: + automated: + # No prune: a bad commit must not be able to delete the storage layer. + prune: false + selfHeal: true + syncOptions: + - CreateNamespace=true + - ServerSideApply=true diff --git a/cluster/sealed-secrets.yaml b/cluster/sealed-secrets.yaml new file mode 100644 index 0000000..9d8ec6e --- /dev/null +++ b/cluster/sealed-secrets.yaml @@ -0,0 +1,26 @@ +# Sealed Secrets controller: decrypts the SealedSecrets committed in apps/. +# Installed as kube-system/sealed-secrets-controller, the name and namespace +# kubeseal looks for by default. +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: sealed-secrets + namespace: argocd +spec: + project: default + source: + repoURL: https://bitnami.github.io/sealed-secrets + chart: sealed-secrets + targetRevision: 2.20.0 + helm: + valuesObject: + fullnameOverride: sealed-secrets-controller + destination: + server: https://kubernetes.default.svc + namespace: kube-system + syncPolicy: + automated: + # No prune: deleting the controller loses its key, and every sealed + # secret in the repo becomes unreadable. Back the key up (README). + prune: false + selfHeal: true diff --git a/cluster/traefik-config.yaml b/cluster/traefik-config.yaml new file mode 100644 index 0000000..0cb9111 --- /dev/null +++ b/cluster/traefik-config.yaml @@ -0,0 +1,19 @@ +# Tweaks to the Traefik that k3s bundles (infra/traefik/). +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: traefik-config + namespace: argocd +spec: + project: default + source: + repoURL: http://10.0.0.14:3000/un-oeil/argocd.git + targetRevision: main + path: infra/traefik + destination: + server: https://kubernetes.default.svc + namespace: kube-system + syncPolicy: + automated: + prune: true + selfHeal: true diff --git a/infra/traefik/helmchartconfig.yaml b/infra/traefik/helmchartconfig.yaml new file mode 100644 index 0000000..a4b1b6f --- /dev/null +++ b/infra/traefik/helmchartconfig.yaml @@ -0,0 +1,16 @@ +# Trust X-Forwarded-For from Caddy (10.0.0.10), so apps see the real client IP +# instead of Caddy's. Requests reach Traefik through ServiceLB, which rewrites +# the source to a node or pod address, hence the two ranges. +apiVersion: helm.cattle.io/v1 +kind: HelmChartConfig +metadata: + name: traefik + namespace: kube-system +spec: + valuesContent: |- + ports: + web: + forwardedHeaders: + trustedIPs: + - 10.0.0.0/24 + - 10.42.0.0/16 diff --git a/scripts/prep-worker.sh b/scripts/prep-worker.sh new file mode 100755 index 0000000..efb1bc8 --- /dev/null +++ b/scripts/prep-worker.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Longhorn prerequisites on one k3s worker. Run on each worker (141-143): +# ssh un-oeil@10.0.0.141 'sudo bash -s' < scripts/prep-worker.sh +# +# Formats /dev/sdb, the worker's own empty 200 GB disk. On the Proxmox host +# that disk is the zvol hdd-mirror/vm-disks/vm-14x-disk-0, separate from +# /hdd-mirror/vzdump-backups and hdd-mirror/restic-repo. The script stops if +# sdb already has a filesystem or partitions. +set -euo pipefail +DISK=/dev/sdb +MNT=/var/lib/longhorn + +[ "$(lsblk -dno SIZE "$DISK" | tr -d " ")" = "200G" ] || { echo "$DISK is not the 200G Longhorn disk, stopping."; exit 1; } +if [ -n "$(blkid -o value -s TYPE "$DISK" 2>/dev/null)" ] || [ "$(lsblk -no NAME "$DISK" | wc -l)" -gt 1 ]; then + if findmnt -no SOURCE "$MNT" | grep -q "^$DISK$"; then + echo "$DISK already mounted on $MNT, skipping the disk part." + else + echo "$DISK already has data and isn't mounted on $MNT, stopping."; exit 1 + fi +else + mkfs.ext4 -L longhorn "$DISK" + mkdir -p "$MNT" + echo "LABEL=longhorn $MNT ext4 defaults,noatime 0 2" >> /etc/fstab + mount "$MNT" +fi + +apt-get update -qq +apt-get install -y -qq open-iscsi nfs-common +systemctl enable --now iscsid +modprobe iscsi_tcp +echo iscsi_tcp > /etc/modules-load.d/longhorn.conf +# multipathd grabs Longhorn's devices if it's running. +systemctl disable --now multipathd.service multipathd.socket 2>/dev/null || true + +df -h "$MNT"; systemctl is-active iscsid diff --git a/scripts/seal-secrets.sh b/scripts/seal-secrets.sh new file mode 100755 index 0000000..8fce1d9 --- /dev/null +++ b/scripts/seal-secrets.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# Generates random passwords for every app and writes them as SealedSecrets +# (apps//sealed-secret.yaml), which are safe to commit. +# +# Needs: kubectl + kubeseal, KUBECONFIG pointing at the cluster, and the +# sealed-secrets controller running (cluster/sealed-secrets.yaml synced). +# +# Existing files are never overwritten: Postgres only reads its passwords on +# the very first start, so regenerating them would lock the apps out. +# To rotate one app's secret, delete its file, rerun, and change the password +# in Postgres by hand. +# +# Optional, for the nightly Postgres backup (apps/postgres/backup-cronjob.yaml): +# RESTIC_REPOSITORY=rest:http://k3s-postgres:@10.0.0.15:8000/k3s-postgres \ +# RESTIC_PASSWORD=... KUMA_PUSH_URL=https://status.un-oeil.fr/api/push/... \ +# scripts/seal-secrets.sh +set -euo pipefail +cd "$(dirname "$0")/.." + +rand() { openssl rand -hex "${1:-24}"; } + +# seal KEY=VALUE... +seal() { + local app=$1 name=$2 out=apps/$1/$3; shift 3 + if [ -e "$out" ]; then echo "skip $out (exists)"; return; fi + local args=() + for kv in "$@"; do args+=(--from-literal="$kv"); done + kubectl create secret generic "$name" -n "$app" "${args[@]}" --dry-run=client -o yaml \ + | kubeseal --format yaml > "$out" + echo "wrote $out" +} + +# Passwords shared between Postgres and the apps. +WIKIJS=$(rand); PAPERLESS=$(rand); LINKWARDEN=$(rand); FIREFLY=$(rand); HEDGEDOC=$(rand) +# If Postgres already has its passwords, a DB app missing its file would get a +# new password Postgres doesn't know. +if [ -e apps/postgres/sealed-secret.yaml ]; then + for a in wikijs paperless linkwarden firefly hedgedoc; do + if [ ! -e apps/$a/sealed-secret.yaml ]; then + echo "apps/$a/sealed-secret.yaml is missing but Postgres already has its password." + echo "Set it by hand: ALTER ROLE $a PASSWORD '...'; then seal the same value. Aborting." + exit 1 + fi + done +fi + +seal postgres postgres-secrets sealed-secret.yaml \ + POSTGRES_PASSWORD="$(rand)" \ + WIKIJS_DB_PASSWORD="$WIKIJS" PAPERLESS_DB_PASSWORD="$PAPERLESS" \ + LINKWARDEN_DB_PASSWORD="$LINKWARDEN" FIREFLY_DB_PASSWORD="$FIREFLY" \ + HEDGEDOC_DB_PASSWORD="$HEDGEDOC" + +seal wikijs wikijs-secrets sealed-secret.yaml DB_PASS="$WIKIJS" +seal paperless paperless-secrets sealed-secret.yaml \ + PAPERLESS_DBPASS="$PAPERLESS" PAPERLESS_SECRET_KEY="$(rand 32)" \ + PAPERLESS_ADMIN_PASSWORD="$(rand 12)" +seal linkwarden linkwarden-secrets sealed-secret.yaml \ + DATABASE_URL="postgresql://linkwarden:$LINKWARDEN@postgres.postgres.svc:5432/linkwarden" \ + NEXTAUTH_SECRET="$(rand 32)" +# Firefly wants exactly 32 characters for both. +seal firefly firefly-secrets sealed-secret.yaml \ + APP_KEY="$(rand 16)" DB_PASSWORD="$FIREFLY" STATIC_CRON_TOKEN="$(rand 16)" +seal hedgedoc hedgedoc-secrets sealed-secret.yaml \ + CMD_DB_URL="postgres://hedgedoc:$HEDGEDOC@postgres.postgres.svc:5432/hedgedoc" \ + CMD_SESSION_SECRET="$(rand 32)" +seal vikunja vikunja-secrets sealed-secret.yaml VIKUNJA_SERVICE_JWTSECRET="$(rand 32)" +seal searxng searxng-secrets sealed-secret.yaml SEARXNG_SECRET="$(rand 32)" +seal n8n n8n-secrets sealed-secret.yaml N8N_ENCRYPTION_KEY="$(rand 32)" + +if [ -n "${RESTIC_REPOSITORY:-}" ]; then + seal postgres postgres-backup-secrets sealed-backup-secret.yaml \ + RESTIC_REPOSITORY="$RESTIC_REPOSITORY" RESTIC_PASSWORD="${RESTIC_PASSWORD:?}" \ + KUMA_PUSH_URL="${KUMA_PUSH_URL:?}" + grep -q sealed-backup-secret.yaml apps/postgres/kustomization.yaml \ + || sed -i 's/^ - sealed-secret.yaml$/&\n - sealed-backup-secret.yaml/' apps/postgres/kustomization.yaml + echo "Now set 'suspend: false' in apps/postgres/backup-cronjob.yaml." +fi + +echo +echo "Done. Commit apps/*/sealed-secret.yaml and push." +echo "Paperless admin password, once synced:" +echo " kubectl -n paperless get secret paperless-secrets -o jsonpath='{.data.PAPERLESS_ADMIN_PASSWORD}' | base64 -d; echo"