argocd/README.md
2026-10-11 21:37:30 +02:00

9 KiB
Raw Blame History

argocd

GitOps for the homelab k3s cluster (k3s-design.md). Argo CD watches this repo on Forgejo and deploys whatever is on main.

Layout

bootstrap/root.yaml   the only thing applied by hand: the "root" Application -> cluster/
cluster/              cluster-level Argo objects (app-of-apps)
  argocd.yaml           Argo CD manages its own install from argocd-install/
  apps.yaml             ApplicationSet: one Application per folder in apps/
  sealed-secrets.yaml   Sealed Secrets controller (kube-system)
  longhorn.yaml         Longhorn storage (longhorn-system)
  traefik-config.yaml   tweaks to k3s's Traefik, from infra/traefik/
argocd-install/       Argo CD install (upstream install.yaml pinned + small patches)
infra/                plain manifests used by cluster/ Applications
apps/<name>/          one app per folder, deployed to namespace <name>
scripts/              prep-worker.sh (Longhorn prerequisites), seal-secrets.sh

Argo reads the repo at http://10.0.0.14:3000/un-oeil/argocd.git (Forgejo's internal address). The repo is public, so no credentials are needed.

Apps

App URL Data Database
homepage home.un-oeil.fr config in git none
ntfy ntfy.un-oeil.fr 2 Gi SQLite
wikijs wiki.un-oeil.fr none Postgres
paperless paperless.un-oeil.fr 2 + 50 + 5 Gi Postgres + Valkey
linkwarden links.un-oeil.fr 20 Gi Postgres
firefly firefly.un-oeil.fr 2 Gi Postgres
hedgedoc md.un-oeil.fr 5 Gi Postgres
freshrss rss.un-oeil.fr 2 Gi SQLite
vikunja tasks.un-oeil.fr 5 Gi SQLite
n8n n8n.un-oeil.fr 2 Gi SQLite
searxng search.un-oeil.fr none none
privatebin paste.un-oeil.fr 1 Gi files
stirling-pdf pdf.un-oeil.fr none none
it-tools tools.un-oeil.fr none none
postgres (internal) postgres.postgres.svc 10 Gi shared by the 5 above
whoami (example, no DNS) none none

Every volume is Longhorn, 2 replicas, on the workers' sdb. Total requests are about 3 GiB RAM and 110 GiB of volumes (×2 replicas = 220 of 600 GiB).

First rollout (in this order)

Over WireGuard, with KUBECONFIG=~/.kube/k3s-homelab.yaml.

1. Prepare the workers for Longhorn. On each of 10.0.0.141, .142, .143:

ssh un-oeil@10.0.0.141 'sudo bash -s' < scripts/prep-worker.sh

It formats /dev/sdb, which is the worker's own empty 200 GB disk (the zvol hdd-mirror/vm-disks/vm-14x-disk-0 on the host). It is not /hdd-mirror/vzdump-backups or hdd-mirror/restic-repo. The script stops if sdb isn't 200 GB or already holds anything. It also installs open-iscsi and nfs-common.

2. Push the cluster layer only:

git add cluster/ infra/ scripts/ README.md
git commit -m "Add Sealed Secrets, Longhorn, Traefik config"
git push

Wait until both are healthy:

kubectl -n kube-system rollout status deploy/sealed-secrets-controller
kubectl -n longhorn-system get pods          # all Running
kubectl get nodes.longhorn.io -n longhorn-system   # 3 workers, Ready

3. Back up the Sealed Secrets key. Without it, nobody can decrypt the secrets in this repo after a cluster rebuild. Keep it out of git, e.g. in Vaultwarden:

kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.yaml

4. Generate and seal the app secrets, then push the apps:

scripts/seal-secrets.sh
git add apps/
git commit -m "Add apps"
git push

Postgres needs to start before the five apps that use it. Those apps restart a few times until it's ready, which is normal.

5. Caddy (LXC 100). One block for every app. Caddy keeps the Host header, and Traefik routes on it:

home.un-oeil.fr, ntfy.un-oeil.fr, wiki.un-oeil.fr, paperless.un-oeil.fr,
links.un-oeil.fr, firefly.un-oeil.fr, md.un-oeil.fr, rss.un-oeil.fr,
tasks.un-oeil.fr, n8n.un-oeil.fr, search.un-oeil.fr, paste.un-oeil.fr,
pdf.un-oeil.fr, tools.un-oeil.fr {
    reverse_proxy 10.0.0.141:80 10.0.0.142:80 10.0.0.143:80 {
        lb_policy round_robin
        fail_duration 30s
        # ntfy and HedgeDoc stream
        flush_interval -1
    }
}

Then env $(cat /etc/caddy/infomaniak.env) caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy. Expect 14 certificate obtained lines in the journal.

6. DNS. Add the 14 A records → 10.0.0.10 at Infomaniak (zone file in ../infomaniak-zone-un-oeil.fr.txt).

After the first deploy

  • Paperless: user admin. Read the password with kubectl -n paperless get secret paperless-secrets -o jsonpath='{.data.PAPERLESS_ADMIN_PASSWORD}' | base64 -d, then change it.
  • Wiki.js, FreshRSS, n8n, Firefly, Linkwarden, HedgeDoc, Vikunja: the first visit creates the admin or owner account. Do it right after the deploy.
  • Vikunja: once your account exists, set VIKUNJA_SERVICE_ENABLEREGISTRATION to "false" in apps/vikunja/deployment.yaml.
  • HedgeDoc: same, with CMD_ALLOW_EMAIL_REGISTER: "false".
  • ntfy: nobody can use it until you create users:
    kubectl -n ntfy exec deploy/ntfy -- ntfy user add --role=admin un-oeil
    kubectl -n ntfy exec deploy/ntfy -- ntfy token add un-oeil   # for Uptime Kuma / scripts
    
  • Uptime Kuma: add the 14 URLs (uptime-kuma-monitors.md §3), plus a Push monitor for the Postgres backup below.

Postgres backups

apps/postgres/backup-cronjob.yaml streams pg_dumpall into restic on LXC 105 every night at 02:45, keeps 7 daily / 4 weekly / 3 monthly, then pings Uptime Kuma. It's suspended until it has credentials:

  1. On LXC 105, add a rest-server user k3s-postgres, named after its repo like the existing ones (--private-repos limits each user to its own path): htpasswd -B /etc/restic-htpasswd k3s-postgres && systemctl restart rest-server
  2. Create a Push monitor "Backup – Postgres (k3s)" in Kuma, interval 90000 s.
  3. Seal the credentials. The script adds them to the kustomization:
    RESTIC_REPOSITORY='rest:http://k3s-postgres:<pw>@10.0.0.15:8000/k3s-postgres' \
    RESTIC_PASSWORD='<new repo password, keep it in Vaultwarden>' \
    KUMA_PUSH_URL='https://status.un-oeil.fr/api/push/<token>?status=up&msg=OK' \
    scripts/seal-secrets.sh
    
  4. Set suspend: false in the CronJob, push, then test it once: kubectl -n postgres create job --from=cronjob/postgres-backup test-backup.

Not covered: the files on Longhorn volumes (Paperless documents, Linkwarden archives, uploads). The workers' sdb disks are excluded from vzdump (backup=0), and Longhorn has no backup target yet. Longhorn's backup target can be an NFS export or S3 bucket, e.g. on the restic LXC. Until then, those files survive a node failure (2 replicas) but not the loss of the pool.

Launching an app

  1. Create apps/<name>/ (copy apps/whoami/). Its contents can be:
    • plain manifests, or
    • a kustomization.yaml (Helm charts work too via helmCharts:), or
    • a Chart.yaml (umbrella chart with the real chart as a dependency + values.yaml).
  2. Volumes: storageClassName: longhorn, and strategy: Recreate on the Deployment (a Longhorn volume attaches to one node at a time).
  3. Secrets: add a seal line to scripts/seal-secrets.sh, or run kubeseal by hand. Never commit a plain Secret.
  4. A Postgres database: create it by hand, because the init script only runs on the first start: kubectl -n postgres exec -it postgres-0 -- psql -U postgres -c "CREATE ROLE x LOGIN PASSWORD '...'" -c "CREATE DATABASE x OWNER x"
  5. Commit and push. Argo picks it up within ~3 min and creates namespace <name>.
  6. Ingress host <name>.un-oeil.fr, add the name to the Caddy block above, and an A record → 10.0.0.10.

Removing a folder deletes the app and its resources, PVCs included. Longhorn, Sealed Secrets and Argo CD itself have prune: false, so a bad commit can't delete them.

Bootstrap (once, already done 2026-10-09)

kubectl apply --server-side -k argocd-install/
kubectl -n argocd rollout status deploy/argocd-server
kubectl apply -f bootstrap/root.yaml

Argo CD UI: https://argocd.un-oeil.fr through Caddy, behind Argo's own login. Caddy can't restrict it to the admin laptop: the WireGuard hub masquerades every peer as 10.0.0.250, so remote_ip never sees 10.10.10.2. Without Caddy: kubectl -n argocd port-forward svc/argocd-server 8080:80. Longhorn UI: kubectl -n longhorn-system port-forward svc/longhorn-frontend 8081:80 (or the same Caddy pattern with a longhorn.un-oeil.fr Ingress).

Before relying on it

  • Image versions were current on 2026-10-11. Several are new major versions (Paperless 3, Vikunja 2, n8n 2, Homepage 2, Stirling-PDF 3), so check their release notes for renamed settings if one misbehaves.
  • Traefik trusts X-Forwarded-For from 10.0.0.0/24 and the pod network, so apps see real client IPs through Caddy. Someone on WireGuard who hits Traefik directly could fake that header. That's acceptable for a private network.
  • Not there yet: Forgejo webhook → Argo for instant syncs, Longhorn backup target, kube-vip on .139.