argocd/README.md
2026-10-11 21:37:38 +02:00

204 lines
9.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# argocd
GitOps for the homelab k3s cluster (`k3s-design.md`). Argo CD watches this repo
on Forgejo and deploys whatever is on `main`.
## Layout
```
bootstrap/root.yaml the only thing applied by hand: the "root" Application -> cluster/
cluster/ cluster-level Argo objects (app-of-apps)
argocd.yaml Argo CD manages its own install from argocd-install/
apps.yaml ApplicationSet: one Application per folder in apps/
sealed-secrets.yaml Sealed Secrets controller (kube-system)
longhorn.yaml Longhorn storage (longhorn-system)
traefik-config.yaml tweaks to k3s's Traefik, from infra/traefik/
argocd-install/ Argo CD install (upstream install.yaml pinned + small patches)
infra/ plain manifests used by cluster/ Applications
apps/<name>/ one app per folder, deployed to namespace <name>
scripts/ prep-worker.sh (Longhorn prerequisites), seal-secrets.sh
```
Argo reads the repo at `http://10.0.0.14:3000/un-oeil/argocd.git` (Forgejo's
internal address). The repo is public, so no credentials are needed.
## Apps
| App | URL | Data | Database |
|---|---|---|---|
| homepage | home.un-oeil.fr | config in git | none |
| ntfy | ntfy.un-oeil.fr | 2 Gi | SQLite |
| wikijs | wiki.un-oeil.fr | none | Postgres |
| paperless | paperless.un-oeil.fr | 2 + 50 + 5 Gi | Postgres + Valkey |
| linkwarden | links.un-oeil.fr | 20 Gi | Postgres |
| firefly | firefly.un-oeil.fr | 2 Gi | Postgres |
| hedgedoc | md.un-oeil.fr | 5 Gi | Postgres |
| freshrss | rss.un-oeil.fr | 2 Gi | SQLite |
| vikunja | tasks.un-oeil.fr | 5 Gi | SQLite |
| n8n | n8n.un-oeil.fr | 2 Gi | SQLite |
| searxng | search.un-oeil.fr | none | none |
| privatebin | paste.un-oeil.fr | 1 Gi | files |
| stirling-pdf | pdf.un-oeil.fr | none | none |
| it-tools | tools.un-oeil.fr | none | none |
| postgres | (internal) `postgres.postgres.svc` | 10 Gi | shared by the 5 above |
| whoami | (example, no DNS) | none | none |
Every volume is Longhorn, 2 replicas, on the workers' `sdb`. Total requests are
about 3 GiB RAM and 110 GiB of volumes (×2 replicas = 220 of 600 GiB).
## First rollout (in this order)
Over WireGuard, with `KUBECONFIG=~/.kube/k3s-homelab.yaml`.
**1. Prepare the workers for Longhorn.** On each of `10.0.0.141`, `.142`, `.143`:
```bash
ssh un-oeil@10.0.0.141 'sudo bash -s' < scripts/prep-worker.sh
```
It formats `/dev/sdb`, which is the worker's own empty 200 GB disk (the zvol
`hdd-mirror/vm-disks/vm-14x-disk-0` on the host). It is **not**
`/hdd-mirror/vzdump-backups` or `hdd-mirror/restic-repo`. The script stops if
`sdb` isn't 200 GB or already holds anything. It also installs `open-iscsi` and
`nfs-common`.
**2. Push the cluster layer only:**
```bash
git add cluster/ infra/ scripts/ README.md
git commit -m "Add Sealed Secrets, Longhorn, Traefik config"
git push
```
Wait until both are healthy:
```bash
kubectl -n kube-system rollout status deploy/sealed-secrets-controller
kubectl -n longhorn-system get pods # all Running
kubectl get nodes.longhorn.io -n longhorn-system # 3 workers, Ready
```
**3. Back up the Sealed Secrets key.** Without it, nobody can decrypt the
secrets in this repo after a cluster rebuild. Keep it out of git, e.g. in
Vaultwarden:
```bash
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.yaml
```
**4. Generate and seal the app secrets**, then push the apps:
```bash
scripts/seal-secrets.sh
git add apps/
git commit -m "Add apps"
git push
```
Postgres needs to start before the five apps that use it. Those apps restart
a few times until it's ready, which is normal.
**5. Caddy (LXC 100).** One block for every app. Caddy keeps the `Host` header,
and Traefik routes on it:
```
home.un-oeil.fr, ntfy.un-oeil.fr, wiki.un-oeil.fr, paperless.un-oeil.fr,
links.un-oeil.fr, firefly.un-oeil.fr, md.un-oeil.fr, rss.un-oeil.fr,
tasks.un-oeil.fr, n8n.un-oeil.fr, search.un-oeil.fr, paste.un-oeil.fr,
pdf.un-oeil.fr, tools.un-oeil.fr {
reverse_proxy 10.0.0.141:80 10.0.0.142:80 10.0.0.143:80 {
lb_policy round_robin
fail_duration 30s
# ntfy and HedgeDoc stream
flush_interval -1
}
}
```
Then `env $(cat /etc/caddy/infomaniak.env) caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy`.
Expect 14 `certificate obtained` lines in the journal.
**6. DNS.** Add the 14 A records → `10.0.0.10` at Infomaniak (zone file in
`../infomaniak-zone-un-oeil.fr.txt`).
## After the first deploy
- **Paperless:** user `admin`. Read the password with
`kubectl -n paperless get secret paperless-secrets -o jsonpath='{.data.PAPERLESS_ADMIN_PASSWORD}' | base64 -d`, then change it.
- **Wiki.js, FreshRSS, n8n, Firefly, Linkwarden, HedgeDoc, Vikunja:** the
first visit creates the admin or owner account. Do it right after the deploy.
- **Vikunja:** once your account exists, set `VIKUNJA_SERVICE_ENABLEREGISTRATION`
to `"false"` in `apps/vikunja/deployment.yaml`.
- **HedgeDoc:** same, with `CMD_ALLOW_EMAIL_REGISTER: "false"`.
- **ntfy:** nobody can use it until you create users:
```bash
kubectl -n ntfy exec deploy/ntfy -- ntfy user add --role=admin un-oeil
kubectl -n ntfy exec deploy/ntfy -- ntfy token add un-oeil # for Uptime Kuma / scripts
```
- **Uptime Kuma:** add the 14 URLs (`uptime-kuma-monitors.md` §3), plus a
Push monitor for the Postgres backup below.
## Postgres backups
`apps/postgres/backup-cronjob.yaml` streams `pg_dumpall` into restic on LXC 105
every night at 02:45, keeps 7 daily / 4 weekly / 3 monthly, then pings Uptime
Kuma. It's suspended until it has credentials:
1. On LXC 105, add a rest-server user `k3s-postgres`, named after its repo like
the existing ones (`--private-repos` limits each user to its own path):
`htpasswd -B /etc/restic-htpasswd k3s-postgres && systemctl restart rest-server`
2. Create a Push monitor "Backup – Postgres (k3s)" in Kuma, interval 90000 s.
3. Seal the credentials. The script adds them to the kustomization:
```bash
RESTIC_REPOSITORY='rest:http://k3s-postgres:<pw>@10.0.0.15:8000/k3s-postgres' \
RESTIC_PASSWORD='<new repo password, keep it in Vaultwarden>' \
KUMA_PUSH_URL='https://status.un-oeil.fr/api/push/<token>?status=up&msg=OK' \
scripts/seal-secrets.sh
```
4. Set `suspend: false` in the CronJob, push, then test it once:
`kubectl -n postgres create job --from=cronjob/postgres-backup test-backup`.
**Not covered:** the files on Longhorn volumes (Paperless documents, Linkwarden
archives, uploads). The workers' `sdb` disks are excluded from vzdump
(`backup=0`), and Longhorn has no backup target yet. Longhorn's backup target
can be an NFS export or S3 bucket, e.g. on the restic LXC. Until then, those
files survive a node failure (2 replicas) but not the loss of the pool.
## Launching an app
1. Create `apps/<name>/` (copy `apps/whoami/`). Its contents can be:
- plain manifests, or
- a `kustomization.yaml` (Helm charts work too via `helmCharts:`), or
- a `Chart.yaml` (umbrella chart with the real chart as a dependency + `values.yaml`).
2. Volumes: `storageClassName: longhorn`, and `strategy: Recreate` on the
Deployment (a Longhorn volume attaches to one node at a time).
Put `enableServiceLinks: false` in the pod spec: otherwise a Service named
like the app injects `<APP>_PORT=tcp://...`, which some apps (Paperless)
read as their own setting.
3. Secrets: add a `seal` line to `scripts/seal-secrets.sh`, or run `kubeseal`
by hand. Never commit a plain Secret.
4. A Postgres database: create it by hand, because the init script only runs on
the first start:
`kubectl -n postgres exec -it postgres-0 -- psql -U postgres -c "CREATE ROLE x LOGIN PASSWORD '...'" -c "CREATE DATABASE x OWNER x"`
5. Commit and push. Argo picks it up within ~3 min and creates namespace `<name>`.
6. Ingress host `<name>.un-oeil.fr`, add the name to the Caddy block above, and
an A record → `10.0.0.10`.
Removing a folder deletes the app **and its resources, PVCs included**.
Longhorn, Sealed Secrets and Argo CD itself have `prune: false`, so a bad
commit can't delete them.
## Bootstrap (once, already done 2026-10-09)
```bash
kubectl apply --server-side -k argocd-install/
kubectl -n argocd rollout status deploy/argocd-server
kubectl apply -f bootstrap/root.yaml
```
Argo CD UI: https://argocd.un-oeil.fr through Caddy, behind Argo's own login.
Caddy can't restrict it to the admin laptop: the WireGuard hub masquerades
every peer as `10.0.0.250`, so `remote_ip` never sees `10.10.10.2`.
Without Caddy: `kubectl -n argocd port-forward svc/argocd-server 8080:80`.
Longhorn UI: `kubectl -n longhorn-system port-forward svc/longhorn-frontend 8081:80`
(or the same Caddy pattern with a `longhorn.un-oeil.fr` Ingress).
## Before relying on it
- Image versions were current on 2026-10-11. Several are new major versions
(Paperless 3, Vikunja 2, n8n 2, Homepage 2, Stirling-PDF 3), so check their
release notes for renamed settings if one misbehaves.
- Traefik trusts `X-Forwarded-For` from `10.0.0.0/24` and the pod network, so
apps see real client IPs through Caddy. Someone on WireGuard who hits Traefik
directly could fake that header. That's acceptable for a private network.
- Not there yet: Forgejo webhook → Argo for instant syncs, Longhorn backup
target, kube-vip on `.139`.