201 lines
9 KiB
Markdown
201 lines
9 KiB
Markdown
# argocd
|
||
|
||
GitOps for the homelab k3s cluster (`k3s-design.md`). Argo CD watches this repo
|
||
on Forgejo and deploys whatever is on `main`.
|
||
|
||
## Layout
|
||
|
||
```
|
||
bootstrap/root.yaml the only thing applied by hand: the "root" Application -> cluster/
|
||
cluster/ cluster-level Argo objects (app-of-apps)
|
||
argocd.yaml Argo CD manages its own install from argocd-install/
|
||
apps.yaml ApplicationSet: one Application per folder in apps/
|
||
sealed-secrets.yaml Sealed Secrets controller (kube-system)
|
||
longhorn.yaml Longhorn storage (longhorn-system)
|
||
traefik-config.yaml tweaks to k3s's Traefik, from infra/traefik/
|
||
argocd-install/ Argo CD install (upstream install.yaml pinned + small patches)
|
||
infra/ plain manifests used by cluster/ Applications
|
||
apps/<name>/ one app per folder, deployed to namespace <name>
|
||
scripts/ prep-worker.sh (Longhorn prerequisites), seal-secrets.sh
|
||
```
|
||
|
||
Argo reads the repo at `http://10.0.0.14:3000/un-oeil/argocd.git` (Forgejo's
|
||
internal address). The repo is public, so no credentials are needed.
|
||
|
||
## Apps
|
||
|
||
| App | URL | Data | Database |
|
||
|---|---|---|---|
|
||
| homepage | home.un-oeil.fr | config in git | none |
|
||
| ntfy | ntfy.un-oeil.fr | 2 Gi | SQLite |
|
||
| wikijs | wiki.un-oeil.fr | none | Postgres |
|
||
| paperless | paperless.un-oeil.fr | 2 + 50 + 5 Gi | Postgres + Valkey |
|
||
| linkwarden | links.un-oeil.fr | 20 Gi | Postgres |
|
||
| firefly | firefly.un-oeil.fr | 2 Gi | Postgres |
|
||
| hedgedoc | md.un-oeil.fr | 5 Gi | Postgres |
|
||
| freshrss | rss.un-oeil.fr | 2 Gi | SQLite |
|
||
| vikunja | tasks.un-oeil.fr | 5 Gi | SQLite |
|
||
| n8n | n8n.un-oeil.fr | 2 Gi | SQLite |
|
||
| searxng | search.un-oeil.fr | none | none |
|
||
| privatebin | paste.un-oeil.fr | 1 Gi | files |
|
||
| stirling-pdf | pdf.un-oeil.fr | none | none |
|
||
| it-tools | tools.un-oeil.fr | none | none |
|
||
| postgres | (internal) `postgres.postgres.svc` | 10 Gi | shared by the 5 above |
|
||
| whoami | (example, no DNS) | none | none |
|
||
|
||
Every volume is Longhorn, 2 replicas, on the workers' `sdb`. Total requests are
|
||
about 3 GiB RAM and 110 GiB of volumes (×2 replicas = 220 of 600 GiB).
|
||
|
||
## First rollout (in this order)
|
||
|
||
Over WireGuard, with `KUBECONFIG=~/.kube/k3s-homelab.yaml`.
|
||
|
||
**1. Prepare the workers for Longhorn.** On each of `10.0.0.141`, `.142`, `.143`:
|
||
```bash
|
||
ssh un-oeil@10.0.0.141 'sudo bash -s' < scripts/prep-worker.sh
|
||
```
|
||
It formats `/dev/sdb`, which is the worker's own empty 200 GB disk (the zvol
|
||
`hdd-mirror/vm-disks/vm-14x-disk-0` on the host). It is **not**
|
||
`/hdd-mirror/vzdump-backups` or `hdd-mirror/restic-repo`. The script stops if
|
||
`sdb` isn't 200 GB or already holds anything. It also installs `open-iscsi` and
|
||
`nfs-common`.
|
||
|
||
**2. Push the cluster layer only:**
|
||
```bash
|
||
git add cluster/ infra/ scripts/ README.md
|
||
git commit -m "Add Sealed Secrets, Longhorn, Traefik config"
|
||
git push
|
||
```
|
||
Wait until both are healthy:
|
||
```bash
|
||
kubectl -n kube-system rollout status deploy/sealed-secrets-controller
|
||
kubectl -n longhorn-system get pods # all Running
|
||
kubectl get nodes.longhorn.io -n longhorn-system # 3 workers, Ready
|
||
```
|
||
|
||
**3. Back up the Sealed Secrets key.** Without it, nobody can decrypt the
|
||
secrets in this repo after a cluster rebuild. Keep it out of git, e.g. in
|
||
Vaultwarden:
|
||
```bash
|
||
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.yaml
|
||
```
|
||
|
||
**4. Generate and seal the app secrets**, then push the apps:
|
||
```bash
|
||
scripts/seal-secrets.sh
|
||
git add apps/
|
||
git commit -m "Add apps"
|
||
git push
|
||
```
|
||
Postgres needs to start before the five apps that use it. Those apps restart
|
||
a few times until it's ready, which is normal.
|
||
|
||
**5. Caddy (LXC 100).** One block for every app. Caddy keeps the `Host` header,
|
||
and Traefik routes on it:
|
||
```
|
||
home.un-oeil.fr, ntfy.un-oeil.fr, wiki.un-oeil.fr, paperless.un-oeil.fr,
|
||
links.un-oeil.fr, firefly.un-oeil.fr, md.un-oeil.fr, rss.un-oeil.fr,
|
||
tasks.un-oeil.fr, n8n.un-oeil.fr, search.un-oeil.fr, paste.un-oeil.fr,
|
||
pdf.un-oeil.fr, tools.un-oeil.fr {
|
||
reverse_proxy 10.0.0.141:80 10.0.0.142:80 10.0.0.143:80 {
|
||
lb_policy round_robin
|
||
fail_duration 30s
|
||
# ntfy and HedgeDoc stream
|
||
flush_interval -1
|
||
}
|
||
}
|
||
```
|
||
Then `env $(cat /etc/caddy/infomaniak.env) caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy`.
|
||
Expect 14 `certificate obtained` lines in the journal.
|
||
|
||
**6. DNS.** Add the 14 A records → `10.0.0.10` at Infomaniak (zone file in
|
||
`../infomaniak-zone-un-oeil.fr.txt`).
|
||
|
||
## After the first deploy
|
||
|
||
- **Paperless:** user `admin`. Read the password with
|
||
`kubectl -n paperless get secret paperless-secrets -o jsonpath='{.data.PAPERLESS_ADMIN_PASSWORD}' | base64 -d`, then change it.
|
||
- **Wiki.js, FreshRSS, n8n, Firefly, Linkwarden, HedgeDoc, Vikunja:** the
|
||
first visit creates the admin or owner account. Do it right after the deploy.
|
||
- **Vikunja:** once your account exists, set `VIKUNJA_SERVICE_ENABLEREGISTRATION`
|
||
to `"false"` in `apps/vikunja/deployment.yaml`.
|
||
- **HedgeDoc:** same, with `CMD_ALLOW_EMAIL_REGISTER: "false"`.
|
||
- **ntfy:** nobody can use it until you create users:
|
||
```bash
|
||
kubectl -n ntfy exec deploy/ntfy -- ntfy user add --role=admin un-oeil
|
||
kubectl -n ntfy exec deploy/ntfy -- ntfy token add un-oeil # for Uptime Kuma / scripts
|
||
```
|
||
- **Uptime Kuma:** add the 14 URLs (`uptime-kuma-monitors.md` §3), plus a
|
||
Push monitor for the Postgres backup below.
|
||
|
||
## Postgres backups
|
||
|
||
`apps/postgres/backup-cronjob.yaml` streams `pg_dumpall` into restic on LXC 105
|
||
every night at 02:45, keeps 7 daily / 4 weekly / 3 monthly, then pings Uptime
|
||
Kuma. It's suspended until it has credentials:
|
||
|
||
1. On LXC 105, add a rest-server user `k3s-postgres`, named after its repo like
|
||
the existing ones (`--private-repos` limits each user to its own path):
|
||
`htpasswd -B /etc/restic-htpasswd k3s-postgres && systemctl restart rest-server`
|
||
2. Create a Push monitor "Backup – Postgres (k3s)" in Kuma, interval 90000 s.
|
||
3. Seal the credentials. The script adds them to the kustomization:
|
||
```bash
|
||
RESTIC_REPOSITORY='rest:http://k3s-postgres:<pw>@10.0.0.15:8000/k3s-postgres' \
|
||
RESTIC_PASSWORD='<new repo password, keep it in Vaultwarden>' \
|
||
KUMA_PUSH_URL='https://status.un-oeil.fr/api/push/<token>?status=up&msg=OK' \
|
||
scripts/seal-secrets.sh
|
||
```
|
||
4. Set `suspend: false` in the CronJob, push, then test it once:
|
||
`kubectl -n postgres create job --from=cronjob/postgres-backup test-backup`.
|
||
|
||
**Not covered:** the files on Longhorn volumes (Paperless documents, Linkwarden
|
||
archives, uploads). The workers' `sdb` disks are excluded from vzdump
|
||
(`backup=0`), and Longhorn has no backup target yet. Longhorn's backup target
|
||
can be an NFS export or S3 bucket, e.g. on the restic LXC. Until then, those
|
||
files survive a node failure (2 replicas) but not the loss of the pool.
|
||
|
||
## Launching an app
|
||
|
||
1. Create `apps/<name>/` (copy `apps/whoami/`). Its contents can be:
|
||
- plain manifests, or
|
||
- a `kustomization.yaml` (Helm charts work too via `helmCharts:`), or
|
||
- a `Chart.yaml` (umbrella chart with the real chart as a dependency + `values.yaml`).
|
||
2. Volumes: `storageClassName: longhorn`, and `strategy: Recreate` on the
|
||
Deployment (a Longhorn volume attaches to one node at a time).
|
||
3. Secrets: add a `seal` line to `scripts/seal-secrets.sh`, or run `kubeseal`
|
||
by hand. Never commit a plain Secret.
|
||
4. A Postgres database: create it by hand, because the init script only runs on
|
||
the first start:
|
||
`kubectl -n postgres exec -it postgres-0 -- psql -U postgres -c "CREATE ROLE x LOGIN PASSWORD '...'" -c "CREATE DATABASE x OWNER x"`
|
||
5. Commit and push. Argo picks it up within ~3 min and creates namespace `<name>`.
|
||
6. Ingress host `<name>.un-oeil.fr`, add the name to the Caddy block above, and
|
||
an A record → `10.0.0.10`.
|
||
|
||
Removing a folder deletes the app **and its resources, PVCs included**.
|
||
Longhorn, Sealed Secrets and Argo CD itself have `prune: false`, so a bad
|
||
commit can't delete them.
|
||
|
||
## Bootstrap (once, already done 2026-10-09)
|
||
|
||
```bash
|
||
kubectl apply --server-side -k argocd-install/
|
||
kubectl -n argocd rollout status deploy/argocd-server
|
||
kubectl apply -f bootstrap/root.yaml
|
||
```
|
||
Argo CD UI: https://argocd.un-oeil.fr through Caddy, behind Argo's own login.
|
||
Caddy can't restrict it to the admin laptop: the WireGuard hub masquerades
|
||
every peer as `10.0.0.250`, so `remote_ip` never sees `10.10.10.2`.
|
||
Without Caddy: `kubectl -n argocd port-forward svc/argocd-server 8080:80`.
|
||
Longhorn UI: `kubectl -n longhorn-system port-forward svc/longhorn-frontend 8081:80`
|
||
(or the same Caddy pattern with a `longhorn.un-oeil.fr` Ingress).
|
||
|
||
## Before relying on it
|
||
|
||
- Image versions were current on 2026-10-11. Several are new major versions
|
||
(Paperless 3, Vikunja 2, n8n 2, Homepage 2, Stirling-PDF 3), so check their
|
||
release notes for renamed settings if one misbehaves.
|
||
- Traefik trusts `X-Forwarded-For` from `10.0.0.0/24` and the pod network, so
|
||
apps see real client IPs through Caddy. Someone on WireGuard who hits Traefik
|
||
directly could fake that header. That's acceptable for a private network.
|
||
- Not there yet: Forgejo webhook → Argo for instant syncs, Longhorn backup
|
||
target, kube-vip on `.139`.
|