Introduction

These documents contain information that relates to my personal setup. They may
work for you, they may not.

My Home Operations Repository :octocat:

... managed with Flux, Renovate, and GitHub Actions 🤖

📖 Overview

This is a repository for my home infrastructure and Kubernetes cluster. I try to adhere to Infrastructure as Code (IaC) and GitOps practices using tools like Terraform, Kubernetes, Flux, Renovate, and GitHub Actions.

⛵ Kubernetes

My Kubernetes cluster is deploy with Talos. This is a cluster of workloads running on Intel NUC devices with all storage with NFS and ISCSI shared mounted via a Synology Rackstation appliance. I also utilize offsite backups to cloudflare r2 blob storage and take hourly snapshots using [Volsync].

There is a template over at onedr0p/cluster-template if you want to try and follow along with some of the practices I use here.

Core Components

• cert-manager: Creates SSL certificates for services in my cluster.
• cilium: Internal Kubernetes container networking interface.
• cloudflared: Enables Cloudflare secure access to certain ingresses.
• external-dns: Automatically syncs ingress DNS records to a DNS provider.
• external-secrets: Managed Kubernetes secrets using 1Password Connect.
• ingress-nginx: Kubernetes ingress controller using NGINX as a reverse proxy and load balancer.
• sops: Managed secrets for Kubernetes and Terraform which are commited to Git.
• spegel: Stateless cluster local OCI registry mirror.
• volsync: Backup and recovery of persistent volume claims.

GitOps

Flux watches the clusters in my kubernetes folder (see Directories below) and makes the changes to my clusters based on the state of my Git repository.

The way Flux works for me here is it will recursively search the kubernetes/apps folder until it finds the most top level kustomization.yaml per directory and then apply all the resources listed in it. That aforementioned kustomization.yaml will generally only have a namespace resource and one or many Flux kustomizations (ks.yaml). Under the control of those Flux kustomizations there will be a HelmRelease or other resources related to the application which will be applied.

Renovate watches my entire repository looking for dependency updates, when they are found a PR is automatically created. When some (minor/patch) PRs are merged Flux applies the changes to my cluster.

Directories

This Git repository contains the following directories under Kubernetes.

📁 kubernetes
├── 📁 apps           # applications
├── 📁 bootstrap      # bootstrap procedures
├── 📁 flux           # core flux configuration
└── 📁 templates      # re-useable components

Flux Workflow

This is a high-level look how Flux deploys my applications with dependencies. Below there are 3 Flux kustomizations postgres, postgres-cluster, and atuin. postgres is the first app that needs to be running and healthy before postgres-cluster and once postgres-cluster is healthy atuin will be deployed.

graph TD;
  id1>Kustomization: cluster] -->|Creates| id2>Kustomization: cluster-apps];
  id2>Kustomization: cluster-apps] -->|Creates| id3>Kustomization: postgres];
  id2>Kustomization: cluster-apps] -->|Creates| id5>Kustomization: postgres-cluster];
  id2>Kustomization: cluster-apps] -->|Creates| id8>Kustomization: radarr];
  id3>Kustomization: postgres] -->|Creates| id4[HelmRelease: postgres];
  id5>Kustomization: postgres-cluster] -->|Depends on| id3>Kustomization: postgres];
  id5>Kustomization: postgres-cluster] -->|Creates| id10[Postgres Cluster];
  id8>Kustomization: radarr] -->|Creates| id9[HelmRelease: radarr];
  id8>Kustomization: radarr] -->|Creates| id11[PersistentVolumeClaim: radarr];
  id8>Kustomization: radarr] -->|Creates| id12[ExternalSecret: radarr-volsync-r2-secret];
  id8>Kustomization: radarr] -->|Creates| id13>ReplicationSource: radarr-r2];
  id8>Kustomization: radarr] -->|Creates| id14>ReplicationDestination: radarr-dst];
  id11>PersistentVolumeClaim: radarr] -->|SourceRef| id13>ReplicationSource: radarr-r2];
  id14>ReplicationDestination: radarr-dst] -->|Depends on| id12[ExternalSecret: radarr-volsync-r2-secret];
  id8>Kustomization: radarr] -->|Depends on| id5>Kustomization: postgres-cluster];
  id9>HelmRelease: radarr] -->|DependsOn| id11[PersistentVolumeClaim: radarr];

Networking

Click here to see my high-level network diagram

☁️ Cloud Dependencies

While most of my infrastructure and workloads are self-hosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about three things. (1) Dealing with chicken/egg scenarios, (2) services I critically need whether my cluster is online or not and (3) The "hit by a bus factor" - what happens to critical apps (e.g. Email, Password Manager, Photos) that my family relies on when I no longer around.

Alternative solutions to the first two of these problems would be to host a Kubernetes cluster in the cloud and deploy applications like HCVault, Vaultwarden, ntfy, and Gatus; however, maintaining another cluster and monitoring another group of workloads would be more work and probably be more or equal out to the same costs as described below.

🌐 DNS

In my cluster there are two ExternalDNS instances deployed. One is deployed with the ExternalDNS webhook provider for UniFi which syncs DNS records to my UniFi router. The other ExternalDNS instance syncs DNS records to Cloudflare only when the ingresses and services have an ingress class name of external and contain an ingress annotation external-dns.alpha.kubernetes.io/target. All local clients on my network use my UniFi router as the upstream DNS server.

🔧 Hardware

Click here to see my server rack

⭐ Stargazers

🤝 Gratitude and Thanks

Thanks to all the people who donate their time to the Home Operations Discord community. A special thanks to onedr0p for the inspiration, templates, and support. Be sure to check out kubesearch.dev for ideas on how to deploy applications or get ideas on what you could deploy.

1Password & Secret Storage

Model: SOPS bootstraps 1Password; 1Password does everything else

SOPS holds only what's needed to bring the cluster up far enough for External Secrets + 1Password Connect to run. Once Connect is up, every other secret comes from 1Password via ExternalSecrets.

SOPS-encrypted (bootstrap-only — must stay):

Everything else (cluster substitution vars, cloudflare/cloudflared/cert-manager tokens, all per-app secrets) is a 1Password ExternalSecret.

Cluster substitution variables (${SECRET_*})

Flux postBuild substitution reads, in order:

1. cluster-settings — a committed, non-secret ConfigMap (kubernetes/components/common/cluster-settings/) holding dummy values. This is what CI (which has no 1Password/SOPS) renders against, and a bootstrap fallback.
2. cluster-secrets — the 1Password ExternalSecret (kubernetes/components/common/cluster-secrets/) with the real values, listed second so it overrides the dummies in-cluster.

Real domains/IPs therefore live only in 1Password (out of git); only fake placeholders are committed.

Bootstrap seeding. bootstrap/resources.yaml.j2 also seeds flux-system/cluster-secrets from the local op CLI, so postBuild substitution has real values from the first bootstrap apply — before onepassword-connect (and the ExternalSecret) is up. The ExternalSecret adopts/refreshes that Secret once Connect is running. This closes the brief from-scratch-bootstrap window where apps would otherwise render against the cluster-settings placeholders.

Required 1Password fields (verify BEFORE merging the SOPS-minimization PR)

Removing the SOPS copies makes 1Password the sole source — these fields must exist or the consuming app breaks:

Item cluster-secrets — every referenced ${SECRET_*}, notably the newly-added SECRET_EXTERNAL_DOMAIN (was missing from the ExternalSecret), plus SECRET_DOMAIN, SECRET_INTERNAL_DOMAIN, SECRET_ACME_EMAIL, SECRET_CIDR, SECRET_CLOUDFLARE_TUNNEL_ID, SECRET_NAS_DOOM/MORDOR, SECRET_NFS_SERVER, SECRET_NFS_PATH_MEDIA, SECRET_VIP_PLEX/SMTP/ZIGBEE, CILIUM_LB_RANGE_START/END.

Item cloudflare:

• CLOUDFLARE_API_TOKEN — used by cert-manager (new ExternalSecret) and external-dns
• CLOUDFLARE_EMAIL — external-dns
• TUNNEL_ID, TUNNEL_CREDENTIALS_JSON — cloudflared

The cloudflared / external-dns ExternalSecrets already existed (the deleted SOPS files were redundant duplicates), so those fields are almost certainly already populated; cert-manager is the only newly-wired consumer.

CI

kustomize build does not run Flux substitution, so kubeconform skips HTTPRoute (its ${SECRET_DOMAIN} hostnames can't pass the strict hostname schema). flux-local substitutes via cluster-settings and validates the real rendered output.

Log Monitoring & Daily Health Review

A unified, cheap log pipeline for container, Talos host, and network/syslog logs — with a self-hosted single pane of glass and a daily Claude health digest.

Goals

• Aggregate container logs, Talos host logs, and network/device syslog into one place.
• Keep storage cheap: every log byte lands in Cloudflare R2, not replicated Ceph.
• Single pane of glass is self-hosted (free); Grafana Cloud is used only for off-cluster alerting (the "is the homelab reachable from outside" safety net).
• A daily Claude Code routine analyses the last 24h and posts a health digest.

Architecture

 Container logs ─► Alloy/Promtail ─┐
 Talos host logs (machine.logging) ─► Vector log-edge ─┤
 UniFi / Synology / UPS·PDU / printers (syslog) ─────► │─► Loki ─► R2 (chunks, cheap)
                                       (parse/drop/                 │
                                        log→metric)                 ├─► index cache on Ceph (20Gi)
                                                                    │
 Self-hosted Grafana (single pane) ◄── Prometheus + Loki datasources┘
 Grafana Cloud  ◄── Alloy (metrics only) ── used ONLY for off-cluster alerting
 Daily Claude CronJob ──LogQL+PromQL──► markdown digest ─► Slack (+Pushover if CRITICAL)

Components

Storage & retention (cost control)

• Loki default retention 30d (retention_period: 720h); chunks always live in R2.
• "Cold" tier ~270d for high-value streams via retention_stream: source=talos|unifi|synology|network and namespace=kube-system|flux-system|security.
• Vector drops health-check/probe noise and emits vector_log_events_total so dashboards/alerts query cheap metrics instead of scanning logs.
• Ceph PVC shrunk to 20Gi (index cache/WAL only).

Syslog source ports (point devices at the Vector syslog LoadBalancer IP)

Configure UniFi (Settings → System → Remote Logging), Synology (Log Center → syslog), CyberPower, and printers to send to the Vector syslog LoadBalancer IP (the io.cilium/lb-ipam-ips value on the vector-aggregator syslog Service).

Network-device analysis & alerting recommendations

Current monitoring is metric-rich but had no log/syslog visibility and several metric-alert gaps. Added/recommended:

• UniFi (unpoller + new syslog): poller-down, device-offline, switch port RX errors; syslog-driven firewall/IDS spikes, repeated auth failures. Tune unpoller_* metric names to your unpoller version.
• Talos hosts (new host logs): kernel/hardware errors (I/O, EXT4/XFS, MCE, NIC link-down), flapping services.
• Synology ×2: SNMP-down alert; syslog volume/disk/SMART/scrub errors.
• CyberPower UPS: estimated-runtime-low (complements existing on-battery/low- capacity/overload rules in terraform/alerting/rules_power.tf).
• WAN/speedtest: download/upload degraded, latency high.
• Log pipeline self-health: Loki down, Vector down, ingest stalled, R2 flush failures.
• Off-cluster (Grafana Cloud): metrics-forwarding-down and all-nodes-unreachable dead-man's switches.

Required 1Password items

Create these in the STONEHEDGES vault (referenced by ExternalSecrets):

Also create the loki-logs bucket in Cloudflare R2 (same account as the existing kube-prometheus-stack Thanos bucket).

Apply / rollout

1. Create the 1Password items and the loki-logs R2 bucket (above).
2. Merge the PR; Flux reconciles the observability namespace.
3. Apply the Talos change (host log shipping) — regenerate and apply machine config to each node, e.g. task talos:apply-node IP=<node-ip> (or your existing talhelper/talosctl workflow). This is required for source=talos logs to flow.
4. Point network devices' remote-syslog at the Vector syslog LoadBalancer IP (see ports above).
5. Visit https://grafana.${SECRET_DOMAIN} and confirm the Logs and Network & Syslog dashboards populate.

Per-app alerting (metrics)

Alerting is moving to co-located per-app PrometheusRules (alongside each app's ServiceMonitor + gatus check + dashboard), so the SPOG assembles itself from each app's own folder. The matching Terraform/Grafana-Cloud rules are kept running in parallel during cutover (delete later once the in-cluster ones are confirmed) — expect temporary duplicate pages.

Added/migrated in-cluster (reusing the exact PromQL from Terraform):

• *Media/arr: Plex, Jellyfin, Sonarr, Radarr, Prowlarr, Readarr, Lidarr, Bazarr, SABnzbd, Seerr (down/health/queue/indexers/storage).
• Infra: Authentik, Blocky, Gatus, Cloudflared.
• Ceph (was Cloud-only — now in-cluster under rook-ceph), UPS/PDU (under snmp-exporter), IoT/Sonos/Chamberlain (under blackbox/unifi-poller).
• New gap alerts: Dragonfly (down + memory), Mosquitto/MQTT telemetry, Home Assistant, Zigbee2MQTT.
• Log-based: ArrDatabaseLocked added to the Loki ruler.

Routing: in-cluster rules use severity + type labels → Alertmanager → Pushover/Slack (existing config). Grafana Cloud keeps only the off-cluster safety nets (NodeDown, AllNodesUnreachable, MetricsForwardingDown, PrometheusDataSourceDown, CloudflareTunnelDown).

Known not-yet-homed / needs-wiring:

• IngressControllerDown / HighErrorRate / HighIngressLatency — no ingress-nginx app dir found; left in Terraform pending a deliberate home.
• Home Assistant & Zigbee2MQTT alerts need their local Prometheus scrape wired (HA is currently scraped only by Alloy→Cloud; scrapeconfig.yaml is commented out). The rules are intentionally dormant (no absent() guard) until then; gatus still covers reachability.

Follow-ups (intentionally deferred)

• Thanos Query + Store Gateway to read the long-term metrics already in R2 (today the sidecar only writes). Local Prometheus retention was bumped to 48h so the pane/daily-job have a window; add Thanos Query for >48h history in Grafana.
• Optionally retire Promtail into Alloy to consolidate container-log collection.