It began innocently enough: the WiFi needed an upgrade. My Archer A7 was wheezing along, and I’d already picked up a Protectli Vault VP2420 to finally get pfSense running. The plan? Replace the router. Maybe tidy things up. No drama.
You know where this is going.
That innocent little swap quickly turned into a full-blown infrastructure refactor. I took one look at the tangle in my network hole—half a dozen Proxmox nodes, scattered services, OMV running bare-metal on a dusty old box—and thought, yeah… it’s time.
I'd been eyeing Ceph for a while. My music collection had ballooned to 1.5 TB, my archives needed better redundancy, and my VM disks were still stuck on local LVM volumes. This was the moment.
So I went down the rabbit hole.
The Cluster Awakens
I spun up six Ceph nodes across the Proxmox cluster. Got 3 MONs and 2 MGRs going. Once the cluster came online, it was almost too easy: OSDs syncing, CRUSH rules doing their thing—Ceph was alive, even on my glorified home network.
I created my first pools: music and archive. That’s when Proxmox got cranky. RBD timeouts, question marks, GUI complaints. After chasing my tail for a bit, I realized the problem: I’d made the pools manually via CLI and forgot to run ceph osd pool application enable. Proxmox wasn't amused.
Classic mistake. Classic me.
So I tore it all down and did it the Proxmox way. This time, I set up tiered pools—ceph-ssd for fast stuff (VMs, music), ceph-hdd for cold storage. CRUSH rules made sure the data landed where it should. Simpler. Smarter.
Enter: OMV
Next step: OpenMediaVault. I spun it up as a VM, attached two RBD-backed disks—one from each Ceph tier—and booted into the GUI. Wiped, formatted, mounted. Easy… until it wasn’t.
The first disk add failed. Proxmox got weird again. KRBD configs, stale metadata, maybe a bad keyring—who knows. I ended up recreating the storage entries, restarted a few daemons, and it all clicked.
OMV saw the disks. I created shared folders—Music and Archive—gave a few users access, and set up Samba. But of course, OMV had one more gotcha: it refused to enable user home directories for SMB until I flipped the global user setting. I did. It worked.
And Still...
Meanwhile, the Protectli Vault—my original reason for all this—is now a node in my cluster. My new pfSense router-to-be. Patient. At some point, I’ll install it.
Right after I finish tuning CRUSH weights, rebalancing PGs, and figuring out why OSD.0 keeps reporting slow operations.
Probably.