Hi there,
I'm planning to build a Proxmox - OMV - HA system. First to just learn how to do it properly and gain experience, later to use it as a fileserver.
3 identical nodes, each has
- i7 CPU
- 32GB RAM
- 7x 1GBit/s NICs
- 2x SATA SSD for OS
- 3x4 TB HDDs, but will scale up as necessary (in production use 12 TB drives will be used)
- additional SSDs for VMs as needed
I do feel comfortable with OMV and ZFS, also Ext4 with LUKS encryption is a routine.
However I have no practical experience with ceph and only rudementary experience with Proxmox so far.
What I'd like to achieve:
- fully encrypted Proxmox (on either one SSD or with two SSDs as Raid1 or similar for HA)
- unlocking via Dropbear for remote SSH access and unlocking the System
- OMV as VM managing storage (some / most / all of it?)
- storage has to be encrypted.
- scalability to be able to add more drives upwards of 80 TB total per node
- couple of VMs if RAM is enough
--> all that on 3 Nodes
What I lack is
- proper understanding of ZFS snapshots and how to use them for backups (usually use rsync / rsnapshot primarily)
--> wondering what a proper Backup would look like
- understanding the storage management in Proxmox and how to properly use that with OMV
- understanding ceph storage and how to properly use that with OMV
- understanding of encryption in ZFS and Ceph and how it compares to LUKS
------------------------------------------------------------------------------------------------------
Questions (some of which may better be suited for proxmox and ceph forums):
a) so far any comments on the setup and experiences?
b) with 32 GB of RAM and the standard recommendation for Ceph 1 GB RAM per 1 TB of storage - will I run just a slower system or are more serious problems to be expected? (Mainboard does not support more then 32 GB)
c) which way is best to manage storage with OMV?
I assume it is wise to let proxmox manage the HW of storage instead of OMV?
In ceph as I understand it Block Level Storage is the way to go yet if I'd want to use ZFS I'd have to use File Level Storage? I did not realy get what is meant by that distinction in practice.
Since encryption is a must ZFS was for a long time not really an option even though I really like the system.
Basically I just would really appreciate any help in getting a better understanding of which way to manage storage in that kind of system. The forums and wikis didn't clearify the open questions since I want to combine it with encryption.
If I let Proxmox handle the drives - does OMV just let me format them as Ext4? If so where to add the encryption layer?I don't get how to do that properly from a security standpoint.
c2) assuming I'd build the same without the Cluster-Ceph-Idea, just a single machine - would the best way be ZFS then? In that case I'd probably go for mirror DEVs mainly because of the expected number of drives and RAM Requirements. This might be the way for the Backup Server.
Would a solution with Snapraid + mergerFS work with Proxmox as well in such a scenario (wich would easily let me keep existing ext4+LUKS drives)?
Obviousy ZFS is much faster if properly setup, that is not a big issue here provided the speed suffices to do backups.
d) proper backup for 80 TB+X; r
sync / rsnapshot are an option still, since not that many changes hapen on the system. But maybe ZFS Snapshots (which I can only use if I choose ZFS over ceph) or Borg or any other solution is just a better way. Any suggestions are much appreciated.
Midterm I'll get a tapelibrary, but for now I need another way to backup, will probably be another of those machines not included in the cluster as single OMV (the current production server)
e) GlusterFS or any other solutions to achive this seem to be less favoured with proxmox, if you have experiences in the area I'm all ears.
Thank you in advance.
kwon