It is highly recommended that you Use ECC RAM.
For general use as a rule of thumb you should have min 1GB RAM per 1TB of disk.
If you are using deduplication feature then a good rule of thumb is 5GB RAM for every 1TB of disk.
The reason is that deduplicated blocks are stored in a DDT table (stored in the ZFS ARC) which is actually stored in RAM.
Many block devices are combined to make VDEV. We generally use HDDs to make VDEV but in theory you can use partitions and the like.
You cannot change number of devices in existing VDEV. So you can’t add new hard drives to a VDEV after it is initially created.
Zpool consists of one or more VDEVs. Zpool can not suravive loss of VDEVs so each VDEV should handle redundency itself.
Enable LZ4 compression (recommended):
zfs set compression=lz4 tank
LZ4 uses multiple CPU cores so having a multicore CPU helps. LZ4 is not just fast enough and will put a max of 20% more CPU strain, but it also is able to fast abort/skip incompressible data therefore compression will be used where appropriate.
ZFS slows down as it fills up past 80% of its storage capacity. So do not fill your filesystems past 80%.
If you don’t follow this advice your ZFS instance may get permanently broken in terms of performance.
You should know that a stripe of mirrors (RAID10), has a higher failure probability compared to RAIDZ2 (or higher) on most common setups by several orders of magnitude. Though a mirror can resilver much faster, and it doesn’t significantly affect the vdev performance while doing so.
URE Risks: 6x8TB RAID10 has a 47.3% chance of encountering a URE during rebuild while A 6x8TB raidz2 has a 0.0002% chance!
So generally you should go for RAIDZ2
Cheap M/B Options discussed here: