I bought the "upgraded" RAID controller when purchasing the server, since I wanted to keep my options open. After thinking a bit harder about hardware RAID, it doesn't really seem that interesting to me. Hardware RAID might be a win if I didn't have tons of RAM to spare, or if I was very CPU constrained. Since neither of those are the case, it seems wiser to use my powerful Xeon CPUs and the large amount of ECC RAM available on the server to do fs checksumming and for caching purposes.

Awesome ZFS features:

• Great reliability features (checksumming in metadata)
• Snapshots
• Very flexible RAID configurations
• Theoretically easy to setup and try out
• Fantastic monitoring tools (check out the influxdb intergrations)
• Great reviews online
• Great arch-wiki documentation
• Apparently no RAID write hole problems despite not having a dedicated write-flush backup battery like HW would have
• many more

Looks great, but not as featureful as ZFS. If I try ZFS out and it doesn't work, I figured it would be easy to switch.

BTRFS was eliminated early as it seems to still be fairly immature.

Getting these drives into the server was easy. Just screw them into the enclosures:

Then pop them into the front mounting slots:

Next up was configuring the RAID controller to get out of the way. I wanted the raid controller to just pass the disks through to the operating system. It also seemed important to make sure that I could access the S.M.A.R.T. status of the devices.

Surprise surprise, the upgraded RAID controller I purchased is not able to do this! Apparently, the lower end model is, but only if you flash the thing with some special alternative firmware that breaks all of the fancy Dell integration.

Regardless, I booted the machine with some of the drives installed to see what would happen. The dell controller was not happy with the consumer drives. It marked a number of them as degraded, and thought that the kingston drives were SAS drives (maybe they actually are? never figured this out). Fortunately, it seemed like all of the drives were at least working.

After a very very large amount of time spent googling around, I found some references that said that, if you get the downgraded Dell H310 mini controller, it is possible to flash the controller to an alternative LSI "IT mode" firmware. The IT mode firmware is supposed to allow you to just pass the disks through to the OS.

Standard flashing procedures won't work though, because Dell looks for some special "I'm a Dell Special Thing" from the device at boot time. If you flash the board incorrectly, the server will refuse to boot in any way when the board is installed (so you can't reflash it).

There's a guy on eBay who will sell you on of these pre-flashed. Search for "Dell H310 mini monolithic K09CJ with LSI 9211-8i P20 IT Mode" then just buy one from him if you want to do this.

I of course didn't go down this path. Instead, I found some PDF file on archive.org that contained some instructions for flashing the controller. Since references to this file seem to all go stale, I'm mirroring it here, although I keep redoing my blog so this link will probably go stale too. I booted an Arch Linux iso through the remote managment interface and configured everything from Arch.

To follow these instructions, you'll have to find the LSI firmware files. Since LSI has been acquired like 30 times, its not entirely clear where to find them. To find these file names, figure out who owns LSI now and go look for their firmware downloads page.

You're looking for:

• 9211_8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows.zip or 9211_8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows.zip. After unzipping, you'll find Firmware/HBA_9211_8i_IT/Firmware/HBA_9211_8i_IT.bin
• UEFI_BSD_P20.zip. After unzipping, you'll find uefi_bsd_rel/Signed/x64sas2.rom

Once you have these, you should be able to follow the remaining instructions in the PDF.

There's a note in the PDF that says:

Should you want to boot off a drive attached to the H310MM, you will also have to flash the appropriate bootrom (mptsas2.rom for BIOS, x64sas2.rom for UEFI).

This is a very true statement and you'll be stuck scrathing your head for a long time if you miss/ignore it. Make sure to also flash the efi firmware to the device.

Since the Dell firmware integration is all broken with the new firmware, I needed to be able to keep track of which drive was which without having being able to easily toggle the chassis LEDs. I booted an Arch ISO and started dd=ing zeros to each disk through =/dev/disk/by-id/, then recording the serial numbers of the disks whose activity LEDs lit up. For some reason, the activity LEDs won't light up on the ADAT disks, so I just popped those in and out and watched the kernel logs.

All of the serial numbers and slot assignments are saved in a safe place. This is probably important to have when disks need replacing.

From the Arch iso, I partioned the disk in the 0th slot, and installed Arch using the standard install guide.

The OS install when smoothly, so I thought I was finally done with this ordeal once Arch finished bootstrapping the system.

Wrong!

Linux consistently failed to boot. I'd get through a GRUB screen, load initrd, then consistently fail to find the root partition. The root partition was on the same drive as GRUB, so this doesn't really make sense.

Apparently, when booting, the EFI system initializes the controller to get the bootloader, Linux initrd, etc. But then, when the initrd starts, something in Linux's drivers causes the SAS controller to reinit. The controller takes a long time to initialize, so Linux will have a hard time finding it's boot disk.

Adding rootdelay=600 to my kernel command line got me passed this problem; now Linux waits for root partition to show up for 5 minutes before giving up on the filesystem.

Just follow the instructions on the Arch Wiki. I installed the DKMS version of ZFS so that I would be able to pacman -Syu and have pacman attempt to rebuild ZFS with the latest kernel.

I setup two zpools. One for my personal files named nas and another for server stuff named server. These are mounted, creatively, at /nas and /server.

Unfortunately, NFS over my 10 GbE network doesn't perform as well as you'd hope.

From an NFS mount over 10 GbE (default mount options, few seem to make a difference but I have more to learn here):

# copy a 5 GiB file of random bytes from /tmp (ramdisk), to the NFS mount
# From switch stats: NFS isn't saturating the link for some reason.
$ dd if=/tmp/test of=test bs=1M
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 10.2211 s, 525 MB/s

# read the file we just copied to nowhere (immediately after writing)
# again, the switch maxed out at 4gbps during this transfer..
#        but mostly was nowhere close to the limit
$ dd if=test of=/dev/null bs=1M
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 16.3145 s, 329 MB/s

# same thing again
# better, this time I'm hitting the page cache on my RYZEN box
$ dd if=test of=/dev/null bs=1M
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 0.614145 s, 8.7 GB/s

# drop page cache, reread same file
# again, same deal
$ dd if=test of=/dev/null bs=1M
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.64609 s, 557 MB/s

As of this time, I haven't attempted to figure out why these rates are so poor. Trivial network tests with iperf3 and some custom code indicate that my NIC drivers and switch are all working properly, so there must be something I need to tune somewhere in the NFS layer.

I can trivally saturate gigabit with these rates, which means I'm also trivialy saturating the uplink through my VPN as well. Since I'm currently spending more of my time connected to the VPN from remote places (with less than gigabit bw), optimizing the NFS has not been a priority.