Cables Matter

No, I don't mean brand, or color, or even connector-type zealotry.. but when sizing solutions and working with customers and partners on new builds, I often find that either no thought goes into the SAS cabling, or only a little bit does. I find this distressing. Let me tell you why.

ZFS Intent Log

The ZFS Intent Log gets a lot of attention, and unfortunately often the information being posted on various forums and blogs and so on is misinformed or makes assumptions about the knowledge level of the reader that if incorrect can lead to danger. Since my ZIL page on the old site is gone now, let me try to reconstruct the knowledge a bit in this post. I'm hesitant to post this - I've written the below and.. it is long. I tend to get a bit wordy, but it is also a subject with a lot of information to consider. Grab a drink and take your time here, and since this is on Blogger now, comments are open so you can ask questions.

If you don't want to read through this entire post, and you are worried about losing in-flight data due to things like a power loss event on the ZFS box, follow these rules:

1. Get a dedicated log device - it should be a very low-latency device, such as a STEC ZeusRAM or an SLC SSD, but even a high quality MLC SSD is better than leaving log traffic on the data vdevs (which is where they'll go without log devices in the pool). It should be at least a little larger than this formula: (maximum possible incoming write traffic in GB * seconds between transaction group commits). Make it much larger if its an SSD, and much much larger if its an MLC SSD - the size will help with longevity. Oh, and seconds between transaction group commits is the ZFS tunable zfs_txg_timeout. Default in older distros is 30 seconds, newer is 5, with even newer probably going to 10.
2. (optional but strongly preferred) Get a second dedicated log device (of the exact same type as the first), and when creating the log vdev, specify it as a mirror of the two. This will protect you from nasty edge cases.
3. Disable 'writeback cache' on every LU you create from a zvol, that has data you don't want to lose in-flight transactions for.
4. Set sync=always on the pool itself, and do not override the setting on any dataset you care about data integrity on (but feel free TO override the setting to sync=disabled on datasets where you know loss of in-transit data will be unimportant, easily recoverable, and/or not worth the cost associated with making it safe; thus freeing up I/O on your log devices to handle actually important incoming data).

Alright, on with the words.

(IPMP vs LACP) vs MPIO

If you're running an illumos or Solaris-based distribution for your ZFS needs, especially in a production environment, you may find yourself wanting to aggregate multiple network interfaces either for performance, redundancy, or both. With Solaris, your choices are not limited to standard LACP.

ZVOL Used Space

One of the more common complaints I hear about with ZFS is when clients are using zvol's to offer up iSCSI or FC block targets to clients, formatting filesystems on them, and then using them. Well, I don't get complaints about that, so much.. what I get are complaints about how, over time, the 'used' space as visible in ZFS is completely out of whack with the used space as visible in the filesystem on the client.

Nexenta - SMTP Server Settings

A common problem for users of NexentaStor, especially home users and people doing a trial evaluation of the software, is that, at least as of 3.1.3.5, it requires a real SMTP server somewhere else in order to send email. I'm often told no email was set up because no SMTP server or account was available (at home, this is often just permanently true, and in the enterprise this is often true during eval phases). This is bad - NexentaStor sends all sorts of alerts via email that it does not display anywhere else. Your appliance may be warning you of something, and you'll have no idea. There is a workaround, however, if the appliance can get to the internet and you have GMail (or can be bothered to set up an account)!