ZVOL Used Space

One of the more common complaints I hear about with ZFS is when clients are using zvol's to offer up iSCSI or FC block targets to clients, formatting filesystems on them, and then using them. Well, I don't get complaints about that, so much.. what I get are complaints about how, over time, the 'used' space as visible in ZFS is completely out of whack with the used space as visible in the filesystem on the client.

The reason for this is pretty simple - ZFS has absolutely no idea about your filesystem, and most filesystems delete files by updating a File Allocation Table or some such construct, not by going out and wiping out the blocks that had made up those files. Without some sort of semantics to understand what has happened, ZFS can't know that some of the blocks in your zvol are no longer being referenced by any file on the filesystem on top.

Enter SCSI Unmap. Support for this has been in COMSTAR for more than a year now, and the whole idea is that if your OS supports it, it can send down a SCSI UNMAP command that we can use to realize the blocks can be freed in the underlying disk. Except there's a problem. Very few operating systems/filesystems currently support this fully. Even when they do, they seem to only do so for normal operations, and often don't send down anything for metadata updates and such, so over time, you still get to a point where your filesystem on your client reads 39% full, and your zvol reads 100% full.

So, what's an admin to do? Well, that depends on the operating system you're dealing with. Also, all of the below absolutely rely on the fact that you have compress=on set on the zvol. Also, all of these are manual methods of clearing up the discrepancy between the zvol and the filesystem. None are perfect (there will pretty much always be some level of difference between the 'used' space on the zvol and the filesystem on top of it), but all of these should get you much closer to parity. If you want them to be automated, I leave that as a task for the reader. Please note that all of these utilities tend to cause a ton of throughput traffic as they do what they're doing - and can take hours and hours to complete depending on available bandwidth. Running one every night is probably impossible.

I also hope I don't have to mention that snapshots will sort of muck up this whole idea. You'll clear up space on the current dataset, but snapshots will just keep referring to the prior blocks until they're destroyed. If you have some sort of automatic snapshot taking/destroying mechanism (like AutoSnap in NexentaStor) then the effects on the overall zpool 'used space' will not be felt until the snapshots have rolled out, replaced by new ones. If you take permanent or semi-permanent snapshots by yourself, you'll have to remember that your zpool won't regain any of the space from this activity until you've killed all the snapshots made prior to running one of the below utilities.

Windows

The free answer is 'sdelete', provided by Microsoft. Link here: http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx

Basically snag the 'sdelete' tool, and as an Administrator, run it with "sdelete -z C:" (replacing C: with whatever disk is the one with a zvol ultimately at the back) in a Command Prompt window.

Linux

The main two I see used are "secure-delete" and "zerofree". On Ubuntu, both are available. With secure-delete, you're looking for the 'sfill' utility, and you'll want to read the man page, I believe there's a way to reduce it to basically just zero filling. However, I use 'zerofree', which is very simple to use - install it, and just run 'zerofree <filesystem>' and it'll do the rest.

Mac OSX

Use the built-in functionality. Check this link: http://support.apple.com/kb/ht3680