Tuesday, March 26, 2013

ZVOL Used Space

One of the more common complaints I hear about with ZFS is when clients are using zvol's to offer up iSCSI or FC block targets to clients, formatting filesystems on them, and then using them. Well, I don't get complaints about that, so much.. what I get are complaints about how, over time, the 'used' space as visible in ZFS is completely out of whack with the used space as visible in the filesystem on the client.

The reason for this is pretty simple - ZFS has absolutely no idea about your filesystem, and most filesystems delete files by updating a File Allocation Table or some such construct, not by going out and wiping out the blocks that had made up those files. Without some sort of semantics to understand what has happened, ZFS can't know that some of the blocks in your zvol are no longer being referenced by any file on the filesystem on top.

Enter SCSI Unmap. Support for this has been in COMSTAR for more than a year now, and the whole idea is that if your OS supports it, it can send down a SCSI UNMAP command that we can use to realize the blocks can be freed in the underlying disk. Except there's a problem. Very few operating systems/filesystems currently support this fully. Even when they do, they seem to only do so for normal operations, and often don't send down anything for metadata updates and such, so over time, you still get to a point where your filesystem on your client reads 39% full, and your zvol reads 100% full.

So, what's an admin to do? Well, that depends on the operating system you're dealing with. Also, all of the below absolutely rely on the fact that you have compress=on set on the zvol. Also, all of these are manual methods of clearing up the discrepancy between the zvol and the filesystem. None are perfect (there will pretty much always be some level of difference between the 'used' space on the zvol and the filesystem on top of it), but all of these should get you much closer to parity. If you want them to be automated, I leave that as a task for the reader. Please note that all of these utilities tend to cause a ton of throughput traffic as they do what they're doing - and can take hours and hours to complete depending on available bandwidth. Running one every night is probably impossible.

I also hope I don't have to mention that snapshots will sort of muck up this whole idea. You'll clear up space on the current dataset, but snapshots will just keep referring to the prior blocks until they're destroyed. If you have some sort of automatic snapshot taking/destroying mechanism (like AutoSnap in NexentaStor) then the effects on the overall zpool 'used space' will not be felt until the snapshots have rolled out, replaced by new ones. If you take permanent or semi-permanent snapshots by yourself, you'll have to remember that your zpool won't regain any of the space from this activity until you've killed all the snapshots made prior to running one of the below utilities.


The free answer is 'sdelete', provided by Microsoft. Link here: http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx

Basically snag the 'sdelete' tool, and as an Administrator, run it with "sdelete -z C:" (replacing C: with whatever disk is the one with a zvol ultimately at the back) in a Command Prompt window.


The main two I see used are "secure-delete" and "zerofree". On Ubuntu, both are available. With secure-delete, you're looking for the 'sfill' utility, and you'll want to read the man page, I believe there's a way to reduce it to basically just zero filling. However, I use 'zerofree', which is very simple to use - install it, and just run 'zerofree <filesystem>' and it'll do the rest.


Use the built-in functionality. Check this link: http://support.apple.com/kb/ht3680


  1. Can client systems use the "TRIM" command, much like SSD drives need to know that blocks are no longer used?
    This is mostly a curiosity question, for which I'm somewhat out of my depth in asking because I don't have any SSDs that I use daily. I thought I had heard (don't have URLs, sorry) that zvols could reclaim space when they received TRIM directives.

  2. COMSTAR in illumos has SCSI UNMAP support, which in theory would do as you suggest. However, you won't often find that helping you, I'm afraid, as most clients don't support it (for instance, last I checked, Windows iSCSI initiator does not send down SCSI UNMAP commands).

  3. your article is captivating and amazing getting material. Much appreciation to you for sharing your ability.
    SAP training in Kolkata
    Best SAP training in Kolkata
    SAP training institute in Kolkata

  4. Thanks for sharing informative post. Keep on sharing such useful information. Looking for best tamil novels free download to improve your language skills, check out the popular novels below
    Muthulakshmi Raghavan Novels
    Mallika Manivannan Novels
    Srikala Tamil Novels
    Ramanichandran Novels
    Sashi Murali novels
    Rajesh Kumar Novels


  5. Bài viết rất hay! Quà tặng 24k chuyên cung cấp:

    Đặt cúp đồng giá rẻ tại Hà Nội

    Quà tặng vinh danh ở đâu chất lượng và uy tín

    Hotline: 0866 866 711

  6. I must thank you for your time to read this wonderful article!! Foreign visitors should apply for Turkey evisa online.

  7. Thank you for this work.. In urgency, the travelers want to apply for e visa, then he/she can apply for Azerbaijan urgent visa through this application you will get visa within few hours.

  8. Attractive component of the material. I just stumbled across your web site and accession capital to say that I really enjoyed your site. With just a few clicks, foreign nationals can apply for a Kenya evisa application from their home. Fill the form with accurate and complete information about the passenger's data.

  9. This is a topic that’s near to my heart… Many thanks! Where are your contact details though? US citizens are eligible to apply for the Indian eVisa. You can apply for an emergency Indian visa for us citizen registration via mobile phone, computer or desk.

  10. I definitely enjoy every little bit of it. It is a good website and good stock. I want to thank you. COVID-19 PCR test requirements to travel to Ethiopia, passengers must present a medical certificate showing a negative COVID-19 test result before boarding a flight to Ethiopia. You can read all info related to COVID-19 test for Ethiopia via visacent website.

  11. I need to thank you for this very good read and i have bookmarked to check out new things from your post. Thank you very much for sharing such a useful article and will definitely save and revisit your site.
    artificial intelligence internship | best final year projects for cse | internship certificate online | internship for mba finance students | internship meaning in tamil

  12. I love to read this, thank you...Foreign nationals can get visas for India online. India visa application can be completed from home with the help of internet connection. In 5 to 10 minutes you can fill your form.

  13. I love what you guys do too. Much clever work and reporting!Keep up the good work guys...Turkish visa for American citizens, You can apply online via Turkish visa website.

  14. Greetings, Sir. Do you wish to apply for an India tourist visa? You can get more information on our information page about the Indian tourist visa. Read here..