Tuesday, March 26, 2013

ZVOL Used Space

One of the more common complaints I hear about with ZFS is when clients are using zvol's to offer up iSCSI or FC block targets to clients, formatting filesystems on them, and then using them. Well, I don't get complaints about that, so much.. what I get are complaints about how, over time, the 'used' space as visible in ZFS is completely out of whack with the used space as visible in the filesystem on the client.

The reason for this is pretty simple - ZFS has absolutely no idea about your filesystem, and most filesystems delete files by updating a File Allocation Table or some such construct, not by going out and wiping out the blocks that had made up those files. Without some sort of semantics to understand what has happened, ZFS can't know that some of the blocks in your zvol are no longer being referenced by any file on the filesystem on top.

Enter SCSI Unmap. Support for this has been in COMSTAR for more than a year now, and the whole idea is that if your OS supports it, it can send down a SCSI UNMAP command that we can use to realize the blocks can be freed in the underlying disk. Except there's a problem. Very few operating systems/filesystems currently support this fully. Even when they do, they seem to only do so for normal operations, and often don't send down anything for metadata updates and such, so over time, you still get to a point where your filesystem on your client reads 39% full, and your zvol reads 100% full.

So, what's an admin to do? Well, that depends on the operating system you're dealing with. Also, all of the below absolutely rely on the fact that you have compress=on set on the zvol. Also, all of these are manual methods of clearing up the discrepancy between the zvol and the filesystem. None are perfect (there will pretty much always be some level of difference between the 'used' space on the zvol and the filesystem on top of it), but all of these should get you much closer to parity. If you want them to be automated, I leave that as a task for the reader. Please note that all of these utilities tend to cause a ton of throughput traffic as they do what they're doing - and can take hours and hours to complete depending on available bandwidth. Running one every night is probably impossible.

I also hope I don't have to mention that snapshots will sort of muck up this whole idea. You'll clear up space on the current dataset, but snapshots will just keep referring to the prior blocks until they're destroyed. If you have some sort of automatic snapshot taking/destroying mechanism (like AutoSnap in NexentaStor) then the effects on the overall zpool 'used space' will not be felt until the snapshots have rolled out, replaced by new ones. If you take permanent or semi-permanent snapshots by yourself, you'll have to remember that your zpool won't regain any of the space from this activity until you've killed all the snapshots made prior to running one of the below utilities.

Windows


The free answer is 'sdelete', provided by Microsoft. Link here: http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx

Basically snag the 'sdelete' tool, and as an Administrator, run it with "sdelete -z C:" (replacing C: with whatever disk is the one with a zvol ultimately at the back) in a Command Prompt window.

Linux


The main two I see used are "secure-delete" and "zerofree". On Ubuntu, both are available. With secure-delete, you're looking for the 'sfill' utility, and you'll want to read the man page, I believe there's a way to reduce it to basically just zero filling. However, I use 'zerofree', which is very simple to use - install it, and just run 'zerofree <filesystem>' and it'll do the rest.

Mac OSX


Use the built-in functionality. Check this link: http://support.apple.com/kb/ht3680

22 comments:

  1. Can client systems use the "TRIM" command, much like SSD drives need to know that blocks are no longer used?
    This is mostly a curiosity question, for which I'm somewhat out of my depth in asking because I don't have any SSDs that I use daily. I thought I had heard (don't have URLs, sorry) that zvols could reclaim space when they received TRIM directives.

    ReplyDelete
  2. COMSTAR in illumos has SCSI UNMAP support, which in theory would do as you suggest. However, you won't often find that helping you, I'm afraid, as most clients don't support it (for instance, last I checked, Windows iSCSI initiator does not send down SCSI UNMAP commands).

    ReplyDelete
  3. your article is captivating and amazing getting material. Much appreciation to you for sharing your ability.
    SAP training in Kolkata
    Best SAP training in Kolkata
    SAP training institute in Kolkata

    ReplyDelete
  4. Thanks for sharing informative post. Keep on sharing such useful information. Looking for best tamil novels free download to improve your language skills, check out the popular novels below
    Muthulakshmi Raghavan Novels
    Mallika Manivannan Novels
    Srikala Tamil Novels
    Ramanichandran Novels
    Sashi Murali novels
    Rajesh Kumar Novels

    ReplyDelete
  5. I must thank you for your time to read this wonderful article!! Foreign visitors should apply for Turkey evisa online.

    ReplyDelete
  6. Thank you for this work.. In urgency, the travelers want to apply for e visa, then he/she can apply for Azerbaijan urgent visa through this application you will get visa within few hours.

    ReplyDelete
  7. This is a topic that’s near to my heart… Many thanks! Where are your contact details though? US citizens are eligible to apply for the Indian eVisa. You can apply for an emergency Indian visa for us citizen registration via mobile phone, computer or desk.

    ReplyDelete
  8. I definitely enjoy every little bit of it. It is a good website and good stock. I want to thank you. COVID-19 PCR test requirements to travel to Ethiopia, passengers must present a medical certificate showing a negative COVID-19 test result before boarding a flight to Ethiopia. You can read all info related to COVID-19 test for Ethiopia via visacent website.

    ReplyDelete
  9. Likewise, a degree in Software engineering is generally required. At the point when you recruit CSS designers, it is likewise smart to register how they fit with the group and corporate culture. All things considered, it is one piece of the entire application improvement cycle and cross-group cooperation is the way to progress. At times individuals will generally befuddle these specializations. Thus, one probably won't know whether they need a UI designer or remote parttime front end engineer for a particular venture.

    This disarray is justifiable as those jobs are comparable yet not indistinguishable.

    A remote independent front end engineer manages programming design on the show layer.

    UI represents UI and manages the plan of cooperations among people and machines. It covers everything that such connections incorporate so a console and a mouse likewise become a piece of UI on the grounds that they are fundamental for certain cooperations.

    In web application improvement, front end and UI mean practically exactly the same thing. Yet, the terms are utilized for various jobs. So a "lesser web engineer front end parttime" is saved for a programming job and a "UI designer" for a plan job. In more modest groups, these jobs could meet or an accomplished independent front-end web designer can satisfy the two of them.

    The greater groups separate those considering contrasts>> freelance front end developer for hire</a.

    ReplyDelete