Before I get into it, let me say that the majority of the information below and analysis of it comes from one of my partners in crime (coworker at Nexenta), Kirill (@kdavyd). All credit for figuring out what was causing the long export times, and reproducing the issue internally (it was initially observed at a customer) after we cobbled together enough equipment, and analysis to date, goes to him. I'm just the messenger, here.
Here's the short version, in the form of the output from a test machine (emphasis is mine):
root@gold:/volumes# time zpool export l2test
real 38m57.023s
user 0m0.003s
sys 15m16.519s
That was a test pool with a couple of TB of L2ARC, filled up, with an average block size on pool of 4K (a worst case scenario) -- and then simply exported, as shown. If you are presently using a pool with a significant amount of L2ARC (100's of GB's or TB's), (update:) and you have multiple (3+) cache vdevs, and have any desire to ever export your pool that has a time limit attached to it before you need to be importing it again to maintain service availability, you should read on. Otherwise this doesn't really affect you. In either case, it should be fixable, long-term, so probably not a panic-worthy piece of information for most people. :)
If you want the workaround, by the way, it is to shoot the box in the head -- panic it, power it off hard, etc, to export the pool. This is obviously very dangerous if you do not have a proper setup with an actually utilized ZIL (preferably through nice log devices) and all your important data arriving synchronously, or if you have other pools on the system you don't want to lose access to as well. Unfortunately if either of those is the case, the only other workaround would be to remove the cache vdevs from the pool, wait out the eviction stuff, and only then export the pool. Better than just exporting it and waiting, since you'd be online while you removed the cache vdevs, but fairly time consuming still.
Here's your culprit, where we sit for potentially minutes, or even hours if you had a sufficiently 'bad' environment for this issue:
PC: _resume_from_idle+0xf1 CMD: zpool export l2test
stack pointer for thread ffffff4b94b44180: ffffff022cdd9820
[ ffffff022cdd9820 _resume_from_idle+0xf1() ]
swtch+0x145()
turnstile_block+0x760()
mutex_vector_enter+0x261()
_l2arc_evict+0x8d()
l2arc_evict+0xb0()
l2arc_remove_vdev+0x9b()
spa_l2cache_drop+0x65()
spa_unload+0xa6()
spa_export_common+0x1d9()
spa_export+0x2f()
zfs_ioc_pool_export+0x41()
zfsdev_ioctl+0x15e()
cdev_ioctl+0x45()
spec_ioctl+0x5a()
fop_ioctl+0x7b()
ioctl+0x18e()
dtrace_systrace_syscall32+0x11a()
_sys_sysenter_post_swapgs+0x149()
Analysis is ongoing, but it appears to be made exponentially worse the more L2ARC you have, the bigger the L2ARC devices are individually, the clock speed of your CPU (as this is a single-CPU-bound task), and your average block size (as that affects how many l2hdr entries you have).
This should be relatively easy to fix, in theory -- ARC evictions were already made asynchronous, but it looks like this wasn't done for L2ARC as well. If it is, this should be less of an issue.
Update 10/10/13:
Well, it appears there is code for an async l2arc eviction, and it's in play:
root@gold:/volumes# echo zfs_l2arc_async_evict::print | mdb -k
0x1 (B_TRUE)
So the investigation continues as to why this is happening anyway.
Update 10/10/13:
Kirill continues to investigate, and believes he's pinned the problem down to the async l2arc eviction not being always actually being asynchronous. In his testing, much of it is actually done asynchronously, but the minute there's more than X number going, the next one gets done synchronously. This is tied to the number of cache vdevs you have. And you cannot export until all the data they referenced is cleared, so the export won't finish until the only remaining tasks are in process (and are asynchronous, otherwise it'll have to wait them out). This explains why the export did complete before this was all actually through -- ongoing tasks were asynchronous and there were no more outstanding. Seemingly there is a 'maxalloc' being set on taskq_create by this line:
arc_flush_taskq = taskq_create("arc_flush_tq",
max_ncpus, minclsyspri, 1, 4, TASKQ_DYNAMIC);
The '4' is the culprit here. In Kirill's words: "Probably at the beginning of export, we are flushing ARC as well, so 2 threads are already busy, leaving us with only 2 for L2ARC. It is evicting asynchronously, just not all drives, because taskq isn’t big enough, since someone probably didn’t consider that there may be more than 2 L2ARC’s on a system :)", then adds, "Also explains why I couldn’t reproduce it with a single 1TB L2ARC - looks like you need at least three."
So the major takeaway here is that the original correlation between size of L2ARC and the size of the individual L2ARC vdevs is actually only of concern if you hit this 'bug' in the first place. To hit it, the size of your L2ARC and L2ARC vdevs isn't important, it is basically the number of them - seemingly in most situations you'll need at least 3 cache vdevs in the pool to ever run into this.
Update 10/14/13:
So apparently the arc_flush_taskq and the asynchronous ARC & L2ARC eviction code is all only in NexentaStor at the moment, having not (yet? I cannot publicly state our intent with this code, I have truly not been informed internally of what it is) been pushed back to illumos. Fortunately this means that on Nexenta 3.x machines, you'll only hit this long export time problem if you have more than 2 L2ARC devices in a pool you're trying to export, in general, and also the amount of ARC related to the pool shouldn't effect export times, either. Long-term we'll of course move to fix this issue, likely by increasing the limit above 4, but only after more thorough analysis internally.
Unfortunately it seems to mean that if you're not running Nexenta 3.x, and are on an older version or using an alternative illumos-derivative, you may very well be susceptible to a long export time even with just a lot of small block data in ARC, as well as any number of L2ARC devices including just 1. I do not have a box of sufficient size with a non-Nexenta illumos distro on it at the moment to do any testing, I'm afraid.
Thanks for the post. We are definitely hitting this bug with 3 SSD drives for L2ARC. Our failover time is typically about 20-25 minutes. to export the pool with a pool size of 8TB use( pool size of 30TB) and swing it over to the other head node. It looks like I'll me removing one of the L2ARC drives before we manually swing over (upgrade, etc.) for now.
ReplyDeleteScott, if you have SLOG devices in your pools and all your datasets have sync set properly (zvols too) then it is perfectly safe to panic one node via uadmin or reboot -p.
ReplyDeleteI've been dealing with this problem for a couple years now. I run OmniOS so I'm not benefiting from the work Nexenta has done on this. My systems are all HA with RSF-1.
ReplyDeleteMy procedure for doing maintenance is to first set sync=always on the pool of the pool I need to export. I then panic the system and the other head takes over the pool. When I'm finished I set sync=standard back on the pool. This has worked many times for me with out issue.
One thing I would like to point out is that removing the cache SSDs from the pool is not helpful here. As soon as you execute "zpool remove tank {cache_ssd}" you get into a blocking situation that too can last a ridiculous amount of time. On one pool I have with 8 400GB SSDs the first remove took 12 minutes, that blocked the I/O the entire time. That was 12 minutes of downtime I was not planning on dealing with.
Because of this issue, I prefer to build my high performance systems with more RAM and less L2ARC.
Hopefully Nexenta will release their work on this to Illumos soon.
I simply want to tell you that I’m all new to blogs and truly liked you’re blog site. Very likely I’m likely to bookmark your site .You surely come with remarkable articles. Cheers for sharing your website page.
ReplyDeleteHome Interiors in Chennai
This comment has been removed by the author.
ReplyDeleteI can get very detailed information from your blog alltime so looking forward for more from your side if anybody interested refer them links to get updates relevant to interior designs and approaches from:
ReplyDeleteBest Architects in India
Turnkey Interior Contractors in Chennai
Architecture Firms in Chennai
Warehouse Architect
Factory Architect Chennai
Office Interiors in Chennai
Rainwater Harvesting chennai
Bạn đã có 1 bài viết tuyệt vời. Chúc bạn một ngày mới tốt đẹp
ReplyDeleteThông tin mới nhất về cửa lưới chống muỗi
Siêu thị cửa chống muỗi
Hé mở thông tin cửa lưới chống muỗi xếp
Phòng chống muỗi cho biệt thư ở miền Nam
thú vị anh
DeleteLều xông hơi – Ngồi chơi cũng giảm cân
Nice blog, Visit Mutual Fund Wala for best mutual fund investment schemes.
ReplyDeleteInvestment Advisor in Delhi
This is an informative blog you share with us, thank you so much for sharing this.
ReplyDeleteLifestyle Magazine India
نئی معلومات، مجھے واقعی پسند ہے.
ReplyDeleteLều xông hơi khô
Túi xông hơi cá nhân
Lều xông hơi hồng ngoại
Mua lều xông hơi
Thật hay và thú vị
Deletehttps://forums.pokemmo.eu/index.php?/profile/131787-cualuoihm/
https://doremir.com/forums/profile/cualuoihm
https://www.wincert.net/forum/profile/100889-cualuoihm/
https://www.goodreads.com/user/show/104133368-cualuoihm
Thank you so much for sharing such an amazing blog. Visit Kalakutir Pvt Ltd for the best Commercial Vehicle Painting & Branding, Godown Floor Marking Painting and Caution & Indication Signages services in delhi, India.
ReplyDeleteGodown Floor Marking Painting
This should be relatively easy to fix, in theory -- ARC evictions were already made asynchronous, but it looks like this wasn't done for L2ARC as well. If it is, this should be less of an issue, please see here : Starup
ReplyDeleteok bạn hay quá
ReplyDeleteBồn ngâm chân
máy ngâm chân
bồn massage chân
may mat xa chan
Bài viết của bạn hết sức tuyệt vời
ReplyDeletemáy tạo hương thơm trong phòng
máy xông tinh dầu bằng điện tphcm
máy xông hương
may xong huong tinh dau
máy đốt tinh dầu điện
hihi hu
ReplyDeleteCông dụng của giảo cổ lam 5 lá
Giá bán hạt methi bao nhiêu tiền?
Hạt methi mua ở đâu Thanh Xuân tốt?
Dùng hạt methi trị tiểu đường tuyp 2
Công dụng giảm béo của giảo cổ lam
ok anh à
ReplyDeletethanh lý phòng net
màn hình máy tính 24 inch cũ
lắp đặt phòng net
giá card màn hình
Thanks for provide great informatic and looking beautiful blog, really nice required information & the things i never imagined and i would request, wright more blog and blog post like that for us. Thanks you once agian
ReplyDeletebirth certificate in delhi
name add in birth certificate
birth certificate in gurgaon
birth certificate correction
birth certificate in noida
birth certificate online
birth certificate in ghaziabad
birth certificate in india
birth certificate apply online
birth certificate in bengaluru
Ngày nay, bàn học không chỉ có chức năng để học nữa mà còn như một vật trang trí trong phòng của các bé. Khác hẳn với bé gái, bàn học dành cho bé trai mang vẻ năng động và tinh nghịch. Vì vậy, hãy tham khảo những mẫu bàn học dành cho bé trai dưới đây để tìm ra mẫu phù hợp cho con mình nhé!
ReplyDeleteok anh ơi mà
ReplyDeletemáy xông tinh dầu bằng điện
máy khuếch tán tinh dầu silent night
máy xông tinh dầu đuổi muỗi
máy khuếch tán tinh dầu hà nội
Your topic shows you are exploring. thank you for sharing: Máy ép dầu thực vật Nanifood, Máy ép tinh dầu Nanifood, Máy ép dầu Nanifood, Máy lọc dầu Nanifood, Máy ép dầu, May ep dau, Máy lọc dầu, Máy ép tinh dầu, Máy ép dầu thực vật, Máy ép dầu gia đình, Máy ép dầu kinh doanh, Bán máy ép dầu thực vật, Giá máy ép dầu, Máy ép dầu lạc, Máy ép dầu lạc mini, .......................
ReplyDeletehy thanks for sharing this information.very helpful and useful thanks
ReplyDeleteVery nice blog and articles. I am realy very happy to visit your blog. Now I am found which I actually want. I check your blog everyday and try to learn something from your blog. Thank you and waiting for your new post. thanks
ReplyDeleteNhững chia sẻ quá hay
ReplyDeleteSự thật về đông trùng hạ thảo Việt Nam bạn nên biết
https://dongtrunghathaonepa.com/
Giải mã bí mật đông trùng hạ thảo là gì?
Tác dụng đông trùng hạ thảo
Đông trùng hạ thảo
Nice Blog !
ReplyDeleteAny issue pops up in this acclaimed accounting software can be fixed in the least possible time by our talented professionals at QuickBooks Customer Service Phone Number 1-(855) 550-7546. Our experts are highly skilled and have years of experience in resolving all the issues of QuickBooks. Our number is open 24/7.
Hey! Good blog. I was facing an error in my QuickBooks software, so I called QuickBooks Support Phone Number (855)756-1077. I was tended to by an experienced and friendly technician who helped me to get rid of that annoying issue in the least possible time.
ReplyDeleteHey! Mind-blowing blog. Keep writing such beautiful blogs. In case you are struggling with issues on QuickBooks software, dial QuickBooks Customer Support Phone Number (877)948-5867. The team, on the other end, will assist you with the best technical services.
ReplyDeletethis is really too useful and have more ideas from yours. keep sharing many techniques. eagerly waiting for your new blog and useful information. keep doing more.
ReplyDeleteQuickBooks Error 6189
Hey! What a wonderful blog. I loved your blog. QuickBooks is the best accounting software, however, it has lots of bugs like QuickBooks Error. To fix such issues, you can contact experts via QuickBooks Customer Service Number
ReplyDeleteHey! Nice Blog, I have been using QuickBooks for a long time. One day, I encountered QuickBooks Customer Service in my software, then I called QuickBooks Customer Service Number. They resolved my error in the least possible time.
ReplyDeleteHey! Mind-blowing blog. Keep writing such beautiful blogs. In case you are struggling with issues on QuickBooks software, dial QuickBooks Customer Support Number . The team, on the other end, will assist you with the best technical services.
ReplyDeleteTật ok
ReplyDeleteMách bạn chi phí thi công nội thất trọn gói tại hà nội
Chi phí thi công nội thất trọn gói bao nhiêu năm 2021
Khám phá chi phí thi công nội thất trọn gói bao nhiêu
Hey! Excellent work. Being a QuickBooks user, if you are struggling with any issue, then dial QuickBooks Phone Number. Our team at QuickBooks will provide you with the best technical solutions for QuickBooks problems.
ReplyDeleteCanada Visa from UK
ReplyDeleteKanada Visum
Hey! Lovely blog. Your blog contains all the details and information related to the topic. In case you are a QuickBooks user, here is good news for you. You may encounter any error like QuickBooks Error, visit at QuickBooks Customer Support Number for quick help.
ReplyDeleteGreat tips! Thanks for sharing useful information... hair salons ues
ReplyDeleteHey! Lovely blog. Your blog contains all the details and information related to the topic. In case you are a QuickBooks user, here is good news for you. You may encounter any error like QuickBooks Error, visit at QuickBooks Customer Service (855)741-3663 for quick help.
ReplyDeleteok hấp dẫn
ReplyDeletetư vấn điện
công ty tư vấn điện
nhiệt điện
Hey! Mind-blowing blog. Keep writing such beautiful blogs. In case you are struggling with issues on QuickBooks software, dial QuickBooks Support Phone Number (888)233-6656. The team, on the other end, will assist you with the best technical services.
ReplyDeleteI am very thankfull to you for sharing this fantastic article
ReplyDeleteinternship request letter | Internship completion letter | internship companies | internship resume objective | internship application letter | Internship with training | internship email | internship experience | What internship means | Internship acknowledgement
Great work! It is the best thing that I have read on the internet today. Moreover, If you encounter any error while working on QuickBooks software , do contact this QuickBooks Customer Service (888)233-6656 number for quick assistance.
ReplyDeleteNhững chia sẻ của anh quá ok
ReplyDeletekhảo sát địa hình
điện sinh khối
trạm biến áp
Hey! Mind-blowing blog. Keep writing such beautiful blogs. In case you are struggling with issues on QuickBooks Enterprise Support (855)756-1077, dial QuickBooks Support Phone Number . The team, on the other end, will assist you with the best technical services.
ReplyDeleteHey! Well-written blog. It is the best thing that I have read on the internet today. Moreover, if you are looking for the solution of QuickBooks for MAC Support , visit at QuickBooks Support Phone Number (888)233-6656 to get your issues resolved quickly.
ReplyDeletenice blog. if you are searching for a quickbook customer service you can contact us on call.+1 866-669-5068
ReplyDeleteNice & Informative Blog !
ReplyDeleteIf you are looking for the best accounting software that can help you manage your business operations. call us at QuickBooks support service Phone Number.+1 888-272-4881
Good contant. we are provide a best service for custumer in Quickbooks support serviceyou can contact us at.+18882724881
ReplyDeletequả thật ok
ReplyDeletebao fibc
bao jumbo 1000kg
công ty bao bì jumbo
ReplyDeleteHi! Excellent blog. I feel great to be here reading your brilliant post. Moreover, if you are an avid QuickBooks user and facing any issue, dial QuickBooks Customer Care Phone Number.+13464148256 and get instant solutions for QuickBooks queries
Hi! Excellent blog. I feel great to be here reading your brilliant post. Moreover, if you are an avid QuickBooks user and facing any issue, dial QuickBooks Customer Care Phone Number.+1 602-325-1557,AZ and get instant solutions for QuickBooks queries
ReplyDeleteKhá hay
ReplyDeletepp jumbo bag scrap
công ty bao bì jumbo
type a fibc
Thanks for sharing amazing blog. Yoga can makes the easy to control your mind ,to understand about your mind and to keep calm your mind. Yoga, types of yoga , yogainfo ,you reach us at
ReplyDeleteIf you need help on correctable errors or issues on your desktop, call us at Quickbooks Customer Service Phone Number+1 855-769-6757 to get the best services. We are here 24/7 and are ready to provide answers to all your QuickBooks questions.
ReplyDeleteThis is beneficial Information for me Thanks For Sharing Such as Information .Kindly Go Through QuickBooks Support Phone Number +1 866-448-6293 For any Best QuickBooks Customer Service
ReplyDeletehay hay hay
ReplyDeleteThiết kế nội thất chung cư uy tín tại hà nội
Thiết kế nội thất chung cư cần lưu ý những gì?