Tuesday, April 2, 2013

ZFS Intent Log

[edited 11/22/2013 to modify formula]

The ZFS Intent Log gets a lot of attention, and unfortunately often the information being posted on various forums and blogs and so on is misinformed or makes assumptions about the knowledge level of the reader that if incorrect can lead to danger. Since my ZIL page on the old site is gone now, let me try to reconstruct the knowledge a bit in this post. I'm hesitant to post this - I've written the below and.. it is long. I tend to get a bit wordy, but it is also a subject with a lot of information to consider. Grab a drink and take your time here, and since this is on Blogger now, comments are open so you can ask questions.

If you don't want to read through this entire post, and you are worried about losing in-flight data due to things like a power loss event on the ZFS box, follow these rules:
  1. Get a dedicated log device - it should be a very low-latency device, such as a STEC ZeusRAM or an SLC SSD, but even a high quality MLC SSD is better than leaving log traffic on the data vdevs (which is where they'll go without log devices in the pool). It should be at least a little larger than this formula, if you want to prevent any possible chance of overrunning the size of your slog: (maximum possible incoming write traffic in GB * seconds between transaction group commits * 3). Make it much larger if its an SSD, and much much larger if its an MLC SSD - the size will help with longevity. Oh, and seconds between transaction group commits is the ZFS tunable zfs_txg_timeout. Default in older distributions is 30 seconds, newer is 5, with even newer probably going to 10. It is worth noting that if your rarely if ever have heavy write workloads, you may not have to size it as large -- it is very preferably from a performance perspective that you not be regularly filling the slog, but if you do it rarely it's no big deal. So if your average writes in [txg_timeout * 3] is only 1 GB, then you probably only need 1 GB of log space, and just understand when you rarely overfill it there will be a performance impact for a short period of time while the heavy write load continues. [edited 11/22/2013 - also, as a note, this logic only applies on ZFS versions that still use the older write code -- newer versions will have the new write mechanics and I will update this again with info on that when I have it]
  2. (optional but strongly preferred) Get a second dedicated log device (of the exact same type as the first), and when creating the log vdev, specify it as a mirror of the two. This will protect you from nasty edge cases.
  3. Disable 'writeback cache' on every LU you create from a zvol, that has data you don't want to lose in-flight transactions for.
  4. Set sync=always on the pool itself, and do not override the setting on any dataset you care about data integrity on (but feel free TO override the setting to sync=disabled on datasets where you know loss of in-transit data will be unimportant, easily recoverable, and/or not worth the cost associated with making it safe; thus freeing up I/O on your log devices to handle actually important incoming data).
Alright, on with the words.

It is important to, first and foremost, clear up a common misconception I see about the ZIL. It is not a write cache. There is no caching of any kind going on in the ZIL. The ZIL's purpose is not to provide you with a write cache. The ZIL's purpose is to protect you from data loss. It is necessary because the actual ZFS write cache, which is not the ZIL, is handled by system RAM, and RAM is volatile.

ZFS absolutely caches writes (usually) - incoming writes are held in RAM and, with a few notable exceptions, only written to disk during transaction group commits, which happen every N seconds. However, that isn't the ZIL. The ZIL is invoked when the incoming write meets certain requirements (most notably, something has tagged it as being a synchronous request), and overrides the 'put in RAM and respond to client that data is written' normal flow of asynchronous data in ZFS to be 'put in RAM, then put on stable media, and only once it is on stable media respond to client that data is written'.

One of the most common performance problems people run into with ZFS is not understanding ZIL mechanics. This comes about because, on every distribution I'm aware of, the default ZFS setup is that the ZIL is enabled -- and if there are no dedicated log devices configured on a pool, the ZIL will use a small portion of the data drives themselves to handle the log traffic. This workload is terrible on spinning media - it is single queue depth random sync write with cache flush - something spinning disks are terrible at. This leads to not only a noticeable performance problem for clients on writes, it has a very disruptive effect on the spinning media's ability to handle normal read requests and normal transaction group commits.

It is just all around a less than stellar situation to be in, and one that any ZFS appliance doing any significant traffic load is going to end up getting bit by (home users often do not - I run a number of boxes at home off a ZFS device with no dedicated log, and it is fine - I simply do not usually do enough I/O for it to be an issue).

So, enter the 'log' vdev type in ZFS. You can specify multiple 'log' virtual devices on a ZFS pool, containing one or more physical devices, just like a data vdev - you can even mirror them (and that's often a good idea). When ZFS sees that an incoming write to a pool is going to a pool with a log device, and that the rules surrounding usage of the ZIL are triggered and the write needs to go into the ZIL, ZFS will use these log virtual devices in a round-robin fashion to handle that write, as opposed to the normal data vdevs.

This has a double win for performance. First, you've just offloaded the ZIL traffic from the data vdevs. Second, your write response times (write latency) to clients will drop considerably not only because you're no longer using media that is being contended by multiple workflows, but because any sane person uses an SSD or non-volatile RAM-based device for log vdevs.

As a minor third benefit, by the way, you might see an additional overall improvement because the lower latency allows for more incoming writes, which has itself two potential performance improvements: one, it means that if the data being written happens to be such that it is writing to the same block multiple times within a single transaction group commit (txg), the txg only has to write the final state of the block to spinning media instead of all the intermediary updates; and second, the increased ability to send sync writes at the pool may mean better utilization of existing pool assets than was possible before (the pool might have been starved for writes, even though the client could send more, as the client was waiting on response before sending more and the pool was slow to send that response because the latency for writes was too high). However, these two benefits are very reliant on the specific environment and workload.

So, to summarize so far, you've got a ZFS Intent Log that is very similar to the log a SQL database uses, write once and forget (unless something bad happens), and you've got an in-RAM write cache and transaction group commits that handle actually writing to the data vdevs (and by the by, the txg commits are sequential, so all your random write traffic that came in between commits is sequential when it hits disk). The write cache is volatile as its in RAM, so the ZIL is in place to store the synchronous writes on stable media to restore from if things go south.

If you've gotten this far, you may have noticed I've kept hedging and saying 'synchronous' and such. This is important. ZFS is making some assumptions about data retention based on how the incoming writes look that many typical users just don't realize it is doing, and are often bitten quite hard because of it. I have seen thousands of ZFS boxes that are in danger of data loss.

The reason is that they are unaware that their clients are not sending data in a manner that triggers the ZIL, and as such, the incoming writes are only going into RAM, where they sit until the next txg commit - some day, when the box inevitably has an issue resulting in power loss, they're going to lose data. The severity of this data loss is directly tied to the workload they're putting on the server. It is extremely common for those environments I see where they're in danger to be utilizing things like iSCSI to provide virtual hard disks to VM's, and this is one of the worst environments to lose a couple seconds of write data in, as that write data is potentially critically important metadata for a filesystem sitting on top of a zvol, that when lost, corrupts the whole thing.

So first, let's talk about what gets you into the ZIL, today. This is pretty complicated, because there's essentially a number of ways that ZFS can handle an incoming write. Note first of all that as far as I'm aware, all incoming writes will be stored in RAM while the transaction group is open or committing to disk (I haven't been able to fully verify this yet), even when they're instantly put on final data vdev (thus, a read on this data should come from RAM). Aside from that, however, any of the following could happen:
  1. Write the data immediately to the log (ZIL) and respond to client OK. Data will be written from RAM to disk during next txg commit, normally. Data in log is only for restoration if power is lost.
  2. Write the data immediately to the data vdevs and store a pointer to the new block in the log (ZIL) then respond to client OK. Pointer to data block in log is used only on recovery if power is lost. On txg commit, just update metadata to point at the already-written new block (the data block itself won't be rewritten on txg commit, merely actually made part of the pool; prior to that, it's not actually referenced by the ZFS pool aside from the pointer in the ZIL).
  3. Write the data immediately to the data vdevs - nothing is written to the log device as this is a full write complete with metadata update, etc - then respond to client OK.
What can lead to these 3 types of workflow is a combination of a number of variables and the characteristics of both the incoming write and the total open transaction group. Sufficed to say, these variables are important:
  • logbias setting on the dataset being written to
  • zfs_immediate_write_sz
  • zil_slog_limit
  • Existence of a log device (method ZFS uses to handle writes will take into account rather the ZIL is on a log or not - it has major effects on the choice of mode used to deal with incoming data)
  • The incoming data has been, in one way or another, tagged as synchronous.
That last bold bullet point is key. None of all of the above stuff matters, and the incoming data will be stored solely in RAM, no ZIL mechanics in play, and written only to disk as part of the upcoming transaction group commit, if the incoming data is considered asynchronous. So. What can make you synchronous? Any of the following.
  1. The dataset being written into is sync=always. The incoming block could even be specifically called ASYNC in some way, and it won't matter, ZFS is going to treat it as a synchronous write.
  2. The dataset being written into is sync=standard and the block was written with a form of SYNC flag set on it.
The sync=standard setting is the default, and important data should be sent SYNC, right? So, surely all your important data is already being set with one of the above, right? Wrong. Different protocols specify sync or honor (or don't) client sync requests in different ways. Different layers in the stack between the client and the ZFS pool may alter a request to be sync or to disregard a sync request. And of course, ZFS itself may choose to interpret the incoming write as sync or async disregarding client semantics.
  • NFS - out of the box, most NFS traffic should be properly set as sync; specifying 'sync' (or the OS equivalent on your platform) on the mount command will guarantee this, specifying 'async' will likely ruin this and lead to most or all of the traffic from that mount not utilize the ZIL
  • CIFS/SMB - somewhat dependent on client - check with it to see what its defaults are
  • iSCSI - default is async, and very dangerously, some intermediary layers commonly found in an iSCSI setup will disregard sync requests from clients - notably, some hypervisor intermediary layers, where the hypervisor is responsible for iSCSI and the VM only sees the disk as presented by the hypervisor may be requesting O_SYNC inside the VM, but the hypervisor is disregarding that based on settings, and the request is sent to ZFS without sync set
  • Local box - this is to say, you're doing tasks directly on the box running the zpool - usually this is going to be asynchronous unless the application has intentionally requested sync writes (some things will, depending on settings, like *SQL databases for example). Generally speaking, however, it will be asynchronous from a client perspective.
If you've got data you want to be sure is being set properly to sync, how you guarantee this is a factor of rather you care about granularity. If you want every last bit of data being written to be sync (as you very often do when you have a dedicated log device, and even more so when the clients are, say, virtual machines using the storage as their primary disks), make sure all your datasets have sync unset (eg: being inherited from parent) and set sync=always on the pool itself. This is a quick and easy way that should guarantee data integrity.

It may seem counter-intuitive, but sometimes, data integrity is trumped by the cost of delivering it. Labs, non-production use-cases, and so on are obvious, but even other times, it is perhaps not important enough to warrant the ongoing performance cost, not to mention the up-front cost of hardware to support it. 

Take, for instance, the aforementioned virtual machine host use-case. The VM's in question may be important, but a good backup system may be in place, the services they offer unlikely to be severely impacted by the loss of a few minutes of data, or even services that essentially do not change, meaning a restore of a backup from the prior day would work just as well. 

If the restoration process only takes an hour, and the time the VM can be offline before it is important is longer, and the VM, once restored, would be of sufficient service (even having lost some amount of recent data), then the costs involved in delivering fully ZIL-backed-up ZFS storage underneath the VM may be higher than they are worth. 

The only time having a ZIL matters is if the ZFS server itself loses power, and once it has been restored, the only data lost would be in-transit data (so, at best, 1 or 2 transaction group commits worth of seconds of data). In most file server situations, you won't have any issue other than recently updated or in-the-process-of-being-updated files would be affected at all. In situations where the storage is hosting things like virtual hard disks for VM's, the filesystems on top of those virtual disks (be they zvols or files within an NFS share) may experience some level of loss. 

Depending on the filesystem sitting on top of those zvols or vhd files, and what was in transit at the time, the damage may be negligible. I've seen VM's come back up without a single warning, and when they do, the very common scenario is that the filesystem merely complains and needs to be fsck'd or chkdsk's, and the data lost is zero or not noticeably important (last few seconds of a log file, for instance).

I'm not suggesting that data integrity is unimportant - but it is worth looking at the overall environment before deciding that the storage in question truly requires ZIL mechanics to keep from losing a few seconds of data. In many environments, it doesn't. Also remember that in such environments, you don't have to go all or none - if you set sync=always on the datasets that matter, and intentionally set sync=disabled on datasets where it does not, a single pool can fulfill both sorts of situations. ZFS itself should (barring serious hardware problems) never have a problem; rather the data in the dataset was ZIL-backed or not, ZFS itself is, due to its atomic nature, always fine after the power is restored - it cannot by its design require a 'fsck'.

In closing, I'd also like to make another point - if you use a log device, and properly configure the pool or your clients to send all important data in such a manner that it makes use of the ZIL, and ZFS' own built-in integrity saves you from almost any disaster.. why would you need backups? Answer: because you need backups! Pools CAN be corrupted beyond reasonable recovery (there are a few very gifted ZFS experts out there willing to help you, but their time is precious, and your data may not be worth enough to afford their rates), and perhaps more importantly, the data on the pool can be destroyed in oh so many ways, some of which are flat out unrecoverable.

Accidental rm -rf or intentionally even? Hacker? Exploit? Client goes nuts and spews bad data and you didn't notice and didn't have a snapshot pre-crazy (or, even if you did, no easy way to recover from it due to environment)? SAN itself explodes? Is melted? Is shot by Gatling gun? Controller goes nuts and spews bad data at disks for hours while you're on vacation?

It is a simple fact of IT sanity that a comprehensive backup strategy (one that handles not only backing up the data, but making it quick and easy to restore as well) is a necessity for any production infrastructure you put up. Since this is a fact, and you are going to do it or rue the day you chose otherwise, you should probably remember that because you have it, you might not actually need a log device nor even ZIL mechanics, at least on some of your datasets (and every dataset you set sync=disabled on is another dataset with a bit more ZIL IOPS available for it to use instead). Carefully weigh risk and potential damage caused by loss of in-flight data as well as time to restore and how critical the service is before determining if ZIL mechanics are necessary.

121 comments:

  1. Hi you. I has some don't understand. When I increase timeout time and set sync=standard, the iops read and write increase lots. Any risk for this?

    ReplyDelete
  2. Hey, thanks for this post. I've been researching the ZFS ZIL aspect for a few days now and this is one of the better well-rounded, well worded articles that I've come across. I'm glad you decided to repost it!

    ReplyDelete
  3. I appreciate your efforts because it conveys the message of what you are trying to say.
    It's a great skill to make even the person who doesn't know about the subject could able to understand the subject .
    Your blogs are understandable and also elaborately described. I hope to read more and more interesting articles from your blog. All the best.
    Java training in Chennai
    Java training in Bangalore
    Java online training
    Java training in Pune
    Java training in Bangalore|best Java training in Bangalore

    ReplyDelete
  4. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  5. شركة تنظيف بالاحساء

    شركة تنظيف بالاحساء هي واحدة من الشركات الكبرى في مجال النظافة ويرجع ذلك الأمر إلى ضرورة توفير خدمات التنظيف لمختلف المؤسسات الحكومية والخدمية وكذلك توفر خدماتها للمنازل بمختلف احجامها وأنواعها لذا تهتم شركة تنظيف بالاحساء بالعمل المستمر على تطوير نفسها بشكل كبير في عملية التنظيف كما أنها اتجهت إلى توفير مجموعة كبيرة من العمال القادرين على تقديم أفضل مستويات خدمة التنظيف وبشكل سريع.

    شركة تنظيف مجالس بالاحساء
    شركة تنظيف كنب بالاحساء
    شركة تنظيف شقق بالاحساء
    شركة تنظيف سجاد بالاحساء
    شركة تنظيف بالاحساء
    شركة تنظيف منازل بالاحساء
    شركة تنظيف فلل بالاحساء
    شركة تنظيف بيوت بالاحساء

    ReplyDelete

  6. تعتبر شركة مكافحة حشرات بالاحساء من أفضل الشركات التي تقدم أكفأ الخدمات لتسهيل عملية التواصل بين الشركة و العميل و الرد على جميع الإستفسارات للعمل على إنتاج الكفاءة العلمية و الخبرة المهنية و الأمانة و الإخلاص في العمل و قد يكون من الصعب التخلص من كافة أنواع الحشرات لكن الطريقة المثلى لمكافحتها هي الحصول على مساعدة مهنية و بشركة متخصصة و رائدة في إدارة عملها بحيث يجري التقني بحثاً دقيقاً جداً للتأكد من وجود الحشرات و أماكن التواجد و يمكن أن يستعمل بعد ذلك مبيدات خاصة أو نظام تجميد سريع للتخلص من شتى الأنواع الطائرة و الزاحفة تواصل معنا الآن .
    شركة مكافحة الصراصير بالاحساء
    شركة مكافحة النمل الابيض بالاحساء
    شركة مكافحة حشرات بالاحساء
    شركة مكافحة بق الفراش بالاحساء
    شركة مكافحة الفئران بالاحساء

    شركة رش مبيدات بالإحساء
    شركة تركيب طارد حمام بالاحساء

    ReplyDelete
  7. تقوم شركة تسليك مجاري بالاحساء بخدمات متعددة منها تسليك المجاري بأحدث أجهزة الشفط حيث تفرغ المجاري كي تقوم بتسليكها من خلال كافة الطرق المعتمدة من أكبر الشركات العالمية و يقوم طاقم العمل بتنظيف المجاري من جميع الأوساخ المتراكمة بها من خلال أيدي عمالة محترفة و ذو خبرة فائقة و كفاءة مهنية عالية الجودة و تقوم الشركة بخدمات التخلص من الحشرات المتجمعة حول المجاري بإستخدام مبيدات و مواد فعالة تقضي تماماً على الحشرات و يقوم العمال بتنظيف و تعقيم المجاري للتخلص نهائياً من الأوبئة و الأمراض التي تسببها مياه المجاري تواصل معنا الآن و ستحصل على كافة الخدمات التي تحتاج إليها بأيدي عمالة محترفة و مدربة على أعلى مستوى اتصل بنا فوراً .




    شركة تنظيف خزانات بالاحساء
    شركة تسليك مجاري بالاحساء

    ReplyDelete


  8. fantastic post from you guys that amazed me as I get lot to learn from your article. It is Really very informative and creative contents for all the person who is looking to gain knowledge from the article. Thanks for sharing in information
    Thanks.
    DedicatedHosting4u.com

    ReplyDelete
  9. Great. It is good to constantly coming up with creative ideas. Provides much needed knowledge. goal oriented blog posts and always tried to find creative ways to meet goals.

    Thanks
    Online affiliates

    ReplyDelete
  10. Good article.
    For Data science training in banglore,visit:
    Data science training in bangalore

    ReplyDelete
  11. Nice Blog.
    For Blockchain training in Bangalore, visit:
    Blockchain training in bangalore

    ReplyDelete
  12. For Data Science training in Bangalore, Visit:
    Data Science training in Bangalore

    ReplyDelete
  13. Visit for Data Science training in Bangalore:
    Data Science training in Bangalore

    ReplyDelete
  14. Visit for Python training in Bangalore:
    Python training in Bangalore

    ReplyDelete
  15. Nice Post thanks for the information, good information & very helpful for others,Thanks for Fantasctic blog and its to much informatic which i never think ..Keep writing and grwoing your self

    duplicate rc in delhi online
    duplicate rc in ghaziabad
    duplicate rc in online
    duplicate rc in greater noida
    duplicate rc in mumbai
    duplicate rc in bangalore
    duplicate rc in faridabad
    duplicate rc in gurgaon
    duplicate rc in noida
    death certificate online

    ReplyDelete
  16. For Data Science training in Bangalore, Visit:
    Data Science training in Bangalore

    ReplyDelete
  17. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
    Python training in bangalore

    ReplyDelete
  18. For Data Science training in Bangalore, Visit:
    Data Science training in Bangalore

    ReplyDelete
  19. Thanku for sharing. For any Kind of Course Visit
    SourceKode Training Institute

    ReplyDelete
  20. DevOps Training course online will help you learn DevOps and master various aspects of software development, operations, continuous integration, continuous delivery, automated build, test and deployment. In this best DevOps online training course, you will learn DevOps tools like Git, Puppet, Jenkins, SVN, Maven, Docker, Ansible, Nagios and more.

    ReplyDelete
  21. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    best angular js online training

    ReplyDelete
  22. I really enjoyed your blog Thanks for sharing such an informative post...
    python training in bangalore - eCare Technologies located in Marathahalli - Bangalore, is one of the best Python Training institute with 100% Placement support. Python Training in Bangalore provided by Python
    Certified Experts and real-time Working Professionals with handful years of experience in real time Python Projects.


    ReplyDelete
  23. I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously in their life, he/she can earn his living by doing blogging.thank you for thizs article. devops online training

    ReplyDelete
  24. Nice Post thanks for the information, good information & very helpful for others,Thanks for Fantasctic blog and its to much informatic which i never think ..Keep writing and grwoing your self

    rc transfer in delhi
    rc transfer in ghaziabad
    rc transfer in noida
    rc transfer in gurgaon
    rc transfer in faridabad
    rc transfer in bangalore
    rc transfer in greater noida
    rc transfer in mumbai
    rc transfer in ranchi
    rc transfer in chennai

    ReplyDelete
  25. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    best microservices online training
    top microservices online training
    microservices online training

    ReplyDelete
  26. I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously in their life, he/she can earn his living by doing blogging.thank you for thizs article. pega online training , best pega online training ,
    top pega online training

    ReplyDelete
  27. Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing. learn devops online

    ReplyDelete
  28. I'm very much inspired when I've visited your blog. Your blog is informative. Hope you will continue with the new article.
    SQL Azure Online Training
    Azure SQL Training
    SQL Azure Training

    ReplyDelete
  29. اهلاً ومرحباً بكم عملائنا الكرام نحن نقدم خدمان منزلية مميزة وذات ضمان وجودة عالية جدا عليكم بالتواصل معنا الأن وسوف نلبي طلباتكم بكافة تفاصيلها عن طريق بعض الروابط الخاص بالموقع الخاص بالشركة :.
    شركة عزل اسطح بابها
    شركة نقل عفش بابها
    شركة عزل خزانات بابها
    شركة تنظيف مجالس بابها
    شركة تنظيف شقق بابها
    شركة مكافحة النمل الأبيض بابها
    شركة ترميم منازل بابها
    شركة عزل اسطح بابها

    ReplyDelete
  30. Attend The Machine Learning course Bangalore From ExcelR. Practical Machine Learning course Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Machine Learning course Bangalore.
    ExcelR Machine Learning course Bangalore

    ReplyDelete
  31. I wanted to thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post.
    Please check this Machine Learning Course in Pune

    ReplyDelete

  32. Excellent Blog! I would like to thank for the efforts you have made in writing this ExcelR Machine Learning Courses post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!

    ReplyDelete
  33. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck... Thank you!!! machine learning courses in Bangalore

    ReplyDelete
  34. Poker online situs terbaik yang kini dapat dimainkan seperti Bandar Poker yang menyediakan beberapa situs lainnya seperti http://62.171.128.49/hondaqq/ , kemudian http://62.171.128.49/gesitqq/, http://62.171.128.49/gelangqq/, dan http://62.171.128.49/seniqq. yang paling akhir yaitu http://62.171.128.49/pokerwalet/. Jangan lupa mendaftar di panenqq salam hoki

    ReplyDelete
  35. Attend The Machine Learning course Bangalore From ExcelR. Practical Machine Learning course Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Machine Learning course Bangalore.
    ExcelR Machine Learning course Bangalore

    ReplyDelete

  36. Wow. That is so elegant and logical and clearly explained. Brilliantly goes through what could be a complex process and makes it obvious.I want to refer about the best tableau online training

    ReplyDelete
  37. NAGAQQ | AGEN BANDARQ | BANDARQ ONLINE | ADUQ ONLINE | DOMINOQQ TERBAIK

    Yang Merupakan Agen Bandarq, Domino 99, Dan Bandar Poker Online Terpercaya di asia hadir untuk anda semua dengan permainan permainan menarik dan bonus menarik untuk anda semua

    Bonus yang diberikan NagaQQ :
    * Bonus rollingan 0.5%,setiap senin di bagikannya
    * Bonus Refferal 10% + 10%,seumur hidup
    * Bonus Jackpot, yang dapat anda dapatkan dengan mudah
    * Minimal Depo 15.000
    * Minimal WD 20.000

    Memegang Gelar atau title sebagai Agen BandarQ Terbaik di masanya

    Games Yang di Hadirkan NagaQQ :
    * Poker Online
    * BandarQ
    * Domino99
    * Bandar Poker
    * Bandar66
    * Sakong
    * Capsa Susun
    * AduQ
    * Perang Bacarrat (New Game)


    Info Lebih lanjut Kunjungi :
    Website : NAGAQQ
    Facebook : NagaQQ Official
    WHATSAPP : +855977509035
    Line : Cs_nagaQQ
    TELEGRAM : +855967014811

    BACA JUGA BLOGSPORT KAMI YANG LAIN:
    agen bandarq terbaik
    Winner NagaQQ
    Daftar NagaQQ
    Agen Poker Online

    ReplyDelete
  38. CROWNQQ I AGEN BANDARQ I BANDARQ ONLINE I ADUQ ONLINE I DOMINOQQ TERBAIK
    Yuk Buruan ikutan bermain di website CrownQQ

    Sekarang CROWNQQ Memiliki Game terbaru Dan Ternama loh...

    9 permainan :
    => Poker
    => Bandar Poker
    => Domino99
    => BandarQ
    => AduQ
    => Sakong
    => Capsa Susun
    => Bandar 66
    => Perang Baccarat (NEW GAME)

    => Bonus Refferal 20%
    => Bonus Turn Over 0,5%
    => Minimal Depo 20.000
    => Minimal WD 20.000
    => 100% Member Asli
    => Pelayanan DP & WD 24 jam
    => Livechat Kami 24 Jam Online
    => Bisa Dimainkan Di Hp Android0619679319
    => Di Layani Dengan 5 Bank Terbaik
    => 1 User ID 9 Permainan Menarik

    Ayo gabung sekarang juga hanya dengan
    mengklick CrownQQ

    Link Resmi CrownQQ:
    ratuajaib.com
    ratuajaib.net
    ratuajaib.info

    BACA JUGA BLOGSPORT KAMI:
    Agen BandarQ Terbaik
    Winner CrownQQ
    Daftar CrownQQ
    Agen Poker Online

    Info Lebih lanjut Kunjungi :
    WHATSAPP : +855882357563
    Line : CS CROWNQQ
    Facebook : CrownQQ Official

    ReplyDelete
  39. Mau Cari Situs Judi Online Aman dan Terpercaya? Banyak Bonusnya?
    Di Sini Tempatnya.

    DIVAQQ, Agen BandarQ, Domino99, Poker Online Terpercaya DI INDONESIA

    100% MUDAH MENANG di Agen DIVAQQ
    Daftar > Depo > Main > Menang BANYAK!
    Deposit minimal Rp. 5.000,-
    Anda Bisa Menjadi JUTAWAN? Kenapa Tidak?
    Hanya Dengan 1 ID bisa bermain 9 Jenis Permainan
    * BandarQ
    * Bandar 66
    * Bandar Poker
    * Sakong Online
    * Domino QQ
    * Adu Q
    * Poker
    * Capsa Susun
    * PERANG BACCARAT (NEW)
    BONUS SETIAP MINGGU NYA
    Bonus Turnover 0,5%
    Bonus Refferal 20%

    MARI GABUNG BERSAMA KAMI DAN MENANGKAN JUTAAN RUPIAH!
    Website : DIVAQQ
    WA : +85569279910

    DIVAQQ AGEN JUDI DOMINO QQ ONLINE TERPERCAYA DI INDONESIA MENERIMA DEPOSIT VIA PULSA 24 JAM
    DIVAQQ | AGEN DOMINO PULSA TELKOMSEL | XL | AXIS 24 JAM
    DIVAQQ | AGEN JUDI DOMINO DEPOSIT BEBAS 24 JAM

    ReplyDelete
  40. I need to to thank you for your time due to this fantastic read!! I definitely enjoyed every bit of it and I have you bookmarked to see new information on your blog.
    Java Training in Bangalore
    Ui Development Training in Bangalore

    ReplyDelete
  41. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    welcome to akilmanati
    akilmanati

    ReplyDelete
  42. I read Your Post and trust me its really helpful for us. Otherwise if anyone Want SALESFORCE Training with Placement So You Can Contact here-9311002620
    Salesforce training institute in delhi
    Salesforce training institute in Noida
    Salesforce training institute in Faridabad

    ReplyDelete
  43. Such an ideal piece of blog. It’s quite interesting to read content like this. I appreciate your blog
    AWS Training
    AWS Online Training
    Amazon Web Services Online Training

    ReplyDelete
  44. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.satta king

    ReplyDelete
  45. Your work is very good and I appreciate you and hopping for some more informative posts. Thank you for sharing great information to us. Satta king

    ReplyDelete
  46. I have read your blog its very attractive and impressive. I like it your blog.
    DevOps Online Training
    DevOps Training
    DevOps Training in Ameerpet

    ReplyDelete
  47. Your blog is splendid, I follow and read continuously the blogs that you share, they have some really important information. M glad to be in touch plz keep up the good work.
    ExcelR Solutions

    ReplyDelete
  48. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

    AWS Online Training
    AWS Certification Training
    AWS Certification Course
    Python Training
    Python Course

    ReplyDelete
  49. I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…
    More Info of Machine Learning

    ReplyDelete
  50. Your article has piqued my interest. This is definitely a thinker's article with great content and interesting viewpoints. I agree in part with a lot of this content. Thank you for sharing this informational material.
    Best Data Science training in Mumbai

    Data Science training in Mumbai

    ReplyDelete
  51. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    mulesoft online training
    best mulesoft online training
    top mulesoft online training

    ReplyDelete
  52. i was willing to use this black theme at Mobile Mall

    ReplyDelete
  53. Excellent blog I visit this blog it's really awesome. The important thing is that in this blog content written clearly and understandable. The content of Java is very informative.

    Java training in chennai | Java training in annanagar | Java training in omr | Java training in porur | Java training in tambaram | Java training in velachery

    ReplyDelete
  54. Excellent! I love to post a comment that "The content of your post is awesome" Great work!
    digital marketing courses mumbai

    ReplyDelete
  55. Effective blog with a lot of information. I just Shared you the link below for Courses .They really provide good level of training and Placement,I just Had PHP Classes in this institute,Just Check This Link You can get it more information about the PHP course.


    Java training in chennai | Java training in annanagar | Java training in omr | Java training in porur | Java training in tambaram | Java training in velachery

    ReplyDelete
  56. Thanks for provide great informatic and looking beautiful blog, really nice required information & the things i never imagined and i would request, wright more blog and blog post like that for us. Thanks you once agian

    religion change
    arya samaj mandir in ghaziabad
    arya samaj mandir in greater bangalore
    name change ads in newspaper
    arya samaj mandir in punjab
    arya samaj mandir in rajasthan
    arya samaj mandir in noida
    arya samaj mandir in gurgaon
    arya samaj mandir in faridabad
    arya samaj mandir in delhi

    ReplyDelete
  57. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!...machine learning courses in bangalore

    ReplyDelete
  58. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!...bangalore digital marketing course

    ReplyDelete
  59. The content that I normally see is nothing like what you have written. This is very well-thought out and well-planned. You are a unique thinker and bring up great individualized points. Please continue your work.
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata
    SAP training institute Kolkata

    ReplyDelete
  60. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!...digital marketing course bangalore

    ReplyDelete
  61. I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more....bangalore digital marketing course

    ReplyDelete
  62. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    DevOps Training in Chennai | DevOps Training in anna nagar | DevOps Training in omr | DevOps Training in porur | DevOps Training in tambaram | DevOps Training in velachery

    ReplyDelete
  63. I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…
    Machine Learning Courses The web site is lovingly serviced and saved as much as date. So it should be, thanks for sharing this with us.

    ReplyDelete
  64. Thanks for sharing this information. I really Like Very Much.
    top devops online training

    ReplyDelete
  65. This is great information and all relevant to me. I know when I engage with my readers on my blog posts, not only does it encourage others to leave comments, but it makes my blog feel more like a community – exactly what I want! DevOps Training in Chennai | DevOps Training in anna nagar | DevOps Training in omr | DevOps Training in porur | DevOps Training in tambaram | DevOps Training in velachery

    ReplyDelete
  66. Thanks for the inspired use of both new image sets, and that sentiment stamp is a must have. Your cards are wonderful; first time on your blog, but I'll be back!!
    top Mobile App Development Companies in India

    ReplyDelete
  67. The MSME Ministry notified that Udyam Registration can be filed online based on self-declaration with no requirement to upload documents, papers, certificates or proof.

    Udyam Registration MSME

    UDYAM Registration Process

    GST Registration Process in India

    MSME Registration In India
    (Eligibility, Registration Process, Benefits, Schemes, Documents Required)
    Benefits Under MSME Registration (Udyog Aadhaar)
    A micro, small and medium enterprise (MSME) will now be known as Udyam Registration in India.

    SEO Expert & Digital Marketing in India
    Best Digital Marketing Company in India

    Property Dealer In Sri Ganganagar Real estate, Property, Rental in Sri Ganganagar

    Website Design and Development Company in USA

    ReplyDelete
  68. One of the best articles ever. Honestly, it deserves more and more depth. I had the pleasure of reading your article and I am proud of you for what you offer in your blog. You are an example of blogs that provide useful content.
    من أفضل المقالات على الإطلاق صراحة تستحق التعمق أكثر واكثر لقد سعدت بقراءة مقالك وانا فخور بك وبما تقدمه في مدونتك ف انت مثال للمدونات التي تقدم محتوي مفيد

    تنظيف موكيت بالدمام
    شركة تنظيف موكيت بالدمام
    شركة تنظيف فلل بالدمام
    شركة تنظيف بيوت بالدمام
    شركة نظافة بالدمام
    شركة تنظيف منازل بالدمام
    شركة نظافة عامة بالدمام
    شركة تنظيف شقق بالدمام
    شركة تنظيف كنب بالدمام
    شركة تنظيف مجالس بالدمام
    شركة تنظيف منازل بالاحساء

    ReplyDelete
  69. Thanks for sharing this information. I really Like Very Much.
    devops online training

    ReplyDelete
  70. Reading this article gave me many things to think about. You have some quality information here that any reader would enjoy. I share many of your views in this article.
    Data Science training in Mumbai
    Data Science course in Mumbai
    SAP training in Mumbai

    ReplyDelete
  71. Thanks for sharing this information. I really Like Very Much.
    devops online training

    ReplyDelete
  72. Thanks for an interesting blog. What else may I get that sort of info written in such a perfect approach? I have an undertaking that I am just now operating on, and I have been on the lookout for such info.
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata

    ReplyDelete