ZFS Essentials: Array Disk Conversion to ZFS or Other Filesystems - No Data Loss, No Parity Break!

Summary

This video is a part of the ZFS series by Spaceinvader One, focusing on converting an array disk to ZFS or any other file system without data loss or breaking parity. In this tutorial, viewers learn to reformat a disk within an Unraid array, utilizing ZFS to enhance their data storage strategies. The process is demonstrated step-by-step, ensuring data integrity through the usage of tools like Unbalance, and viewers are guided through potential pitfalls and how to overcome them. The tutorial is tailored towards users who have already engaged with previous content in the series, aiming to upgrade their storage without requiring additional hardware purchases, while also setting the stage for future explorations into ZFS's advanced features. By the end, users are prepared to integrate a single ZFS disk into their Unraid setups effectively.

Highlights

• Add a ZFS disk to your Unraid array without buying new disks. 🎉
• Ensure your data is safe by moving it using the Unbalance plugin. 🔄
• Understand the importance of keeping parity intact during conversions. ⚖️
• Leverage basic ZFS features like compression and encryption. 🔒
• Look forward to using ZFS as a replication target for backups. 🔄

Key Takeaways

• Convert your array disks to ZFS or other file systems without losing data! 🔄
• No need for new disks; just ensure enough free space in your array. 💽
• Use the Unbalance plugin to safely move data off the disk to be reformatted. 📦
• Maintain parity and data integrity throughout the reformatting process. ✅
• Explore ZFS's advanced features and get ready for future enhancements! 🚀

Overview

In this engaging video, Spaceinvader One takes us through the process of converting an array disk to ZFS in an Unraid setup, without the need for additional disks. He highlights the importance of maintaining data safety while introducing viewers to the impressive capabilities of ZFS, even when using a single disk setup. The video demystifies ZFS's advanced features, readying viewers for more complex configurations.

Step by step, viewers learn to utilize tools such as the Unbalance plugin to safely relocate data, ensuring no data loss occurs during the disk reformatting. Spaceinvader One emphasizes the crucial role of parity in protecting data integrity, explaining the XOR algorithm in a simple, digestible manner. This video is a goldmine for anyone looking to safely upgrade their storage capabilities without compromising on data safety.

As the tutorial progresses, the groundwork for future ZFS exploration is laid. Spaceinvader One concludes by setting the stage for using ZFS disks as replication targets, inviting viewers to dive deeper into the capabilities of this powerful file system. This video serves as both a practical guide and an invitation to harness the full potential of ZFS in your storage solutions.

Chapters

• 00:00 - 01:00: Introduction and Video Overview The chapter "Introduction and Video Overview" begins with an expressive introduction featuring sounds and music, meant to capture the audience's attention. The host then addresses the viewers in a friendly and welcoming manner. The chapter is part of a video series that focuses on ZFS on the Unraid platform. It elaborates on the process of converting a pool device to a ZFS format. Additionally, it discusses techniques for automatically converting top-level folders into datasets and the option of manual conversion.
• 01:00 - 02:30: Automating Snapshots and Adding ZFS Pool In this chapter, the focus is on automating snapshots and adding a ZFS pool. The chapter begins by addressing feedback from previous videos, indicating viewer interest in learning how to automate snapshot creation and replicate data across ZFS pools. The content also touches on using ZFS replication, a topic well-appreciated by many users with multiple ZFS pools. However, for those who do not have more than one ZFS pool, the chapter reassures them of the value of understanding and getting at least one ZFS pool running.
• 02:30 - 04:00: Formatting a Disk without Losing Data This chapter covers the process of adding a second Z pool to a server using ZFS, without the need to purchase new disks. It emphasizes the importance of having more free space than the smallest data drive in the array, as the data will be moved off one disk onto the others in the array before proceeding.
• 04:00 - 06:00: Installing Necessary Plugins This chapter explains how to handle a single disk stripe in ZFS without breaking parity or losing data by incorporating it into an unraid array, which provides protection through unraid parity.
• 06:00 - 09:00: Using Unbalance to Move Data This chapter discusses the trade-offs involved in using ZFS with Unraid. While features like automatic data healing and high speed might not be available, ZFS can still offer compression and encryption options. It can also serve as a replication target for backups. The speaker emphasizes the importance of having at least one ZFS disk in your Unraid array for these benefits, suggesting that the explanations are best understood through practical demonstration.
• 09:00 - 12:00: Reformatting the Disk with ZFS In this chapter titled 'Reformatting the Disk with ZFS', the focus is on configuring a test server with three XFS disks. The instructor demonstrates how to reformat one of the disks to use the ZFS file system. Initially, the disk contains some shares, and the process involves clearing all existing data to prevent data loss during reformatting. This chapter provides a practical example of disk management and server configuration.
• 12:00 - 15:00: Setting Up ZFS Backups In the chapter titled 'Setting Up ZFS Backups,' the process begins with the need to install a new plugin. The user navigates to the apps tab to search for and install the plugin named 'unbalance.' Once installed, the window is closed. Before using the plugin, the user goes to the docker tab to ensure all containers are stopped, ensuring nothing is utilizing the disk at that moment.
• 15:00 - 16:00: Conclusion and Next Steps The chapter begins with a reminder to ensure that all VMS are stopped before proceeding with any disk reformatting operations. This step is crucial if the disk in question had a system share present on it as it requires stopping both docker and VM services. However, it's also noted that typically, the system share is not found in these scenarios, mitigating this concern for most users. The chapter concludes by noting the necessity of installing an additional plugin, indicating further steps or considerations are necessary for completing the procedure.

ZFS Essentials: Array Disk Conversion to ZFS or Other Filesystems - No Data Loss, No Parity Break! Transcription

• 00:00 - 00:30 foreign [Applause] [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] hi there guys so I hope you're all doing well now let's continue the series of videos where we're looking at ZFS on unrained now we've seen how we can convert a pool device to be formatted in ZFS and we've seen how within the data set we can Auto convert the top level folders into data sets and manually take
• 00:30 - 01:00 a snapshot from the comments received in the last video there's a lot of people looking forward to the next video where they're going to be automating snapshots and then replicating data from one's equal to another now I love CFS replication but some guys out there some of you you may not have two z-pools now if you followed along in this series then you've probably at least got one C pull at the moment your cash Drive is likely to be ZFS and with any luck all
• 01:00 - 01:30 of your app data that's been converted to data sets so you're ready to start using some of zfs's more advanced features so in this video so we're going to add a second Z pull to the server and for this you're not going to need to buy any new disks for the server the only thing you're going to have to have is more free space than the size of the smallest Data Drive in your array because what we're going to do is move all of the data off this disk onto the other drives in the array and then we're
• 01:30 - 02:00 going to reformat this drive but we're going to do it without breaking parity or without losing data and a z pull with just one disk is generally known as a single disk stripe in normal ZFS this doesn't have any redundancy at all but because this drive is actually going to be part of the unraid array the unraid parity is actually going to protect the disk should it fail now obviously a z pull with only one drive you do lose a
• 02:00 - 02:30 lot of the features of ZFS you don't get automatic data healing you don't get super fast speed but we can use things like compression encrypt option and we can use it as a ZFS replication Target to back up things so in my opinion it's really great to be able to have at least one ZFS disk inside your unraid array okay I'm going to stop trying to explain now I think it's much easier if we just jump right in do it and I'll show you as we go along
• 02:30 - 03:00 okay so here we are back on the test server and we've got these three xfs disks making up the array now obviously you can see there's not much data on them this is just a test server but what I want to do is I want to format disk one and have this as the ZFS disk now at the moment if we look there are a few shares that are on this drive so basically we need to clear everything off because if I go and format this drive now well I'm going to lose this
• 03:00 - 03:30 data so we're going to need to install another plugin so we need to go across to the apps tab here and we want to do a search for unbalance so let's install this plugin and with that done we can close this window now before I go and use this plugin I'm just going to go to the docker tab here and I'm going to make sure that all of these containers have stopped I don't want anything using that disk
• 03:30 - 04:00 one at all and also the same with VMS just make sure all of those are stopped as well now if for any reason the disk that you want to reformat if it had the system share on it well just like when we reformatted the cache Drive you'd have to actually stop the docker service and stop the VM service but normally you won't find the system share here so I don't have to worry about it now there's another plugin we need to make sure we've got installed and I'm
• 04:00 - 04:30 sure most of you will probably have this anyway it's to fix common problems plugin so just make sure that's installed now you might be wondering why we need this well if we go across to the tools tab here now installing fixed common problems gives the stock a safe new permissions we're going to click onto that and then I'm going to run this just to make sure all the permissions on my Shares are as they should be okay so that didn't take very long for
• 04:30 - 05:00 me with a lot of data obviously it will take quite a long time but when that's done just click onto done then click onto settings scroll down here and here we can see unbalance let's click onto its icon and at the moment it's not enabled so let's change no TS click apply and now we can open its web GUI here so let's click onto the green writing and here we can see the drives I have in the server now what I want to do is use the setting
• 05:00 - 05:30 here called scatter and the drive I want to move everything off is disk one so I'm going to check here from and then in the column here too we can choose which Drive the files from the from are scattered to now I don't want anything going on to the cache Drive I just want it on the other array drives so these two drives are selected here now you see I've selected the drive I also need to select the shares inside of there as
• 05:30 - 06:00 well so I've selected everything that's in disk one I'm going to click onto plan now and what planning does is just make sure it can fit everything where it needs to go and check the permissions are correct so everything looks good here to me so I'm going to untick dry run because I don't just want it to go through and do nothing and we don't want to copy the files we want to actually move them so I'm going to click on to move here and then we just let unbalance do its thing we can see here for me it's not going to
• 06:00 - 06:30 take very long because there's only a tiny amount of data but obviously for you guys there's going to be more data so it's going to take a lot longer okay so we can see it's all finished now it took 17 minutes all of the files are copied across so I can close the unbalanced window now and let's go back to the main tab click onto disk one and we can see some folders are left over going inside them well there's nothing there because all the data's been moved so I'm just going to delete the folders
• 06:30 - 07:00 if you want to be 100 sure there's nothing in it select them all click on to calculate and you can be double sure by checking the size there's nothing there so I'm going to click on to delete and now that drive has been emptied so what we can do now is we go back to the main tab okay so now it's time to shut down the array but before I do that I'm actually going to reset all of the disk stats here this isn't any part of actually reformatting the drives I just want to clear the stats here so
• 07:00 - 07:30 everything's set to zero so I can just mention how parity works and how reformatting the drives is not going to affect the parity and so all of the reading and writes are set to zero because when we reformat disk one in a different file system the way parity works it works on What's called the xor algorithm which basically just calculates whether the ones and zeros across all of the disks are either odd or even if you want to know more about parity then see my video here so as far
• 07:30 - 08:00 as the parity is concerned reformatting is no different than if we added a bunch of files to disk one and we can see that by seeing the various reads and writes as we go through the process anyway let's scroll down and actually stop the array and so with the array stop now I'm going to scroll down here and I'm going to click maintenance mode and then start this will start the array but without
• 08:00 - 08:30 mounting the disks now I'm going to select the disk which I want to format now obviously make double sure you select the correct disk mine's disk one so I'm going to click onto the name of the disk here because we're in maintenance mode here we're able to erase the disk now I suggest you do that and click onto erase now here we have to type the name so I'm going to type disk one and click on to proceed and now we've erased the disk there's no
• 08:30 - 09:00 file format on it we can see that it says the file system type is auto so now I'm going to click onto done I'm going to scroll back down and now I'm going to stop the array here and now with the array stopped I'm going to go back to the disk here click onto its name and now because it's not in maintenance mode I can choose a file system now remember you can choose any file system you want it doesn't just have to be CFS maybe you want to reformat your
• 09:00 - 09:30 array disks to be encrypted and you want to use xfs encrypted maybe you want your CFS disk to be Luke's encrypted so you'd select this one I want mine just to be regular ZFS and I'm going to enable the compression so with that done I'm going to click on to apply and done we can see here the file system is going to be CFS so let's scroll down and start the array now as the array starts it's going to ask us to format the disk that's
• 09:30 - 10:00 perfectly normal so check the check box here you can ignore this warning and click onto format and this will now format for me my drive in ZFS we can see here that it's formatting and we'll see the rights jump right up on this drive in a minute we can see here the reads and writes changing on this disk and we can see the parity rights go up I'm not sure if you saw that there so our disc is now ZFS our parity is valid and we've lost no data now if you want
• 10:00 - 10:30 to you could just run a parity sync if you wanted to be 100 sure I'm not actually going to do that but I'm just going to quickly run maybe just a minute or two just to show you that there's no errors when it first runs [Music] see here there's no sync errors because we erase the disk and how we went through the process so that means parity is valid and everything's good now if you did see any sync errors here just let the parity sync finish all the way through and it will correct them if you've done
• 10:30 - 11:00 everything properly then hopefully you won't see any at all anyway I'm going to cancel this and so basically that's it it's really easy to reformat a disk in the array so now if I scroll down here in CFS Master we can see there's now two different z-pools we got the one called cache here because it's the name of the cash pool here and the second pool is called disk one taking its name from obviously disk one at the top here
• 11:00 - 11:30 now at the moment obviously there's no data sets here because it's a freshly formatted drive so I'm going to create run ready for the next video to do that I'm going to go to shares here I'm going to click add share and I'm going to call it ZFS hyphen backups primary storage I want it to be the array now for included disks I only want it to be my ZFS disk one that's important so I'm going to select that I don't want any secondary storage and I'm going to click add share
• 11:30 - 12:00 okay so with that done obviously let's click on done and let's go to the main tab again and if we scroll down here now we can see we've got our ZFS backups data set and in the other Z Pool we've got all of these data sets here which in the next video we're going to replicate across into this data set here therefore backing up all of our app data onto the array but that's the next video anyway guys that's it for now I really
• 12:00 - 12:30 hope you enjoyed this video and you found it useful now I'm really looking forward to showing you the next video with the CFS replication but if you enjoyed this video then well please as always hit that like button as all YouTubers say apparently it helps the algorithm and if you know anyone who might also find this video useful and like it then please share it with them and I want to give a big thank you to all of my patrons and supporters out there without you guys I really wouldn't be able to make these videos and I just want to really thank
• 12:30 - 13:00 you for all of your help and support anyway guys it's time for me to go now but whatever you're up to for the rest of the day I hope it's good and I'll catch you in the next video