RAID 6 --> ZFS pool

Today I completed the transition of my NAS machine from RAID 6 to a ZFS pool.  To document the process for my future self and any semi-interested rando who stumbles across this, here is an outline of how I did it.

Part 1 - Research

I found this 'ZFS for Dummies' post really useful to re-familiarise myself with the core concepts.  Once I'd done that, the Ubuntu examples here were really useful as well as this post with more detailed information than I would ever need.

My RAID was nowhere near full, which meant I could manage the process without having to restore everything from backups (although I made damn sure I had working backups before I messed with it).

Part 2 - Degrading the RAID and pooling the freed-up drives

First, I marked two drives from the RAID as failed so I could use them in the zfs pool.  I changed their partition type with cfdisk but I had trouble making a pool out of them because they kept on being picked up and re-started as a raid device.  In the end, I stopped this happening by using dd to nuke the start of each partition:

dd if=/dev/zero of=/dev/sdc1 bs=4M count=32

Then zfs would finally shut up and make a mirror out of those two partitions.  I chose to use partitions instead of whole disks because I want more flexibility with the size/brand of the actual disks I use.  I also used partition IDs rather than device names because names like /dev/sdb can get swapped around if they end up cabled into different SATA ports, but partition IDs should remain constant.

An example command for this is this:

zpool create -f sharepool mirror /dev/disk/by-partuuid/0008fc4d-01 /dev/disk/by-partuuid/000b5420-01 

Part 3 - Copying the RAID data across

As I've already mentioned, this was only possible because I was using a tiny fraction of the RAID.  However, it made this step as simple as copying from the degraded RAID to the new zfs pool:

cp -rp /raid /sharepool

Part 4 - Euthanise the RAID

With the data safely copied across, I didn't need to mark drives as failed any more.  I just stopped the raid, changed the partition types and nuked the start of each partition as in Part 2.

Part 5 - Add further drives to the zfs pool

Again, I used partition IDs rather than device names, but this was as simple as using 'zpool add' instead of 'zpool create'.  Now the pool is made up of two vdevs, each made up of two mirrored drives so I have redundancy and still more than enough space.

Part 6 - Sorting the backup drive

My RAID setup was really quite weird: 4 x 1.5TB drives and a 2TB drive (with a 1.5TB partition on it).  The remaining space on the 2TB drive was previously used as a temporary backup whilst monkeying around with the other drives.

I wanted to set up the 2TB drive to hold a rolling backup of some of the stuff on the zfs pool, but I didn't want to lose the stuff in the last 0.5TB of it.

Doing this was not strictly necessary because I already had all the RAID data safely on the new zfs pool, but I am paranoid.

So I made a new filesystem on the 1.5TB partition of the 2TB drive.  Then I copied the stuff backed up in the last 0.5TB down to the new filesystem.  I could then delete the old 0.5TB partition, expand the 1.5TB partition to the full 2TB and resize2fs to expand the file system to fill the partition.

Part 7 - Setting up regular backups

Most of the data on my NAS was music ripped from CDs that I don't want to have to rip again (gods that would take ages :S).  Another whole bunch of files is years and years of irreplaceable photos.  I don't need to rsync these because the only way their contents will change is if the entire file has changed (rsync only copies the updated parts of each file).  

Instead, I'm using cp -R -u -p.  The -u bit is the important one: it is the 'update' flag which makes it only copy files that either don't exist in the target directory, or are newer than the target directory's version.

So I tested the command works, then added it via crontab -e and added a test file to the pool so I can check tomorrow that it got copied across!





Comments

Post a Comment

Popular posts from this blog

Micro:Bit and SPI display

I want to explore the full potential of the BBC Micro:bits my school has. Making rock-paper-scissors games and the like is all well and good, and the students love it (for the most part), but it always feels a little bit gimmicky.  But when students really get engaged and suggest ideas for them (e.g. games, compass display), it all gets a little bit cramped and feels like it could  kinda  work, but not really . Then I found out that all the tiny strips on the edge are actually GPIO pins, not just the fat 0, 1 and 2 ones.  Since then, I've wanted to explore what I can do with it. Hence, over the last few days I have been trying to get the microbit to control a TFT display. I first went for this 2.4" display with a touchscreen.  The SPI interface is just for the microSD, not for the screen but I didn't think that would be a problem because the micro:bit has so many pins.  After much frustration, I discovered that there are many of the pins you can't reliably use extern
Read more

DCS World with TrackIR under Ubuntu

This is a record of my experience of setting this up.  It's meant more as a record for my future self than a tutorial, but maybe somebody will find it useful. 1 - Install DCS First step, install DCS World via the Steam client as normal.  2 - Install LinuxTrack Linuxtrack Download here: https://github.com/uglyDwarf/linuxtrack/wiki/Downloads   Instructions here: https://github.com/uglyDwarf/linuxtrack/wiki/universal-Linuxtrack-package   Using the gui (ltr_gui) requires qt4 libraries, which I could not find in my standard Ubuntu repositories, so I had to use the ppa instructions here: https://ubuntuhandbook.org/index.php/2020/07/install-qt4-ubuntu-20-04 Run ltr_gui from linuxtrack's bin directory to check everything works so far before moving on to get it running under Proton.  3 - Install LinuxTrack into the DCS World prefix NOTE: Attempting to install for Wine games WON'T WORK e.g. here https://github-wiki-see.page/m/uglyDwarf/linuxtrack/wiki/Wine-Games   This is because Ste
Read more

Cardboard Mock-up of USB Joystick

Image
For years now (literally years ), I have been playing around with the idea of making a USB joystick.  You may have seen my previous experiment in building a throttle control . I wanted to use a similar idea to the throttle, taking the angular motion of the handle and magnify it when applying it to the potentiometer.  e.g. move the handle through 45 degrees and the potentiometer gets moved through 90 degrees. I also wanted it anchored at the rear, so I can build a mount to attach it to my computer chair. In the video you can see a cardboard mock-up I made today.  I'm amazed it worked at all, but it even provides a solid foundation for designing the proper plywood one. The process of making a cardboard mock-up was really useful.  There were three fatal flaws in my sketch design which I would not have found out until I tried to make it: Pushing the joystick left/right when it was already pulled full back makes it catch on the rear support. The internal supports in the
Read more