ZFS Degraded Pool

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • bookie56 wrote:

      I have redone another server....and would like to transfer files from my zfs server....Yes, I can do that via network....but takes too long when it is several TB'sI was wondering about mounting the zfs pool from a Debian installation...I have a flash drive with Debian 9 and zfs installed...just not sure how I could best mount this from my Debian installation so that file transfer to another drive connected to the server can work.....
      I have mounted mdadm raid 5 from my Debian USB but never tried zfs....
      Got any suggestions?
      bookie56
      I'm not sure what you're asking - exactly.
      ______________________________________________

      Where ZFS is concerned, I wouldn't export a pool. If the OMV ZFS plugin is installed (we'll say on a 2nd USB boot drive), if the boot drive boots up with a foreign ZFS pool attached it would likely import it. However, while I can't specify the exact reasons, I would see that as being somewhat risky. (I.E. I wouldn't do it without full backup which is what I think you're trying to achieve, not something you currently have.) My concern would be that the 2nd boot drive may write a change to the pool that might be incompatible or not recognized by the first boot drive. (Hey,, when it comes to data without solid full backup, I'm extremely risk adverse.)

      Are you in a hurry? If it was me I'd use the network. You won't have to alter configurations to do it that way - a straight server to server transfer. It's safe. You could do it one share at a time and let it run all night or over a weekend. (Friday evening to Monday morning is a big chunk of time.) With the "keep partially transferred files" flag set, even if there was an error along the way, eventually it would get done.

      (Interestingly, after looking, it appears that's there's no straight forward way of stopping an rsync job other than killing the running process.)
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • This BTRFS uncorrectable errors situation doesn't look good. Since scrub results are based on file checksums, it would seem the solution would be - find the files that are corrupted, delete them and replicate them from the (bitrot protected) source again. However, finding the names of the files themselves is not straight forward. What a PITA.

      I guess it's time to thoroughly test the disk and dive into the syslog.
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • flmaxey wrote:

      This BTRFS uncorrectable errors situation doesn't look good. Since scrub results are based on file checksums, it would seem the solution would be - find the files that are corrupted, delete them and replicate them from the (bitrot protected) source again. However, finding the names of the files themselves is not straight forward. What a PITA.

      I guess it's time to thoroughly test the disk and dive into the syslog.
      Good luck with that....A while ago I had an electrician put a separate line in my office at home to give me protection in case of problem anywhere else in the flat....only problem is I didn't utilise it one hundred procent....yesterday the cooker decided to be over sensitive and the main switch went off....problem was that one of my servers had run for several hours backing up my windows work computer.....The Windows computer was on the right line but not the linux one and I had to start again....ringing electrician today to get the office line extended.....help!

      flmaxey wrote:

      Are you in a hurry? If it was me I'd use the network. You won't have to alter configurations to do it that way - a straight server to server transfer. It's safe. You could do it one share at a time and let it run all night or over a weekend. (Friday evening to Monday morning is a big chunk of time.) With the "keep partially transferred files" flag set, even if there was an error along the way, eventually it would get done.

      (Interestingly, after looking, it appears that's there's no straight forward way of stopping an rsync job other than killing the running process.)
      I will do as you suggest and use the network.....
      Thanks mate!

      bookie56
    • I just ran a scrub and had:

      Source Code

      1. scrub device /dev/md0 (id 1) history
      2. scrub started at Tue Apr 24 06:49:35 2018 and finished after 00:34:51
      3. total bytes scrubbed: 1020.60GiB with 1 errors
      4. error details: csum=1
      5. corrected errors: 0, uncorrectable errors: 1, unverified errors: 0


      I have added lots of partition backups and not done a scrub after adding each....
      But I did find this command to get some idea of the problem...

      Source Code

      1. dmesg| grep -e "BTRFS warning.*path:" | sed -e 's/^.*path\: //'
      After running this I got:

      Source Code

      1. root@simson:~# dmesg| grep -e "BTRFS warning.*path:" | sed -e 's/^.*path\: //'
      2. Media1/DN/DN-Users(E)-Partition-Simson/DN-Users(E)-Partition-Simson/DN-Users(E)-Partition-Simson1.adi)
      Well, when I added that partition image the computer went down as I said....so I restarted the backup and then removed the unfinished backup afterwards...I will now clean up the partition on the computer and then run that back up again and see what gives....


      bookie56
    • bookie56 wrote:

      I have added lots of partition backups and not done a scrub after adding each....
      But I did find this command to get some idea of the problem...


      Source Code

      1. root@simson:~# dmesg| grep -e "BTRFS warning.*path:" | sed -e 's/^.*path\: //'
      2. Media1/DN/DN-Users(E)-Partition-Simson/DN-Users(E)-Partition-Simson/DN-Users(E)-Partition-Simson1.adi)
      I've found command lines to apply to dmesg, similar to yours, that yielded nothing.

      Cutting it down something simple, I got the following (duplicate lines removed):


      root@omv-rpi:~# dmesg| grep -e "BTRFS"
      [ 15.999653] BTRFS: device label WDMYP4TB devid 1 transid 11896 /dev/sda1
      [ 17.602127] BTRFS info (device sda1): disk space caching is enabled
      [ 17.610050] BTRFS info (device sda1): has skinny extents
      [ 18.111079] BTRFS info (device sda1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 0, gen 9
      [ 24.803414] BTRFS info (device sda1): checking UUID tree
      [ 25.222142] BTRFS error (device sda1): parent transid verify failed on 935958413312 wanted 4029 found 4027
      [ 25.247679] BTRFS warning (device sda1): btrfs_uuid_scan_kthread failed -5

      I've seen the "wanted xxxx found xxxx" before. It's usually from a dirty shutdown but "umount"ing and zeroing the log doesn't fix it. Even after reviewing the entire dmesg dump, there's nothing to indicate which 4 files have errors Running zgrep against /var/log/syslog* files doesn't provide a clue either.

      On to the dreaded btrfs check --repair command. Here's to hoping you don't have these problems. Again, for your scenario, an UPS is a real good idea.
      ___________________________________________________________________________

      I might even look into running ZFS on the R-PI :) . That would be a trick with 1 (one) whole GB of ram. ZFS allows "copies=2" which provides for real checksummed bitrot protection, for critical filesystems, even when using a single disk.
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • flmaxey wrote:

      I might even look into running ZFS on the R-PI :) . That would be a trick with 1 (one) whole GB of ram. ZFS allows "copies=2" which provides for real checksummed bitrot protection, for critical filesystems, even when using a single disk.
      Sounds like fun.... :)
      Had problems on my works computer....Windows 7....building a new updated .wim image in vmware at the moment and then I am going to add Samsung nvme drivers for the install usb.....always something to do even when you don't go looking for it....lol

      bookie56
    • I couldn't find a process or command line that worked in my case, to reveal the names of the 4 files with errors.
      Other than the 4 uncorrectable errors, there were three other problems.

      - parent transid verify failed" on 14265458688 wanted wanted 464230 found 464221
      - free space inode generation (0) did not match free space cache generation (12376) mismatched cache error
      - some oddball error regarding the UUID of the drive.

      I've fixed the first error type before with - btrfs-zero-log /dev/sda1
      It didn't work this time around.

      So, the repair process was:

      umount /dev/sda1
      btrfs check --repair


      Then to clear the free space error, I mounted with:
      mount -a clear_cache

      All was clear after that. I ran a scrub to follow up that was clean.
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119

      The post was edited 1 time, last by flmaxey: edit ().

    • Hi mate!
      Thanks for sharing that.....
      I tried to create a partition backup and for some reason that kept on showing the csum error..not sure why...
      There is no problem with that drive....
      Anyway, after removing it for the second time and a scrub came up clean....I did a file back up of the same drive and the scrub showed no errors.....
      Back to backing up files... :)

      bookie56
    • Well, my only misgiving with the solution is that btrfs check --repair command may have simply reset the checksums on the 4 corrupted files. On the other hand, the errors, if they are still there, are probably in non-critical files. For backup purposes (this is 1 of 2 backups), I'll live with it for now. However, the OCD in me will recopy the irreplaceable data the road.
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119

      The post was edited 1 time, last by flmaxey: edit ().

    • Hi mate!
      Well, got my work computer on line with everything how it was and can concentrate on other things...
      I did, however, start my work server and had direct access to the backups I wanted to back up to the old server, so I put them on an external drive and then booted from my usb Debian installation on the OMV server and copied them over to the server...
      I did a btrfs scrub no problems....so now I can do some other work that I have been putting off....
      I back up my computers on a regular basis but don't have a separate directory to make things easier when redoing stuff....another thing on my list of to do thingies.....


      bookie56
    • I'm real interested in how the older server does with BTRFS. As I remember, it took awhile for the errors to crop up with ZFS.

      I can't imagine a specified file system having issues on specific hardware but stranger things have happened.
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119