Harddrive Failure and Data Recovery

    • OMV 4.x
    • btrfs check /dev/sda

      Checking filesystem on /dev/sda
      UUID: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      checking extents
      checking free space cache
      cache and super generation don't match, space cache will be invalidated
      checking fs roots
      checking csums
      checking root refs
      found 131072 bytes used err is 0
      total csum bytes: 0
      total tree bytes: 131072
      total fs tree bytes: 32768
      total extent tree bytes: 16384
      btree space waste bytes: 123735
      file data blocks allocated: 0
      referenced 0
    • henfri wrote:

      you can try the same again on the original:
      mount with the two options
      btrfs check
      btrfs info
      Original drive:

      btrfs check /dev/sda

      Checking filesystem on /dev/sda
      UUID: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      checking extents
      checking free space cache
      checking fs roots
      checking csums
      checking root refs
      found 393216 bytes used err is 0
      total csum bytes: 0
      total tree bytes: 131072
      total fs tree bytes: 32768
      total extent tree bytes: 16384
      btree space waste bytes: 124162
      file data blocks allocated: 262144
      referenced 262144

      I'm hoping you meant "btrfs filesystem show /dev/sda" as there is no "info" token.

      btrfs filesystem show /dev/sda

      Label: 'sdadisk1' uuid: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      Total devices 1 FS bytes used 384.00KiB
      devid 1 size 931.51GiB used 2.04GiB path /dev/sda


      As this was a SnapRaid drive, I'm dismayed that it is only showing 2.04GiB used. There should be at a minimum 300-400 GiB used. But even if we could get to that 2.04GiB, it would be better than nothing.

      Sigh ...
    • You did not try to mount (which I wrote) and scrub (which I did not write).

      Can you try that please?

      What do you mean by SnapRaid drive? There are two types: Data and Parity.
      Are we working on the Parity or Data?
      In any case, we can also try the other drive (if we are now on Parity, then we would do the data-drive now).
      You can try to mount it with the recovery option(s) and see with ls if there is data on it.

      If you do, please show me the output of dmesg -T| tail after mounting.

      Greetings,
      Hendrik
    • Hi Hendrik,
      This is where I'm "dumb" with linux things. When I put the original drive back in and did a look at btrfs, I thought it just automatically was in "mount" status, like where it was before. I'm so stupid.

      Anyway, I'll mount it and do it again.

      I have just always assumed we were working with the data drive, not the parity drive. Is there a way to tell if a drive is one or the other?

      I'll get going on the instructions ...

      So I ran: "mount -t btrfs -o recovery,nospace_cache,clear_cache /dev/sda /srv/test" and it returned to the prompt, which I would expect. Verified by running the command again and it said "already mounted".


      henfri wrote:

      you can try the same again on the original:
      mount with the two options
      btrfs check
      btrfs info
      After re-reading this, I'm not sure I understand. Do I just run "btrfs check /dev/sda" and btrfs info /dev/sda"? That doesn't seem right, as you indicate they are options. (told you, I'm linux stupid. I have to keep going back to all of the posts in our thread to figure out what to type into the commands. Yeesh. Wish I was better at this.)
      I ran "btrfs check /dev/sda" and it said "/dev/sda is currentyly mounted. Aborting." So I must not being looking at the right instructions.

      The post was edited 2 times, last by curious1 ().

    • Hello,

      openmediavault may automatically mount the btrfs-drives. But we want to mount with the options recovery,nospace_cache,clear_cache,
      So, if omv mounts automatically, you need to unmount and mount manually with the options.
      Then you need to do an

      Source Code

      1. ls /srv/test





      to see if the data is there. Also you need to run the dmesg command


      Source Code

      1. dmesg -T| tail
      If the data is not there, we need to see what is wrong:


      Source Code

      1. btrfs scrub start /dev/sda
      2. btrfs scrub status /dev/sda
      When completed

      Source Code

      1. umount /dev/sda
      2. btrfs check /dev/sda

      All this you can repeat on each of the broken drives (disk1, disk2, ...).
      Based on the output of ls we can determine what drive is what. To be able to associate what happened on which drive, we need

      Source Code

      1. btrfs filesystem show /dev/sda
      Because that gives us the unique identifier of the drive, whereas /dev/sda can change on each boot to /dev/sdb. Thus, you always need to check what drive-"letter" currently is associated to the drive that you want to check. You get the label of the drive in the omv web-interface (disk1, disk2, ...)


      So,
      disk1 may be UUID: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      disk2 may be UUID: xxxxxx

      and so on.

      In the OMV Webinterface of Snapraid you should also find which disk[1,2,3] is parity and which is data.

      Greetings,
      Hendrik
    • Hi Hendrik,
      Here is the original disk (wonder if, in fact, this is the parity disk - I'm looking into that)

      unmounted /dev/sda

      mount -t btrfs -o recovery,nospace_cache,clear_cache /dev/sda /srv/test

      ls /srv/test (returned nothing)

      dmesg -T | tail
      [Fri Jan 17 17:17:41 2020] ata1: EH complete
      [Fri Jan 17 17:18:31 2020] logitech-hidpp-device 0003:046D:400A.0004: HID++ 2.0 device connected.
      [Sat Jan 18 11:40:29 2020] BTRFS warning (device sda): 'recovery' is deprecated, use 'usebackuproot' instead
      [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): trying to use backup root at mount time
      [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): disabling disk space caching
      [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): force clearing of disk cache
      [Sun Jan 19 11:58:24 2020] BTRFS warning (device sda): 'recovery' is deprecated, use 'usebackuproot' instead
      [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): trying to use backup root at mount time
      [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): disabling disk space caching
      [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): force clearing of disk cache

      btrfs scrub start /dev/sda
      scrub started on /dev/sda, fsid fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16 (pid=19881)

      btrfs scrub status /dev/sda
      scrub status for fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      scrub started at Sun Jan 19 12:03:35 2020 and finished after 00:00:00
      total bytes scrubbed: 256.00KiB with 0 errors

      umount /dev/sda

      btrfs check /dev/sda
      Checking filesystem on /dev/sda
      UUID: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      checking extents
      checking free space cache
      cache and super generation don't match, space cache will be invalidated
      checking fs roots
      checking csums
      checking root refs
      found 131072 bytes used err is 0
      total csum bytes: 0
      total tree bytes: 131072
      total fs tree bytes: 32768
      total extent tree bytes: 16384
      btree space waste bytes: 123986
      file data blocks allocated: 0
      referenced 0

      btrfs filesystem show /dev/sda
      Label: 'sdadisk1' uuid: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      Total devices 1 FS bytes used 128.00KiB
      devid 1 size 931.51GiB used 4.10GiB path /dev/sda


      Way back when we started this, I was directed to create a new boot, which I did. So SnapRaid is no longer part of this installation.

      The post was edited 2 times, last by curious1 ().

    • Hi Hendrik,
      Here is the original disk (wonder if, in fact, this is the parity disk - I'm looking into that)

      unmounted /dev/sda

      mount -t btrfs -o recovery,nospace_cache,clear_cache /dev/sda /srv/test

      ls /srv/test (returned nothing)

      dmesg -T | tail
      [Fri Jan 17 17:17:41 2020] ata1: EH complete
      [Fri Jan 17 17:18:31 2020] logitech-hidpp-device 0003:046D:400A.0004: HID++ 2.0 device connected.
      [Sat Jan 18 11:40:29 2020] BTRFS warning (device sda): 'recovery' is deprecated, use 'usebackuproot' instead
      [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): trying to use backup root at mount time
      [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): disabling disk space caching
      [Sat Jan 18 11:40:29 2020] BTRFS info (device sda): force clearing of disk cache
      [Sun Jan 19 11:58:24 2020] BTRFS warning (device sda): 'recovery' is deprecated, use 'usebackuproot' instead
      [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): trying to use backup root at mount time
      [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): disabling disk space caching
      [Sun Jan 19 11:58:24 2020] BTRFS info (device sda): force clearing of disk cache

      btrfs scrub start /dev/sda
      scrub started on /dev/sda, fsid fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16 (pid=19881)

      btrfs scrub status /dev/sda
      scrub status for fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      scrub started at Sun Jan 19 12:03:35 2020 and finished after 00:00:00
      total bytes scrubbed: 256.00KiB with 0 errors

      umount /dev/sda

      btrfs check /dev/sda
      Checking filesystem on /dev/sda
      UUID: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      checking extents
      checking free space cache
      cache and super generation don't match, space cache will be invalidated
      checking fs roots
      checking csums
      checking root refs
      found 131072 bytes used err is 0
      total csum bytes: 0
      total tree bytes: 131072
      total fs tree bytes: 32768
      total extent tree bytes: 16384
      btree space waste bytes: 123986
      file data blocks allocated: 0
      referenced 0

      btrfs filesystem show /dev/sda
      Label: 'sdadisk1' uuid: fdce5ae5-fd6d-46b9-8056-3ff15ce9fa16
      Total devices 1 FS bytes used 128.00KiB
      devid 1 size 931.51GiB used 4.10GiB path /dev/sda


      Way back when we started this, I was directed to create a new boot, which I did. So SnapRaid is no longer part of this installation.
    • You had two broken drives.
      We have now worked on the original and the copy of one.

      Please connect all the drives and do a

      Source Code

      1. btrfs fi show

      You will get something like

      Source Code

      1. Label: 'sdadisk' uuid: 303256f2-6901-4564-892e-cdca9dda50e3
      2. Total devices 1 FS bytes used 17.88GiB
      3. devid 1 size 52.16GiB used 29.05GiB path /dev/sda
      4. Label: 'sdbdisk' uuid: 1212312313-6901-4564-892e-cdca9dda50e3
      5. Total devices 1 FS bytes used 17.88GiB
      6. devid 1 size 52.16GiB used 29.05GiB path /dev/sdb
      7. Label: 'sdcdisk' uuid: 34ff34234-6901-4564-892e-cdca9dda50e3
      8. Total devices 1 FS bytes used 17.88GiB
      9. devid 1 size 52.16GiB used 29.05GiB path /dev/sdc

      That should help you identify the drives. But also please post the output here.
      Can you please repeat (all) the steps that we last did on the other broken drive(s)?
      When mounting, please use these options:

      Source Code

      1. mount -t btrfs -o recovery,nospace_cache,clear_cache,subvolid=0 /dev/sda /srv/test
      After mounting, in addition to what you did before please also show me the output of


      Source Code

      1. btrfs subvolume list /mnt/test
      and of

      Source Code

      1. mount | grep /dev/sd

      Are you sure that you did not format the drive(s)?


      Regards,
      Hendrik

      The post was edited 3 times, last by henfri ().

    • Hi Hendrik,

      I'll endeavor to work on all the drives. (I was away for a few days and just got back).

      Absolutely certain I did not format the drives.

      henfri wrote:

      Can you please repeat (all) the steps that we last did on the other broken drive(s)?
      I'm going to need to put together a list of all the steps ... to make sure that I have it all in proper sequence (also, because of the nature of the forum postings, I have to jump back and forth to know which ones to use, then I get lost a little ... sorry). For instance, I think I understand that I'm not doing a ddrescue on any other drives at this point (??). Anyway, I'll take a shot at the proper sequence, and then list it here to get your 'thumbs up' before I proceed.

      Of course, unless you want to put that list together for me ... LOL. :D But I'll work on it.

      Steve
    • Hi Hendrick.

      I had a couple of things that took some intense focus from me, but am now looking to get back on this.

      Just a recap, I had 4 hardrives (one parity, and three data) that were in my Snapraid. Two of those drives won't even spin (those I may have to send to data recovery and take my chances - certainly can't do anything with them otherwise). Of the two that will spin, you and I have been working with only one drive (a Seagate) that so far hasn't given us much - certainly no data (so far). So it would seem that maybe we can get the second drive looked at (a Toshiba).

      So, I'll put both drives back into the box and do the following on each (xxx will be sda and sdb):

      1. umount /dev/xxx
      2. mount -t btrfs -o recovery,nospace_cache,clear_cache,subvolid=0 /dev/xxx /srv/test
      3. btrfs subvolume list /mnt/test
      4. mount | grep /dev/xxx
      5. ls /srv/test
      6. dmesg -T | tail
      7. btrfs scrub start /dev/xxx
      8. btrfs scrub status /dev/xxx
      9. umount /dev/xxx
      10. btrfs check /dev/xxx
      11. btrfs filesystem show /dev/xxx
      12. btrfs fi show


      Does this look right?

      I don't know if I mentioned this before, but maybe we should perform the ddrescue on the second drive (the Toshiba).

      Let me know if this list looks right, and I'll proceed.

      Thanks,
      Steve