ext4 drive now shows no filesystem or files after adding USB drive (Odroid-HC4, Armbian)

Brief history:

Did 90% of OMV setup 1 year ago

2 SATA drives (ext4) in ODROID HC4, OMV installed on microSD card. Snapraid installed but I don't think I installed it correctly (didn't set it to run or do a first run I think).

Dusted off my setup

Updated everything

Added USB HDD (ntfs)

Immediately, OMV can't recognize one of my SATA drives. I see a new UUID for the same drive. No file systems that I can find. I put the drive in another PC and booted into kubuntu 20.04.3 live and still couldn't find any filesystems. Tried searching for superblocks and failed.

I know almost nothing about Linux HDD recovery so don't assume I tried something already. It could be very basic. Or maybe the drive is dead.

I looked at this guide but didn't see much relevant. I still need to run lspci in verbose mode. I'll have logs after work but wanted to see if it's a simple issue.

System check and harddisk/raid speed tests in a nutshell

I ran testdisk and then reboot. It's been unresponsive for hours. Rebooted my router and none of the dhcp leases are for the odroid. The Network LED for the odroid is blinking and there are vibrations and noise from the drive(s). Would align partition take hours?

/dev/sda: LBA, HPA, LBA48 support

/dev/sda: size 3907029168 sectors

/dev/sda: user_max 3907029168 sectors

/dev/sda: native_max 3907029168 sectors

Using locale 'C.UTF-8'.

Thu Jan 20 18:37:15 2022

Command line: TestDisk /log /dev/sda

TestDisk 7.0, Data Recovery Utility, April 2015

Christophe GRENIER <grenier@cgsecurity.org>

http://www.cgsecurity.org

OS: Linux, kernel 5.10.81-meson64 (#21.08.6 SMP PREEMPT Mon Nov 22 11:21:51 UTC 2021) aarch64

Compiler: GCC 8.2

ext2fs lib: 1.44.5, ntfs lib: libntfs-3g, reiserfs lib: none, ewf lib: none, curses lib: ncurses 6.1

Hard disk list

Disk /dev/sda - 2000 GB / 1863 GiB - CHS 243201 255 63, sector size=512 - ST2000DM008-2FR102, S/N:ZK201FE8, FW:0001

Partition table type (auto): EFI GPT

Disk /dev/sda - 2000 GB / 1863 GiB - ST2000DM008-2FR102

Partition table type: EFI GPT

Analyse Disk /dev/sda - 2000 GB / 1863 GiB - CHS 243201 255 63

Bad GPT partition, invalid signature.

Trying alternate GPT

1 P Unknown 2048 3907029134 3907027087

Current partition structure:

Bad GPT partition, invalid signature.

Trying alternate GPT

1 P Unknown 2048 3907029134 3907027087

search_part()

Disk /dev/sda - 2000 GB / 1863 GiB - CHS 243201 255 63

MS Data 2048 3907029127 3907027080 [seagate2tb]

ext4 blocksize=4096 Large_file Sparse_SB, 2000 GB / 1863 GiB

interface_write()

1 P MS Data 2048 3907029127 3907027080 [seagate2tb]

write!

No extended partition

You will have to reboot for the change to take effect.

New options :

Dump : No

Align partition: Yes

Expert mode : No

Interface Advanced

1 P MS Data 2048 3907029127 3907027080 [seagate2tb]

ext4 blocksize=4096 Large_file Sparse_SB, 2000 GB / 1863 GiB

ext2fs_read_inode(ino=11) failed with error 2133571474.

ext2fs_read_inode(ino=5242881) failed with error 1.

ext2fs_read_inode(ino=17) failed with error 2133571474.

ext2fs_read_inode(ino=16) failed with error 2133571474.

ext2fs_read_inode(ino=110624769) failed with error 1.

ext2fs_read_inode(ino=114425857) failed with error 1.

ext2fs_read_inode(ino=110755841) failed with error 1.

ext2fs_read_inode(ino=110886913) failed with error 1.

ext2fs_read_inode(ino=111017985) failed with error 1.

ext2fs_read_inode(ino=113246209) failed with error 1.

ext2fs_read_inode(ino=114425857) failed with error 1.

Directory /

2 drwxrwsr-x 0 1001 4096 19-Jan-2022 08:59 .

2 drwxrwsr-x 0 1001 4096 19-Jan-2022 08:59 ..

18 -rw------- 0 1001 250125 19-Jan-2022 08:59 snapraid.content

X 18 -rw------- 0 1001 250125 19-Jan-2022 08:59 snapraid.content.tmp

TestDisk exited normally.

Photorec / TestDisk on an x86 linux distro seems to be able to recover the files. It's going to take several days for 2 TB. This is a huge error and I still don't know what caused it. https://www.tecmint.com/photor…eted-lost-files-in-linux/

Quote from RidgeRacer

Photorec / TestDisk on an x86 linux distro seems to be able to recover the files. It's going to take several days for 2 TB. This is a huge error and I still don't know what caused it. https://www.tecmint.com/photor…eted-lost-files-in-linux/

I'm not concerned about the drive I just don't want this flaw to come back once I turn this into a real NAS and not a toy system. I still don't plan on making the NAS my single point of failure, but recovering from backup errors is annoying, takes time, and puts strain on my other drives.

Phonerec was taking forever and doesn't give me filenames, so I killed it. I have the original data so I'll copy it again.

- The added NTFS hard drive - is it USB powered (by the HC4)? Or is it in a drive dock or, maybe, powered by a USB hub? (The reason why this is mentioned is powering hard drives can be problematic with SBC's. An HC4 recommends a 4amp supply, minimum, for two on-board drives.

USB powered, usb3 "p10" WD black https://www.amazon.com/WD_Blac…20BBK-WESN/dp/B07VMTNDMK/

The HC4 has 2 onboard drives (3.5" desktop HDDs) and the USB drive (2.5 inch) all drawing power. The PSU is 15V/4A Power Supply × 1 from Ameridroid.

- Have you looked look at SMART data for the drive you think is / has failed?

Everything looks fine. It's almost a brand new drive (albeit I could be hitting the early part of the bathtub curve).

Ran this on a kubuntu live USB on a totally different x86 device (not OMV).

kubuntu@kubuntu:~$ sudo skdump /dev/sda
Device: sat16:/dev/sda
Type: 16 Byte SCSI ATA SAT Passthru
Size: 1907729 MiB
Model: [ST2000DM008-2FR102]
Serial: [ZK201FE8]
Firmware: [0001]
SMART Available: yes
Quirks:
Awake: yes
SMART Disk Health Good: yes
Off-line Data Collection Status: [Off-line data collection activity was never started.]
Total Time To Complete Off-Line Data Collection: 0 s
Self-Test Execution Status: [The previous self-test routine completed without error or no self-test has ever been run.]
Percent Self-Test Remaining: 0%
Conveyance Self-Test Available: yes
Short/Extended Self-Test Available: yes
Start Self-Test Available: yes
Abort Self-Test Available: yes
Short Self-Test Polling Time: 1 min
Extended Self-Test Polling Time: 207 min
Conveyance Self-Test Polling Time: 2 min
Bad Sectors: 0 sectors
Powered On: 3.1 months
Power Cycles: 296
Average Powered On Per Power Cycle: 7.5 h
Temperature: 30.0 C
Attribute Parsing Verification: Bad
Overall Status: GOOD
ID# Name                        Value Worst Thres Pretty      Raw            Type    Updates Good Good/Past  1 raw-read-error-rate          82    64     6   163297192   0xa8b7bb090000 prefail online  yes  yes   3 spin-up-time                 96    96     0   n/a         0x000000000000 prefail online  n/a  n/a   4 start-stop-count            100   100    20   370         0x720100000000 old-age online  yes  yes   5 reallocated-sector-count    100   100    10   0 sectors   0x000000000000 prefail online  yes  yes   7 seek-error-rate              80    60    45   91519515    0x1b7a74050000 prefail online  yes  yes   9 power-on-hours               98    98     0   3.1 months  0xad080000a820 old-age online  n/a  n/a  10 spin-retry-count            100   100    97   0           0x000000000000 prefail online  yes  yes  12 power-cycle-count           100   100    20   296         0x280100000000 old-age online  yes  yes 183 runtime-bad-block-total     100   100     0   0           0x000000000000 old-age online  n/a  n/a 184 end-to-end-error            100   100    99   0           0x000000000000 old-age online  yes  yes 187 reported-uncorrect          100   100     0   0 sectors   0x000000000000 old-age online  n/a  n/a 188 command-timeout             100   100     0   0           0x000000000000 old-age online  n/a  n/a 189 high-fly-writes             100   100     0   0           0x000000000000 old-age online  n/a  n/a 190 airflow-temperature-celsius  70    53    40   30.0 C      0x1e00141e0000 old-age online  yes  yes 191 g-sense-error-rate          100   100     0   0           0x000000000000 old-age online  n/a  n/a 192 power-off-retract-count     100   100     0   62          0x3e0000000000 old-age online  n/a  n/a 193 load-cycle-count            100   100     0   1889        0x610700000000 old-age online  n/a  n/a 194 temperature-celsius-2        30    47     0   30.0 C      0x1e0000001200 old-age online  n/a  n/a 195 hardware-ecc-recovered       82    64     0   163297192   0xa8b7bb090000 old-age online  n/a  n/a 197 current-pending-sector      100   100     0   0 sectors   0x000000000000 old-age online  n/a  n/a 198 offline-uncorrectable       100   100     0   0 sectors   0x000000000000 old-age offline n/a  n/a 199 udma-crc-error-count        200   200     0   0           0x000000000000 old-age online  n/a  n/a 240 head-flying-hours           100   253     0   2.8 months  0xbc070000f97f old-age offline n/a  n/a 241 total-lbas-written          100   253     0   165533.598 TB 0x1bf70b260100 old-age offline n/a  n/a 242 total-lbas-read             100   253     0   611433.316 TB 0x35ae1f3e0400 old-age offline n/a  n/a kubuntu@kubuntu:~$ 

Reinstalled basically everything. Fresh armbian, fresh bootloader (petitboot), fresh OMV 5. Formatted all 3 drives to GPT and ext4. Also made one of them 1 big partition. (Before, it was two partitions: 2TB and 1TB).

I notice that the 'disks' field of the omv webui says "/dev/sda" , b, c. Before, I thought some/all of them were /dev/disk/by-uuid/...

Maybe the clue to the bug is in there somewhere? I also switched the 'problem' drive to the other SATA slot on the odroid hc4. We'll see if the drive swap makes the 'good' drive start misbehaving.

When plugged into a monitor, the boot sequence still seems to look for 2 or 3 drives by uuid. I opened up config.xml and couldn't find where it was pulling that from. One of them showed up in the disks webUI and I removed it.