fsck on large 6Tb disk fails

sandoz · 18. März 2019

today i noticed my server was not booting. After connecting a monitor, i noticed it was in emergency mode.
Rebooted it, and i saw it was fsck that 6tb disk, but it doesn;t complete, and hangs at 70% every time.
I can't say i force stopped my server or so, so no idea what has happenend.

I tried booting from a gparted live cd, but then it also stops at 70%, with e2fsck using 100% cpu

What can i do? My server is not booting currently

Disconnecting the drive doesn't work either, its also not booting, because of the missing disk

tkaiser · 18. März 2019

Zitat von sandoz

Rebooted it, and i saw it was fsck that 6tb disk, but it doesn;t complete, and hangs at 70% every time

'Every time'? So you already killed the fsck multiple times? What about some patience since e2fsck can take several days without a problem?

sandoz · 18. März 2019

i see. But why does it go to 70%? And then it halts

tkaiser · 18. März 2019

Zitat von sandoz

But why does it go to 70%? And then it halts

"Halts" means what exactly? At the link provided above the reply contained strace and ltrace suggestions to get a clue what's going on...

sandoz · 18. März 2019

it stops fsck with:
Failed to start File system check on /dev/disk-by-label/6tbfdisk -l

After the system is in emergencymode:
systemctl status "systemd-fsck@dev-disk-by\\x2dlabel-6tb.service

6tb contains a file system with errors, forced check
a double or bad block is being used
Multiple claimed block (or blocks) in inode 75826141: 34897920--3512725--35651583
(there are 2 inodes with multiple claimed blocks)

tkaiser · 19. März 2019

Zitat von sandoz

a double or bad block is being used

A web search for this exact phrase "a double or bad block is being used" results in one single hit and that's your own post above. How should anyone be able to help you if you do not even take care to provide exact error messages?

Anyway: the filesystems seems to be severely damaged so most probably restoring from backup is the best option. And getting a clue what happened of course to prevent such things from happening again. The reasons why filesystems can become corrupt are endless, especially on commodity hardware often used with OMV (underpowering, bad cabling, crappy SATA controllers onboard or in PCIe slots, whatever).

Adoby · 19. März 2019

Yes, re-partitioning, formatting and THEN running fsck and finally restoring from backup may be much faster.

Another option could be to stick the drive in a fast computer, to a dedicated SATA controller, with a crazy amount of RAM and run fsck there.

Yet another to make sure the computer running the fsck has plenty of swap enabled on a NVME SSD.

Jetzt mitmachen!