File System Quota Check running for 12 hours after unexpected shutdown

TrickyTrix · 20. März 2019

I have nvme SSD on which I installed Debian and OMV on top of it and two WD Red 4 TB drives in OMV RAID 1 config. Everything worked without any problems for a week and then I had two unexpected shutdowns. When I started my machine after unexpected shutdown, File System Quota Check was running for 12 hours after it finally loaded Debian OS. Is this delay normal? And interesting thing is that /var/log/syslog doesn't have any log input for that time period. Input in syslog starts only after Debian OS loads!
Memory isn't the problem (memtest pic below) and same PicoPSU that I use, worked for over a year without any problems in another machine (and it consumes only 20W).

tkaiser · 21. März 2019

Zitat von TrickyTrix

two WD Red 4 TB drives in OMV RAID 1 config ... PicoPSU

Also called 'recipe for disaster'

If you want to play RAID you need reliable hardware, especially powering is essential. You could try to run quotacheck manually to see if the problem persists and I would immediately also run a forced fsck on your RAID1 (of course you might suffer from silent data corruption even if the filesystem isn't corrupted or can be repaired)

TrickyTrix · 21. März 2019

But it is original PicoPSU-150XT and like I said, it worked for over a year on another system without any problems.
Is it normal for quotacheck to work for 12 hours and then show all logs in the same second (the moment system booted)?
Part of the log is below... how does it look?

Code

...
Mar 20 07:44:40 omv systemd[1]: Found device /dev/disk/by-label/vaultX.
Mar 20 07:44:40 omv systemd[1]: Started MD array monitor.
Mar 20 07:44:40 omv systemd[1]: Starting File System Check on /dev/disk/by-label/vaultX...
Mar 20 07:44:40 omv systemd[1]: mdmonitor.service: Main process exited, code=exited, status=1/FAILURE
Mar 20 07:44:40 omv systemd[1]: mdmonitor.service: Unit entered failed state.
Mar 20 07:44:40 omv systemd[1]: mdmonitor.service: Failed with result 'exit-code'.
Mar 20 07:44:40 omv systemd[1]: Reached target Sound Card.
Mar 20 07:44:40 omv systemd-fsck[496]: vaultX: clean, 4241455/244187136 files, 677531248/976721872 blocks
Mar 20 07:44:40 omv systemd[1]: Started File System Check on /dev/disk/by-label/vaultX.
Mar 20 07:44:40 omv systemd[1]: Mounting /srv/dev-disk-by-label-vaultX...
Mar 20 07:44:40 omv systemd[1]: Mounted /srv/dev-disk-by-label-vaultX.
Mar 20 07:44:40 omv systemd[1]: Mounting Mount shared folder Bekap to /sharedfolders/Bekap...
Mar 20 07:44:40 omv systemd[1]: Mounting /export/Bekap...
Mar 20 07:44:40 omv systemd[1]: Mounting Mount shared folder Download to /sharedfolders/Download...
Mar 20 07:44:40 omv systemd[1]: Mounting Mount shared folder Sertifikati to /sharedfolders/Sertifikati...
Mar 20 07:44:40 omv systemd[1]: Mounting Mount shared folder VirtualBox to /sharedfolders/VirtualBox...
Mar 20 07:44:40 omv systemd[1]: Mounting Mount shared folder Instalacije to /sharedfolders/Instalacije...
Mar 20 07:44:40 omv systemd[1]: Starting File System Quota Check...
Mar 20 07:44:40 omv systemd[1]: Mounting Mount shared folder Media to /sharedfolders/Media...
Mar 20 07:44:40 omv systemd[1]: Mounted /export/Bekap.
Mar 20 07:44:40 omv systemd[1]: Mounted Mount shared folder Instalacije to /sharedfolders/Instalacije.
Mar 20 07:44:40 omv systemd[1]: Mounted Mount shared folder Download to /sharedfolders/Download.
Mar 20 07:44:40 omv systemd[1]: Mounted Mount shared folder VirtualBox to /sharedfolders/VirtualBox.
Mar 20 07:44:40 omv systemd[1]: Mounted Mount shared folder Sertifikati to /sharedfolders/Sertifikati.
Mar 20 07:44:40 omv systemd[1]: Mounted Mount shared folder Bekap to /sharedfolders/Bekap.
Mar 20 07:44:40 omv systemd[1]: Mounted Mount shared folder Media to /sharedfolders/Media.
Mar 20 07:44:40 omv systemd[1]: Started File System Quota Check.
Mar 20 07:44:40 omv systemd[1]: Starting Enable File System Quotas...
Mar 20 07:44:40 omv systemd[1]: Started Enable File System Quotas.
Mar 20 07:44:40 omv systemd[1]: Reached target Local File Systems.
Mar 20 07:44:40 omv systemd[1]: Starting Cleanup PHP session files...
Mar 20 07:44:40 omv systemd[1]: Starting Preprocess NFS configuration...
Mar 20 07:44:40 omv systemd[1]: Starting Set console font and keymap...
Mar 20 07:44:40 omv systemd[1]: Starting Cleanup monit state file...
Mar 20 07:44:40 omv systemd[1]: Starting Create Volatile Files and Directories...
Mar 20 07:44:40 omv systemd[1]: Starting Raise network interfaces...
Mar 20 07:44:40 omv systemd[1]: Started Cleanup PHP session files.
Mar 20 07:44:40 omv systemd[1]: Started Cleanup monit state file.
Mar 20 07:44:40 omv systemd[1]: Started Preprocess NFS configuration.
...

Alles anzeigen

tkaiser · 21. März 2019

Zitat von TrickyTrix

mdmonitor.service: Main process exited, code=exited, status=1/FAILURE

You should check this. And as already written I would strongly recommend to run a forced fsck on your RAID1 --> fsck -f

Code

-f     Force checking even if the file system seems clean.

TrickyTrix · 21. März 2019

Is it possible to to make the command run before starting OS like chkdsk in Windows???
In order to run it in terminal as I know it, I would have to unmount the drive, and in order to unmount the drive I would have to delete all the shares created earlier...

macom · 22. März 2019

You could boot a live distro like SystetmRescueCD which is available in omv-extras (Kernel tab). That should work for sure.

I am not sure if you could unmount the drive from CLI, check the disk and reboot ("mount -a" might be enough).

TrickyTrix · 22. März 2019

I found out that if I run 'touch /forcefsck' that it will check file system after reboot.
In syslog I got 'Mar 22 07:22:14 omv systemd-fsck[449]: vaultX: clean, 4241516/244187136 files, 679887432/976721872 blocks' so I guess that everything is fine.
Can Docker container which is running in privileged mode cause system crash? Zoneminder for video surveillance is my suspect.

tkaiser · 22. März 2019

Zitat von TrickyTrix

I found out that if I run 'touch /forcefsck' that it will check file system after reboot

It will check the rootfs which is totally unrelated to your mdraid.

TrickyTrix · 23. März 2019

I installed SystemRescueCD but it isn't working. I can select it in the boot menu but it is giving me errors. I logged via SFTP on OMV machine and the ISO is there so what can be the problem?

Jetzt mitmachen!