Failed to mount Pool(?), unable to boot into OS

Dejh0n · 29. Juli 2023

Hi OMV team,

I have an issue I have been trying to troubleshoot, however I am fairly useless at debian and cannot for the life of me figure out what is breaking and how to fix it.

Last week Friday I found I had no access to my data - my MergerFS volume appeared to be down.
So I rebooted the server and it did not come back up like it usually does when I have an issue.

I've got it hooked up to a screen and keyboard and it appears that OMV cannot mount the MergerFS volume (I'm guessing)

Here is the boot screen and result of the command it recommended I type to see what the error is:

Code

0.607555]
ipmi:dmi: Base address is zero, assuming no IPMI interface
2.8714831 usb 2-1-port7: over-current condition
3.091634] usb 2-1-port8: over-current condition
/dev/sdo1: clean, 479495/30466048 files, 16846273/121846272 blocks
SELinux:
Could not open policy file <= /etc/selinux/targeted/policy/policy.33:
No such file or directory
[FAILED] Failed to mount /srv/94e87264-55f0-49c3-b699-0efa2dbf f 704.
see
systemct1 status "srv-94e8726411x2d55f011x2d49c31\x2db69911x2d0efa2dbff704.mount" for details.
[DEPEND]
Dependency failed for /export/Backup.
[DEPENDI
Dependency failed for /export/resume.
[DEPEND]
Dependency failed for Local File Systems.
[DEPEND]
Dependency failed for /export/iPhoto.
[DEPEND]
Dependency failed for /export/Media.
[DEPENDI
Dependency failed for /export/Software.
Starting Preprocess NFS configuration...
Starting Set console font and keymap...
Starting Raise network interfaces...
]
Stopped Dispatch Password Requests to Console Directory Watch.
Starting Beep after system start...
Starting Cleanup PHP session files...
OK
]
Closed syslog socket.
Starting Cleanup monit state file...
OK
Reached target Timers.
OK
] Reached target Sockets.
Starting Create Volatile Files and Directories...
00K0K
]
Started Emergency Shell.
Reached target Login Prompts.
0K
Reached target Emergency Mode.
Stopped Forward Password Requests to Wall Directory Watch.
OK
Started Preprocess NFS configuration.
OK
Started Set console font and keymap.
OK
Started Raise network interfaces.
OK
Started Beep after system start.
OK
Started Cleanup PHP session files.
OK
]
Started Cleanup monit state file.
Started Create Volatile Files and Directories.
Starting Network Name Resolution...
Starting Update UTMP about System Boot/Shutdown.
Starting RPC bind portmap service...
[
OK
1 Reached target NFS client services.
OK
]
Starting NFSv4 ID-name mapping service...
Started RPC bind portmap service.
OK
Reached target RPC Port Mapper.
OK
Started Update UTMP about System Boot/Shutdown.
OK
]
Started NFSv4 ID-name mapping service.
Starting Update UTMP about System Runlevel Changes...
OK
Started Update UTMP about System Runlevel Changes.
OK
Started Network Name Resolution.
OK
Reached target Host and Network Name Lookups.
OK
Reached target Network.
Starting NFS status monitor for NFSv2/3 locking....
Starting NFS Mount Daemon...
OK
Started NFS Mount Daemon.
]
Started NFS status monitor for NFSv2/3 locking..
Starting NFS server and services...
LI
OK
0K
77
Started Enable File System Quotas.
Started NFS server and services.
Starting Notify NFS peers of a restart.
OK
Reached target Remote File Systems (Pre).
OK
Reached target Remote File Systems.
OK
] Started Notify NFS peers of a restart.
You are in emergency mode.
type "journalct1 -xb" to view
system logs,
"systemct1 reboot"
to reboot,
"exit"
to boot into default mode.
Give root password for maintenance (or press Control-D to continue):
root@Dpt imus:*# systemct1 status "srv-94e872641\x2d55f01 \x2049c31\x2db6991 \x2d0efa2dbf f704.mount"
• srv-94e87264\x2d55f01x2d49c3\x2db699\x2d0efa2dbff704. mount - /srv/94e87264-55f0-49c3-b699-0efa2dbf† 704
Loaded: loaded (/etc/fstab; generated)
Active: failed (Result: exit-code) since Sat 2023-07-29 21:17:32 NZST; 1min 6s ago
Where:
/srv/94e87264-55f0-49c3-b699-0efa2dbff 704
What: /srv/dev-disk-by-uuid-2c95f714-7151-4085-8c98-45696e6b28ca:/srv/dev-disk-by-uuid-b9995e0d-4daf-42a0-8c25-04e41e701ae9:
Docs: man:fstab (5)
man: systemd-fstab-generator (8)
Jul 29 21:17:32 Optimus systemd[1]: Mounting /srv/94e87264-55f0-49c3-b699-0efa2dbff 704...
Jul 29 21:17:32 Optimus mount [1358]: fuse: mountpoint is not empty
Jul 29 21:17:32 Optimus mount [1958]: fuse: if you are sure this is safe, use the 'nonempty' mount option
วัน! 29 21:17:32 opt inus systend (1): sry-94687264 x2055101x2049c31x20b6991x2d0efa2dbff704, mount : Mount process exited, code exitec
Jul 29 21:17:32 Optimus systemd[1]: srv-94e87264\x2d55f0\x2049c31x2db6991x2d0efa2dbff704. mount: Failed with result exit-code'.
11 39 21817:32 Optimus systemd[1]: Failed to mount /srv/94e87264-55f0-49c3-b699-0efa2dbff704.

Alles anzeigen

I could really use some help to get my server back up and running again please!

Dejh0n · 29. Juli 2023

I tried editing out the drive I thought was causing the error from FSTAB (made a backup first!)- this made things worse.
So I put it back in

All drives appear mounted when typing 'mount'

Happy to take any advice or commands here

macom · 29. Juli 2023

Code

Jul 29 21:17:32 Optimus mount [1358]: fuse: mountpoint is not empty

Maybe this is the issue. You can shutdown the server, disconnect the drives and reboot. After that check if the directory /srv/94e87264-55f0-49c3-b699-0efa2dbff704 exist and is not empty.

Dejh0n · 29. Juli 2023

Okay,

So just to double check.

I disconnect all non OS drives,

Navigate to that folder (cd /srv/xxx)

Do a Ls -a and there should hopefully be nothing?

Question: if there is nothing- what next?

If there is something- what next?

Dejh0n · 30. Juli 2023

So following up - gave this a go.

Disconnected all drives except my boot drive, turned the server back on and navigated to that folder.

It showed quite a few errors (obviously) during boot up.

So I navigated to the folder in question which was there.

There was quite a few folders in there (of names and structures I was familiar with), however it appeared there was 0 files in there.

I used the ls -a command and there seemed to be nothing in any of the folders and subfolders.

I navigated to the root folder again /srv/94e87264-55f0-49c3-b699-0efa2dbff704 and did a rm -r of the 2 sub folders in that directory,

rebooted the server and it came online!

I will have to continue to monitor for any oddities that may be in place - I did notice that during this boot there was an error saying something about unable to rotate the log files(?)?

macom · 30. Juli 2023

Zitat von Dejh0n

I did notice that during this boot there was an error saying something about unable to rotate the log files(?)?

That seems unrelated. If it comes up again, post the message here.

Failed to mount Pool(?), unable to boot into OS

Jetzt mitmachen!

Tags