The ZFS modules are not loaded after reboot

huangwb8 · 1. Juli 2020

Today I found my NAS stop working.

Here are some feature of the problems:

1. ZFS module not loaded since Jul 1 07:00:33

Code

$ less /var/log/syslog|grep 'The ZFS modules are not loaded'
Jul  1 07:00:33 openmediavault zpool[505]: The ZFS modules are not loaded.
Jul  1 07:00:33 openmediavault zfs[507]: The ZFS modules are not loaded.
Jul  1 07:00:33 openmediavault zvol_wait[508]: The ZFS modules are not loaded.
...

It must be the problem about reboot, because I set a routine reboot (in /etc/crontab) at 7:00 am in first day of every month. It never showed error till today.

It seems that ZFS module collapsed as I reboot (Maybe ZFS is working then. I had set an auto snapshot at 3:00 am and destory at 5:00 am every day)

Other informations were also support the ZFS-collapse hypothesis. For example, there are some error imformation in web GUI:

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; zfs list -H -t snapshot -o name,used,refer 2>&1' with exit code '1': The ZFS modules are not loaded. Try running '/sbin/modprobe zfs' as root to load them.

When try /sbin/modprobe zfs

Code

$ /sbin/modprobe zfs
modprobe: FATAL: Module zfs not found in directory /lib/modules/5.5.0-0.bpo.2-amd64

At the same time, S.M.A.R.T was also stopped, but I'm not sure the relationship between them:

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; smartctl -x '/dev/sda' 2>&1' with exit code '1': smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.5.0-0.bpo.2-amd64] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org /var/lib/smartmontools/drivedb/drivedb.h(5775): Syntax error, '"' expected

2. Disks are not damaged.

Considering reboot sometimes may damage disks, I try fdisk -l:

Code

$ fdisk -l
Disk /dev/sda: 14.9 GiB, 16013942784 bytes, 31277232 sectors
Disk model: faspeed K5M-16G 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5978c26a

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1  *        2048 23111679 23109632   11G 83 Linux
/dev/sda2       23113726 31277055  8163330  3.9G  5 Extended
/dev/sda5       23113728 31277055  8163328  3.9G 82 Linux swap / Solaris


Disk /dev/sdd: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68N
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 01789BDB-17E3-6443-AEC1-C1E58485189F

Device          Start        End    Sectors  Size Type
/dev/sdd1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdd9  7814019072 7814035455      16384    8M Solaris reserved 1
...

Alles anzeigen

They were all be identified by system, so I think the disk is OK.

3. system information of my NAS:

Code

Linux openmediavault.local 5.5.0-0.bpo.2-amd64 #1 SMP Debian 5.5.17-1~bpo10+1 (2020-04-23) x86_64 GNU/Linux

Conclusion:

ZFS is not working now, causing all files not available for visit. It's mostly likely to be the loss of ZFS module.

How to fix it?

ryecoaaron · 1. Juli 2020

Use the proxmox kernel instead of the debian kernel. There is a button to install the proxmox kernel in omv-extras. This kernel has the module built-in.

huangwb8 · 2. Juli 2020

Zitat von ryecoaaron

Use the proxmox kernel instead of the debian kernel. There is a button to install the proxmox kernel in omv-extras. This kernel has the module built-in.

Thank you, it exactly works for ZFS coming back. Thank you very much!!

However, something creepy still exist.

First, check storage system

Code

$ df -lh
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 383M 7.3M 376M 2% /run
/dev/sda1 11G 5.2G 5.1G 51% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 1.9G 12K 1.9G 1% /tmp
tmpfs 383M 0 383M 0% /run/user/0

Where's /nas (6.83TiB), position of my ZFS storage?

After some searching, I solved the problem. Here's the solustion:

First, check the mount status of zfs.

Code

$ systemctl status zfs-mount
● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2020-07-02 08:24:05 CST; 33min ago
     Docs: man:zfs(8)
  Process: 1295 ExecStart=/sbin/zfs mount -a (code=exited, status=1/FAILURE)
 Main PID: 1295 (code=exited, status=1/FAILURE)

Jul 02 08:24:05 openmediavault.local systemd[1]: Starting Mount ZFS filesystems...
Jul 02 08:24:05 openmediavault.local zfs[1295]: cannot mount '/nas': directory is not empty
Jul 02 08:24:05 openmediavault.local systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Jul 02 08:24:05 openmediavault.local systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
Jul 02 08:24:05 openmediavault.local systemd[1]: Failed to start Mount ZFS filesystems.

Alles anzeigen

According to this message, it's most likely that the fake "/nas" in the system disk is not empty, which stopped ZFS, the virtual disk, from mounting.

The most feature of fake "/nas" is the minimal size (several bp or kb, small anyway), which's not consistent with the original ones. The /nas virtual disk had not been mounted but the ZFS software still working, resulting in some fake folders in system disk.

Second, based on the answer I found in ZFS not mounting - OMV 4.x, I did steps following:

1. Uninstall ZFS plugin

2. Delete all the fake(empty) folders in /nas.

3. Install ZFS plugin. It would apper the error information like "Failed to read from socket: Connection reset by peer", but it doesn't matter.

4. reboot.

Because the /nas had been empty, so the ZFS could be mounted automatically. Finally it works!

Check disk status again:

Code

$ df -h
Filesystem                                                            Size  Used Avail Use% Mounted on
udev                                                                  1.9G     0  1.9G   0% /dev
tmpfs                                                                 383M  7.6M  376M   2% /run
/dev/sda1                                                              11G  4.9G  5.4G  48% /
tmpfs                                                                 1.9G     0  1.9G   0% /dev/shm
tmpfs                                                                 5.0M     0  5.0M   0% /run/lock
tmpfs                                                                 1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs                                                                 1.9G  4.0K  1.9G   1% /tmp
nas                                                                   6.9T  2.1T  4.8T  30% /nas
tmpfs                                                                 383M     0  383M   0% /run/user/0

Alles anzeigen

/nas was just back!

Code

$ systemctl status zfs-mount
● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2020-07-02 09:08:14 CST; 7min ago
     Docs: man:zfs(8)
  Process: 1019 ExecStart=/sbin/zfs mount -a (code=exited, status=0/SUCCESS)
 Main PID: 1019 (code=exited, status=0/SUCCESS)

Jul 02 09:08:14 openmediavault.local systemd[1]: Starting Mount ZFS filesystems...
Jul 02 09:08:14 openmediavault.local systemd[1]: Started Mount ZFS filesystems.

zfs-mount.service now is running. Happy ending.

The ZFS modules are not loaded after reboot

Jetzt mitmachen!

Tags