OMV5: Cannot find cause or resolve error: syslog kernel - EDAC pnd2: Failed to register device with error -22

Hello everyone, I returned to my hometown where I have a small headless Intel NUC that runs OMV 5.6.26-1 (i.e. latest).

I performed a BIOS update to this NUC (model NUC6CAYH) and upon plugging it back to a monitor+keyboard, I saw an error that keeps repeating upon every boot, that I had not noticed before (4 entries in each boot, not anymore later):

$ grep -E "error" /var/log/syslog

May  8 11:48:43 NUC-NAS watchdog[1357]: error retry time-out = 60 seconds
May  8 11:50:33 NUC-NAS kernel: [    9.562856] EXT4-fs (sdb1): re-mounted. Opts: errors=remount-ro
May  8 11:50:33 NUC-NAS kernel: [   11.463860] EDAC pnd2: Failed to register device with error -22.
May  8 11:50:33 NUC-NAS kernel: [   11.502157] EDAC pnd2: Failed to register device with error -22.
May  8 11:50:33 NUC-NAS kernel: [   11.566280] EDAC pnd2: Failed to register device with error -22.
May  8 11:50:33 NUC-NAS kernel: [   11.618265] EDAC pnd2: Failed to register device with error -22.
May  8 11:50:53 NUC-NAS watchdog[1187]: error retry time-out = 60 seconds
[...]
May  8 14:24:24 NUC-NAS kernel: [    8.800715] EXT4-fs (sdb1): re-mounted. Opts: errors=remount-ro
May  8 14:24:24 NUC-NAS kernel: [   10.580472] EDAC pnd2: Failed to register device with error -22.
May  8 14:24:24 NUC-NAS kernel: [   10.615192] EDAC pnd2: Failed to register device with error -22.
May  8 14:24:24 NUC-NAS kernel: [   10.675325] EDAC pnd2: Failed to register device with error -22.
May  8 14:24:24 NUC-NAS kernel: [   10.767136] EDAC pnd2: Failed to register device with error -22.
May  8 14:24:44 NUC-NAS watchdog[1178]: error retry time-out = 60 seconds
[...]
May  8 14:40:34 NUC-NAS kernel: [    9.218320] EXT4-fs (sdb1): re-mounted. Opts: errors=remount-ro
May  8 14:40:34 NUC-NAS kernel: [   11.254685] EDAC pnd2: Failed to register device with error -22.
May  8 14:40:34 NUC-NAS kernel: [   11.297516] EDAC pnd2: Failed to register device with error -22.
May  8 14:40:34 NUC-NAS kernel: [   11.345416] EDAC pnd2: Failed to register device with error -22.
May  8 14:40:34 NUC-NAS kernel: [   11.393533] EDAC pnd2: Failed to register device with error -22.
May  8 14:40:45 NUC-NAS watchdog[1181]: error retry time-out = 60 seconds

I tried to find out on other Linux-related forums for information on identifying EDAC and pnd2 device, I only found out that EDAC stands for Error Detection And Correction (EDAC). Most people talk about ECC memory and of course, this Intel NUC does not have ECC memory inside (nor supporting it).

The Intel NUC Visual BIOS has no settings for memory, as perhaps on typical motherboards. And unfortunately, Intel does not allow rolling-back i.e. downgrade BIOS.

The advice to resolve this "EDAC pnd2 failed to register error -22" found was a) update BIOS first, then b) run a previous kernel. The output of my hardware configurations is the following:

$ lscpu

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       39 bits physical, 48 bits virtual
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               92
Model name:          Intel(R) Celeron(R) CPU J3455 @ 1.50GHz
Stepping:            9
CPU MHz:             798.873
CPU max MHz:         2300.0000
CPU min MHz:         800.0000
BogoMIPS:            2995.20
Virtualization:      VT-x
L1d cache:           24K
L1i cache:           32K
L2 cache:            1024K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault cat_l2 ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts md_clear arch_capabilities

$ dmidecode --type memory

Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x002E, DMI type 16, 23 bytes
Physical Memory Array
    Location: System Board Or Motherboard
    Use: System Memory
    Error Correction Type: None
    Maximum Capacity: 32 GB
    Error Information Handle: Not Provided
    Number Of Devices: 4

Handle 0x002F, DMI type 17, 40 bytes
Memory Device
    Array Handle: 0x002E
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 4096 MB
    Form Factor: SODIMM
    Set: None
    Locator: ChannelA-DIMM0
    Bank Locator: BANK 0
    Type: DDR3
    Type Detail: Synchronous
    Speed: 1866 MT/s
    Manufacturer: Kingston
    Part Number: KHX1866C11S3L/4G  
    Rank: Unknown
    Configured Memory Speed: 1866 MT/s
    Minimum Voltage: 44.975 V
    Maximum Voltage: 44.975 V
    Configured Voltage: 1.5 V

Handle 0x0030, DMI type 17, 40 bytes
Memory Device
    Array Handle: 0x002E
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 4096 MB
    Form Factor: SODIMM
    Set: None
    Locator: ChannelB-DIMM0
    Bank Locator: BANK 1
    Type: DDR3
    Type Detail: Synchronous
    Speed: 1866 MT/s
    Manufacturer: Kingston
    Part Number: KHX1866C11S3L/4G  
    Rank: Unknown
    Configured Memory Speed: 1866 MT/s
    Minimum Voltage: 1.35 V
    Maximum Voltage: 1.5 V
    Configured Voltage: 1.5 V

$ lsmem

RANGE                                 SIZE  STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff   2G online       yes  0-15
0x0000000100000000-0x000000027fffffff   6G online       yes 32-79

Memory block size:       128M
Total online memory:       8G
Total offline memory:      0B

My Intel NUC has 2x4GB Kingston memory modules (same mode each). Not sure why lsmem reports this as 2+6GB...

I could not find to what relates this pnd2 device, but if anyone from you has encountered this error and understands it or can confirm it's related to ECC memory checking, can you kindly help me fix this error?

Thank you all in advance.

Hi ryecoaaron  macom have you gents come across in your experience with this error? Any understanding as to what could possibly be causing it? I'd be happy to provide any other log or info I can... thank you.

Thank you for your time ryecoaaron and your reply.

I am using the latest kernel available for OMV5.x and I understand it's EOL.

Thus, I will try upgrading to OMV6.x on this existing installation, via the guide(s) available in the forum, and will report back.

Many thanks!

Apologies ryecoaaron you are right asking about the kernel flavour. Since I am not knowledgeable on Linux kernels, I use the vanilla one that came with the last OMV5.x update, namely kernel 5.10.0-0.deb10.16-amd64

I found this guide for in-place, on-device upgrade of OMV major version, so I will attempt this, this coming weekend. I assume now OMV6 is mature without the possible hiccups of the past for such an upgrade:

Thema

Is there a guide to in place upgrade?

Hi all

Reading bits and pieces in different threads regarding the CLI upgrade path from 5 to 6, but wondered if there's a thread that covers the steps in one place that I'm struggling to find?

My basic understanding is:

1. Backup OS drive
2. Check any used plugins have been ported to OMV6 and if they haven't install via Docker instead
3. Temporarily remove data drives
4. Disable running services
5. Go in via CLI (Putty in my case) and run the couple of update commands (when I find them again!)
6. Wait
…

Adrian E

7. März 2022

I am booting from USB whilst keeping the SSD inside the NUC for 1 single data partition, so what remains is to find a way to perfectly clone my USB installation, first

Thanks again for your replies!

Zitat von crashtest

Did you notice that the error is related to memory AND did you see the notable difference between memory modules in the output of

dmidecode ?
Have you tried popping out one SODIMM at a time?

Wow! That totally escaped me. Where on earth is this coming from! I will try using one SO-DIMM at a time, this weekend. Thanks mate.

Could this be a stupid BIOS firmware bug (latest update) ? It is curious... I will see inside VISUAL BIOS what's been reported.

Zitat von ryecoaaron

You must not have installed OMV using the ISO or you disabled backports.

The first release of OMV 6 was almost two years ago. I would say most hiccups that people are run into have threads about them on this forum. Not even windows can do upgrades without hiccups though. So, no one can guarantee that. I have done many of them without issues.

I most certainly created a USB installer from the original 5.x ISO from the repository. I remember because the installer could not detect the SATA controller at first, had to reboot many times + re-do the installation USB (via Rufus, I think, it's been 2 years) to be sure it had not failed.

With regard to "backports" I do not remember tampering with these except adding OMV-Extras like everyone else:

Why did you suspect the backports ryecoaaron ? I mean, was I supposed to get a newer kernel on OMV5.x ?

I am curious. Thanks, nevertheless.

From your post in another thread, I ran this command so nothing new exists, it seems:

$ apt-get -t buster-backports install linux-image-amd64

Reading package lists... Done
Building dependency tree       
Reading state information... Done
linux-image-amd64 is already the newest version (5.10.127-2~bpo10+1).
0 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.

Give me time, you won't need to ever remember it, mate! (i.e. to remember that I was on OMV5)

UPDATE: OK ryecoaaron I launched sudo omv-release-upgrade process after the needed checks. All went smoothly. Rebooted, did omv-upgrade, rebooted, system is now at 6.3.10-2 (Shaitan) and vanilla kernel: Linux 6.1.0-0.deb11.6-amd64. P.S. I don't see any method to switch kernels like in OMV5's Web UI.

Rebooting now does show the same error, but only once:

$ grep -E "EDAC" /var/log/syslog

May 11 17:20:42 nuc-nas kernel: [   14.921612] EDAC pnd2: Failed to register device with error -22.
[...]
May 11 17:26:47 nuc-nas kernel: [    0.271817] EDAC MC: Ver: 3.0.0
May 11 17:26:47 nuc-nas kernel: [   22.636883] EDAC pnd2: b_cr_tolud_pci=080000001 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.636932] EDAC pnd2: b_cr_touud_lo_pci=080000000 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.636974] EDAC pnd2: b_cr_touud_hi_pci=000000002 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.637016] EDAC pnd2: b_cr_asym_mem_region0_mchbar=000000000 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.637058] EDAC pnd2: b_cr_asym_mem_region1_mchbar=000000000 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.637099] EDAC pnd2: b_cr_mot_out_base_mchbar=000000000 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.637141] EDAC pnd2: b_cr_mot_out_mask_mchbar=000000000 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.637222] EDAC pnd2: b_cr_slice_channel_hash=80000d9480001248 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.637264] EDAC pnd2: b_cr_asym_2way_mem_region_mchbar=000000000 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.637306] EDAC pnd2: d_cr_drp0=0e000c005 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.637347] EDAC pnd2: d_cr_drp0=000000000 ret=2
May 11 17:26:47 nuc-nas kernel: [   22.637389] EDAC pnd2: d_cr_drp0=0e000c005 ret=0
May 11 17:26:47 nuc-nas kernel: [   22.637430] EDAC pnd2: d_cr_drp0=000000000 ret=2
May 11 17:26:47 nuc-nas kernel: [   22.637432] EDAC pnd2: ECC disabled on channel 0
May 11 17:26:47 nuc-nas kernel: [   22.637433] EDAC pnd2: ECC disabled on channel 2
May 11 17:26:47 nuc-nas kernel: [   22.637434] EDAC pnd2: Failed to register device with error -22

(no other errors at 17:32:10 logged)

I will check those SODIMMs like crashtest suggested. Just wanted to let you know of the results of my efforts. I understand it's a kernel issue not OMV6, so any advice or guidance, is welcome.

Cheers.

Zitat von ryecoaaron

I had to separate the kernel features of omv-extras into a separate plugin - openmediavault-kernel

Are these causing any problems or you just want to fix the message?

(thanks for the tip on the kernels plugin. Good idea to have it like that ; I know it's late to comment on it )

Ideally this error is not seemingly causing issues, the server boots OK, all services run OK via systemctl list-units --type=service --no-pager and systemctl list-units -all --state=failed

In my desire and process of learning and seeing/judging the criticality of errors (and now this one) I looked around and was hoping to understand its source. It can be ignored, I guess.

(don't go thinking this OMV installation is pushed hard, it's for home use with a handful of docker containers, like Syncthing, Homebridge etc.)

Hi ryecoaaron and everyone, apologies for replying few days later, it's been a busy and away-from-keyboard days for me.

So tonight I opened up my NUC, removed one SO-DIMM, swapped the memory slots, swapped SO-DIMMs, set BIOS settings to "defaults" and the result remains the same:

a) In all cases and with 1 SO-DIMM installed, using dmidecode --type memory I get still (in either SO-DIMM slot!) the same voltage result:

Minimum Voltage: 44.975 V
Maximum Voltage: 44.975 V

Configured Voltage: 1.5 V

(interesting because the absence of the other module makes either one report 44V!)

Please note that my SO-DIMMs are identical Kingston HyperX IMPACT model HX318LS11IB/4 (1866 MHz, DDR3L, CL11, SODIMM 1.35V) at 1.35V and not 1.5V, as required by Intel for this NUC! In Intel Visual BIOS screen, no warning appears on the memory screen or tab.

b) In all cases, I get the "EDAC pnd2: Failed to register device with error -22" error

This error is registered once, per boot, it seems.

So my options are the following, as I can see it:

a) Not sure this is related to the actual physical SO-DIMM modules (i.e. they didn't get bad);

b) Perhaps the latest BIOS firmware update has an issue, I will open a ticket to Intel Support although I SERIOUSLY doubt they will accept there's a bug there (especially coz I am using Debian) let alone fix it, as this is an older model;

c) Configure the kernel, if possible, to not deal with ECC?

d) To live long and prosper, by ignoring this error (but learning more, is interesting)

ryecoaaron is it possible for e.g. grub to not take into consideration ECC?

(or is this a n00b question/mistake? Apologies, I am not too knowledgeable in such deeper Debian tweaks)

Also, the message reference you posted (in that URL) where this "pnd2 fail" text should be marked as INFO instead of ERROR, dates back from 2020 and not sure if it was implemented, eventually? (from the last reply in the mail thread, it's not clear as of the fate of this reported issue)... I assume OMV is dependent on these kernels, and doesn't build their own, yes?

I understand this is not related to OMV, so your time/insight spent on this, is welcome. At least this will be available for others to consult, if it arrives again.

Many thanks.

Thank you ryecoaaron for your time, it's appreciated as I ended up learning a few things, so greater good achieved

If there is any development for any reason, I may come back and post here. Cheers.

So much for "help"...

Zitat

Thank you for contacting Intel Customer Support.

Please be advised that this NUC model has not been tested and validated by Intel to work with linux. Therefore using it with a linix distribution it may or may not work or as in your situation may have some unexpected behavior:

https://www.intel.com/content/…/000005499/intel-nuc.html

This NUC has not been validated to work with Linux. Taking this in consideration, we cannot comment on diagnostics or reporting under the specific OS.

Since the BIOS does not report anything abnormal, the NUC should work just fine with the supported operating systems.

Alles anzeigen

No Linux is found under my NUC6CAYH model. Not sure why they don't even support or test even 1 distro...

I will run memtext86 and hope all goes well. Thanks crashtest