checksum errors in a ZFS pool only in OMV not in BSD

raulfg3 · 19. März 2015

I expose my really weird case to others power user to try to identify the problem.

I install and create a ZFS pool and notice estrange down on trhoughtput when cpoy my initial data ( about 6TB of movies), so I scrub the pool and meet this:

Code

Pool status (zpool status):


  pool: Rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 435M in 1h33m with 1 errors on Sun Mar 15 22:20:56 2015
config:


    NAME                                  STATE     READ WRITE CKSUM
    Rpool                                 ONLINE       0     0     1
      raidz1-0                            ONLINE       0     0    36
        ata-TOSHIBA_DT01ACA300_Z2P592PAS  ONLINE       0     0   490
        ata-TOSHIBA_DT01ACA300_Z2P4LS0GS  ONLINE       0     0   357
        ata-TOSHIBA_DT01ACA300_Z2P5909AS  ONLINE       0     0   494
        ata-TOSHIBA_DT01ACA300_Z2P58ZSAS  ONLINE       0     0   320
      raidz1-1                            ONLINE       0     0    23
        ata-TOSHIBA_DT01ACA300_63NZK2UGS  ONLINE       0     0   452
        ata-TOSHIBA_DT01ACA300_63NZKLSKS  ONLINE       0     0   264
        ata-TOSHIBA_DT01ACA300_63NZ4Z9GS  ONLINE       0     0   513
        ata-TOSHIBA_DT01ACA300_63QZR0TGS  ONLINE       0     0   290
      raidz1-2                            ONLINE       0     0   107
        ata-TOSHIBA_DT01ACA300_94OL4R2GS  ONLINE       0     0 1,97K
        ata-TOSHIBA_DT01ACA300_94OL4S4GS  ONLINE       0     0 1,90K
        ata-TOSHIBA_DT01ACA300_94OL228GS  ONLINE       0     0 1,87K
        ata-TOSHIBA_DT01ACA300_94OL343GS  ONLINE       0     0 1,58K


errors: 1 data errors, use '-v' for a list

Alles anzeigen

so I suspect a RAM corruption and shutdown my NAS and boot a memtesrt Cd, all test was passed so no RAM problem.

my next test was to do a new scrub , this time I do not copy any file to the pool so I expect no more checksum error, surprise, more checksum errors where this time corrected:

Code

Pool status (zpool status):


  pool: Rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 315M in 1h33m with 1 errors on Mon Mar 16 18:56:51 2015
config:


    NAME                                  STATE     READ WRITE CKSUM
    Rpool                                 ONLINE       0     0     1
      raidz1-0                            ONLINE       0     0    28
        ata-TOSHIBA_DT01ACA300_Z2P592PAS  ONLINE       0     0   300
        ata-TOSHIBA_DT01ACA300_Z2P4LS0GS  ONLINE       0     0   245
        ata-TOSHIBA_DT01ACA300_Z2P5909AS  ONLINE       0     0   267
        ata-TOSHIBA_DT01ACA300_Z2P58ZSAS  ONLINE       0     0   227
      raidz1-1                            ONLINE       0     0    17
        ata-TOSHIBA_DT01ACA300_63NZK2UGS  ONLINE       0     0   358
        ata-TOSHIBA_DT01ACA300_63NZKLSKS  ONLINE       0     0   206
        ata-TOSHIBA_DT01ACA300_63NZ4Z9GS  ONLINE       0     0   305
        ata-TOSHIBA_DT01ACA300_63QZR0TGS  ONLINE       0     0   223
      raidz1-2                            ONLINE       0     0    60
        ata-TOSHIBA_DT01ACA300_94OL4R2GS  ONLINE       0     0 1,41K
        ata-TOSHIBA_DT01ACA300_94OL4S4GS  ONLINE       0     0 1,15K
        ata-TOSHIBA_DT01ACA300_94OL228GS  ONLINE       0     0 1,59K
        ata-TOSHIBA_DT01ACA300_94OL343GS  ONLINE       0     0 1,32K


errors: 1 data errors, use '-v' for a list

Alles anzeigen

So something goes wrong whit my pool, my last test where to install a FreeNAS in a new boot disk and import the pool.

Import was sucessfull and I can configure SMB shares on FreeNAS and do a new scrub and susrprise no checksum errors:

Code

root@RNAS] ~# zpool status
  pool: Rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub in progress since Mon Mar 16 23:19:49 2015
        272G scanned out of 4.61T at 845M/s, 1h29m to go
        0 repaired, 5.75% done
config:


        NAME        STATE     READ WRITE CKSUM
        Rpool       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da3p1   ONLINE       0     0     0
            da0p1   ONLINE       0     0     0
            da1p1   ONLINE       0     0     0
            da2p1   ONLINE       0     0     0
          raidz1-1  ONLINE       0     0     0
            da7p1   ONLINE       0     0     0
            da4p1   ONLINE       0     0     0
            da5p1   ONLINE       0     0     0
            da6p1   ONLINE       0     0     0
          raidz1-2  ONLINE       0     0     0
            ada5p1  ONLINE       0     0     0
            ada4p1  ONLINE       0     0     0
            ada3p1  ONLINE       0     0     0
            ada6p1  ONLINE       0     0     0

Alles anzeigen

I copy a new amount of data by SMB and repeat a new scrub, and no errors are in checksum:

Code

[root@RNAS] ~# zpool status -v
  pool: Rpool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 1h32m with 0 errors on Wed Mar 18 00:06:21 2015
config:


        NAME        STATE     READ WRITE CKSUM
        Rpool       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da3p1   ONLINE       0     0     0
            da0p1   ONLINE       0     0     0
            da1p1   ONLINE       0     0     0
            da2p1   ONLINE       0     0     0
          raidz1-1  ONLINE       0     0     0
            da7p1   ONLINE       0     0     0
            da4p1   ONLINE       0     0     0
            da5p1   ONLINE       0     0     0
            da6p1   ONLINE       0     0     0
          raidz1-2  ONLINE       0     0     0
            ada5p1  ONLINE       0     0     0
            ada4p1  ONLINE       0     0     0
            ada3p1  ONLINE       0     0     0
            ada6p1  ONLINE       0     0     0

Alles anzeigen

My last test is to boot from OMV and import the pool, the pool was sucessfully imported, and do a new scrub ( remember that I do a previos scrub on FreeNAS and the pool was clear):
this time a new checksum errors appears:

Code

Pool status (zpool status):


  pool: Rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 5,05G in 5h0m with 2 errors on Thu Mar 19 20:33:16 2015
config:


    NAME                                  STATE     READ WRITE CKSUM
    Rpool                                 ONLINE       0     0     2
      raidz1-0                            ONLINE       0     0   170
        ata-TOSHIBA_DT01ACA300_Z2P592PAS  ONLINE       0     0 3,82K
        ata-TOSHIBA_DT01ACA300_Z2P4LS0GS  ONLINE       0     0 6,70K
        ata-TOSHIBA_DT01ACA300_Z2P5909AS  ONLINE       0     0 4,04K
        ata-TOSHIBA_DT01ACA300_Z2P58ZSAS  ONLINE       0     0 5,57K
      raidz1-1                            ONLINE       0     0   189
        ata-TOSHIBA_DT01ACA300_63NZK2UGS  ONLINE       0     0 5,23K
        ata-TOSHIBA_DT01ACA300_63NZKLSKS  ONLINE       0     0 5,62K
        ata-TOSHIBA_DT01ACA300_63NZ4Z9GS  ONLINE       0     0 4,32K
        ata-TOSHIBA_DT01ACA300_63QZR0TGS  ONLINE       0     0 5,61K
      raidz1-2                            ONLINE       0     0 1,11K
        ata-TOSHIBA_DT01ACA300_94OL4R2GS  ONLINE       0     0 15,5K
        ata-TOSHIBA_DT01ACA300_94OL4S4GS  ONLINE       0     0 24,9K
        ata-TOSHIBA_DT01ACA300_94OL228GS  ONLINE       0     0 16,0K
        ata-TOSHIBA_DT01ACA300_94OL343GS  ONLINE       0     0 24,3K


errors: 2 data errors, use '-v' for a list


Pool details (zpool get all):

Alles anzeigen

So my questions are:

1 - why the same pool in the same hardaware have checksums errors with ZFS in OMV and do not have errors in FreeNAS?.

2 - If you have a ZFS pool , can you please do a scrub and post result ( I'll try to identify if this happens only to me or happenhs to others ZFS users).

Thanks.

raulfg3 · 26. März 2015

Hello, sorry for bump this topic, but.....

anyone have interest to do a scrub in his ZFS pool to answer my question?

raulfg3 · 17. Mai 2015

still interested on some test by others users.

blublub · 17. Mai 2015

My scrub is clean, no error reported.

Was the zfs partition created in OMV Linux or with BSD?

blublub · 17. Mai 2015

Please also post complete system specs. Thx

raulfg3 · 17. Mai 2015

ZFS pool created on Linux=OMV using ZFS Pluging as described here: [HOWTO] Instal ZFS-Plugin & use ZFS on OMV

System spec on my signature

To complete info, I notice that my boot disk change between rebots, sometimes is /dev/sda and sometimes is /dev/sdm this is supposed that do not affect to ZFS , and really do not affect the pool when I mount in Nas4Free or FreeNAS systems ( based on BSD).

blublub · 17. Mai 2015

Yeah that should not matter as ZFS uses UID's to identify drives.

blublub · 17. Mai 2015

phew that really is strange.

Questions:

1) For how long did you let Memtest run?
2) Do you use ECC or regular non correcting RAM?

raulfg3 · 17. Mai 2015

Zitat

1) For how long did you let Memtest run?

several hours (12 or more hours), no errors detected.

Zitat

2) Do you use ECC or regular non correcting RAM?

Regular RAM ( Not ECC), But as I say I use same hardware to test the pool, checksun errors only happens on ZFS-Onlinux=OMV and do not happens in BSD=FreeNAS.

blublub · 18. Mai 2015

12h is not enough 24 or 48h are needed.

Using ZFS and non correcting RAM kinda spoils the advantages of ZFS.
There really is a reason why ECC RAM is needed for ZFS.

BSD and OMV Linux might use ram differently so that it does not happen on BSD.

Options:
1 build a proper rig with ECC

2 run memtest longer hoping you find the culprit

3 remove your ram modules one by one and do checksums/scrubs

raulfg3 · 21. Mai 2015

72 h of RAM test and NO errors.

Remember I do the test in the same NAS only difference is Boot OS :

- OMV=Debian = Multiple checksum errors
- FreeNAS=BSD = No errors.

So my point is that shift disk syndrome is the cause. (diferent order of disk in the boot)

If you have a working pool,can you please shutdown, interchange 2 disk and test copying 100 or 200 GB of data, if my suspect are correct you must show same checksum errors ( you do not lose your data, only need to recover disk to original order or SATA port and do a scrub).

ryecoaaron · 21. Mai 2015

Do you dual boot between Freenas and OMV? If so, why?

raulfg3 · 22. Mai 2015

Zitat von ryecoaaron

Do you dual boot between Freenas and OMV? If so, why?

I have 2 boot disk , and boot from FreeNAs or boot from OMV acording to my need.

I prefer OMV, But actually I can't use because checksum errors on my ZFS pool.

Same hardware, pool, & disk , in same order, works fien when boot FreeNAS , so actually I use FreeNAS, but I want to discover what is the problem to return to use OMV, as I say I suspect that disk are not detected in boot in same order or otrher problem ( not sure what problem is it).

blublub · 22. Mai 2015

Mhh I had my drive detection order change twice while exchanging all my drives and I did not get any errors. I am a bit reluctant to test that on my production machine.

What about an incompatibility between ZFS in Freenas and OMV - so that Freenas changes something in the file system that Linux ZFS can't handle - do you get errors on every scrub in OMV or ONLY in the first scrub after reboot/change from Freenas to OMV?

raulfg3 · 22. Mai 2015

on every scrub after a copy of data to pool, if I do a scrub, and do not copy data and do a second scrub, this second do not show errors, only happens if I write data to pool.

ryecoaaron · 22. Mai 2015

Have you created the pool in OMV or is it always created in freenas?

raulfg3 · 22. Mai 2015

I create the pool in OMV as describe here: [HOWTO] Instal ZFS-Plugin & use ZFS on OMV

I know that is strange, and no other have the same problem, so my heart says that is my problem in my hardware, but as I say previosly, same hardware and pool and disk , and only difference is boot disk, works, so i'll try to identify the exact problem.

Perhaps Disk by path are alias of /dev/sdx ?

In my opinnion /dev/sdx must be aliases of diskbypath not viceversa

ryecoaaron · 22. Mai 2015

That is old motherboard if you are using the P5Q. Maybe FreeNAS actually has better drivers for the board?? Are you using the 3.2 or 3.16 kernel?

raulfg3 · 22. Mai 2015

3.16 kernel

blublub · 22. Mai 2015

Mhh try 3.2 .

What OS the BIOS setting for the SATA controller - AHCI vor IDE?

checksum errors in a ZFS pool only in OMV not in BSD

Jetzt mitmachen!

Tags