ZFS Degraded Pool

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • Well, if you will, install curl (very small) and run the curl command line on one of those drives. It would be good to get full detail on at least one of those drives.
      __________________________________

      In attempts to reproduce the behavior in your accounting, I noticed the following:

      - If a drive faults in a serious way, with autoreplace "on", ZFS automatically reformats and starts resilvering.
      - After this process is complete (with a new drive or, actually, a drive with a new ID) the checksum on the drive is "5". (Screen capture above.)
      - What I can't simulate in a VM is a large string of continuing errors that results in kicking a drive out of the pool - "Too many errors". (This limitation is irrelevant in that you seem to have something else going.)

      If the drive(s) are healthy, you may have an actual hardware problem. It's certainly not beyond possibility. I've read ZFS accountings where others had unnoticed hardware problems, while admins / users weren't monitoring their servers / workstations. ZFS masked the problem, and it went on for months or until there was a complete failure.

      While this is speculation, but based on actual experience, this wouldn't be the first time I've seen issues crop up from ESD damage, intermittent or full time. (Ungrounded fingers,, power transients that get through a PS?) It happens, and it could have happened when your Mobo was new, with the actual symptoms just starting to show up years later. I'm sure, in your business, you've seen Mobo's do inexplicable things, for no apparent reason, that diagnostics couldn't detect. I pitched one recently for just this reason.
      In this forum there was another thread where a user with an mdadm array was experiencing crc errors on single port. He replaced the cable with the same effect, continuing errors. And despite the issue being isolated to a single port, he didn't think it could be a hardware issue.

      So what I'm getting at is, if you do a full rebuild from scratch, maybe use another Mobo?

      But before wiping everything out, if you have another sata port available, I'd consider moving the new drive to a different port and using the replace command. (It wouldn't hurt anything, but 7 sata ports would be a rare Mobo.)
      ________________________________

      In any case, I know this is a PITA.
      Regrets....
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119

      The post was edited 2 times, last by flmaxey: edit ().

    • if ZFS detected an error on new disk, please revise SATA cabes and/or power cables.

      is not a waste of time detect and error
      OMV 3.0.96 x64 on a HP T510, 16GB CF as Boot Disk & 32GB SSD 2,5" disk for Data, 4 GB RAM, CPU VIA EDEN X2 U4200 is x64 at 1GHz

      Post: HPT510 SlimNAS ; HOWTO Install Pi-Hole ; HOWTO install MLDonkey ; HOHTO Install ZFS-Plugin ; OMV_OldGUI ; ShellinaBOX ;
      Dockers: MLDonkey ; PiHole ;
      Videos: @TechnoDadLife
    • bookie56 wrote:

      HTML Source Code

      1. root@rocky:~# for disk in /dev/sdg ; do smartctl -x $disk ; done | curl -F 'sprunge=<-' http://sprunge.us
      2. <html>
      3. <head>
      4. <title>500 Internal Server Error</title>
      5. </head>
      6. <body>
      7. <h1>500 Internal Server Error</h1>
      8. The server has either erred or is incapable of performing the requested operation.<br /><br />
      Comes up as above?
      Crappers. Something seems to have changed with curl - probably in a recent update.
      (Posting your drive data to a sprunge URL would be convenient, however.....)

      The same info can be had in the GUI under Storage, SMART, the Devices Tab, click on the device, Information, Extended Information. Copy all in the window into a *.txt file and attach.

      Based on what you've posted already, I don't think the drive(s) are dying but it won't hurt to look at extended drive stat's.
      ________________________________________________________________________

      On ZFS, at a guess, I don't think the problem you're observing has anything to do with the filesystem itself. ZFS was born in SUN Microsystems - with servers and their operating system, Solaris, as their bread and butter. ZFS, even on IBM compatibles, is mature and has been vetted in numerous field deployments worldwide.
      But to balance that out:
      ZFS on Linux is enabled by a "kernel module" and I've seen an instance where, if something in ZFS is grossly misconfigured, the kernel displays bizarre behavior. (Still, I can't imagine a software issue where a single sata port would be affected.)

      When you get to the "point" - before trashing the pool and starting over:
      Consider exporting the pool and uninstalling ZFS. Then reinstall ZFS again, VIA the GUI plugin this time around, and import the pool. It wouldn't take long and it won't hurt to try. (Unfortunately, if the existing pool has odd or unsupported properties, importing the pool won't change those properties.)

      If you built your pool from the command line the first time around, the second time, I would stick with the ZFS GUI plugin and create the pool from there.
      ________________________________________________________________________

      I'm planning to explore the ZFS video tutorial you posted, to see what configuration variables it recommends.
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • So,, I did the tutorial and noted a difference in installation sources:

      When installing per the tutorial on zfsonlinux.org , their instructions for Debian Jessie,(OMV3) included adding a line to source.list -> ftp.debian.org/debianjessie-backports main

      With ZFS, as installed on OMV, the backports repository noted above is not in source.list. The closest is ftp.us.debian.org/debian/jessie main The ZFS packages may be identical between the two sources, or they may be different. I don't know.

      ______________________________________________

      As I've come to find, there's really no clear method of uninstalling ZFS packages, so..:
      ______________________________________________

      - Per the tutorial Video, I created a pool on the command line. Then, I exported it.
      - I removed the jessie-backports repository from source.list
      - It was necessary to reboot (to unload ZFS)
      - I installed ZFS using the OMV plugin.
      (All appeared to install properly, with a few minutes needed to compile the kernel module.)
      - The pool auto imported and was displayed in the GUI plugin.

      I compared my server's pool to the VM pool. I didn't find a difference between the server and the VM's zfs or zpool attributes, but one of the first things I did was apply the changes for ACL's.

      I don't know if this will fix anything (I tend to doubt it) but the process is quick and a determination could be made after a resilver and reboot.
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • Hi guys!
      I am not counting my chickens yet but...
      I added a new RM750x Moduler PSU and 6 new sata cables and I didn't even get an initial problem after restart



      I am going to run a scrub to see what gives...


      This is just after starting the scrub:



      I am getting scrub errors on the same three drives....but no error from last drive.....I am not willing to believe that I have three more new drives that are faulty and haven't seen a sign that one is faulty yet.....but if a scrub keeps bringing up the same errors ..what do I do?

      Not really willing to believe in hardware problems....just wondering if I back up the files again and remove and readd the pool and just add a few files to see what gives.....

      This has already cost time and money and it is supposed to give us piece of mind....When!!!

      bookie67

      The post was edited 2 times, last by bookie56 ().

    • I am at a loss tu understand why this motherboard is showing errors with zfs going t get the files on a external drive and then start again with just a few files...if the same problem comes up again despite new sata cables and psu then I will have to build a new rig or buy something reliable....
      I still think zfs is over rated!!
      Got any recommendations to new rig please say....


      bookie56
    • bookie56 wrote:

      I am at a loss tu understand why this motherboard is showing errors with zfs going t get the files on a external drive and then start again with just a few files...if the same problem comes up again despite new sata cables and psu then I will have to build a new rig or buy something reliable....
      I still think zfs is over rated!!
      Got any recommendations to new rig please say....


      bookie56
      I won't deny what it seems like but, look at it from another angle:
      If you have a Mobo that actually has emerging issues and you were running ext4 over mdadm RAID you might not see anything wrong at all. That is until you try to use an older image for restoring. (And just a bit of corruption in the restored image might not manifest itself in your shop...) From your posts, so far, it seems that ZFS is correcting data corruption and the pool is intact.

      Of the name brand Mobo's I've bought (not PC OEM boards), I've went with Asus. I don't have a lot of experience with Asus Mobo's, but so far, there have been no issues. (Also, in many cases, Asus supports ECC.) The main gripe, as it seems, is Asus puts out a board and the drivers it supplies, at that time, tend to be "it". Accordingly, Windows users complain as M$ changes Windows versions and Asus doesn't supply drivers endlessly.

      Have you run memtest86, afterhours?
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • While good to have, ECC is not a hard requirement. How about pulling an older, unused, customer box with at least 6 sata ports?

      If you statically addressed OMV and are using a USB thumb drive to boot:
      Since OMV isn't driver sensitive, it might be a matter of booting up. The only real concerns would be the platform 64bit and the NIC. (I know of at least one forum user that built on one box and transferred to another.) In any case, a new OMV build doesn't take long and, as I've noticed, a ZFS pool will import readily.

      I know it might not be as simple as the above and there are at least minimal risks to consider, but you have backup. (Always the smart thing to do.)
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • For safety, I'd considering cloning the boot drive before booting it on another box.
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • bookie56 wrote:

      Been looking at this motherboard

      bookie56
      Holy Toledo! That is one expensive Mobo - dual Xeon's..!!!

      But, and this is an opinion, do you really need anything more than an i3? (Unless you're doing something else I'm not aware of:) If you're just moving files around, and most of it is static storage (not transactional), hot processors are just going to be idle. In your case as I understand it, the network would be the bottleneck, not the CPU.

      For a client build a few months ago, I used an AMD FX8320 (PassMark around 8000) and this Mobo. It supports ECC. (Since this is one generation older than current tech, it might be harder to find but if you can the price will be good.) I bought this mobo/cpu pair and this case as a successor for my current server if/when it needs to be replaced. For any straight file server application, even this is way overkill.

      _______________________________________

      If you still suspect it's a ZFS issue:
      If you can find a old box with 6 sata ports (that could be a problem) I might try it out before dropping a lot of cash. (But I have more time than you do.)

      (On your import - to prevent an array reassembly issue - note the exact order of your drives and match them to the new sata ports.)
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • flmaxey wrote:

      bookie56 wrote:

      Been looking at this motherboard

      bookie56
      Holy Toledo! That is one expensive Mobo - dual Xeon's..!!!
      But, and this is an opinion, do you really need anything more than an i3? (Unless you're doing something else I'm not aware of:) If you're just moving files around, and most of it is static storage (not transactional), hot processors are just going to be idle. In your case as I understand it, the network would be the bottleneck, not the CPU.

      For a client build a few months ago, I used an AMD FX8320 (PassMark around 8000) and this Mobo. It supports ECC. (Since this is one generation older than current tech, it might be harder to find but if you can the price will be good.) I bought this mobo/cpu pair and this case as a successor for my current server if/when it needs to be replaced. For any straight file server application, even this is way overkill.

      _______________________________________

      If you still suspect it's a ZFS issue:
      If you can find a old box with 6 sata ports (that could be a problem) I might try it out before dropping a lot of cash. (But I have more time than you do.)

      (On your import - to prevent an array reassembly issue - note the exact order of your drives and match them to the new sata ports.)
      All good and sound info....going to check out your links for reference!
      Thanks for your time!!

      bookie56
    • Hi guys!
      Well, I have been busy and finally got the components together for a new rig with ECC ram....but that was easier said than done....
      When I ordered the memory for some reason I couldn't order both at the same time....anyway, the first one arrived and didn't work....so had to wait for the second one and time just got the better of me...
      I now have the following components:
      Chassis: Fractal Design Define XL R2 Black Pearl
      Mobo: ASUS X99-M WS/SE, Socket-2011-3
      CPU: Intel Xeon E5-2620 V4 Processor
      Memory: Samsung - DDR4 - 32 GB - DIMM 288-pin 2400 MHz / PC4-19200, CL17, 1.2 V, ECC
      System Drive: Samsung 960 EVO 250GB M.2 PCIe SSD
      Cooling CPU: Noctua NH-D15

      PSU: XFX 850W Semi Modular
      That was just for the main installation and then I have the drives from the old installation and after setting up OMV for ZFS I could add my zpool no problems....
      I didn't get any errors on zpool status so I forced a scrub to make sure things are as they should be....and that is where I am at the moment....
      The scrub hasn't thrown up any errors at all and I will update this when it is finished....
      Finally things are starting to look up....

      bookie56
    • I have to say, I'm relieved. Problems like those you had before can be maddening.
      (Even for someone like you - who knows how to deal with it. :) Issues like those you were having would help you to get a customers perspective on PC issues. )

      Here's to hoping for a clean scrub or, if there are a few errors found because of the old hardware, one corrective scrub then clean again.
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • New problem... :)
      When I installed zfs and then connected my 6 original 3TB drives the original pool mounted and was there every time I rebooted until adding the two 4TB drives and creating a new pool that things went wron...
      As I understand zpools are automatically mounted when created and that isn't a problem for the new pool because it was created on this machine...but the original one wasn't....
      so, my thinking is I need to mount the original pool automatically everytime I boot...
      Anyone got any info on that?

      bookie56
    • Ok! Panic over...God! I wish I was hit on the back of the head with a 4x2 sometimes...

      The situation was when I first installed ZFS and then added the OMV plugin my origianl pool was recognised straight away and I didn't need to import it.....bit strange...but didn't think any more about it untill I added the new pool....

      So, as you know the code for import is:

      Source Code

      1. # zpool import Rocky


      And after that I ran the:

      Source Code

      1. # zfs mount -a
      Now my original Rocky pool is available...and before anyone asks why Rocky...one of my cats two cats that had to be put down last year....after developing complications at the ripe old ages of 16...Rocky and Simson were both Holy Birma cats and named by the young boy of the cat breeder we used here in Sundsvall Sweden...Rocky after Sylvester Stallone's Rocky and Simson after the series Simsons....Rocky lived up to his name but Simson became Sir Simson because of his graceful way....

      Sorry, getting carried away there....working from home I had their company day in day out for 16 years and it is a bit quiet around here without them...

      Anyway, back to computers....

      Now both of my zpools are automatically listed as shown below:




      I can now concentrate on file transfers from customers....


      bookie56