Unexpected disk errors seen with PCIE card when adding more than 2 disks

1st Official Post

    I thought I would write up my notes of using a cheap PCIE card to add extra disks to my Open Media Vault setup. I have had a few problems with this type of disk controller but could not categorically prove it was at fault. Maybe other people who are using a similar controller might find these notes helpful. I have found I can use two disks but if I try to add more I get I/O errors even on the two disks that work fine when the later disk were not added.


    The controller I picked was this quite cheap one:

    BEYIMEI PCIE SATA Card 8 Port, 6 Gbit/s SATA 3.0 PCI-E Card 4X with 8 SATA Cables, Power Splitter Cable, JMB575 + ASM1166 PCIe to SATA Controller Expansion Card, Non-Raid, Boot as System Hard Drive
    BEYIMEI PCIE SATA Card 8 Port, 6 Gbit/s SATA 3.0 PCI-E Card 4X with 8 SATA Cables, Power Splitter Cable, JMB575 + ASM1166 PCIe to SATA Controller Expansion…
    www.amazon.co.uk

    BEYIMEI PCIE SATA Card 8 Port, 6 Gbit/s SATA 3.0 PCI-E Card 4X

    Chipset: ASMedia ASM1166


    From an OMV / Debian / Linux perspective we must be using it through this SATA controller


    Code
    root@omv23:~# lspci | grep ASM
01:00.0 SATA controller: ASMedia Technology Inc. Device 1166 (rev 02)
04:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller


    Verbose mode


    My Open Media Vault version is up to date:

    Code
    root@omv23:~# dpkg -l | grep openmediavault
ii  openmediavault                 6.3.2-2                        all          openmediavault - The open network attached storage solution
ii  openmediavault-kernel          6.4.5                          all          kernel package
ii  openmediavault-keyring         1.0                            all          GnuPG archive keys of the OpenMediaVault archive
ii  openmediavault-omvextrasorg    6.1.1                          all          OMV-Extras.org Package Repositories for OpenMediaVault
ii  openmediavault-zfs             6.0.12                         amd64        OpenMediaVault plugin for ZFS

    The original two disks I added look like this

    Code
    lsblk -o NAME,MODEL,SERIAL,WWN,HCTL,MOUNTPOINT,SIZE,FSTYPE,UUID,PARTUUID


sdh    TOSHIBA_HDWE160         10GAK1QDFB8G   0x50000399cbb81261 36:0:0:0                5.5T
├─sdh1                                        0x50000399cbb81261                         5.5T zfs_member 17579695501947784507                 de287461-af87-2849-a0d6-1cd58c90a959
└─sdh9                                        0x50000399cbb81261                           8M                                                 0547f32f-48e5-4345-896f-b1a3b733724d
sdi    TOSHIBA_HDWE160         10GXK1B6FB8G   0x50000399cc700b52 37:0:0:0                5.5T
├─sdi1                                        0x50000399cc700b52                         5.5T zfs_member 17579695501947784507                 6d50b91d-825a-1647-84dd-23005db0caaf
└─sdi9                                        0x50000399cc700b52                           8M                                                 d0671e81-7697-7247-9fd1-84cad7c99b9b

    The additional fours disk I was trying to add are very old: three SAMSUNG SpinPoint F1 DT, one SAMSUNG SpinPoint F3.

    Code
    sdd    SAMSUNG_HD103UJ         S13PJDWS228239 0x50000f0000fe02b9 10:0:0:0              931.5G
sde    SAMSUNG_HD103UJ         S13PJDWS228237 0x50000f0000fe02ab 11:0:0:0              931.5G
sdf    SAMSUNG_HD103SJ         S246JDWZ525473 0x50024e900392fa0f 34:0:0:0              931.5G
sdg    SAMSUNG_HD103UJ         S13PJDWS228229 0x50000f0000fe026b 35:0:0:0              931.5G


    I tested with the two Toshiba drives for weeks with no issues at all. When I add the four old Samsung disks (they were unformatted from a QNAP NAS which were working perfectly fine) we see within a few minutes the IO errors. These IO errors are seen on the Toshiba drive which made me think this is a Controller issue.


    There are many errors but to summarize they look like this




    I restarted the machine with just the four Samsung disks attached and they were fine on reboot. The sd d,e,f,g labels are now rearranged

    Code
    lsblk -o NAME,MODEL,SERIAL,WWN,HCTL,MOUNTPOINT,SIZE,FSTYPE,UUID,PARTUUID

...
sdd    SAMSUNG_HD103UJ          S13PJDWS228239 0x50000f0000fe02b9 10:0:0:0              931.5G
sde    SAMSUNG_HD103UJ          S13PJDWS228237 0x50000f0000fe02ab 11:0:0:0              931.5G
sdf    SAMSUNG_HD103SJ          S246JDWZ525473 0x50024e900392fa0f 34:0:0:0              931.5G
sdg    SAMSUNG_HD103UJ          S13PJDWS228229 0x50000f0000fe026b 35:0:0:0              931.5G

    and they did not detect any problems in their smart reports. NB That these SAMSUNG SpinPoint disks make an almost comic "Whoooop" sound on startup and have a fair bit of clicking - but I understand for these disks this is normal and they have always done this.


    At some point we do get IO errors reported and smartctl reports the following for all but the sdd disk on 10.0.0.0 where that's the HCTL: Host:Channel:Target:Lun identifier.

    Code
    root@omv23:~# smartctl --device=ata -H /dev/sdf
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-6.1.10-1-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: Input/output error

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.


    I bought brand new SATA leads and re-seated all power leads and was able to once again run long smart tests with no issue. After many hours I added the Toshiba disks and once again I got within ten minutes IO errors.


    I've read reports that some people think PCIE SATA cards are not a good idea

    They state they

    • can have buggy firmware
    • have issues with more than two SATA ports

    but I could find no backing for this in any other sources.

    They recommend instead: Serial Attached SCSI (SAS) Host Bus Adapter (HBA). I could not find many examples of these for sale.


    For the time being I'm using my motherboard's four SATA ports and two on the PCIE controller. Everything is working well. As my Samsung drives are very old I think I will just give up on them but I'm quite confident they are fine.

  • KM0201

    Approved the thread.

    Getting a SAS HBA instead is good advice, especially since you can find good used cards at prices comparable to new PCIE SATA cards.


    I have purchased several used LSI HBA cards on ebay. The most recent was an LSI SAS9207-8e that I paid $35US for last November. I have bought similar cards on Amazon for similar money.


    Have a look around on ebay. Here's a few pages of them for little money:


    Code
    https://www.ebay.com/sch/i.html?_from=R40&_trksid=p3814320.m570.l1313&_nkw=LSI+SAS9207-8i&_sacat=0


    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.


    Thanks 1

    Is it better because of the controller or something else? My mobo only has 1 PCIe x1 anh 1 PCIe x16, and I have another plan for the x16 slot, so I really want to use the x1 slot for extending extra HDDs. Could you give me any recommend?

    Well, LSI has been in the RAID/HBA card business a long time and has well known Linux compatibility. I don't think you can say the same thing about a lot of the no name SATA cards.


    You should match the slot you want to use with an available card.


    First LSI card was an LSI 9200-8E that I used in my ASRock server eight years ago. That machine has only one PCIe2.0 x8 slot and it worked perfectly.

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.


    Quote from andrzejls

    I use two of these for couple of years without any issues.


    I paid $2.00 more for an eight port LSI SAS9207-8 card.

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.


    I also care about the bandwidth of each disk while parallel operating. If I use an extended card with PCIe x1 slot, how many disks should I attach with this card?

    Quote from Destruct

    I also care about the bandwidth of each disk while parallel operating. If I use an extended card with PCIe x1 slot, how many disks should I attach with this card?

    I had 4 (four) Seagate 4TB 3.5 HD`s on it.

    If you talking "bandwidth" then you could to look into Bond0 network interface and managed switch supporting LACP.

    Arch Linux, Linux Mint 22, FreeBSD KDE Plasma6

    OMV7 NAS 10GB Fiber, Fractal Design Define R5 Case, Kodi "Omega", pfSense Plus firewall/router

    Edited once, last by andrzejls ().

    Quote from andrzejls

    I had 4 (four) Seagate 4TB 3.5 HD`s on it.

    If you talking "bandwidth" then you could to look into Bond0 network interface and managed switch supporting LACP.

    I mean if I use x1 slot with 4 disks, could they run full 6Gb/s in one time?

    Quote from Destruct

    I also care about the bandwidth of each disk while parallel operating. If I use an extended card with PCIe x1 slot, how many disks should I attach with this card?

    The bandwidth of each disk is not something I am overly concerned with. My LSI 9200-8E in the PCIe2.0 x8 slot had eight drives attached to it. I had no complaints about the performance.


    On my current server, the LSI controller is on the motherboard, it is an LSI SAS 2008 that is more than ten years old, and uses all eight ports. No complaints.

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.


    Like 1
    Quote from andrzejls

    You got a great deal.

    Still plenty of those out there.

    --
    Google is your friend and Bob's your uncle!


    A backup strategy is worthless unless you have a verified to work by testing restore strategy.


    OMV AMD64 7.x on headless Chenbro NR12000 1U Intel Xeon CPU E3-1230 V2 @ 3.30GHz 32GB ECC RAM.


    Quote from Destruct

    I mean if I use x1 slot with 4 disks, could they run full 6Gb/s in one time?

    The actual throughput is governed by HD capabilities. SATA III is capable of 6Gb/s, but HD MAX. transfer rate is what you are going to get. I get HD to HD transfer rate of ~105-116 MB/s on Seagate 4TB mechanical drives.

    Arch Linux, Linux Mint 22, FreeBSD KDE Plasma6

    OMV7 NAS 10GB Fiber, Fractal Design Define R5 Case, Kodi "Omega", pfSense Plus firewall/router

    For these SAS HBA models, I guess I would need the 8i variant rather than 8e: LSI SAS9207-8i if I want the cables to actually go to disks inside the case!

    My LSI 9207-8 arrived a couple of days ago. This was via ebay for GBP48.80 / USD56.27 and included the 2*4 cables. It was reported as actually being unused and I didn't see any evidence that this old card had been. I switched to it from the BEYIMEI PCIE card for my two drives and all was okay. I then added the further four Samsung disks which had been a problem for the BEYIMEI. This time a single disk is reported as having a problem and no other disks. (As reported above errors were seen across multiple disks last time.)


    Code
     [Sat Mar 11 15:28:55 2023] sd 0:0:5:0: task abort: SUCCESS scmd(0x000000008e71f524)
2[Sat Mar 11 15:28:57 2023] sd 0:0:5:0: [sdf] tag#7569 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
3[Sat Mar 11 15:28:57 2023] sd 0:0:5:0: [sdf] tag#7569 Sense Key : Not Ready [current]
4[Sat Mar 11 15:28:57 2023] sd 0:0:5:0: [sdf] tag#7569 Add. Sense: Logical unit not ready, cause not reportable
5[Sat Mar 11 15:28:57 2023] sd 0:0:5:0: [sdf] tag#7569 CDB: Read(10) 28 00 74 70 6d 00 00 00 08 00
6[Sat Mar 11 15:28:57 2023] I/O error, dev sdf, sector 1953524992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2

    I took this sdf disk out and no further problems have been reported. So it seems that my cheaper PCIE controller is bad at handling which disk has the problem.


    This card is seen in lspci as

    01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)


    $ sudo dmesg -T | grep LSI

    [Sat Mar 11 15:58:15 2023] mpt2sas_cm0: LSISAS2308: FWVersion(20.00.07.00), ChipRevision(0x05), BiosVersion(07.39.02.00)


    It completely rearranges the HCTL: Host:Channel:Target:Lun values putting its own disks on Host 0 before the motherboard hosted disks starting here at line 8.


    • Official Post

    In case anyone with the same problem reads this thread.


    Cards with the Asmedia ASM1166 chip sometimes have outdated firmware and this can cause problems with some Intel systems. This can be solved by updating the card firmware. Here's a way to do it. https://docs.phil-barker.com/p…1166-firmware-for-unraid/


    Aside from this possible firmware update issue, the ASM1166 chip for 6-port cards should not cause any problems.


    ____________________________________________________________________________________________


    There may be several reasons to choose these cards over a SAS HBA, such as:

    - There is no pcie x8 port on the system (or it is already occupied).

    - Lower consumption.

    - Impossibility of getting one in your country or prices that are too high on available shopping sites.

    - Preference for a new product instead of a used one.

    - I only need two more SATA ports, combined with everything above.


    In general, never buy cards with port multiplier chips, they are the ones that cause problems especially in Raid configurations.


    The following chips from Asmedia and JMicron may be suitable. They can be found on cards that use these chips with connection to PCIe or mini PCIe ports:


    - 6 SATA ports: ASM1166 PCIe 3.0 x4

    - 5 SATA ports: JMB585 PCIe 3.0 x4

    - 4 SATA ports: ASM1164 PCIe 3.0 x4

    - 4 SATA ports: ASM1064 PCIe 3.0 x1

    - 2 SATA ports: ASM1062 PCIe 2.0 x1

    - 2 SATA ports: ASM1061 PCIe 2.0 x1

    - 2 SATA ports: JMB582 PCIe 3.0 x1


    If the card has more ports than specified, it probably also has a multiplier chip, avoid it.


    You can find cards with these chips and a x1 pcie port instead of x4. The total bandwidth will depend on the port used on the card. x1 is enough for two ports. x4 is enough for up to 6 ports. If you do not have a pcie x4 port available and must choose a pcie x1 port to install a card with more than two SATA ports, keep in mind that the total bandwidth will decrease. The bandwidth will be less than the total speed of the disks added together. That does not mean that it will not work, simply the performance will be lower.

    Quote from gderf

    I paid $2.00 more for an eight port LSI SAS9207-8 card.

    Followed your advise and got same card on ebay for $37 with free shipping. Works great. I do not know why I waited so long. THANKS!

    Arch Linux, Linux Mint 22, FreeBSD KDE Plasma6

    OMV7 NAS 10GB Fiber, Fractal Design Define R5 Case, Kodi "Omega", pfSense Plus firewall/router

    I bought this M.2 to SATA adapter with ASM1166 chip. https://www.amazon.de/dp/B0CQN…o2ov_dt_b_product_details

    I checked the firmware and it is already on the above mentioned version.


    I can report that it works but i get the impression that it was not the best idea to buy this.


    1. It only gets connection sometimes.

    So it seems like that M2 connector is not very well produced. At first i thought the card does not work with my Motherboard because i was not able to see any drives connected to it. Taking the card out a few times and putting it in again, at some point it suddenly worked.

    I could reproduce this later. I have to get it in about 10 times before i am lucky and it works as it should.


    2. I see read errors, from time to time:

    Code
    [ 2617.395215] smbd(10403): Attempt to set a LOCK_MAND lock via flock(2). This support has been removed and the request ignored.
[ 4598.653166] ata4.00: exception Emask 0x11 SAct 0x400 SErr 0x0 action 0x6 frozen
[ 4598.653188] ata4.00: irq_stat 0x48000008, interface fatal error
[ 4598.653197] ata4.00: failed command: READ FPDMA QUEUED
[ 4598.653204] ata4.00: cmd 60/08:50:78:df:04/00:00:08:00:00/40 tag 10 ncq dma 4096 in
                        res 41/84:ff:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 4598.653227] ata4.00: status: { DRDY ERR }
[ 4598.653233] ata4.00: error: { ICRC ABRT }
[ 4598.653240] ata4: hard resetting link
[ 4598.965646] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)


    So up to here i cannot really recommend it.


    I have now contacted the chinese company that i bought it from. They will send another one from China.

    I wil check if it gets any better, but i doubt it.


    Might have to switch to an LSI card then.

    - OMV7 on Asus Prime N100 -

    Snapraid on 2 Data drives & 1 Parity

    6.8 proxmox kernel

    • Official Post
    Quote from monsen

    Might have to switch to an LSI card then.

    I have no issues with five drives on this JBM585-based card on CM4 boards, amd64 boards, and Friendly CM3588:

    https://www.amazon.com/gp/product/B09MQKW6DP/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1

    omv 7.7.3-1 sandworm | 64 bit | 6.11 proxmox kernel

    plugins :: omvextrasorg 7.0.2 | kvm 7.1.2 | compose 7.4.4 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.1


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login