3 ssd disks that I use in a zfs pool - the 3 with Problem

sismondi · 12. November 2021

Hi guys.

I hope you are very well.

I have 3 1TB ssd disks that I use in a zfs pool for less than 8 months.

I have analyzed the disks with scrutiny.

The results are very rare.

First disk:

https://ibb.co/XJhDjS0

Second Disc:

https://ibb.co/smTwR8x

Third Disc:

https://ibb.co/y5bbLyx

The three ssd are about to die in a near moment?

Aditional info:

fdisk -l

Code

Disk /dev/sdb: 953.9 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: Patriot P200 1TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Code

Disk /dev/sdd: 953.9 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: Patriot P200 1TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 35489ADB-1CD9-4A28-9D5B-BDE5DB2E4AC9


Device     Start        End    Sectors   Size Type
/dev/sdd1   2048 2000409230 2000407183 953.9G Linux filesystem

Alles anzeigen

Code

Disk /dev/sde: 953.9 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: Patriot P200 1TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 056A3B2F-3D26-FB4B-8D54-AA810C85B67B


Device          Start        End    Sectors   Size Type
/dev/sde1        2048 2000392191 2000390144 953.9G Solaris /usr & Apple ZFS
/dev/sde9  2000392192 2000408575      16384     8M Solaris reserved 1

Alles anzeigen

chente · 12. November 2021

Can you show the full result of a short SMART test on one of them?

sismondi · 12. November 2021

I also have the problem that it is impossible to delete these partitions

Device Start End Sectors Size Type

/dev/sde1 2048 2000392191 2000390144 953.9G Solaris /usr & Apple ZFS

/dev/sde9 2000392192 2000408575 16384 8M Solaris reserved 1

this is what you need?

Code

sudo smartctl -a /dev/sdb
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-5.4.143-1-pve] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     Patriot P200 1TB
Serial Number:    AA000000000000000301
Firmware Version: S0424A0
User Capacity:    1,024,209,543,168 bytes [1.02 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Nov 12 04:24:09 2021 -03
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
No failed Attributes found.

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  120) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
                                        entering power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       36
  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       80
  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       5803
 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       22
160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       5
161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       0
163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       9
164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       6868
165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       33
166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       1
167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       14
168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       7000
169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       100
175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0
176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0
177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       0
178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       80
181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       15
194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       37
195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       273899
196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       5
197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       80
198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       5
199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       7
232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       0
241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       91073
242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       15822
245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       209223

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 2

ATA Error Count: 0
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error -4 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 00 00 00 00 00 00

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d0 01 00 4f c2 00 08      00:00:00.000  SMART READ DATA
  b0 d1 01 01 4f c2 00 08      00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  b0 da 00 00 4f c2 00 08      00:00:00.000  SMART RETURN STATUS
  b0 d5 01 00 4f c2 00 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 00 08      00:00:00.000  SMART READ LOG

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      5800         -
# 2  Short offline       Completed: read failure       50%      5795         14739744
# 3  Short offline       Completed: read failure       50%      5775         14739744
2 of 2 failed self-tests are outdated by newer successful extended offline self-test # 1

Selective Self-tests/Logging not supported

Alles anzeigen

chente · 12. November 2021

SMART overall-health self-assessment test result: FAILED!

Drive failure expected in less than 24 hours. SAVE ALL DATA.

5 Reallocated_Sector_Ct 0x0032 100 100 050 Old_age Always - 80

197 Current_Pending_Sector 0x0032 100 100 050 Old_age Always - 80

198 Offline_Uncorrectable 0x0032 100 100 050 Old_age Always - 5

These values indicate that the drive is going to fail and should be replaced.

It is surprising that it has very few hours of use.

9 Power_On_Hours 0x0032 100 100 050 Old_age Always - 5803

On the other hand, SMART does not seem fully compatible with this model. And some values that could be relevant are not reported, such as 187 and 188.

Device is: Not in smartctl database [for details use: -P showall]

If you don't have a backup, I would try to make one immediately. And I would replace these disks, preferably with server-class disks. There are SSDs prepared for 24/7 operation.

I can't find any reason for all three units to fail so soon. Maybe it is related to the applications you use and the amount of data written. Maybe someone will see something else.

sismondi · 12. November 2021

Zitat von chente

SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
5 Reallocated_Sector_Ct 0x0032 100 100 050 Old_age Always - 80
197 Current_Pending_Sector 0x0032 100 100 050 Old_age Always - 80
198 Offline_Uncorrectable 0x0032 100 100 050 Old_age Always - 5

These values indicate that the drive is going to fail and should be replaced.

It is surprising that it has very few hours of use.

9 Power_On_Hours 0x0032 100 100 050 Old_age Always - 5803

On the other hand, SMART does not seem fully compatible with this model. And some values that could be relevant are not reported, such as 187 and 188.

Device is: Not in smartctl database [for details use: -P showall]

If you don't have a backup, I would try to make one immediately. And I would replace these disks, preferably with server-class disks. There are SSDs prepared for 24/7 operation.

I can't find any reason for all three units to fail so soon. Maybe it is related to the applications you use and the amount of data written. Maybe someone will see something else.

Alles anzeigen

do you know how to delete these partitions? Traditional methods don't work.

Device Start End Sectors Size Type

/dev/sde1 2048 2000392191 2000390144 953.9G Solaris /usr & Apple ZFS

/dev/sde9 2000392192 2000408575 16384 8M Solaris reserved 1

ryecoaaron · 12. November 2021

sudo wipefs -a /dev/sde might have to run it many times.

sismondi · 12. November 2021

I did it many times, but dont word.

Disk /dev/sde: 953.9 GiB, 1024209543168 bytes, 2000409264 sectors

Disk model: Patriot P200 1TB

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: gpt

Disk identifier: 056A3B2F-3D26-FB4B-8D54-AA810C85B67B

Device Start End Sectors Size Type

/dev/sde1 2048 2000392191 2000390144 953.9G Solaris /usr & Apple ZFS

/dev/sde9 2000392192 2000408575 16384 8M Solaris reserved 1

Zitat von ryecoaaron

sudo wipefs -a /dev/sde might have to run it many times.

sismondi · 12. November 2021

sudo fdisk /dev/sde option "d", it doesn't work either

chente · 12. November 2021

Just curious if you allow me the question. What is the reason to delete these partitions now?

sismondi · 12. November 2021

Zitat von chente

Just curious if you allow me the question. What is the reason to delete these partitions now?

I want to use the disk again to investigate more.

There is no chance that all 3 are broken.

The disks, since they were installed, had less than 5 TB of writing.

They were used for a z1 zfs pool and nextcloud was installed.

Nothing else.

chente · 12. November 2021

Yes. It's certainly strange that the three of them split up so quickly for no apparent reason.

Once a backup is done, I would try to find out if they are really broken. There must be some disk recovery software that can help.

That's why I asked, I really don't understand the reason for deleting those partitions. It's not going to help you much now.

sismondi · 12. November 2021

Zitat von chente

Yes. It's certainly strange that the three of them split up so quickly for no apparent reason.
Once a backup is done, I would try to find out if they are really broken. There must be some disk recovery software that can help.
That's why I asked, I really don't understand the reason for deleting those partitions. It's not going to help you much now.

if I don't delete those partitions, it won't let me create a new partition. I get an error:

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; sgdisk --new=1:0:0 --typecode=1:8300 --print '/dev/sde' 2>&1' with exit code '4': Found valid GPT with corrupt MBR; using GPT and will write new
protective MBR on save.
Could not create partition 1 from 34 to 2047
Disk /dev/sde: 2000409264 sectors, 953.9 GiB
Model: Patriot P200 1TB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 056A3B2F-3D26-FB4B-8D54-AA810C85B67B
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 2000409230
Partitions will be aligned on 2048-sector boundaries
Total free space is 2669 sectors (1.3 MiB)
Number Start (sector) End (sector) Size Code Name 1 2048 2000392191 953.9 GiB 8300 zfs-d24dfc456f10bc9a 9 2000392192 2000408575 8.0 MiB BF07
Error encountered; not saving changes.

Alles anzeigen

the same problem had other people:

https://www.reddit.com/r/zfs/c…8s/wiping_zfs_disks_help/

ryecoaaron · 12. November 2021

Zitat von sismondi

I did it many times, but dont word.

Is zfs mounted and/or still active?

sismondi · 12. November 2021

Zitat von ryecoaaron

Is zfs mounted and/or still active?

root@NAS:~# zpool status ncdisk

cannot open 'ncdisk': no such pool

root@NAS:~# zpool import ncdisk

cannot import 'ncdisk': no such pool available

root@NAS:~# zpool destroy ncdisk

cannot open 'ncdisk': no such pool

root@NAS:~# zpool list

no pools availableroot@NAS:~# zpool clear ncdisk

cannot open 'ncdisk': no such pool

the pool was degraded and lost.

as I had backup of everything, I did not worry and simply erased the three disks.

Two discs were erased without problem.

The third there is no way to erase the partitions and I cannot create a new filesystem.

ryecoaaron · 12. November 2021

sudo dd if=/dev/zero of=/dev/sde bs=1M count=5000

reboot

sismondi · 12. November 2021

Zitat von ryecoaaron

sudo dd if=/dev/zero of=/dev/sde bs=1M count=5000
reboot

dont work

root@NAS:~# sudo dd if=/dev/zero of=/dev/sde bs=1M count=5000

5000+0 records in

5000+0 records out

5242880000 bytes (5.2 GB, 4.9 GiB) copied, 12.0442 s, 435 MB/s

root@NAS:~# sudo reboot

fdisk -l

Disk /dev/sde: 953.9 GiB, 1024209543168 bytes, 2000409264 sectors

Disk model: Patriot P200 1TB

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: gpt

Disk identifier: 056A3B2F-3D26-FB4B-8D54-AA810C85B67B

Device Start End Sectors Size Type

/dev/sde1 2048 2000392191 2000390144 953.9G Solaris /usr & Apple ZFS

/dev/sde9 2000392192 2000408575 16384 8M Solaris reserved 1

ryecoaaron · 12. November 2021

Something is very strange here. Zero the entire disk then. Just remove the count parameter. If you write zeroes to the entire disk, it cannot still be a zfs disk unless something on your system is re-importing/re-creating it.

Personally, I would boot a rescue disk that I knew wasn't going to support zfs and use wipefs and dd.

sismondi · 12. November 2021

Zitat von ryecoaaron

Something is very strange here. Zero the entire disk then. Just remove the count parameter. If you write zeroes to the entire disk, it cannot still be a zfs disk unless something on your system is re-importing/re-creating it.

Personally, I would boot a rescue disk that I knew wasn't going to support zfs and use wipefs and dd.

I installed debian 10 and OMV again.

I tried to fill the entire disk with zeros, but it didn't work:

Code

root@NAS:~# sudo dd if=/dev/zero of=/dev/sde bs=1M
dd: error writing '/dev/sde': No space left on device
976763+0 records in
976762+0 records out
1024209543168 bytes (1.0 TB, 954 GiB) copied, 1897.19 s, 540 MB/s

root@NAS:~# sudo reboot


Disk /dev/sde: 953.9 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: Patriot P200 1TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 056A3B2F-3D26-FB4B-8D54-AA810C85B67B

Device          Start        End    Sectors   Size Type
/dev/sde1        2048 2000392191 2000390144 953.9G Solaris /usr & Apple ZFS
/dev/sde9  2000392192 2000408575      16384     8M Solaris reserved 1

Alles anzeigen

After that, I removed the disk and connected it to a Windows PC.

With the HD TUNE PRO program I again filled the disk with zeros.

Still not working.

But it gave me the following error.

https://ibb.co/sq4jdTg

Would it be correct to interpret that the disk does not allow to overwrite some sectors?

ryecoaaron · 13. November 2021

Zitat von sismondi

Would it be correct to interpret that the disk does not allow to overwrite some sectors?

No. Between wipefs and dd, nothing can survive that. I would boot systemrescuecd and run the commands again. If it somehow comes back, something in your setup is changing it. I have never seen anything like this. So, I don't know what your system is doing.

ryecoaaron · 13. November 2021

If the other two disks are still in a pool, maybe zfs is adding the disk back in every time you reboot. I don't use zfs.

3 ssd disks that I use in a zfs pool - the 3 with Problem

sismondi 12. November 2021

Jetzt mitmachen!