With iotop i was able to determine that fragmentation probably isn't the problem.
same drive, NAS to windows: 80 MBps
to android tablet SD card: 8-10 MBps
in VLC on android tablet (samba network streaming, guest/anonymous mode): 2-3 MBps
Conclusion: VLC samba streaming is slow.
My 3 NAS drives show this level of fragmentation with
sudo fsck.ext2 -fn /dev/sd_1
2.8%
7.7%
17.0%
ext4 is believed to be unfragmentable, but I'm wondering if 17% frag is why that disk is showing slow performance.
sudo fsck.ext2 -fn /dev/sdb1
e2fsck 1.44.5 (15-Dec-2018)
Warning! /dev/sdb1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Inode 164364376 extent tree (at level 1) could be narrower. Optimize? no
Inode 164364378 extent tree (at level 1) could be narrower. Optimize? no
Inode 164364404 extent tree (at level 1) could be narrower. Optimize? no
Inode 164364488 extent tree (at level 1) could be narrower. Optimize? no
Inode 164364543 extent tree (at level 1) could be narrower. Optimize? no
Inode 164364560 extent tree (at level 1) could be narrower. Optimize? no
Inode 164364621 extent tree (at level 1) could be narrower. Optimize? no
Inode 164495635 extent tree (at level 1) could be narrower. Optimize? no
Inode 164496749 extent tree (at level 1) could be narrower. Optimize? no
Inode 164496798 extent tree (at level 1) could be narrower. Optimize? no
Inode 164496882 extent tree (at level 2) could be narrower. Optimize? no
Inode 164626683 extent tree (at level 1) could be narrower. Optimize? no
Inode 164757580 extent tree (at level 1) could be narrower. Optimize? no
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (335041402, counted=311261078).
Fix? no
Free inodes count wrong (183143227, counted=183143222).
Fix? no
asdf: 5317/183148544 files (17.0% non-contiguous), 397524870/732566272 blocksOne 25 GB file shows 51 extants which supposedly isn't supposed to have perf impact:
filefrag -v /qwer/zxcv.mkv
Filesystem type is: ef53
File size of /qwer/zxcv.mkv is 25021423581 (6108746 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 63487: 34816.. 98303: 63488:
1: 63488.. 96255: 100352.. 133119: 32768: 98304:
2: 96256.. 116735: 133120.. 153599: 20480:
3: 116736.. 124927: 155648.. 163839: 8192: 153600:
4: 124928.. 129023: 81889280.. 81893375: 4096: 163840:
5: 129024.. 131071: 81895424.. 81897471: 2048: 81893376:
6: 131072.. 135167: 81901568.. 81905663: 4096: 81897472:
7: 135168.. 137215: 81907712.. 81909759: 2048: 81905664:
8: 137216.. 139263: 81911808.. 81913855: 2048: 81909760:
9: 139264.. 143359: 82010112.. 82014207: 4096: 81913856:
10: 143360.. 167935: 81985536.. 82010111: 24576: 82014208:
11: 167936.. 217087: 82018304.. 82067455: 49152: 82010112:
12: 217088.. 264191: 82069504.. 82116607: 47104: 82067456:
13: 264192.. 266239: 82147328.. 82149375: 2048: 82116608:
14: 266240.. 270335: 108972032.. 108976127: 4096: 82149376:
15: 270336.. 272383: 108922880.. 108924927: 2048: 108976128:
16: 272384.. 276479: 216598528.. 216602623: 4096: 108924928:
17: 276480.. 280575: 216606720.. 216610815: 4096: 216602624:
18: 280576.. 284671: 216614912.. 216619007: 4096: 216610816:
19: 284672.. 288767: 216623104.. 216627199: 4096: 216619008:
20: 288768.. 294911: 216643584.. 216649727: 6144: 216627200:
21: 294912.. 299007: 216633344.. 216637439: 4096: 216649728:
22: 299008.. 303103: 216653824.. 216657919: 4096: 216637440:
23: 303104.. 309247: 216664064.. 216670207: 6144: 216657920:
24: 309248.. 315391: 216676352.. 216682495: 6144: 216670208:
25: 315392.. 323583: 216688640.. 216696831: 8192: 216682496:
26: 323584.. 329727: 216705024.. 216711167: 6144: 216696832:
27: 329728.. 337919: 216717312.. 216725503: 8192: 216711168:
28: 337920.. 344063: 216733696.. 216739839: 6144: 216725504:
29: 344064.. 352255: 216748032.. 216756223: 8192: 216739840:
30: 352256.. 360447: 216764416.. 216772607: 8192: 216756224:
31: 360448.. 368639: 216780800.. 216788991: 8192: 216772608:
32: 368640.. 372735: 216565760.. 216569855: 4096: 216788992:
33: 372736.. 376831: 216573952.. 216578047: 4096: 216569856:
34: 376832.. 380927: 216582144.. 216586239: 4096: 216578048:
35: 380928.. 385023: 216590336.. 216594431: 4096: 216586240:
36: 385024.. 387071: 241104896.. 241106943: 2048: 216594432:
37: 387072.. 450559: 241108992.. 241172479: 63488: 241106944:
38: 450560.. 942079: 241205248.. 241696767: 491520: 241172480:
39: 942080.. 1433599: 241729536.. 242221055: 491520: 241696768:
40: 1433600.. 1925119: 242253824.. 242745343: 491520: 242221056:
41: 1925120.. 2416639: 242778112.. 243269631: 491520: 242745344:
42: 2416640.. 2779135: 243302400.. 243664895: 362496: 243269632:
43: 2779136.. 2906111: 243666944.. 243793919: 126976: 243664896:
44: 2906112.. 3397631: 243826688.. 244318207: 491520: 243793920:
45: 3397632.. 3889151: 244350976.. 244842495: 491520: 244318208:
46: 3889152.. 4380671: 244875264.. 245366783: 491520: 244842496:
47: 4380672.. 4872191: 245399552.. 245891071: 491520: 245366784:
48: 4872192.. 5363711: 245923840.. 246415359: 491520: 245891072:
49: 5363712.. 5855231: 246448128.. 246939647: 491520: 246415360:
50: 5855232.. 5959679: 246972416.. 247076863: 104448: 246939648:
51: 5959680.. 6108745: 247078912.. 247227977: 149066: 247076864: last,eof
/qwer/zxcv.mkv: 51 extents foundMy NAS is mostly a media server with typical single files running 10-70 GB. I either write the files from Windows with SMB or write them with nzbget. the system is an ARM odroid hc4.
I know I am on omv 5.x but I intend to upgrade very soon. While I'm upgrading I'd like to see if I can prevent this level of fragmentation, possibly by tuning ext4 parameters. Or am I totally barking up the wrong tree? Web sources on ext4 fragmentation say it doesn't even exist, so I can't trust them to be authoritative on its impact (especially on an rpi-3 equivalent arm box).
I did find some interesting links:
The recycle bin for SMB/Samba works great. But it can fill up the whole drive. Has anyone made a script that will delete the oldest deleted files first if space gets low?
I'm thinking of making a script that moves everything out of .recycle to another folder, then deletes file(s) if the free space drops below ~10 GB. But I don't know how easy it would be to delete by age of deletion, not age of the file. Maybe change the 'last modified' on move.
Reinstalled basically everything. Fresh armbian, fresh bootloader (petitboot), fresh OMV 5. Formatted all 3 drives to GPT and ext4. Also made one of them 1 big partition. (Before, it was two partitions: 2TB and 1TB).
I notice that the 'disks' field of the omv webui says "/dev/sda" , b, c. Before, I thought some/all of them were /dev/disk/by-uuid/...
Maybe the clue to the bug is in there somewhere? I also switched the 'problem' drive to the other SATA slot on the odroid hc4. We'll see if the drive swap makes the 'good' drive start misbehaving.
When plugged into a monitor, the boot sequence still seems to look for 2 or 3 drives by uuid. I opened up config.xml and couldn't find where it was pulling that from. One of them showed up in the disks webUI and I removed it.
Photorec / TestDisk on an x86 linux distro seems to be able to recover the files. It's going to take several days for 2 TB. This is a huge error and I still don't know what caused it. https://www.tecmint.com/photor…eted-lost-files-in-linux/
I'm not concerned about the drive I just don't want this flaw to come back once I turn this into a real NAS and not a toy system. I still don't plan on making the NAS my single point of failure, but recovering from backup errors is annoying, takes time, and puts strain on my other drives.
Phonerec was taking forever and doesn't give me filenames, so I killed it. I have the original data so I'll copy it again.
- The added NTFS hard drive - is it USB powered (by the HC4)? Or is it in a drive dock or, maybe, powered by a USB hub? (The reason why this is mentioned is powering hard drives can be problematic with SBC's. An HC4 recommends a 4amp supply, minimum, for two on-board drives.
USB powered, usb3 "p10" WD black https://www.amazon.com/WD_Blac…20BBK-WESN/dp/B07VMTNDMK/
The HC4 has 2 onboard drives (3.5" desktop HDDs) and the USB drive (2.5 inch) all drawing power. The PSU is 15V/4A Power Supply × 1 from Ameridroid.
- Have you looked look at SMART data for the drive you think is / has failed?
Everything looks fine. It's almost a brand new drive (albeit I could be hitting the early part of the bathtub curve).
Ran this on a kubuntu live USB on a totally different x86 device (not OMV).
kubuntu@kubuntu:~$ sudo skdump /dev/sda
Device: sat16:/dev/sda
Type: 16 Byte SCSI ATA SAT Passthru
Size: 1907729 MiB
Model: [ST2000DM008-2FR102]
Serial: [ZK201FE8]
Firmware: [0001]
SMART Available: yes
Quirks:
Awake: yes
SMART Disk Health Good: yes
Off-line Data Collection Status: [Off-line data collection activity was never started.]
Total Time To Complete Off-Line Data Collection: 0 s
Self-Test Execution Status: [The previous self-test routine completed without error or no self-test has ever been run.]
Percent Self-Test Remaining: 0%
Conveyance Self-Test Available: yes
Short/Extended Self-Test Available: yes
Start Self-Test Available: yes
Abort Self-Test Available: yes
Short Self-Test Polling Time: 1 min
Extended Self-Test Polling Time: 207 min
Conveyance Self-Test Polling Time: 2 min
Bad Sectors: 0 sectors
Powered On: 3.1 months
Power Cycles: 296
Average Powered On Per Power Cycle: 7.5 h
Temperature: 30.0 C
Attribute Parsing Verification: Bad
Overall Status: GOOD
ID# Name Value Worst Thres Pretty Raw Type Updates Good Good/Past 1 raw-read-error-rate 82 64 6 163297192 0xa8b7bb090000 prefail online yes yes 3 spin-up-time 96 96 0 n/a 0x000000000000 prefail online n/a n/a 4 start-stop-count 100 100 20 370 0x720100000000 old-age online yes yes 5 reallocated-sector-count 100 100 10 0 sectors 0x000000000000 prefail online yes yes 7 seek-error-rate 80 60 45 91519515 0x1b7a74050000 prefail online yes yes 9 power-on-hours 98 98 0 3.1 months 0xad080000a820 old-age online n/a n/a 10 spin-retry-count 100 100 97 0 0x000000000000 prefail online yes yes 12 power-cycle-count 100 100 20 296 0x280100000000 old-age online yes yes 183 runtime-bad-block-total 100 100 0 0 0x000000000000 old-age online n/a n/a 184 end-to-end-error 100 100 99 0 0x000000000000 old-age online yes yes 187 reported-uncorrect 100 100 0 0 sectors 0x000000000000 old-age online n/a n/a 188 command-timeout 100 100 0 0 0x000000000000 old-age online n/a n/a 189 high-fly-writes 100 100 0 0 0x000000000000 old-age online n/a n/a 190 airflow-temperature-celsius 70 53 40 30.0 C 0x1e00141e0000 old-age online yes yes 191 g-sense-error-rate 100 100 0 0 0x000000000000 old-age online n/a n/a 192 power-off-retract-count 100 100 0 62 0x3e0000000000 old-age online n/a n/a 193 load-cycle-count 100 100 0 1889 0x610700000000 old-age online n/a n/a 194 temperature-celsius-2 30 47 0 30.0 C 0x1e0000001200 old-age online n/a n/a 195 hardware-ecc-recovered 82 64 0 163297192 0xa8b7bb090000 old-age online n/a n/a 197 current-pending-sector 100 100 0 0 sectors 0x000000000000 old-age online n/a n/a 198 offline-uncorrectable 100 100 0 0 sectors 0x000000000000 old-age offline n/a n/a 199 udma-crc-error-count 200 200 0 0 0x000000000000 old-age online n/a n/a 240 head-flying-hours 100 253 0 2.8 months 0xbc070000f97f old-age offline n/a n/a 241 total-lbas-written 100 253 0 165533.598 TB 0x1bf70b260100 old-age offline n/a n/a 242 total-lbas-read 100 253 0 611433.316 TB 0x35ae1f3e0400 old-age offline n/a n/a kubuntu@kubuntu:~$ Photorec / TestDisk on an x86 linux distro seems to be able to recover the files. It's going to take several days for 2 TB. This is a huge error and I still don't know what caused it. https://www.tecmint.com/photor…eted-lost-files-in-linux/
I ran testdisk and then reboot. It's been unresponsive for hours. Rebooted my router and none of the dhcp leases are for the odroid. The Network LED for the odroid is blinking and there are vibrations and noise from the drive(s). Would align partition take hours?
/dev/sda: LBA, HPA, LBA48 support
/dev/sda: size 3907029168 sectors
/dev/sda: user_max 3907029168 sectors
/dev/sda: native_max 3907029168 sectors
Using locale 'C.UTF-8'.
Thu Jan 20 18:37:15 2022
Command line: TestDisk /log /dev/sda
TestDisk 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>
OS: Linux, kernel 5.10.81-meson64 (#21.08.6 SMP PREEMPT Mon Nov 22 11:21:51 UTC 2021) aarch64
Compiler: GCC 8.2
ext2fs lib: 1.44.5, ntfs lib: libntfs-3g, reiserfs lib: none, ewf lib: none, curses lib: ncurses 6.1
Hard disk list
Disk /dev/sda - 2000 GB / 1863 GiB - CHS 243201 255 63, sector size=512 - ST2000DM008-2FR102, S/N:ZK201FE8, FW:0001
Partition table type (auto): EFI GPT
Disk /dev/sda - 2000 GB / 1863 GiB - ST2000DM008-2FR102
Partition table type: EFI GPT
Analyse Disk /dev/sda - 2000 GB / 1863 GiB - CHS 243201 255 63
Bad GPT partition, invalid signature.
Trying alternate GPT
1 P Unknown 2048 3907029134 3907027087
Current partition structure:
Bad GPT partition, invalid signature.
Trying alternate GPT
1 P Unknown 2048 3907029134 3907027087
search_part()
Disk /dev/sda - 2000 GB / 1863 GiB - CHS 243201 255 63
MS Data 2048 3907029127 3907027080 [seagate2tb]
ext4 blocksize=4096 Large_file Sparse_SB, 2000 GB / 1863 GiB
interface_write()
1 P MS Data 2048 3907029127 3907027080 [seagate2tb]
write!
No extended partition
You will have to reboot for the change to take effect.
New options :
Dump : No
Align partition: Yes
Expert mode : No
Interface Advanced
1 P MS Data 2048 3907029127 3907027080 [seagate2tb]
ext4 blocksize=4096 Large_file Sparse_SB, 2000 GB / 1863 GiB
ext2fs_read_inode(ino=11) failed with error 2133571474.
ext2fs_read_inode(ino=5242881) failed with error 1.
ext2fs_read_inode(ino=17) failed with error 2133571474.
ext2fs_read_inode(ino=16) failed with error 2133571474.
ext2fs_read_inode(ino=110624769) failed with error 1.
ext2fs_read_inode(ino=114425857) failed with error 1.
ext2fs_read_inode(ino=110755841) failed with error 1.
ext2fs_read_inode(ino=110886913) failed with error 1.
ext2fs_read_inode(ino=111017985) failed with error 1.
ext2fs_read_inode(ino=113246209) failed with error 1.
ext2fs_read_inode(ino=114425857) failed with error 1.
Directory /
2 drwxrwsr-x 0 1001 4096 19-Jan-2022 08:59 .
2 drwxrwsr-x 0 1001 4096 19-Jan-2022 08:59 ..
18 -rw------- 0 1001 250125 19-Jan-2022 08:59 snapraid.content
X 18 -rw------- 0 1001 250125 19-Jan-2022 08:59 snapraid.content.tmp
TestDisk exited normally.
Brief history:
Did 90% of OMV setup 1 year ago
2 SATA drives (ext4) in ODROID HC4, OMV installed on microSD card. Snapraid installed but I don't think I installed it correctly (didn't set it to run or do a first run I think).
Dusted off my setup
Updated everything
Added USB HDD (ntfs)
Immediately, OMV can't recognize one of my SATA drives. I see a new UUID for the same drive. No file systems that I can find. I put the drive in another PC and booted into kubuntu 20.04.3 live and still couldn't find any filesystems. Tried searching for superblocks and failed.
I know almost nothing about Linux HDD recovery so don't assume I tried something already. It could be very basic. Or maybe the drive is dead.
I looked at this guide but didn't see much relevant. I still need to run lspci in verbose mode. I'll have logs after work but wanted to see if it's a simple issue.