Slow file polling with OMV 3.x

Welcome to the club...

So...dump 3.x...too many problems...if I read the forum, there are most problems with that, not with 2.x
But maybe there are a lot more installations 3.x?

Zitat von Backupmaster

On 2.1 polling some 38000 files took only a few minutes. On 3.x it now takes 3 hours.

What does this mean? 'Polling'?

Zitat von Backupmaster

OMV 3.0.81 (Intel version since installation of AMD version would fail at the stage of getting the apt get sources)

What does this mean? Are you talking about amd64 vs. i386?

Will 2.x run on installed wheezy, upgraded to jessie, maybe to stretch?

Zitat von ryecoaaron

And it ruins my theory about it only affecting 32 bit installs...

Nope...

Zitat von Backupmaster

Intel version since installation of
AMD version would fail at the stage of getting the apt get sources

Zitat von Backupmaster

With the Intel version I mean i386 which I am currently using, being driven by a AMD Sempron processor.

Zitat von Backupmaster

Polling means: when wanting to backup files, the backup program scans (polls) all source files and all files already in the backup directory to see if they are the same or not. Only files not having the same time stamp are then copied from the source to the backup directory. It is this process which takes much more time on OMV 3.x compared to OMV 2.1.

Thanks for confirming (also that you're using the 32-bit version), so it's about directory enumeration and/or checking file metadata (timestamps if the program really only relies on this). It would be great if you can provide a full screenshot from Helios LanTest with both 'Gigabit Ethernet' and 'Enterprise networks -- 10 GbE' settings: http://www.helios.de/web/EN/news/LanTest-6-released.html

Zitat von Dropkick Murphy

if I read the forum, there are most problems with that, not with 2.x

Of course since 2.x installations are older (so you find similar 'most problems with 2.x, not with 1.x' threads when you go back in time) and large updates are always stress for the hardware (in professional support I've seen a significant correlation between 'latest OS update broke the machine' and filesystem corruption already present before and then being the root cause for updates failing).

Also updates raise awareness for problems (typical example) so it's pretty normal to get more problems reported for the version users currently upgrade to. This should be analyzed (asking for logs and debug info) instead of making 'obvious assumptions' (it's often not what it looks like )

Looks I should stop writing and supporting

Zitat von Backupmaster

Please find enclosed the reports you requested.

Thank you. Only sequential throughput numbers are a bit low but those shouldn't affect directory enumeration. Are you able to upload debug logs (somewhere in the web UI, I hope @ryecoaaron or someone else can assist here since I managed to brick all my 3 OMV test installations today).

@ryecoaaron is this a 'common' 3.x problem? Slow directory enumeration or just 'problems'? Would be interesting to sniff packets if there's nothing in the logs.

Zitat von Dropkick Murphy

Will 2.x run on installed wheezy, upgraded to jessie, maybe to stretch?

Nope. It will not. The dist-upgrade kicks OMV and it is never re-installable...so I am caught here...

Zitat von Backupmaster

With "debug logs" you mean "system logs"?

No idea how it's called in OMV, @ryecoaaron just told me months ago that there would be something easily available. The purpose is to just look into the log files to see what's happening (dmesg, syslog, Samba logs). I had many occurences of performance problems that were clearly 'documented' in log files so that's always the first I would try (I even managed to create performance problems and a 'no space left on device' situation due to activated extensive Samba logging).

Zitat von Dropkick Murphy

The dist-upgrade kicks OMV and it is never re-installable

Of course. Welcome to 'dependency hell'. There's no way back to OMV 2.x and the best alternative is to immediately identify all 3.x problems.

Zitat von Dropkick Murphy

Nope...

To prove whats up, I build an i386 stretch machine with just a samba share. Writing large files leads exact to the same slow perf (and high load) what is mentioned here so oft...so I suppose:
- OMV is innocent
- The samba maintainer broke smb since jessie and even in stretch, at least with i386

Solution:
Kill the mainainer of smb
Kick samba, go with NFS to OMV 3.x
Leave OMV (and Debian), stay with samba and go to other *ix/BSD stuff

HTH

Zitat von Dropkick Murphy

Writing large files

BTW: that's just exactly the opposite of

Zitat von Backupmaster

the backup program scans (polls) all source files and all files already in the backup directory to see if they are the same or not. Only files not having the same time stamp are then copied from the source to the backup directory. It is this process which takes much more time on OMV 3.x compared to OMV 2.1.

Usual/logical steps would be to test local storage first (throughput and also random IO) to see whether here's a bottleneck or not (since at least @'Backupmaster''s sequential throughput numbers are somewhat low). This can be done by an 'apt install iozone3', then chdir to the share in question and run at least

iozone -e -I -a -s 100M -r 4k -r 128k -r 1024k -r 16384k -i 0 -i 1 -i 2
iozone -e -I -a -s 4000M -r 128k -r 1024k -i 0 -i 1

Then a comparison with an amd64 installation on the same hardware or exactly same virtual machine would be interesting doing the exact same set of tests.

I don't run OMV on x86 boards but at least on ARM (armhf vs. arm64) there's no performance difference between both Samba variants.

@Backupmaster it would be interesting if you redo the LanTest or simply do an Explorer copy job with large file sizes reporting both transfer speeds and how CPU utilization looks like (especially the %iowait percentage, please see for the graphs in the threads below). And then such a local iozone test would be great to further nail the problem down.

I just did a quick check and since at least one Core 2 Duo machine running the amd64 installation is also affected the '32 bit theory' is somewhat questionable:

• Core2 Duo E7500 running amd64 version: Slow Samba, Fast FTP
• Atom N280 running 32-bit version: Extreme slow Networkspeed
• AMD Sempron 2600+ incapable of AM64 extensions running 32-bit version here

So if anyone already looking into that could collect more threads with reports of the SAME symptoms (high %iowait percentage and poor performance with Debian Jessie's Samba version) it would be great to also report installation variant (32-bit vs. 64-bit) and exact CPU type (since at least the three above are rather old/slow ones -- we had a similar issue a decade ago with a commercial fileserver daemon on Solaris horribly low performing on older CPUs and there a simple Solaris tweak fixed it)