HC2 internal file transfers going through LAN :-(

ryecoaaron · 6. November 2018

Zitat von OMV_HC2

What is that telling me? The native write speed to the drive?

That is telling you that your HC2 cannot write to your ntfs filesystem any faster than that. I'm sure it could write faster to ext4 on the same drive.

OMV_HC2 · 6. November 2018

OK, good to know in more detail what is going on. Thanks for that.
I wanted to know the read speed of the drive, so I looked for some sample code online.
I cleared the cache with sudo /sbin/sysctl -w vm.drop_caches=3

Then ran
dd if=/sharedfolders/Transfers/testfile.dd of=/dev/null bs=1M count=1024

That resulted in :
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.99122 s, 154 MB/s

Does that mean the HC2 can read at 154 MB/s from the disk, but only write at 23?
If it can read at 154 MB/s, why would my read transfer speeds on the network fluctuate so much? They are sequential reads (typically 2+GB ISO, ZIP and media files)?

macom · 6. November 2018

As reference: this is what I get with my HC2 on my drive with BTRFS:

Code

5151653888 bytes (5.2 GB, 4.8 GiB) copied, 29.0046 s, 178 MB/s
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 31.5811 s, 166 MB/s

JohnStiles · 6. November 2018

Zitat von macom
As reference: this is what I get with my HC2 on my drive with BTRFS:
Code
5151653888 bytes (5.2 GB, 4.8 GiB) copied, 29.0046 s, 178 MB/s
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 31.5811 s, 166 MB/s

HDD 5400RPM(Use%86) + EXT4

Code

5182062592 bytes (5.2 GB, 4.8 GiB) copied, 72.0075 s, 72.0 MB/s
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 74.5859 s, 70.3 MB/s

OMV_HC2 · 6. November 2018

Are those read or write speeds?
And what speeds do you get across the network?

I used BTRFS on a Synology at work when I was abroad - loved it. Didn't know the HC2 could play nice with it. I guess there isn't too much overhead.

Does drive size play a part in the overall speeds?

Adoby · 6. November 2018

Writing to an ext4 12TB IronWolf from HC2:

Code

adoby@nas1:~$ dd if=/dev/zero of=/home/adoby/testfile.dd bs=1M count=5000 conv=fdatasync status=progress && sync
5221908480 bytes (5.2 GB, 4.9 GiB) copied, 26.0001 s, 201 MB/s
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 26.7838 s, 196 MB/s

Writing to an ext4 512GB EVO 960 SSD from a HC2:

Code

adoby@ctrl:~$ dd if=/dev/zero of=/home/adoby/testfile.dd bs=1M count=5000 conv=fdatasync status=progress && sync
5022679040 bytes (5.0 GB, 4.7 GiB) copied, 21 s, 239 MB/s
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 22.0455 s, 238 MB/s

Copying a file via nfs from ext4 SSD on one HC2 to another HC2 with ext4 IronWolf HDD:

Code

adoby@ctrl:~$ time cp /home/adoby/testfile.dd /media/nfs/nas1/shared_media


real	0m55.244s
user	0m0.038s
sys	0m36.427s


adoby@ctrl:~$ ls --block-size=1M --size testfile.dd
5001 testfile.dd
adoby@ctrl:~$ echo $(expr 5001 / 55) MB/s
90 MB/s

Alles anzeigen

Adoby · 6. November 2018

Zitat von OMV_HC2

Are those read or write speeds?
And what speeds do you get across the network?

I used BTRFS on a Synology at work when I was abroad - loved it. Didn't know the HC2 could play nice with it. I guess there isn't too much overhead.

Does drive size play a part in the overall speeds?

Write speeds.
My last measurement was SSD read+network transfer+HDD write. 95 MiB/s.
Drive size has a minor influence. Often a HDD gets slower when it gets full. My IronWolf 12TB above was 71% full.

Good news: It seems SMB has a minor effect on the low performance you reported,
Bad news: It seem NTFS has a huge negative effect.

OMV_HC2 · 6. November 2018

Zitat von Adoby

Write speeds.My last measurement was SSD read+network transfer+HDD write. 95 MiB/s.
Drive size has a minor influence. Often a HDD gets slower when it gets full. My IronWolf 12TB above was 71% full.

Good news: It seems SMB has a minor effect on the low performance you reported,
Bad news: It seem NTFS has a huge negative effect.

Yeah, the drives can get full quickly. I'm at 79% with the 10TB.
I might have to figure something out with a couple of external 5TB drives then I could switch over to ext4 or BTRFS. Looks like ntfs is just going to be too slow for me.

Will try to figure out how to do some kind of experiment. Will have to wipe my backup external drives to do it though... Probably worth it.

Thanks for all the feedback and tips everyone.

ryecoaaron · 7. November 2018

Zitat von OMV_HC2

I wanted to know the read speed of the drive

In your post, you only mentioned writes which is why I gave that snippet.

Zitat von OMV_HC2

Does that mean the HC2 can read at 154 MB/s from the disk, but only write at 23?
If it can read at 154 MB/s, why would my read transfer speeds on the network fluctuate so much? They are sequential reads (typically 2+GB ISO, ZIP and media files)?

Seems that way. Just because dd reads at 154 doesn't mean a network transfer will. dd doesn't have the overhead of networking and samba along with ntfs in the mix.

JohnStiles · 7. November 2018

If you need a test with network overhead and SMB use HELIOS LanTest will give you a preview of the real situation of use.

But what exactly is the problem at the end? Locally, you have too low transfers or are they low on the network?
Try to avoid operations between /sharedfolders and /srv. Check the transfers directly in /srv

Operations such as "mv" in "/srv" should act immediately without delay, "mv" operations that take place between /srv and /sharedfolders will have silly gag and will behave like a slow cp.

OMV_HC2 · 7. November 2018

Zitat von JohnStiles

But what exactly is the problem at the end? Locally, you have too low transfers or are they low on the network?

The original issue was my surprise at both of these things. I was expecting 85-100 MB/s network transfer speeds and had read many reports that HC2's were providing such speeds. I was mostly getting 40-75 MB/s with occasional bursts into the 80 MB/s range on read, and 18-23 MB/s for writes. All sequential, single large files. Coming from a dedicated Linux server (with multiple ext4 disks).

In trying to figure that out I found the low internal copy speeds from different shares on the same drive. Windows and full Linux handle these with ease. As long as you are on the same partition, file moves are instantaneous. On the little HC2 they were running at 20 MB/s or less locally. That was the second surprise.

To reorganize my drive, I hooked it up to a USB enclosure and moved the other shares into a large general share. This was instantaneous on Windows. Then hooked the drive back up to the HC2, and any folder / file moved within the 1 large Shared Folder were now instantaneous.

So, thanks to the advice on this thread, I've learned a number of things.

ntfs, while supported on the HC2, is not really a suitable OS for handling large files. It is much more limited than native filesystems on this platform.
Instantaneous local transfers are possible if sub folders are created within a single large Shared Folder.

My next tests will be checking things out on a non-ntfs drive once I manage to move all my media around a combination of internal drives and an external one. I'll choose between ext3, ext4, or btrfs for the filesystem. Open to recommendations from HC2 users.

Adoby · 7. November 2018

I prefer ext4. It's something of a "standard" for Linux. Robust. And perhaps the fastest. Not really any point to use ext3.

Go for btrfs if you want to use the COW-features. Might be fun. Or zfs?

But ext4, zfs and btrfs are most likely all fast enough for the network transfer to be the bottleneck.

OMV_HC2 · 9. November 2018

OK, was able to update my backups and reformat the drive. It is now ext4 formatted.
I am currently in the process of restoring all the files. It takes a long time to populate 8TB of data, even at speeds of 90MB/s - 175MB/s!

I'm using a linux live USB stick to do it from my Windows gaming PC since that has a 4TB Sata backup drive in it. Process should be complete this evening, or through the night. Has been running since last night.

Anyway, I had to run Nemo with elevated privileges in order to copy over all the files. I'm hoping that won't leave me with permission issues when I hook the drive back up to the HC2. Once it is all up and running and I will report back with speed test results. Should be a good comparison as it is the same data, running on the same platform, held by the same drive. The only difference will be the filesystem.

Adoby · 9. November 2018

On the Shared Folders page in OMV there is a tab for resetting permissions. Use that after you have created and populated the shared folder.

OMV_HC2 · 9. November 2018

Thanks for the tip on resetting the permissions.
In the end the Samba wasn't picking up the shared folders. When I went back in, the disk hadn't been remounted. After mounting it manually, the shares still weren't connecting.

I had to remap each one (by simply reselecting the Drive) which had now changed because I gave it a new Disk label. Once I had done that, everything was working again. So then the testing!

Same hardware (HC2) with same 10TB HGST Disk but now formatted as ext4 yielded the following results across the network, using the same desktop as all the other speed tests I have performed.

Read = 105 MB/s
Write = 88 MB/s

Much better than the previous figures, and completely in line with what I had expected.

Of course the HC2 has only been up and running for a few minutes, so I will run multiple tests over the next few days to verify that performance remains in the same ballpark.

Thanks to all who contributed and helped me with this!

OMV_HC2 · 10. November 2018

And in what is hopefully the last report in this thread, speed 17 hours later is as follows.

Read = 100 MB/s
Write = 92 MB/s

That is with some other network activity taking place - 1 kid streaming from the disk, another from YouTube.

Now on to solve my next issue for a which a new thread will be started...
Thanks again for the help.

Jetzt mitmachen!