HC2 internal file transfers going through LAN :-(

    • OMV_HC2 wrote:

      What is that telling me? The native write speed to the drive?
      That is telling you that your HC2 cannot write to your ntfs filesystem any faster than that. I'm sure it could write faster to ext4 on the same drive.
      omv 4.1.13 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • OK, good to know in more detail what is going on. Thanks for that.
      I wanted to know the read speed of the drive, so I looked for some sample code online.
      I cleared the cache with sudo /sbin/sysctl -w vm.drop_caches=3

      Then ran
      dd if=/sharedfolders/Transfers/testfile.dd of=/dev/null bs=1M count=1024

      That resulted in :
      1024+0 records in
      1024+0 records out
      1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.99122 s, 154 MB/s

      Does that mean the HC2 can read at 154 MB/s from the disk, but only write at 23?
      If it can read at 154 MB/s, why would my read transfer speeds on the network fluctuate so much? They are sequential reads (typically 2+GB ISO, ZIP and media files)?
    • As reference: this is what I get with my HC2 on my drive with BTRFS:

      Source Code

      1. 5151653888 bytes (5.2 GB, 4.8 GiB) copied, 29.0046 s, 178 MB/s
      2. 5000+0 records in
      3. 5000+0 records out
      4. 5242880000 bytes (5.2 GB, 4.9 GiB) copied, 31.5811 s, 166 MB/s
      Odroid HC2 - armbian - Seagate ST4000DM004 - OMV4.x
      Asrock Q1900DC-ITX - 16GB - 2x Seagate ST3000VN000 - Intenso SSD 120GB - OMV4.x
      :!: Backup - Solutions to common problems - OMV setup videos - OMV4 Documentation - user guide :!:
    • macom wrote:

      As reference: this is what I get with my HC2 on my drive with BTRFS:

      Source Code

      1. 5151653888 bytes (5.2 GB, 4.8 GiB) copied, 29.0046 s, 178 MB/s
      2. 5000+0 records in
      3. 5000+0 records out
      4. 5242880000 bytes (5.2 GB, 4.9 GiB) copied, 31.5811 s, 166 MB/s
      HDD 5400RPM(Use%86) + EXT4


      Source Code

      1. 5182062592 bytes (5.2 GB, 4.8 GiB) copied, 72.0075 s, 72.0 MB/s
      2. 5000+0 records in
      3. 5000+0 records out
      4. 5242880000 bytes (5.2 GB, 4.9 GiB) copied, 74.5859 s, 70.3 MB/s
    • Writing to an ext4 12TB IronWolf from HC2:

      Source Code

      1. adoby@nas1:~$ dd if=/dev/zero of=/home/adoby/testfile.dd bs=1M count=5000 conv=fdatasync status=progress && sync
      2. 5221908480 bytes (5.2 GB, 4.9 GiB) copied, 26.0001 s, 201 MB/s
      3. 5000+0 records in
      4. 5000+0 records out
      5. 5242880000 bytes (5.2 GB, 4.9 GiB) copied, 26.7838 s, 196 MB/s
      Writing to an ext4 512GB EVO 960 SSD from a HC2:

      Source Code

      1. adoby@ctrl:~$ dd if=/dev/zero of=/home/adoby/testfile.dd bs=1M count=5000 conv=fdatasync status=progress && sync
      2. 5022679040 bytes (5.0 GB, 4.7 GiB) copied, 21 s, 239 MB/s
      3. 5000+0 records in
      4. 5000+0 records out
      5. 5242880000 bytes (5.2 GB, 4.9 GiB) copied, 22.0455 s, 238 MB/s
      Copying a file via nfs from ext4 SSD on one HC2 to another HC2 with ext4 IronWolf HDD:

      Source Code

      1. adoby@ctrl:~$ time cp /home/adoby/testfile.dd /media/nfs/nas1/shared_media
      2. real 0m55.244s
      3. user 0m0.038s
      4. sys 0m36.427s
      5. adoby@ctrl:~$ ls --block-size=1M --size testfile.dd
      6. 5001 testfile.dd
      7. adoby@ctrl:~$ echo $(expr 5001 / 55) MB/s
      8. 90 MB/s
      OMV 4, 5 x ODROID HC2, 2 x 12TB, 2 x 8 TB, 1 x 500 GB SSD, GbE, WiFi mesh

      The post was edited 2 times, last by Adoby ().

    • OMV_HC2 wrote:

      Are those read or write speeds?
      And what speeds do you get across the network?

      I used BTRFS on a Synology at work when I was abroad - loved it. Didn't know the HC2 could play nice with it. I guess there isn't too much overhead.

      Does drive size play a part in the overall speeds?
      Write speeds.
      My last measurement was SSD read+network transfer+HDD write. 95 MiB/s.
      Drive size has a minor influence. Often a HDD gets slower when it gets full. My IronWolf 12TB above was 71% full.

      Good news: It seems SMB has a minor effect on the low performance you reported,
      Bad news: It seem NTFS has a huge negative effect.
      OMV 4, 5 x ODROID HC2, 2 x 12TB, 2 x 8 TB, 1 x 500 GB SSD, GbE, WiFi mesh

      The post was edited 2 times, last by Adoby ().

    • Adoby wrote:

      Write speeds.My last measurement was SSD read+network transfer+HDD write. 95 MiB/s.
      Drive size has a minor influence. Often a HDD gets slower when it gets full. My IronWolf 12TB above was 71% full.

      Good news: It seems SMB has a minor effect on the low performance you reported,
      Bad news: It seem NTFS has a huge negative effect.
      Yeah, the drives can get full quickly. I'm at 79% with the 10TB.
      I might have to figure something out with a couple of external 5TB drives then I could switch over to ext4 or BTRFS. Looks like ntfs is just going to be too slow for me.

      Will try to figure out how to do some kind of experiment. Will have to wipe my backup external drives to do it though... Probably worth it.

      Thanks for all the feedback and tips everyone.
    • OMV_HC2 wrote:

      I wanted to know the read speed of the drive
      In your post, you only mentioned writes which is why I gave that snippet.

      OMV_HC2 wrote:

      Does that mean the HC2 can read at 154 MB/s from the disk, but only write at 23?
      If it can read at 154 MB/s, why would my read transfer speeds on the network fluctuate so much? They are sequential reads (typically 2+GB ISO, ZIP and media files)?
      Seems that way. Just because dd reads at 154 doesn't mean a network transfer will. dd doesn't have the overhead of networking and samba along with ntfs in the mix.
      omv 4.1.13 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • If you need a test with network overhead and SMB use HELIOS LanTest will give you a preview of the real situation of use.


      But what exactly is the problem at the end? Locally, you have too low transfers or are they low on the network?
      Try to avoid operations between /sharedfolders and /srv. Check the transfers directly in /srv

      Operations such as "mv" in "/srv" should act immediately without delay, "mv" operations that take place between /srv and /sharedfolders will have silly gag and will behave like a slow cp.
    • JohnStiles wrote:

      But what exactly is the problem at the end? Locally, you have too low transfers or are they low on the network?
      The original issue was my surprise at both of these things. I was expecting 85-100 MB/s network transfer speeds and had read many reports that HC2's were providing such speeds. I was mostly getting 40-75 MB/s with occasional bursts into the 80 MB/s range on read, and 18-23 MB/s for writes. All sequential, single large files. Coming from a dedicated Linux server (with multiple ext4 disks).

      In trying to figure that out I found the low internal copy speeds from different shares on the same drive. Windows and full Linux handle these with ease. As long as you are on the same partition, file moves are instantaneous. On the little HC2 they were running at 20 MB/s or less locally. That was the second surprise.

      To reorganize my drive, I hooked it up to a USB enclosure and moved the other shares into a large general share. This was instantaneous on Windows. Then hooked the drive back up to the HC2, and any folder / file moved within the 1 large Shared Folder were now instantaneous.

      So, thanks to the advice on this thread, I've learned a number of things.
      1. ntfs, while supported on the HC2, is not really a suitable OS for handling large files. It is much more limited than native filesystems on this platform.
      2. Instantaneous local transfers are possible if sub folders are created within a single large Shared Folder.
      My next tests will be checking things out on a non-ntfs drive once I manage to move all my media around a combination of internal drives and an external one. I'll choose between ext3, ext4, or btrfs for the filesystem. Open to recommendations from HC2 users.

      The post was edited 1 time, last by OMV_HC2 ().

    • I prefer ext4. It's something of a "standard" for Linux. Robust. And perhaps the fastest. Not really any point to use ext3.

      Go for btrfs if you want to use the COW-features. Might be fun. Or zfs?

      But ext4, zfs and btrfs are most likely all fast enough for the network transfer to be the bottleneck.
      OMV 4, 5 x ODROID HC2, 2 x 12TB, 2 x 8 TB, 1 x 500 GB SSD, GbE, WiFi mesh
    • OK, was able to update my backups and reformat the drive. It is now ext4 formatted.
      I am currently in the process of restoring all the files. It takes a long time to populate 8TB of data, even at speeds of 90MB/s - 175MB/s!

      I'm using a linux live USB stick to do it from my Windows gaming PC since that has a 4TB Sata backup drive in it. Process should be complete this evening, or through the night. Has been running since last night.

      Anyway, I had to run Nemo with elevated privileges in order to copy over all the files. I'm hoping that won't leave me with permission issues when I hook the drive back up to the HC2. Once it is all up and running and I will report back with speed test results. Should be a good comparison as it is the same data, running on the same platform, held by the same drive. The only difference will be the filesystem.
    • Thanks for the tip on resetting the permissions.
      In the end the Samba wasn't picking up the shared folders. When I went back in, the disk hadn't been remounted. After mounting it manually, the shares still weren't connecting.

      I had to remap each one (by simply reselecting the Drive) which had now changed because I gave it a new Disk label. Once I had done that, everything was working again. So then the testing!

      Same hardware (HC2) with same 10TB HGST Disk but now formatted as ext4 yielded the following results across the network, using the same desktop as all the other speed tests I have performed.

      Read = 105 MB/s
      Write = 88 MB/s

      Much better than the previous figures, and completely in line with what I had expected.

      Of course the HC2 has only been up and running for a few minutes, so I will run multiple tests over the next few days to verify that performance remains in the same ballpark.

      Thanks to all who contributed and helped me with this!
    • And in what is hopefully the last report in this thread, speed 17 hours later is as follows.

      Read = 100 MB/s
      Write = 92 MB/s

      That is with some other network activity taking place - 1 kid streaming from the disk, another from YouTube.

      Now on to solve my next issue for a which a new thread will be started...
      Thanks again for the help.