Fluctuating Transferspeeds on Nextcloud

  • I'm running OMV 4 on an Odroid UX4 with 3 WD Reds (ext4) connected over 2 USB3 ports (1 usb Hub so two drives share a USB3 port)


    The 3 drives are connected using Unified Filesystem which basically merges the three drives into one.

    I'm also using snapraid.


    On top of this I'm running nextcloud.


    Now when I upload a folder with some files to nextcloud using drag and drop, the uploadspeed fluctuates between 150mbit/s and 0 it's constantly going up and down.

    otjyN.png


    I used iperf to check the network speed between my laptop and my ODROID which is running nextcloud and I get a decent 906Mbits/s:



    Code
    [ ID] Interval Transfer Bandwidth
    [ 3] 0.0-10.0 sec 1.05 GBytes 906 Mbits/sec



    dd if=/dev/zero of=/sharedfolders/test/ bs=8k count=1000k; rm -f /sharedfolders/test/


    returns

    (8.4 GB, 7.8 GiB) copied, 153.854 s, 54.5 MB/s


    I was monitoring that transfer, there the write speed was constant.

  • dd if=/dev/zero of=/sharedfolders/test/ bs=8k count=1000k; rm -f /sharedfolders/test/

    Read from one drive to another, especially from the two connected to the same USB slot. Check that speed.


    Greetings
    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

  • Seems strange but the speeds are about the same. I'll post the code to run this operation because I'm not sure I did this right:

    To do this test I created the directories directly on those drives so I'm not using the union filesystem.

    drive A and C share the same usb port.

    msedge-d3-Zox-OUqc-J.png

    Code
    #create a bigfile on folder test_A
    dd if=/dev/zero of=/sharedfolders/test_A/bigfile bs=8k count=1000k
    echo 3 | sudo tee /proc/sys/vm/drop_caches
    #copy bigfile from test_A to test_C
    time dd if=/sharedfolders/test_A/bigfile of=/sharedfolders/test_C/bigfile bs=8k

    Here are the results:

    (8.4 GB, 7.8 GiB) copied, 160.579 s, 52.2 MB/s



    Edit:


    When I do the dd from the initial post I'm actually getting double the write speed when writing directly to the drive not using union filesystem

    so probably a read from drive A to C using the union filesystem will get me about 25mb/s.


    Also using the same hub had no impact on write speed, I tried the same operation - copying from drive A to B and had about the same speeds.


    Edit:


    I deleted all folders on the Union Filesystem

    changed the union filesystem settings from:

    defaults,allow_other,direct_io,use_ino


    to:

    defaults,allow_other,direct_io,use_ino,cache.files=off,dropcacheonclose=true

    These settings are probably not perfect.

    These are mergerfs settings, which union fileystem is setting up in the background.

    restarted the dockers mariadb and nextcloud.


    now I get a solid 100mbit/s upload speed on nextcloud which is not perfect but I can live with that.

  • davidh2k

    Added the Label resolved
  • Glad you could fix it.


    Greetings

    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

  • The speeds worked for a short time but then went to ok-ish, they were still dropping in regular intervals just not to zero.


    But now I changed a bunch of stuff:

    I added a reverse proxy: traefik, changed the nextcloud image to the original one, now I'm also using cloudflare and added portainer... the traefik stuff I don't really understand... But I did not change any settings in OMV.


    Now it's getting kinda crazy the upload speeds start looking like waveform from soundcloud or something:

    Taskmgr-6-LOy-JNEic-U.png


    Taskmgr-UXts0v-Wi-Iv.png


    The CPU and ram usage is low


    probably not on omv issue

  • Probably not related, but I"m experiencing the same thing with my USB controller as it keeps resetting during writing and the syslog generates about 500 pages a day (the graph looks identical). Is there nothing in your syslog explaining anything at all?

  • I added a reverse proxy: traefik, changed the nextcloud image to the original one, now I'm also using cloudflare and added portainer... the traefik stuff I don't really understand... But I did not change any settings in OMV.

    What don't you understand? Happy to help/explain on the traefik front.


    Greetings

    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

  • This might be a unrelated issue but I was looking through those syslogs and one thing that occuerd every 30 seconds was:


    Code
    'mountpoint_srv_dev-disk-by-id-usb-SanDisk_Ultra_4C530001021028114151-0-0' status failed (1) -- /srv/dev-disk-by-id-u sb-SanDisk_Ultra_4C530001021028114151-0-0 is not a mountpoint

    This is a USB stick I had attached in the beginning but never used. But in OMV I can't remove that filesystem. When I go to filesystems, select that usbdrive filesystem and click delete, it asks me twice if I'm ok with deleting, but nothing happens.

  • What don't you understand? Happy to help/explain on the traefik front.


    Greetings

    David


    Thank you I would much appreciate if you explained some basic things about traefik.


    My main questions are about the labels.

    There are a lot of labels that have to be set, and every tutorial seems to be setting those labels a little differently and they are not really explaining them.


    There seems to be a lot of middlewares, I don't really know where traefik gets them from, for example I have nextcloud-dav and nextcloud-https.


    I don't understand what endpoints are.


    And I have no clue what these regexes are for:

    ^/.well-known/ca(l|rd)dav


    I also don't understand the loadbalance.sever.port


    Also I have traefik mapped to port 8180 for the web gui but I can't access that for some reason.


    here my docker compose for traefik:

    and my traefik.toml

    dynamic.yaml

  • This is a USB stick I had attached in the beginning but never used. But in OMV I can't remove that filesystem. When I go to filesystems, select that usbdrive filesystem and click delete, it asks me twice if I'm ok with deleting, but nothing happens.


    I'm a bit rusty about removing missing stuff. Maybe ryecoaaron can jump in here.




    My main questions are about the labels.

    There are a lot of labels that have to be set, and every tutorial seems to be setting those labels a little differently and they are not really explaining them.


    The usage of labels depens on how you config traefik itself. You usually use labels to be able to set the variables the way you want. Espcially when using stacks the domain for example can be kinda messy.


    There seems to be a lot of middlewares, I don't really know where traefik gets them from, for example I have nextcloud-dav and nextcloud-https.


    I'm not using either of them. The only one I use is basic auth for the traefik webinterface. Think of them as a man-in-the-middle. It depends on what you want to do. nextcloud-https sounds useless to me, but thats without me looking at the actual purpose of it.


    I don't understand what endpoints are.

    https://docs.traefik.io/v2.0/operations/api/#endpoints


    And I have no clue what these regexes are for:

    ^/.well-known/ca(l|rd)dav

    They probably match the any-domain.tld/.wll-known/caldav and anydomain.tld/.well-known/carddav

    You're currently using SSL authentication, i'll later on suggest you to take a look at DNS authentication. This kicks ass.


    I also don't understand the loadbalance.sever.port

    Every service (nextcloud, teamspeak, whatever) needs to have a loadbalancer in front of it. This is due to the new design in traefik 2.0. All you do there is to tell traefik which port it should use to access the service inside the container.


    Also I have traefik mapped to port 8180 for the web gui but I can't access that for some reason.

    You don't need to have the port mapped to be able to access it. I set it to 8099 inside the traefik.yml and created a router for it inside the rules.yml





    Greetings

    David

    "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"


    Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.

    Upload Logfile via WebGUI/CLI
    #openmediavault on freenode IRC | German & English | GMT+1
    Absolutely no Support via PM!

  • 'm a bit rusty about removing missing stuff.

    I would just wipe it from the command line with wipefs -a /dev/sdX

    omv 5.6.13 usul | 64 bit | 5.11 proxmox kernel | omvextrasorg 5.6.2 | kvm plugin 5.1.6
    omv-extras.org plugins source code and issue tracker - github


    Please read this before posting a question.
    Please don't PM for support... Too many PMs!

  • thank you for your replies.


    I figured out what the issue was with these fluctuations.

    It turns out, it's expected behavior. When you upload to nextcloud, you do it in chunks of a specified size (default is 10 or 5mb). Since I'm running through cloudflare, I set my size to 100mb. It will upload the files in chunks and assamble them after.


    https://help.nextcloud.com/t/n…-already-hopeless/70719/5


    It would be cool if it could upload the chunks in parallel but I don't have the beefiest server anyway.


    Since these speeds are a little low, I'm looking into OpenVPN and then have a samba server I can use for bigger files and use nextcloud only for documents and such.

  • I'll probably be flamed for this but, I think Nextcloud is sort of shitty :-/. I haven't stated that before because it appears to be extremely popular (especially here), but in no way was I feeling it was useful when I used it.


    Keep in mind that I haven't "stepped up" and created something myself, so looking at the gift horse and all, but....


    1. If you want to transfer actual files, is sending data via some HTTP POST'ish scheme a viable solution or just a convenient option?

    2. Does the ability to run plugins with it make it better or is that just some bootstrap method that can be accomplished many other ways?

    3. With the presented options, is multiple users really a good idea or just an after thought adding needless complexity?

    4. Why run a separate aggregating DB layer on top of files that should be already known?


    I dunno. I see a whole lot of people mentioning Nextcloud, but I don't see a whole lot of people mentioning what Nextcloud does better. I'm really not trying to be a pessimist, but for me there seems to be better ways at doing everything it does. Somewhere else I read people liking the Drag & Drop functionality, however at that point I wonder why not create a small dedicated webpage for doing just that and side step 99% of the Nextcloud's overhead.


    There's the old idea of "Do 1 thing and do it well", but for me Nextcloud doesn't have 1 thing to boot... :-(.

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!