Fixed upload rate for ALL users or the ProFTPd service itself

    • OMV 1.0
    • Resolved
    • Fixed upload rate for ALL users or the ProFTPd service itself

      Hello OMV-Community!

      I'm getting crazy... Searching for a possibility to limit the FTP uploadspeed to a fixed amount, because if multiple users or one user with multiple connections is/are downloading, they reach my max. upload speed which is about 1280kB/s in total. Then my ping raises like crazy and I cant play properly anymore. I tried multiple things, for example: the linux tool trickle and 2 different QoS. Trickle won't affect proftpd, I dont know why - it just wont limit it. DD-WRT QoS also did not work for the wired server. Only for the wireless clients (I dont know if its a bug or wrong configuration of the router itself, but I just can't see wired clients in the interface of DD-WRT (we use it as Access Point, main router is a Fritzbox 7390 with deavtivated WiFi. DHCP in DD-WRT is turned off or used as forwarder).

      OMV-Server---Gigabit LAN--->DD-WRT---Gigabit LAN---> Fritzbox.

      Fritzbox QoS also doesn't work at all, not even with own, newly added rules with the ports which some games use (CS:GO for example)

      So, this is why I want to limit the ProFTPd transferspeed in overall, not per "TransferRate", as this only limits the speed per connection.
      proftpd.org/docs/directives/li…fig_ref_TransferRate.html

      The OMV-WebUI does use the same command.
      If I limit the connections to 1, the users can't browse anymore when downloading (when they are using a normal browser for downloading, firefox for example), if they use a FTP client like Filezilla, they can't download anymore, because the server always uses permanently 1 connection for the directories. Also if multiple users download at the same time, they always would reach the max. uploadrate, no matter how much connections per user are set.

      subzero79 tried to help me over IRC, but its too hard for me as a linux newbie, to compile and install ProFTPd with the trafficshaper-module.
      Is there a way to request this feature to be implemented in one of the next versions? Who must I ask for this?
      The module: proftpd.org/docs/contrib/mod_shaper.html

      Thank you in advance!
      OpenMediaVault 1.12 on Debian Wheezy 7.8 64bit | 3.16 Backport Kernel | OMV-Extras.org 1.10
      ASRock Q1900-ITX | 1x4GB Corsair Vengeance SODIMM DDR3 | 1xWD Red 3TB | 1xApacer 1,8" SATA1 SSD 32GB as system drive | be quiet! Pure Power L8 350W | SilverStone Milo ML04 Black

      The post was edited 1 time, last by Nyctophilia ().

    • You should create a feature request at bugtracker.openmediavault.org/

      And all your attempts to limit the transfer rates didn't work? Well, I made the experience that proftp running as a plugin on OMV is a litte fussy if you want to change the standard settings.
      For example, if you enter a command like DefaultRoot in the extra options of the plugin, this command is saved to the proftpd.conf file, but the server doesn't use it. Why? After checking the proftpd.conf I saw that the DefaultRoot direcctive appears twice.... the first one is the standard setting (For me this seems to be hardcoded) "DefaultRoot /srv/ftp" and some lines below my command "DefaultRoot ~", but the second one seems to be ignored.
      The only way to tell proftpd that it should use the tilde for leading users to their home dirs is to enter a line in the OMV default config file at /etc/default/openmediavault using the correct syntax and then it works.

      To make a long story short, did you try to use the correct settings for limiting the transfer rates in the default file? I would enter a line like OMV_PROFTPD_TRANSFERRATE="RETR 20.0", run "omv-mkconfig proftpd" and "service proftpd restart" from the CLI.
      If you then enter some limitations for the number of connections, say max. 5 clients and max. 2 connections per client this should end up in a max. transfer rate of 200kb overall. Give it a shot and tell if that works.
      Homebox: Bitfenix Prodigy Case, ASUS E45M1-I DELUXE ITX, 8GB RAM, 5x 4TB HGST Raid-5 Data, 1x 320GB 2,5" WD Bootdrive via eSATA from the backside
      Companybox 1: Standard Midi-Tower, Intel S3420 MoBo, Xeon 3450 CPU, 16GB RAM, 5x 2TB Seagate Data, 1x 80GB Samsung Bootdrive - testing for iSCSI to ESXi-Hosts
      Companybox 2: 19" Rackservercase 4HE, Intel S975XBX2 MoBo, C2D@2200MHz, 8GB RAM, HP P212 Raidcontroller, 4x 1TB WD Raid-0 Data, 80GB Samsung Bootdrive, Intel 1000Pro DualPort (Bonded in a VLAN) - Temp-NFS-storage for ESXi-Hosts
    • You need the dev package from proftpd

      apt-get install proftpd-dev

      Noe we grab the source from debian same version for wheezy

      packages.debian.org/wheezy/proftpd-basic
      ftp.de.debian.org/debian/pool/…-dfsg_1.3.4a-5+deb7u2.dsc

      git clone git://git.debian.org/pkg-proftpd/proftpd-dfsg.git

      cd proftpd-dfsg/contrib

      prxs -c -i -d mod_shaper.c

      That's it your finish. The shaper goes in this simple way in extra options

      Source Code

      1. ShaperLog /var/log/proftpd/proftpd.shaper
      2. ShaperTable /etc/proftpd/shaper.tab
      3. ShaperEngine on
      4. ShaperAll downrate 15 uprate 15


      For advanced options read the proftpd doc website. Is sleeping material but....
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server

      The post was edited 1 time, last by subzero79 ().

    • datadigger wrote:

      You should create a feature request at bugtracker.openmediavault.org/

      And all your attempts to limit the transfer rates didn't work? Well, I made the experience that proftp running as a plugin on OMV is a litte fussy if you want to change the standard settings.
      For example, if you enter a command like DefaultRoot in the extra options of the plugin, this command is saved to the proftpd.conf file, but the server doesn't use it. Why? After checking the proftpd.conf I saw that the DefaultRoot direcctive appears twice.... the first one is the standard setting (For me this seems to be hardcoded) "DefaultRoot /srv/ftp" and some lines below my command "DefaultRoot ~", but the second one seems to be ignored.
      The only way to tell proftpd that it should use the tilde for leading users to their home dirs is to enter a line in the OMV default config file at /etc/default/openmediavault using the correct syntax and then it works.

      To make a long story short, did you try to use the correct settings for limiting the transfer rates in the default file? I would enter a line like OMV_PROFTPD_TRANSFERRATE="RETR 20.0", run "omv-mkconfig proftpd" and "service proftpd restart" from the CLI.
      If you then enter some limitations for the number of connections, say max. 5 clients and max. 2 connections per client this should end up in a max. transfer rate of 200kb overall. Give it a shot and tell if that works.



      I used the commands with the "Extra Options" field. Writing it directly into the ProFTPd config did not help at all, because it reseted when restarting the server or reapplying the config per WebUI.

      The thing is - I want my users to able to download with as much speed per file as possible - no weird combining with connections to get a total rate :P They should be able to download one file with max. bandwidth, indepentend from how much users and connectons are used. That's why the shaper_module is probably there. But I'm too stupid for Linux yet. Luckily subzero79 wrote me a step by step tutorial. I'm gonna try it now :)

      Im also gonna add the request to OMV-Bugtracker - maybe there is interest for it :)
      OpenMediaVault 1.12 on Debian Wheezy 7.8 64bit | 3.16 Backport Kernel | OMV-Extras.org 1.10
      ASRock Q1900-ITX | 1x4GB Corsair Vengeance SODIMM DDR3 | 1xWD Red 3TB | 1xApacer 1,8" SATA1 SSD 32GB as system drive | be quiet! Pure Power L8 350W | SilverStone Milo ML04 Black
    • subzero79 wrote:

      You need the dev package from proftpd

      <code class="inlineCode">apt-get install proftpd-dev</code>

      Noe we grab the source from debian same version for wheezy

      packages.debian.org/wheezy/proftpd-basic
      ftp.de.debian.org/debian/pool/…-dfsg_1.3.4a-5+deb7u2.dsc

      git://git.debian.org/pkg-proftpd/proftpd-dfsg.git

      cd proftpd-dfsg/contrib

      prxs -c -i -d mod_shaper.c

      That's it your finish. The shaper goes in this simple way in extra options



      Quellcode



      ShaperLog /var/log/proftpd/proftpd.shaper
      ShaperTable /etc/proftpd/shaper.tab
      ShaperEngine on
      ShaperAll downrate 15 uprate 15



      For advanced options read the proftpd doc website. Is sleeping material but....
      <code class="inlineCode"></code>…


      apt-get install proftpd-dev worked so far - no problem - but I dont know how to grab the source? What exactly should I do with these 2 links? Downloaded the source now with wget.
      packages.debian.org/wheezy/proftpd-basic
      ftp.de.debian.org/debian/pool/…-dfsg_1.3.4a-5+deb7u2.dsc

      how do i get the git:// link? Or what else am I supposed to do with it.

      Sorry, im really a CLI noob and I only know a few basic linux/unix commands :/
      Using Linux for about 3 weeks or so.

      Thank you for helping me
      OpenMediaVault 1.12 on Debian Wheezy 7.8 64bit | 3.16 Backport Kernel | OMV-Extras.org 1.10
      ASRock Q1900-ITX | 1x4GB Corsair Vengeance SODIMM DDR3 | 1xWD Red 3TB | 1xApacer 1,8" SATA1 SSD 32GB as system drive | be quiet! Pure Power L8 350W | SilverStone Milo ML04 Black

      The post was edited 1 time, last by Nyctophilia ().

    • You can request at bugtracker but remember that proftpd is a package distributed by official debian repos. This feature does not come with the package by default. So a feature to enable this is gonna be useless if the user doesn't build or add this module by his own.
      And is for this reason that we have extra options in global and shares.

      @datadigger The other day I pointed you to use the ~ in DefaultRoot is still not working for you in the environmental variable?

      I tested this the other day and it worked perfectly
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • Okay, I did it exactly as you wrote it down, but still it doesnt shape it to the wanted amount :/


      OpenMediaVault 1.12 on Debian Wheezy 7.8 64bit | 3.16 Backport Kernel | OMV-Extras.org 1.10
      ASRock Q1900-ITX | 1x4GB Corsair Vengeance SODIMM DDR3 | 1xWD Red 3TB | 1xApacer 1,8" SATA1 SSD 32GB as system drive | be quiet! Pure Power L8 350W | SilverStone Milo ML04 Black
    • Loaded modules:
      mod_vroot/0.9.2
      mod_ifsession/1.1
      mod_ifversion/0.1
      mod_deflate/0.5.5
      mod_copy/0.4
      mod_unique_id/0.2
      mod_facl/0.4
      mod_sftp_pam/0.2
      mod_sftp/0.9.8
      mod_site_misc/1.5
      mod_ratio/3.3
      mod_shaper/0.6.6
      mod_exec/0.9.11
      mod_dynmasq/0.4
      mod_wrap2_file/1.2
      mod_wrap2/2.0.6
      mod_ban/0.6
      mod_load/1.0.1
      mod_rewrite/0.9
      mod_wrap/1.2.4
      mod_quotatab_radius.c
      mod_quotatab_file.c
      mod_quotatab/1.3.0
      mod_radius/0.9.1
      mod_tls/2.4.3
      mod_ctrls_admin/0.9.7
      mod_lang/0.9
      mod_ctrls/0.9.4
      mod_cap/1.1
      mod_readme/1.0
      mod_auth_pam/1.1
      mod_ident/1.0
      mod_dso/0.5
      mod_facts/0.3
      mod_delay/0.7
      mod_site.c
      mod_log.c
      mod_ls.c
      mod_auth.c
      mod_auth_file/0.9
      mod_auth_unix.c
      mod_xfer.c
      mod_core.c



      After compiling it also said:

      To load your newly installed module into proftpd, be sure
      to edit your proftpd.conf and add the following:

      <IfModule mod_dso.c>
      LoadModule mod_shaper.c
      </IfModule>

      and then restart your proftpd server, so that the config change
      becomes live.



      But I can't add it, because it always resets the config when reapplying/restarting the server - if i put it into the "Extra Options" tab, there is an error. If try to fix this error by commenting #LoadModule mod_tls_memcache.c, it still gets me the same error.

      Error
      #4000:
      exception 'OMVException' with message 'Failed to execute command 'export
      LANG=C; invoke-rc.d 'proftpd' start 2>&1': Starting ftp server:
      proftpdNAS proftpd[9902]: ?mod_dso/0.5: module 'mod_shaper.c' already
      loaded
      ? <--- Why is it not working then?
      NAS proftpd[9902]: Fatal: LoadModule: error loading module
      'mod_shaper.c': Operation not permitted on line 67 of
      '/etc/proftpd/proftpd.conf'
      failed!
      invoke-rc.d: initscript proftpd, action "start" failed.' in
      /usr/share/php/openmediavault/initscript.inc:176
      Stack trace:
      #0 /usr/share/php/openmediavault/initscript.inc(141):
      OMVSysVInitScript->invoke('start')
      #1 /usr/share/php/openmediavault/initscript.inc(61):
      OMVSysVInitScript->start()
      #2 /usr/share/openmediavault/engined/module/proftpd.inc(79):
      OMVSysVInitScript->exec()
      #3 /usr/share/openmediavault/engined/rpc/config.inc(206):
      OMVModuleProftpd->startService()
      #4 [internal function]: OMVRpcServiceConfig->applyChanges(Array,
      Array)
      #5 /usr/share/php/openmediavault/rpcservice.inc(125):
      call_user_func_array(Array, Array)
      #6 /usr/share/php/openmediavault/rpcservice.inc(158):
      OMVRpcServiceAbstract->callMethod('applyChanges', Array, Array)
      #7 /usr/share/openmediavault/engined/rpc/config.inc(224):
      OMVRpcServiceAbstract->callMethodBg('applyChanges', Array, Array)
      #8 [internal function]: OMVRpcServiceConfig->applyChangesBg(Array,
      Array)
      #9 /usr/share/php/openmediavault/rpcservice.inc(125):
      call_user_func_array(Array, Array)
      #10 /usr/share/php/openmediavault/rpc.inc(79):
      OMVRpcServiceAbstract->callMethod('applyChangesBg', Array, Array)
      #11 /usr/sbin/omv-engined(500): OMVRpc::exec('Config', 'applyChangesBg',
      Array, Array, 1)
      #12 {main}

      (I understood why there is this error now, because it's already loaded.)

      OpenMediaVault 1.12 on Debian Wheezy 7.8 64bit | 3.16 Backport Kernel | OMV-Extras.org 1.10
      ASRock Q1900-ITX | 1x4GB Corsair Vengeance SODIMM DDR3 | 1xWD Red 3TB | 1xApacer 1,8" SATA1 SSD 32GB as system drive | be quiet! Pure Power L8 350W | SilverStone Milo ML04 Black

      The post was edited 2 times, last by Nyctophilia ().

    • Is already loaded as you can check also in /etc/proftpd/modules.conf

      You have your module now in proftpd is time for you to understand how to configure it properly. The example I posted was very simple. This module maybe it works maybe it doesn't, Is hard to find examples of proftpd since is very old. But the doc website indicates many possible configurations
      Check also the log at /var/log/proftpd/proftpd.shape to see whats going on

      Other thing would be now to use traffic shaping rules by port or something. Traffic shaping rules is not something that and I am familiar with.

      The utility trickle can limit bandwidth to commands but not for PID unfortunately
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server

      The post was edited 1 time, last by subzero79 ().

    • I can't use proftpd.org/docs/contrib/mod_shaper.html#shaper_info - Is this needed for the usage/configuration?

      Installation
      To install mod_shaper, copy the mod_shaper.c file
      into:
      proftpd-dir/contrib/

      after unpacking the latest proftpd-1.2.x or proftpd-1.3.x source
      code. Then follow the usual steps for using third-party modules in proftpd,
      making sure to include the --enable-ctrls configure option,
      which mod_shaper requires:
      ./configure --enable-ctrls --with-modules=mod_shaper


      To build mod_shaper as a DSO module:
      ./configure --enable-ctrls --enable-dso --with-shared=mod_shaper

      Then follow the usual steps:
      make
      make install

      Alternatively, if your proftpd was compiled with DSO support, you can
      use the prxs tool to build mod_shaper as a shared
      module:

      prxs -c -i -d mod_shaper.c


      OpenMediaVault 1.12 on Debian Wheezy 7.8 64bit | 3.16 Backport Kernel | OMV-Extras.org 1.10
      ASRock Q1900-ITX | 1x4GB Corsair Vengeance SODIMM DDR3 | 1xWD Red 3TB | 1xApacer 1,8" SATA1 SSD 32GB as system drive | be quiet! Pure Power L8 350W | SilverStone Milo ML04 Black
    • Found a perfectly working solution.
      Followed this tutorial: cyberciti.biz/faq/linux-traffi…ng-ftp-server-port-21-22/

      If you are using normal ftp without passive ports, then you should only set rules for port 20 and 21 (This is the reason why the shaping per iptables and tc never worked for me... I did not know that ACTUALLY FTP PORT 20 is used for the filetransfer. If some guys on IRC.freenode.net ##networking would not have told me that, I still wouldn't know that... Right now, I am using a passive range and it works fine.
      Limited to 1000kB/s outgoing, automatic/dynamic bandwidth dividing and no ping rises at all. Finally!

      Adapted these 2 lines and changed them to my needs:

      Source Code

      1. /sbin/tc class add dev eth0 parent 11: classid 11:1 htb rate 1000kbps ceil 1000kbps quantum 2048
      2. /sbin/tc class add dev eth0 parent 11:1 classid 11:101 htb rate 500kbps ceil 1000kbps prio 0 quantum 2048

      To make everything permanent and working after a reboot, people in ##networking suggested me to do:

      Source Code

      1. /sbin/iptables-save > /etc/firewall.conf; echo "/sbin/iptables-restore < /etc/firewall.conf" >> /etc/rc.local; echo "ENTER YOUR TC COMMANDS HERE" >> /etc/rc.local

      Then open up /etc/rc.local with an editor and it should look like that (exit 0 must be at the very bottom, atleast that's what people told me):

      Shell-Script

      1. #!/bin/sh -e
      2. #
      3. # rc.local
      4. #
      5. # This script is executed at the end of each multiuser runlevel.
      6. # Make sure that the script will "exit 0" on success or any other
      7. # value on error.
      8. #
      9. # In order to enable or disable this script just change the execution
      10. # bits.
      11. #
      12. # By default this script does nothing.
      13. /sbin/iptables-restore < /etc/firewall.conf
      14. /sbin/tc qdisc add dev eth0 root handle 11: htb default 500 r2q 1
      15. /sbin/tc class add dev eth0 parent 11: classid 11:1 htb rate 1000kbps ceil 1000kbps quantum 2048
      16. /sbin/tc class add dev eth0 parent 11:1 classid 11:101 htb rate 500kbps ceil 1000kbps prio 0 quantum 2048
      17. /sbin/tc qdisc add dev eth0 parent 11:101 handle 1001: sfq
      18. /sbin/tc filter add dev eth0 parent 11: protocol ip handle 101 fw classid 11:101
      19. exit 0
      Display All


      This works fine for me, but If you guys have some better and more efficient solution, just post it!

      Some guy on IRC.freenode.net #proftpd brought me to the idea to shape it per iptables and tc, as it is much more efficient. I just love linux and the community. Getting more and more used to it and I'm having some serious fun by learning and actually doing some useful stuff with it! Thread solved!
      OpenMediaVault 1.12 on Debian Wheezy 7.8 64bit | 3.16 Backport Kernel | OMV-Extras.org 1.10
      ASRock Q1900-ITX | 1x4GB Corsair Vengeance SODIMM DDR3 | 1xWD Red 3TB | 1xApacer 1,8" SATA1 SSD 32GB as system drive | be quiet! Pure Power L8 350W | SilverStone Milo ML04 Black
    • Yeah, it is kinda useless. The shaping was awful when I got it to work. The iptable rules are the same as in the guide, I'm just using other ports for the passive range.
      OpenMediaVault 1.12 on Debian Wheezy 7.8 64bit | 3.16 Backport Kernel | OMV-Extras.org 1.10
      ASRock Q1900-ITX | 1x4GB Corsair Vengeance SODIMM DDR3 | 1xWD Red 3TB | 1xApacer 1,8" SATA1 SSD 32GB as system drive | be quiet! Pure Power L8 350W | SilverStone Milo ML04 Black

      The post was edited 2 times, last by Nyctophilia ().