Pinned [Tutorial][Experimental][Third-party Plugin available]Reducing OMV's disk writes, also to install it on USB flash

    • Sergio wrote:

      Don't use micro Sd cards for this. That's your problem. Only use full size SD cards.

      You can't use a full size SD card in an RPi2 or Odroid-C1. They only have a micro SD slot :)
      omv 4.1.14 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • Sergio wrote:

      ryecoaaron wrote:

      Here are the last three types of cards I purchased:
      Transcend 16GB MicroSDHC Class10 UHS-1
      PNY Turbo Performance 16GB High Speed MicroSDHC Class 10 UHS-1
      Kingston Digital 16 GB microSDHC Class 10 UHS-1


      Don't use micro Sd cards for this. That's your problem. Only use full size SD cards.

      I have my XBMC/Kodi running on a SD from the first day without any problem. It does not use a centralized mysql db or anything, all it's saved n the Sd card, temporary files, covers art, any transcoding. It has been used almost every day and I forgot when I built that setup. Perhaps three years... or more.

      If you use mobile class equipment it's normal to expect them dying. I only use storage/industrial class SDs. And USB sticks are always a lottery, so I don't use them. Well... I have my domoticz running out of a separate OMV installation on a random 2GB USB in a diskless eePC for a year. I have it there with me at work. Perhaps I could issue some command to retrieve the past writes numbers or the media age, if someone tells how.


      Both use the same type of flash. So that isn't the reason...

      But would you mind telling me if you use OpenElec, or which distro do you use?

      Greetings
      David
      "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"

      Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.


      Upload Logfile via WebGUI/CLI
      #openmediavault on freenode IRC | German & English | GMT+1
      Absolutely no Support via PM!

      I host parts of the omv-extras.org Repository, the OpenMediaVault Live Demo and the pre-built PXE Images. If you want you can take part and help covering the costs by having a look at my profile page.
    • I would need a little advice.

      After installing the plugin, i started to receive the following messages:
      Failed to fetch file:/var/cache/openmediavault/archives/Packages File not found

      I actually made a mistake after installing the plugin, i rebooted without enabling it, so it might've been the cause. But now, i have no idea how to fix this...

      Thanks!
    • Hit apt-clean in omvextras. ;)

      Greetings
      David
      "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"

      Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.


      Upload Logfile via WebGUI/CLI
      #openmediavault on freenode IRC | German & English | GMT+1
      Absolutely no Support via PM!

      I host parts of the omv-extras.org Repository, the OpenMediaVault Live Demo and the pre-built PXE Images. If you want you can take part and help covering the costs by having a look at my profile page.
    • gyroland wrote:

      I would need a little advice.

      After installing the plugin, i started to receive the following messages:
      Failed to fetch file:/var/cache/openmediavault/archives/Packages File not found


      /var/cache is set to keep_folder_structure which does not retain the contents of the directory. At first I ignored the error. I realized that no updates were being pulled because the updater expects the Packages file to be present, even if empty. Execute this command to create it:

      touch /var/cache/openmediavault/archives/Packages


      This needs to be done on every boot. I created an init.d script to do that, but in hindsight it would've been simpler to add it to rc.local.
      ryecoaaron - what do you think about adding that? An alternative would be to change that directory in fs2ram.conf to keep_file_structure. However since it's a cache folder there could be a lot of cruft in there.
    • chefp wrote:

      touch /var/cache/openmediavault/archives/Packages

      This needs to be done on every boot. I created an init.d script to do that, but in hindsight it would've been simpler to add it to rc.local.
      ryecoaaron - what do you think about adding that? An alternative would be to change that directory in fs2ram.conf to keep_file_structure. However since it's a cache folder there could be a lot of cruft in there.

      Thanks for finding out why people were losing that file :) I rarely reboot my system so I never noticed it being gone. I agree there is too much in that directory to keep the files. While I don't really like adding things to rc.local, I can't think of any other way to do it. I would also prefer that adding it to rc.local is done manually instead of the plugin adding it. I will try to find a better place to put it.
      omv 4.1.14 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      chefp wrote:

      touch /var/cache/openmediavault/archives/Packages

      This needs to be done on every boot. I created an init.d script to do that, but in hindsight it would've been simpler to add it to rc.local.
      ryecoaaron - what do you think about adding that? An alternative would be to change that directory in fs2ram.conf to keep_file_structure. However since it's a cache folder there could be a lot of cruft in there.

      Thanks for finding out why people were losing that file :) I rarely reboot my system so I never noticed it being gone. I agree there is too much in that directory to keep the files. While I don't really like adding things to rc.local, I can't think of any other way to do it. I would also prefer that adding it to rc.local is done manually instead of the plugin adding it. I will try to find a better place to put it.

      I don't think you should fix it that way or in the plugin. To me this is a bug in OMV. I messaged you before about the paths and as stated it says:
      /var/cache is intended for cached data from applications. Such data is locally generated as a result of time-consuming I/O or calculation. The application must be able to regenerate or restore the data. Unlike /var/spool, the cached files can be deleted without data loss. The data must remain valid between invocations of the application and rebooting the system.

      Where "The application must be able to regenerate or restore the data." is the key. OMV should be able to detect the missing file(s) and regenerate it. Put it on the bugtracker.
    • Good point :)

      Filed bug 1283. To fix I created: /etc/apt/apt.conf.d/06localpackages

      with the following content in it:

      APT::Update::Pre-Invoke { "touch /var/cache/openmediavault/archives/Packages"; };
      omv 4.1.14 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!

      The post was edited 2 times, last by ryecoaaron ().

    • Thank you for creating the plugin, aaron!

      Anybody tested it with a BananaPi and BanaNAS so far?
      I think I might give it a try when I'm doing a backup of my BanaNAS SD Card and switch to a full size SD Card instead of using a microSD + Adaptor.
      OMV 1.19 (OMV-Extras.org 1.34) | BanaNAS | 16GB microSD (using FlashMemory plugin) | 1TB SATA WD Blue Mobile
    • Not the BananaPi but I am using it on RPi2 and Odroid-C1. They should be the same.
      omv 4.1.14 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      Not the BananaPi but I am using it on RPi2 and Odroid-C1. They should be the same.

      Fine, thank you!

      So to wrap it up I have to:
      [- optional: Do a cat /sys/fs/ext4/sdX1/session_write_kbytes or a awk '/sd/ {print $3"\t"$10 / 2 / 1024}' /proc/diskstats to see how many data has been written since last reboot.]
      - Enable the Extras-Testing repo as source (if not already enabled)
      - Install the openmediavault-flashmemory plugin
      - Apply changes to the /etc/fstab (if necessary)
      - Enable the plugin (and get logged out afterwards)
      - Reboot and pray
      [- optional: Wait a couple of days, repeat checking the written bytes again and compare to the former values]

      Sounds manageable ;)

      If I want to disable the plugin it goes the somewhat other way round then, I expect (re-edit fstab, disable plugin, reboot)?
      OMV 1.19 (OMV-Extras.org 1.34) | BanaNAS | 16GB microSD (using FlashMemory plugin) | 1TB SATA WD Blue Mobile
    • I just installed the plugin and I had a problem, if you do not activate the plugin and we redémare the machine while plant!
      Indeed fs2ram starts but does not have a copy of the data ...

      And the fonction to log the write does not work on one of my system because of the filsystem type (ext2 because of my uboot does not work with ext4).

      Also I think it would better separate the /var/cache, which contains apt folder that can be very big if you do not think to do the update and do a "apt-get clean" from time to time .

      Edit: I had not seen the "keep_folder_structure" so I guess for the / var / cache is not serious.
      So it is possible to limit the size of tmpfs 5 or 10MB? Because with 512MB of ram it can be a bit limited.
      NSA325v2 on Debian Wheezy + Openmediavault 1.16, 1.6Ghz, 512MB, 1TB + 500GB.
      D2-Network on Debian Wheezy + Openmediavault 1.16, 1.2Ghz, 256MB, 500GB.
      And a custom made NAS/Server on Debian Wheezy + Openmediavault 1.16, Core2Duo 2.4Ghz, 4GB, 4*2TB Raid5 + 4*3TB Raid5.

      The post was edited 1 time, last by Val532 ().

    • Val532 wrote:

      I just installed the plugin and I had a problem, if you do not activate the plugin and we redémare the machine while plant! Indeed fs2ram starts but does not have a copy of the data ...

      What does redemare mean? I guess I didn't test installing the plugin but not using it. When installing using the plugin and fs2ram is not installed yet, fs2ram creates a default config. Maybe I will just get rid of the enable button. Because there are no settings, people can just uninstall the plugin instead of disabling.

      Val532 wrote:

      And the fonction to log the write does not work on one of my system because of the filsystem type (ext2 because of my uboot does not work with ext4).
      OMV's default installer and the RPi/Odroid images use ext4. It is a simple display that I just intended to support standard installs.

      Val532 wrote:

      Also I think it would better separate the /var/cache, which contains apt folder that can be very big if you do not think to do the update and do a "apt-get clean" from time to time .Edit: I had not seen the "keep_folder_structure" so I guess for the / var / cache is not serious.So it is possible to limit the size of tmpfs 5 or 10MB? Because with 512MB of ram it can be a bit limited.

      The wiki does say 1 GB is required.
      omv 4.1.14 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      Val532 wrote:

      I just installed the plugin and I had a problem, if you do not activate the plugin and we redémare the machine while plant! Indeed fs2ram starts but does not have a copy of the data ...

      What does redemare mean? I guess I didn't test installing the plugin but not using it. When installing using the plugin and fs2ram is not installed yet, fs2ram creates a default config. Maybe I will just get rid of the enable button. Because there are no settings, people can just uninstall the plugin instead of disabling.


      Sorry, I meant to say restart not "redémarer".
      And yes the enable button is probably not essential.

      ryecoaaron wrote:

      Val532 wrote:

      And the fonction to log the write does not work on one of my system because of the filsystem type (ext2 because of my uboot does not work with ext4).
      OMV's default installer and the RPi/Odroid images use ext4. It is a simple display that I just intended to support standard installs.


      If it does not a problem ok, but in my case i can not use any thing else than ext2 file system (because of one of my "test nas").

      ryecoaaron wrote:

      Val532 wrote:

      Also I think it would better separate the /var/cache, which contains apt folder that can be very big if you do not think to do the update and do a "apt-get clean" from time to time .Edit: I had not seen the "keep_folder_structure" so I guess for the / var / cache is not serious.So it is possible to limit the size of tmpfs 5 or 10MB? Because with 512MB of ram it can be a bit limited.

      The wiki does say 1 GB is required.


      Yes i know that, but it's like RPi/Odroid, i can not choose the amount of RAM on this system ^^.

      But I want to say that these are just simple questions to optimize a specific case (mine ^^).
      NSA325v2 on Debian Wheezy + Openmediavault 1.16, 1.6Ghz, 512MB, 1TB + 500GB.
      D2-Network on Debian Wheezy + Openmediavault 1.16, 1.2Ghz, 256MB, 500GB.
      And a custom made NAS/Server on Debian Wheezy + Openmediavault 1.16, Core2Duo 2.4Ghz, 4GB, 4*2TB Raid5 + 4*3TB Raid5.
    • Well, I will have to see if I can get the same data from ext2 that I get from
      ext4.

      I don't know if you can limit tmpfs. I do think 5 to 10 mb is not enough. A php security update or kernel update will easily fill that.

      If you find better optimizations for the config, I will add them to the plugin if they benefit all systems.
      omv 4.1.14 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • OpenElec barely generates any writes to your usb stick. No wonder it lives that long.

      Greetings
      David
      "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"

      Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.


      Upload Logfile via WebGUI/CLI
      #openmediavault on freenode IRC | German & English | GMT+1
      Absolutely no Support via PM!

      I host parts of the omv-extras.org Repository, the OpenMediaVault Live Demo and the pre-built PXE Images. If you want you can take part and help covering the costs by having a look at my profile page.
    • No, I don't agree. Every cached metadata file is in there. Thumbs, actors, dbs... etc. Zillions of small files.I know for sure, because time ago xbmc was doing strange things and getting the same van helsing pic for every media. I had to manually browse all the cache dirs and delete the file. If I leaved just one copy, it was using it again.But if anyone knows a procedure for getting the SD writes I would be glad to post the results.