Symlink /var to another disk (non usb-thumb-drive) so that openmediavault-flashmemory plugin not necessary possible?

  • My OMV-System for some reason was crashing in the last days. But I cannot visit logs, since they are not synced to disk and when it crashes, nothing is synced.

    I was anyways thinking to symlink /var to a ssd drive, so that it is not on my OMV-System usb drive to make the USB-stick survive longer.

    Is there anything speaking against this procedure? As I understood the usb-flashmemory plugin is taking frequently writing processes and writes them to a tmpfs and then syncing them to disk after it accumulated a while. So when I move logs somewhere else, will there still be some processes like that write a lot to the OMV-System so that I still shouldnt disable the flashmemory plugin even after having the logs somewhere else?

    Also: Is there some manual how to revert what has been done in the beginning when setting up the flashmemory plugin? I cannot find the procedures anymore that were shown when setting up the flashmemory plugin (where it shows you also how to disable the swap etc)


    Greetings

    • Official Post

    As I understood the usb-flashmemory plugin is taking frequently writing processes and writes them to a tmpfs and then syncing them to disk after it accumulated a while.

    The flashmemory plugin is just mounting various mountpoints in /var to tmpfs. It only syncs those back to the location on disk on shutdown unless you schedule it to be more often.

    My OMV-System for some reason was crashing in the last days. But I cannot visit logs, since they are not synced to disk and when it crashes, nothing is synced.

    Run folder2ram -syncall every few hours from scheduled tasks.

    I was anyways thinking to symlink /var to a ssd drive, so that it is not on my OMV-System usb drive to make the USB-stick survive longer.

    I think this is avoiding the problem. If you have an ssd, why not put the entire OS on it? If you are killing usb drives, maybe you need to move the load causing all of the writes to another disk. The OS is not doing this if you have the flashmemory plugin installed. I current have a system running on a usb stick for four years.

    So when I move logs somewhere else, will there still be some processes like that write a lot to the OMV-System so that I still shouldnt disable the flashmemory plugin even after having the logs somewhere else?

    There is no reason to disable the flashmemory plugin. I use it even with nvme.


    Also: Is there some manual how to revert what has been done in the beginning when setting up the flashmemory plugin? I cannot find the procedures anymore that were shown when setting up the flashmemory plugin (where it shows you also how to disable the swap etc)

    Uninstalling the plugin will revert the changes. If you followed the optional instructions on how to disable swap, there is no need to re-enable unless your system is running out of memory. swap isn't really related to flashmemory plugin. Those instructions were just another way to reduce writes to the system disk.

    omv 6.2.0-1 Shaitan | 64 bit | 6.1 proxmox kernel

    plugins :: omvextrasorg 6.1.1 | kvm 6.2.8 | compose 6.6.1 | cputemp 6.1.3 | mergerfs 6.3.5 | zfs 6.0.12


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I cannot find the procedures anymore that were shown when setting up the flashmemory plugin (where it shows you also how to disable the swap etc)

    I'll only help with this one since I don't see the need for all the prior intention.


    You only need to know where to look:

    https://github.com/OpenMediaVa…mory/Settings.js#L63-L106

  • I think this is avoiding the problem. If you have an ssd, why not put the entire OS on it? If you are killing usb drives, maybe you need to move the load causing all of the writes to another disk. The OS is not doing this if you have the flashmemory plugin installed. I current have a system running on a usb stick for four years.


    I have SSDs attached for other things than the system. The system runs on a 32GB USB-Stick. Since I wanted a dedicated drive for the system and not just some partition on a SSD like it was once recommended to me.
    And the main reason for my idea of mounting the logs there is not primarily for the USB to not die, but to be able to see the logs even when OMV crashes, since they will be on disk.
    Syncing them via a cronjob is possible, but every hour would surely not be enough, and probably even when setting the cronjob to sync every minute , then it somehow defeats the purpose of what the flashmemory plugin does as I assume, while the last minute of logs that may have lead to the crash will still be gone, since they will not be synced.

    Thanks for the answer. I will see how it works mounting them on the SSD but keeping flashmemory plugin

    • Official Post

    even when OMV crashes

    OMV isn't crashing though. Linux is.

    Syncing them via a cronjob is possible, but every hour would surely not be enough, and probably even when setting the cronjob to sync every minute , then it somehow defeats the purpose of what the flashmemory plugin does as I assume, while the last minute of logs that may have lead to the crash will still be gone, since they will not be synced.

    I only recommended that because it seemed like it wasn't an instant crash and you could see logs leading up to the crash.


    If only the OS is on the usb stick and other things like docker are running from data disks/ssds and the flashmemory plugin is installed, the only reason the system should crash is a junk usb stick or running out or memory or hardware is unstable.

    omv 6.2.0-1 Shaitan | 64 bit | 6.1 proxmox kernel

    plugins :: omvextrasorg 6.1.1 | kvm 6.2.8 | compose 6.6.1 | cputemp 6.1.3 | mergerfs 6.3.5 | zfs 6.0.12


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • OMV isn't crashing though. Linux is.

    I only recommended that because it seemed like it wasn't an instant crash and you could see logs leading up to the crash.


    If only the OS is on the usb stick and other things like docker are running from data disks/ssds and the flashmemory plugin is installed, the only reason the system should crash is a junk usb stick or running out or memory or hardware is unstable.


    Thats why I want to see the logs to determine what would use up the RAM.


    Anyways, so if I will symlink /var to another drive, the flashmemory plugin will still create the folder2ram fs, is that correct? So in order to have them written directly, I would still have to disable the plugin ( may it even just be for the period until I found out what crashes my "linux"

    • Official Post

    Thats why I want to see the logs to determine what would use up the RAM.

    There are no logs showing individual process ram use. You will need to look at top periodically to see that. It may not be one process but many small processes as well.


    Anyways, so if I will symlink /var to another drive, the flashmemory plugin will still create the folder2ram fs, is that correct? So in order to have them written directly, I would still have to disable the plugin ( may it even just be for the period until I found out what crashes my "linux"

    If you make the symlink, I recommend removing the flashmemory plugin before that.


    What kind of system? What are you running on the system?

    omv 6.2.0-1 Shaitan | 64 bit | 6.1 proxmox kernel

    plugins :: omvextrasorg 6.1.1 | kvm 6.2.8 | compose 6.6.1 | cputemp 6.1.3 | mergerfs 6.3.5 | zfs 6.0.12


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • yes, but sometimes it is possible to see that a process was tried to be killed because it uses too much ram or other things,

    I'm using pretty many containers (around 30-40) so maybe one of them is doing something it shouldnt. I also check my usage with node_exporter and grafana, but shortly before the system crashed, there was still enough RAM... unfortunately, also there, there was the last 10 minutes missing.


    Im still pretty sure it was a RAM issue, since in the OMV CLI (via a screen attached to the server), it was lagging pretty much, wasnt able to type anything, but it still reacted sometimes on keystrokes.


    My solution to check what could be the cause now, is to periodically write the output of the top command of the top 5 memory consuming processes via cron to another server. With this I may be able to determine what could be the cause if it happens again

    • Official Post

    yes, but sometimes it is possible to see that a process was tried to be killed because it uses too much ram or other things,

    The OOMkiller doesn't always kill the process using the most ram especially when docker is involved.


    I'm using pretty many containers (around 30-40) so maybe one of them is doing something it shouldnt.

    Are any of them java-based?

    Im still pretty sure it was a RAM issue

    Then add swap back. This is what swap is supposed to protect you from. Just don't put it on cheap flash based storage.

    My solution to check what could be the cause now, is to periodically write the output of the top command of the top 5 memory consuming processes via cron to another server. With this I may be able to determine what could be the cause if it happens again

    I would do more than 5. No need to write to another server if the server reboots fine. Just don't write to tmpfs filesystem.

    omv 6.2.0-1 Shaitan | 64 bit | 6.1 proxmox kernel

    plugins :: omvextrasorg 6.1.1 | kvm 6.2.8 | compose 6.6.1 | cputemp 6.1.3 | mergerfs 6.3.5 | zfs 6.0.12


    omv-extras.org plugins source code and issue tracker - github


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!