Posts by no-clu

    I was having the runaway heartbeat some time ago with a version-one M4 and finally figured I had scheduled a remote Rsync to my backup server which coincided with Plex running it’s scheduled maintenance. I moved the schedule a couple hours later and the problem went away.

    I missed this reply. Thanks. This was on a fresh install, all I'd done was, install Docker and Portainer (source on dedicated spinning disk HDD) and created a shared folder on a second spinning HDD and setup SMB share for same folder.


    I've had armbian running for over 1 week with no problems. I even filled the SDCard 100% to see if it causes an problems. All runnign smoothly regarding system loads. Stage two is install OMV using install script for ARM devices, leave as is, i.e. totally fresh, excpet change admin password and wait for the system loads to go crazy, then investigate further as not an armbian issue.


    Thanks, will update again as I discover and require assistance :)


    Here is the output from Login, Uptime and df

    I'm still looking for a way to disable the pretty graphs in OMV while still enabling the collectd service.


    As I say, first thoughts are to comment out this conf file /etc/collectd/collectd.conf.d/rrdcached.conf


    Is this the way to go or anyone have soe better suggestions. Thanks.

    According to some similiar questions on serverfault.com a user suggested that the load average does not need to be linked to explicit load of the CPU but requests to get a CPU timeslot. Even File System waits seem to be able to cause such issues. To be honest, until now I did not know that either of such things can cause a high load average. Yours also seems extremely high.


    Greetings
    David

    Thanks davidh2k an interesting read and led me to a few useful places. I'd seen that post but dismissed it due to the OS. But on second look there were some linked posts which were very useful. Thanks


    My action plan is as follows.

    1. Wipe SDCard and reinstall armbian buster for nanopom4v2, leave this running a few days to see if load stays normal or problem persists.

    2. If step 1. passes. Install OMV using script for ARM installs.

    3. If/When system loads increase use some commands from these sources to try and gain further insight to cause

    --a. serverfault.com

    --b. brendangregg.com


    I'll post back after step 3. Although I'm happy to read any other thoughts in the meantime. Thanks.

    Okay... this morning I'm seeing the high load again here are is the output from uptime, top and syslog. I cannot run htop.

    2 days of running fine, similar timeframe to last one.


    Bash
    root@npim4v2:~# uptime
    07:00:47 up 2 days, 10:53, 1 user, load average: 52.26, 48.29, 38.36

    and 15 mins later load averages increasing...

    Bash
    root@npim4v2:~# uptime
    07:17:57 up 2 days, 11:10, 1 user, load average: 70.37, 66.41, 55.81


    Cannot see anything hogging with top.


    In the last few minutes of the log below something starts to go wrong, but i don't know what. About 05:49 things seem to start going pear shaped followed shortly by the nginx issues.

    --syslog is attached--

    syslog.txt

    It looked to be similiar to your logs, thus I suggested the look at it.

    Sorry, I totally appreciated the direction. While the error in syslog was the same, I was able to access the WebUI etc.. just with a reboot until the next runaway system load increase. The problems, to me, seem unconnected with different symptoms. Although certainly it was worth me looking into, which I did. What I should have said was, my conclusion after reading that thread was that I didn't think it applied to this problem.


    Do you by any chance see what is using the ressources in top or htop?

    When I had the high system load you can see the output of top in my first post above. I don't see anything.

    Load averages were all in the 600's (nearly 700's) and the CPU usage was mostly ideal. Biggest use was from top. Although interestingly my system is currently running okay and top only takes 1% of CPU compared to the 6% seen when load was abnormal.


    I could not run htop while system load was abnormal, SSH just sat with a blinking cursor until I hit Ctl+C. htop runs okay when the system is behaving.

    End goal is docker with grafana and influxdb, just so you know the goal. I've tested this setup and have had a working system. However I want to do it right and such that it persists.


    So I want to create a file in /etc/collectd/collectd.conf.d called "network.conf" with my settings.


    I also want to disable the pretty graphs in OMV WebUI while still using collectd, my understanding is that this is RRDCACHED at work. So my thought was to comment out everything in /etc/collectd/collectd.conf.d/rrdcached.conf. My goal here is 1. I don't need these as I'll be using influxdb. 2. Save system resources from doing something I don't need doing.


    Will modification persist in OMV or do I need to go about this in different way rather than directly changing files?

    There may be other files but let's start here :P

    Take a deeper look at this thread: 502 Bad Gateway nginx


    Greetings

    David

    Appreciate the feedback, thanks. I've now read through the suggested thread. I don't see how it directly applies tbh? I've tried some of the commands to see if packages were missing but seems fine. If I reboot everything is working fine, I can access the WebUI, change settings etc... But after what appears to be a random undefined time period the system starts getting the high system loads.


    Currently, after last reboot it's still running smoothly.

    Code
    root@npim4v2:~# uptime
    14:56:25 up 18:48, 1 user, load average: 0.00, 0.06, 0.07


    I expect however that within the next 12 hours he system load will start to increase and then just runaway. Just need to know where I should look to to try and find out the cause. Then obviously try to resolve.

    Hm... syslog has lots of nginx entries stopping and restarting?? And I cannot connect via WebGUI any more, just SSH. Reboot fixes until it all happens again.

    I've installed "Armbian buster with Linux 5.4.49-rockchip64" on my nanopim4v2, straight to SDCard. I ran apt update and upgrade. Then used the install script for ARM devices and everything went fine.

    I've been able to login to the webUI fine. I created 1 user. Then I created 2 shares. One temporary test share for SMB. One for Docker and any containers. I set the path for Docker. Then installed Docker and Portainer all using the WebUI. All working great.


    Now to problem. I simply leave the system on (doing nothing) over night and next morning a see the 'heartbeat' light flashing away. I wondered why so SSH in and ah... high system load, but low CPU. I don't know how to tackle this but I've tried to include some info below. Let me know what else you might need in order to suggest something. Thanks, any help appreciated.


    BTW this is a totally clean install. I've wiped the SDCard, followed ARM install guide and the result is the same, it's fine initially but after some times (hours) the system load ramps up and up.


    Code
    root@nanopim4v2:~# uptime
    18:56:38 up 2 days, 12:03, 1 user, load average: 689.18, 685.17, 674.69
    Code
    root@nanopim4v2:~# ps -e v
    PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
    1 ? Ss 1:24 86 1149 165654 10136 0.2 /sbin/init
    2 ? S 0:00 0 0 0 0 0.0 [kthreadd]
    3 ? I< 0:00 0 0 0 0 0.0 [rcu_gp]
    4 ? I< 0:00 0 0 0 0 0.0 [rcu_par_gp]
    8 ? I< 0:00 0 0 0 0 0.0 [mm_percpu_wq]
    9 ? S 0:01 0 0 0 0 0.0 [ksoftirqd/0]

    Thanks for the reply Adoby, I'll give your recommendation of variation 2 a trial run and will try to tackle some of the issues for me that remain with it.


    If anyone else wants to chime in please feel free.


    In addition I'll post back any findings as it may help others.

    I would recommend the second variant; Boot and rootfs on SD card, Docker and Docker data on HDD.


    The first variant wears out the SD card. The third variant is messy and doesn't provide improved performance. Except possibly if you use a SSD instead of a HDD.


    I use the second variant. I also have one OMV4 HC1 with just boot on the SD card and the rest on a 2TB hybrid SSD/HDD, it works fine, but is a mess to backup compared to just cloning a SD card. When I upgrade it I will put boot and rootfs back on the SD card, and docker on the hybrid SSD/HDD.

    Thanks Adoby I'm not fussed about performance improvements using HDD as rootfs. You point regarding a messy backup/restore solution, I agree totally. While I await any other kind replies I'd like to ask about your use of/or lack of collectd and the OMV monitoring. I tested using telegraf in Docker (instead of collectd) and had some stumbling blocks for some of the plugins which I didn't want to take further, but it's not totally ruled out.


    So, if I use the 2nd variant as you suggest, yes I can see how to work it for Docker, but how about collectd and monitoring (which writes lots to disk). My thoughts on this so far have been to use the Network plugin to send data to influxdb running as container, and possibly disabled rrdcache (not sure how yet - remove this plugin config maybe?). Not sure what to do about the collectd 'buffer' though (enter proper term), since collectd only sends the data to influxdb every x time interval after it's buffered some data up. I'm not sure where this gets stored, obviously I don't want it to be temporarily stored on the SDCard.

    The other thing is logs, I want these kept, but based on limited research so far It looks like I can copy to SDCard or somewhere else using cron. Not sure how I go about flushing the ram though so it doesn't get bogged down after a long time? Would this be the correct path or do I need to look into another option?

    Hi all,


    I've been experimenting with OMV for last few weeks in between working and other things. You can skip the setups I've boringly described below if you like but they provide some context if you want to read them. The question I have is as follows. I've got an arm device (nanopim4v2) with 4xsata ports (4xsata hat), I want NAS type stats monitoring (and to collect data from other network devices), I want to run Docker & Portainer with about 5 containers, I need 2x sata ports for data and thus I've 2x spare, I'm not fussed about size.

    • Shall I use an SDCard for OMV, Docker etc... as seems typical
    • Shall I use an SDCard for OMV and a partitioned (or not) data disk for Docker plus data
    • Shall I use an SDCard for boot only and have rootfs on HDD THIS IS MY PREFERED OPTION based on experiments noted below but should it be?

    My worries

    • Using collectd (comes with OMV) will write lots of data to SDCard unless I use rootfs on HDD
    • Using Docker containers like inlfuxdb will wirte lots of data to SDCard unless I use HDD as data disk for Docker containers
    • By using SDCard as boot only and rootfs on HDD cloning becomes more tedious to create backups and restore (I need to investigate this topic more also)
    • All experimental setups I tried below had some stability issues of varying degree although setup 3 seems worse, albeit the setup I've messed with the most so may I probably broke it myself


    First I want to select an installation to move forward with (the real question here), then I'll go about trying to fix the issues I have with that setup in separate threads as needed. Your help, experiences and comments are looked forward to.


    ---

    My experimenting to date.

    1st Setup - Keeping it simple as it's all new to me:

    - Install armbian Buster Server

    - run apt update && upgrade

    - Follow scripted installation for flash/SD storage using the guide Installing OMV5 on Armbian Buster found here.

    OUTCOMES: Got it working okay. I then installed Docker and Portainer using OMV WebUI and started to learn about writing to flash devices and the problems that come with this. I want to run Grafana (with influxdb and maybe telegraf/collectd) to monitor this device as well as some others on my network but would like Grafana and Influxdb running on this device via Docker. Problem now being I end up with lots of writes to SDCard. Or to ram (Flash Memory Plugin) but I want to keep this data.


    2nd Setup - Keep it simple ver2

    - Install armbian Buster Server

    - run apt update && upgrade

    - Use armbian-configure to install OMV via the user interface 3rd party application installer (I was hoping this didn't implement the Flash Memory Plugin as the 1st Setup did)

    OUTCOMES: Work a treat installing but still installed Flash Memory Plugin. Same comments as above, once I enable monitoring services and start using Docker I start saving lots of stuff to the SDCard. Or ram which I'll lose.


    3rd Setup - Adding some complexity

    - Install armbian Buster Server

    - run apt update && upgrade

    - run nand-sata-install to move rootfs to a HDD

    - clear SDCard of now old fs, keeping /boot only

    - Use armbian-configure to install OMV via the 3rd party application installer

    - Disable Flash Storage Plugin

    OUTCOMES: Works well-ish (see last sentnce), OMV WebUI seems no slower than when FS was on SDCard and it's plenty fast enough. There are still some folders in ram as part of the armbian install, but I believe these get flushed and copied to the rootfs periodically, I'm content with that. I've installed Docker and tried telegraf, influxdb and grafana with some success but not as I'd like. So I switched to using collectd which comes with OMV and sent the data to influxdb using network plugin, this works okay but the system is now not stable. The load spirals out of control in a very short period after an random amount of time, be it a few hours or a few days.