var/log is running out of space

    • OMV 4.x

    • var/log is running out of space

      Hello,

      tried to get an answer already on an existing thread. Unfortunately the thread was already tagged as solved. So it seems that no one was reading it.Hope a new thread won't be seen as spam from my side.

      On my OMV 4.1.26 installation var/log is running out of space after a while. Available space is only 50M. When the space is out I receive error messages when I want to do a USB back up.

      So here is what I found out so far after some debugging trials via shell:

      Source Code

      1. root@Netzwerkspeicher:~# df -h
      2. Filesystem Size Used Avail Use% Mounted on
      3. udev 931M 0 931M 0% /dev
      4. tmpfs 200M 19M 182M 10% /run
      5. /dev/mmcblk1p2 7.4G 1.6G 5.2G 24% /
      6. tmpfs 999M 8.0K 999M 1% /dev/shm
      7. tmpfs 5.0M 8.0K 5.0M 1% /run/lock
      8. tmpfs 999M 0 999M 0% /sys/fs/cgroup
      9. tmpfs 999M 0 999M 0% /tmp
      10. /dev/mmcblk1p1 58M 23M 33M 42% /boot
      11. /dev/sda1 2.8T 748G 2.0T 27% /srv/dev-disk-by-label-Backupintern
      12. armbian-ramlog 50M 50M 0M 100% /var/log
      13. folder2ram 999M 24M 976M 3% /var/log.hdd
      14. folder2ram 999M 0 999M 0% /var/tmp
      15. folder2ram 999M 260K 999M 1% /var/lib/openmediavault/rrd
      16. folder2ram 999M 1.4M 998M 1% /var/spool
      17. folder2ram 999M 15M 984M 2% /var/lib/rrdcached
      18. folder2ram 999M 8.0K 999M 1% /var/lib/monit
      19. folder2ram 999M 4.0K 999M 1% /var/lib/php
      20. folder2ram 999M 106M 893M 11% /var/lib/netatalk/CNID
      21. folder2ram 999M 420K 999M 1% /var/cache/samba
      22. tmpfs 200M 0 200M 0% /run/user/0
      Display All

      Source Code

      1. root@Netzwerkspeicher:~# folder2ram -status
      3. /var/log is mounted
      4. /var/tmp is mounted
      5. /var/lib/openmediavault/rrd is mounted
      6. /var/spool is mounted
      7. /var/lib/rrdcached is mounted
      8. /var/lib/monit is mounted
      9. /var/lib/php is mounted
      10. /var/lib/netatalk/CNID is mounted
      11. /var/cache/samba is mounted
      Display All

      Source Code

      1. root@Netzwerkspeicher:~# armbianmonitor -u
      2. System diagnosis information will now be uploaded to http://ix.io/1ZVr
      3. Please post the URL in the forum where you've been asked for.
      Disabling the armbian-ram-logging service in /etc/default/armbian-ramlog is changing it to folder2ram. Will there be any issues in the future disabling the log service?
      Is there any other workaround for this topic how to switch from ram2log to folder2ram vor not running out of space for var/log?

      Solved the topic temporarily with increasing the space from 50M to 100M. But won't solve the cause of it.

      Would be great If someone can help me on that! :thumbsup:
      :!: Odroid HC2 - OMV 4 - 3TB Hard Drive :!:

    • How about taking a look inside the logs to see what is filling them? And do something about that.

      If you can figure out what is writing to the log files, and fix that, I suspect that you will find that Bob is your uncle. Or at least a very close friend.
      OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

    • Adoby wrote:

      How about taking a look inside the logs to see what is filling them? And do something about that.

      If you can figure out what is writing to the log files, and fix that, I suspect that you will find that Bob is your uncle. Or at least a very close friend.
      Done!

      Debug has 24MB and has a lot of entries with the following:

      Source Code

      1. Oct 24 22:52:00 Netzwerkspeicher kernel: [100557.897144] database(12390): WRITE block 3306944 on mmcblk1p2 (32 sectors)


      syslog.1 has 22MB and showing also the same message:

      Source Code

      1. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.831540] database(10904): WRITE block 3587456 on mmcblk1p2 (64 sectors)
      2. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.831554] database(10904): WRITE block 4111744 on mmcblk1p2 (64 sectors)
      3. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.831652] database(10904): WRITE block 3587520 on mmcblk1p2 (32 sectors)
      4. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.831660] database(10904): WRITE block 4111808 on mmcblk1p2 (32 sectors)
      5. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.834372] database(10904): WRITE block 128 on mmcblk1p2 (8 sectors)
      6. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.835425] database(10904): WRITE block 2806584 on mmcblk1p2 (8 sectors)
      7. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.835450] database(10904): WRITE block 2806984 on mmcblk1p2 (8 sectors)
      8. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.838861] database(10904): WRITE block 3587552 on mmcblk1p2 (32 sectors)
      9. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.838875] database(10904): WRITE block 4111840 on mmcblk1p2 (32 sectors)
      10. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.838940] database(10904): WRITE block 3587584 on mmcblk1p2 (32 sectors)
      11. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.838947] database(10904): WRITE block 4111872 on mmcblk1p2 (32 sectors)
      12. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.839052] database(10904): WRITE block 3587616 on mmcblk1p2 (32 sectors)
      13. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.839059] database(10904): WRITE block 4111904 on mmcblk1p2 (32 sectors)
      14. Oct 24 22:31:00 Netzwerkspeicher kernel: [99297.842131] database(10904): WRITE block 128 on mmcblk1p2 (8 sectors)
      15. Oct 24 22:31:01 Netzwerkspeicher kernel: [99299.511456] cron(11797): dirtied inode 871373 (loginuid) on proc
      16. Oct 24 22:31:01 Netzwerkspeicher kernel: [99299.557123] systemd-journal(322): dirtied inode 871370 (exe) on proc
      17. Oct 24 22:31:11 Netzwerkspeicher kernel: [99309.471521] runc(11819): dirtied inode 790156 (runc-process134886885) on tmpfs
      18. Oct 24 22:31:11 Netzwerkspeicher kernel: [99309.494179] exe(11825): dirtied inode 871395 (exe) on proc
      19. Oct 24 22:31:11 Netzwerkspeicher kernel: [99309.572233] containerd-shim(2835): dirtied inode 870026 (c423579de76e2375bd964bab821ffc0001cb86cf3824f2352029acf9e70d92d5.pid) on tmpfs
      20. Oct 24 22:31:11 Netzwerkspeicher kernel: [99309.604641] containerd-shim(2835): dirtied inode 871389 (c423579de76e2375bd964bab821ffc0001cb86cf3824f2352029acf9e70d92d5-stdout) on tmpfs
      21. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.735691] dockerd(2050): dirtied inode 167520 (.tmp-config.v2.json144611812) on mmcblk1p2
      22. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.737024] dockerd(2050): dirtied inode 167521 (.tmp-hostconfig.json217170419) on mmcblk1p2
      23. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.737953] dockerd(2050): WRITE block 3587648 on mmcblk1p2 (64 sectors)
      24. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.737979] dockerd(2050): WRITE block 4111936 on mmcblk1p2 (64 sectors)
      25. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.738036] dockerd(2050): WRITE block 3587712 on mmcblk1p2 (32 sectors)
      26. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.738047] dockerd(2050): WRITE block 4112000 on mmcblk1p2 (32 sectors)
      27. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.738232] dockerd(2050): WRITE block 3587744 on mmcblk1p2 (32 sectors)
      28. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.738244] dockerd(2050): WRITE block 4112032 on mmcblk1p2 (32 sectors)
      29. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.747690] dockerd(2050): WRITE block 128 on mmcblk1p2 (8 sectors)
      30. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.749592] kworker/u16:5(28851): WRITE block 1540320 on mmcblk1p2 (8 sectors)
      31. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.750847] dockerd(2050): WRITE block 3587776 on mmcblk1p2 (64 sectors)
      32. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.750867] dockerd(2050): WRITE block 4112064 on mmcblk1p2 (64 sectors)
      33. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.751057] dockerd(2050): WRITE block 3587872 on mmcblk1p2 (32 sectors)
      34. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.751071] dockerd(2050): WRITE block 4112160 on mmcblk1p2 (32 sectors)
      35. Oct 24 22:31:12 Netzwerkspeicher kernel: [99309.759792] dockerd(2050): WRITE block 128 on mmcblk1p2 (8 sectors)
      36. Oct 24 22:31:14 Netzwerkspeicher kernel: [99311.827076] kworker/u16:5(28851): WRITE block 1540328 on mmcblk1p2 (8 sectors)
      37. Oct 24 22:31:34 Netzwerkspeicher kernel: [99332.196448] lighttpd(4162): dirtied inode 14080 (access.log) on mmcblk1p2
      38. Oct 24 22:31:34 Netzwerkspeicher kernel: [99332.325504] lighttpd(4162): dirtied inode 17127 (lighttpd-upload-4Clwq4) on mmcblk1p2
      39. Oct 24 22:31:42 Netzwerkspeicher kernel: [99339.981942] runc(11920): dirtied inode 790163 (runc-process730293760) on tmpfs
      40. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.003959] exe(11926): dirtied inode 873535 (exe) on proc
      41. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.088080] containerd-shim(12892): dirtied inode 870071 (90b88c7ab746dd9745b4bc920f76c01c03d694165d7776d47412ef6cc14f2da4.pid) on tmpfs
      42. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.122102] containerd-shim(12892): dirtied inode 810960 (90b88c7ab746dd9745b4bc920f76c01c03d694165d7776d47412ef6cc14f2da4-stdout) on tmpfs
      43. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.246560] dockerd(4314): dirtied inode 167522 (.tmp-config.v2.json891656886) on mmcblk1p2
      44. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.247965] dockerd(4314): dirtied inode 167523 (.tmp-hostconfig.json947899037) on mmcblk1p2
      45. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.249184] dockerd(4314): WRITE block 3587904 on mmcblk1p2 (64 sectors)
      46. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.249225] dockerd(4314): WRITE block 4112192 on mmcblk1p2 (64 sectors)
      47. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.249321] dockerd(4314): WRITE block 3588032 on mmcblk1p2 (32 sectors)
      48. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.249337] dockerd(4314): WRITE block 4112320 on mmcblk1p2 (32 sectors)
      49. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.249534] dockerd(4314): WRITE block 3588064 on mmcblk1p2 (32 sectors)
      50. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.249549] dockerd(4314): WRITE block 4112352 on mmcblk1p2 (32 sectors)
      51. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.256960] dockerd(4314): WRITE block 128 on mmcblk1p2 (8 sectors)
      52. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.258970] kworker/u16:3(3148): WRITE block 1540336 on mmcblk1p2 (8 sectors)
      53. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.260139] dockerd(4314): WRITE block 3588096 on mmcblk1p2 (64 sectors)
      54. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.260154] dockerd(4314): WRITE block 4112384 on mmcblk1p2 (64 sectors)
      55. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.260298] dockerd(4314): WRITE block 3588160 on mmcblk1p2 (32 sectors)
      56. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.260306] dockerd(4314): WRITE block 4112448 on mmcblk1p2 (32 sectors)
      57. Oct 24 22:31:42 Netzwerkspeicher kernel: [99340.262838] dockerd(4314): WRITE block 128 on mmcblk1p2 (8 sectors)
      Display All

      Unfortunately also uncle Bob can't help me on that.

      Why are there that many entries in the log files?
      :!: Odroid HC2 - OMV 4 - 3TB Hard Drive :!:

      The post was edited 1 time, last by ChrisBuzz ().

    • I installed the image from sourceforge.net
      And the flash memory plugin was already installed and my understanding was that I don t have to change anything on that?!
      Isn't that correct? DO I have to change anything via fstab?

      fstab has the following input:

      Source Code

      1. UUID=d0da7bbe-e3af-4588-8715-aa5c4478eb88 / btrfs defaults,noatime,nodiratime,commit=600,compress=lzo 0 1
      2. UUID=5bab0a55-56f1-4443-8cac-297e1181425c /boot ext4 defaults,commit=600,errors=remount-ro 0 2
      3. tmpfs /tmp tmpfs defaults,nosuid 0 0
      4. # >>> [openmediavault]
      5. /dev/disk/by-label/Backupintern /srv/dev-disk-by-label-Backupintern xfs defaults,nofail,noexec,usrquota,grpquota 0 2
      6. /srv/dev-disk-by-label-Backupintern/Backupintern/Eigene\040Dateien/Christoph/30\040Back\040Ups/20\040ioBroker /export/BackUpioBroker none bind,nofail 0 0
      7. # <<< [openmediavault]
      :!: Odroid HC2 - OMV 4 - 3TB Hard Drive :!:

    • A normal install of the image you used, will not cause any problems like the ones you mention. I have several HC2 running fine without filling any log files. I don't use any extra config to rotate the logs. It isn't needed.

      Something is wrong. You need to figure out what.

      One of the most obvious and simplest ways is a fresh install, from scratch, on a new card. And check after you configure or add anything if the logs start growing again. Then you know what is causing the problem with the growing log files.

      I used to have a lot of strange problems with my HC2s. I figured out that if I used or activated ANY of the physical disk properties, then everything went haywire. So I tried installing and not touching the physical disk properties. And now my HC2s are rock steady. It took me at least 5 full reinstall from scratch before I figured it out. But then I am pretty slow.
      OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

    • ajaja wrote:

      gderf wrote:

      Find out what is flooding your logs and fix that.
      It is not uncommon here that this is because PLEX DB is filling your boot drive. So, yes, find out what is bogarting your space.
      That is usually an issue with the Plex DB filling /var/lib/docker and is not related to Plex logs (that I have ever seen)

      It appears there is an issue with his sdcard and the constant logging of these issues is filling the drive.

      I would buy a new sdcard.
      Air Conditioners are a lot like PC's... They work great until you open Windows.

    • I am not using Plex. So seems that this can't be the reason for filling up the space in my syslog and debug log file.

      Still don't get it! After I opened both files yesterday no additional space was filled. So it seems that log rotate is working correctly now?!

      Can someone explain me how to check if there is an issue with logrotate?

      Found something in an other thread that a workaround is to use the following common in cron job:

      Source Code

      1. find /var/log/ -type f -mtime 90 -exec rm -rf {} \';

      If i put this command in shell I get the following and it seems not working right:

      Source Code

      1. root@Netzwerkspeicher:~# find /var/log/ -type f -mtime 90 -exec rm -rf {} \';
      2. find: missing argument to `-exec'
      3. Try 'find --help' for more information.
      Is it only possible to execute via cron?

      Hope that is not required to use a new SD card and start from scratch. But I agree with Adoby that I deactivated every physical disk properties and everything is working fine so far. ;)
      :!: Odroid HC2 - OMV 4 - 3TB Hard Drive :!:

    • KM0201 wrote:

      It doesn't matter if log rotate is working or you have Plex or not (I don't think anyone suggested you did)...

      Those log entries suggest there's an issue with your sdcard...

      cabrio_leo wrote:

      There is a lot of fake storage media on the way. New SD cards, USB sticks etc. should always be checked with Hwtest2, Etcher or similar tools before using.
      Thanks, guys!
      So since yesterday there are no more logs like there was the ones before. Really don't know what is different now. But hopefully that topic is solved and won't come back in future. :thumbup:
      I am using a brand new SanDisk Extreme 32GB ordered from Amazon. Is there really a chance to get a faked one? 8|
      :!: Odroid HC2 - OMV 4 - 3TB Hard Drive :!: