error during OMV 6 version update: "dpkg: cannot write to log file '/var/log/dpkg.log': No space left on device"

  • Searching the forum didn't give hits, maybe its a new error.


    OMV6.0.6 reported a "new version available" so I started "Upgrade system" and get many error messages:


    Processing triggers for openmediavault (6.0.7-1) ...

    dpkg: cannot write to log file '/var/log/dpkg.log': No space left on device


    root cause is filesystem folder2ram is 100% full, how would I fix this?

    I thought log files would be cleaned up automatically


    Currently the docs for v6  are not available (error: SORRY This page does not exist yet.)

    Searching for folder2ram on Wiki gave no hits either.

    pi@nas64-6:~ $ df

    Filesystem 1K-blocks Used Available Use% Mounted on

    /dev/root 30406220 3273672 25867424 12% /

    devtmpfs 1777356 0 1777356 0% /dev

    tmpfs 1943052 0 1943052 0% /dev/shm

    tmpfs 777224 89300 687924 12% /run

    tmpfs 5120 4 5116 1% /run/lock

    tmpfs 1943052 56 1942996 1% /tmp

    /dev/mmcblk0p1 258095 30393 227702 12% /boot

    folder2ram 1943052 1943052 0 100% /var/log

    pi@nas64-6:/var/log $ ls -l

    total 1106804

    -rw-r--r-- 1 root root 208 Jan 2 14:24 alternatives.log

    -rw-r--r-- 1 root root 5262 Dec 31 01:05 alternatives.log.1

    -rw-r--r-- 1 root root 857 Nov 28 23:23 alternatives.log.2.gz

    drwxr-xr-x 2 root root 180 Jan 4 14:41 apt

    -rw-r----- 1 root adm 28672 Jan 4 15:09 auth.log

    -rw-r----- 1 root adm 91748 Dec 31 01:05 auth.log.1

    -rw-r----- 1 root adm 7926 Dec 18 19:28 auth.log.2.gz

    -rw-r--r-- 1 root root 0 Oct 30 14:43 bootstrap.log

    -rw-rw---- 1 root utmp 0 Jan 2 13:37 btmp

    -rw-rw---- 1 root utmp 800 Dec 30 21:23 btmp.1

    drwxr-x--- 2 _chrony _chrony 40 Nov 28 22:37 chrony

    drwxr-xr-x 2 clamav clamav 80 Nov 28 23:05 clamav

    drwxr-xr-x 2 root root 100 Jan 3 17:37 cron-apt

    -rw-r----- 1 root adm 348160 Jan 4 15:08 daemon.log

    -rw-r----- 1 root adm 933158 Jan 2 13:38 daemon.log.1

    -rw-r----- 1 root adm 75530 Dec 19 16:13 daemon.log.2.gz

    -rw-r----- 1 root adm 1531 Jan 2 14:24 debug

    -rw-r----- 1 root adm 10826 Dec 31 01:05 debug.1

    -rw-r----- 1 root adm 2370 Dec 18 19:28 debug.2.gz

    -rw-r--r-- 1 root root 0 Jan 4 14:42 dpkg.log

    -rw-r--r-- 1 root root 27157 Dec 30 20:28 dpkg.log.1

    -rw-r--r-- 1 root root 12877 Nov 28 23:10 dpkg.log.2.gz

    -rw-r--r-- 1 root root 32064 Dec 18 16:20 faillog

    -rw-r--r-- 1 root root 484 Nov 28 22:36 fontconfig.log

    drwxr-sr-x+ 4 root systemd-journal 80 Nov 28 22:29 journal

    -rw-r----- 1 root adm 40960 Jan 4 11:15 kern.log

    -rw-r----- 1 root adm 278450 Dec 31 01:05 kern.log.1

    -rw-r----- 1 root adm 56013 Dec 18 19:28 kern.log.2.gz

    -rw-rw-r-- 1 root utmp 296592 Jan 4 14:42 lastlog

    -rw-r----- 1 root adm 1864 Jan 4 14:17 mail.info

    -rw-r----- 1 root adm 481 Dec 31 00:17 mail.info.1

    -rw-r----- 1 root adm 315 Dec 18 19:17 mail.info.2.gz

    -rw-r----- 1 root adm 1864 Jan 4 14:17 mail.log

    -rw-r----- 1 root adm 481 Dec 31 00:17 mail.log.1

    -rw-r----- 1 root adm 315 Dec 18 19:17 mail.log.2.gz

    -rw-r----- 1 root adm 535744512 Jan 4 15:09 messages

    -rw-r----- 1 root adm 20709743 Dec 31 01:05 messages.1

    -rw-r----- 1 root adm 8318561 Dec 18 19:28 messages.2.gz

    drwxr-xr-x 2 minidlna minidlna 80 Dec 19 16:13 minidlna

    -rw-r----- 1 root adm 353 Nov 28 22:40 monit.log

    drwxr-xr-x 2 root adm 280 Jan 4 00:00 nginx

    drwxr-xr-x 2 root root 40 Nov 26 13:15 openmediavault

    -rw-r--r-- 1 root root 357 Dec 22 14:52 phoronix-test-suite-benchmark.log

    -rw------- 1 root root 337 Jan 2 14:24 php7.4-fpm.log

    -rw------- 1 root root 2023 Dec 31 01:05 php7.4-fpm.log.1

    -rw------- 1 root root 407 Dec 18 19:28 php7.4-fpm.log.2.gz

    drwx------ 2 root root 40 Oct 30 14:43 private

    drwxr-xr-x 2 root root 100 Dec 19 16:13 proftpd

    drwxr-xr-x 3 root root 60 Oct 30 14:30 runit

    drwxr-xr-x 2 root root 100 Dec 31 00:00 salt

    drwxr-x--- 3 root adm 360 Jan 4 12:33 samba

    -rw-r----- 1 root adm 536096768 Jan 4 15:09 syslog

    -rw-r----- 1 root adm 21664980 Jan 2 13:38 syslog.1

    -rw-r----- 1 root adm 8432049 Dec 19 16:13 syslog.2.gz

    -rw-r--r-- 1 root root 0 Nov 28 22:39 tallylog

    -rw-r----- 1 root adm 8192 Jan 4 08:17 user.log

    -rw-r----- 1 root adm 3916 Dec 31 01:05 user.log.1

    -rw-r----- 1 root adm 381 Dec 18 19:28 user.log.2.gz

    drwxr-xr-x 2 root root 40 Jan 3 2020 watchdog

    -rw-rw-r-- 1 root utmp 38000 Jan 4 14:42 wtmp

    pi@nas64-6:/var/log $

    omv 6.9.6-2 (Shaitan) on RPi CM4/4GB with 64bit Kernel 6.1.21-v8+

    2x 6TB 3.5'' HDDs (CMR) formatted with ext4 via 2port PCIe SATA card with ASM1061R chipset providing hardware supported RAID1


    omv 6.9.3-1 (Shaitan) on RPi4/4GB with 32bit Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 3TB 3.5'' HDDs (CMR) formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

    • Offizieller Beitrag

    Not really an OMV issue. As with any filesystem that is filled, you need to delete old files. I would delete and .1 or .gz log files in /var/log.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.6 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • right that cured the issue, I wonder if log files can be configured to limit their size.

    The real root cause was SMB file access auditing was enabled and I performed massive file moves over the last 2 days.

    As the result log files grew to a size never seen before and exhausted all available space on file system folder2ram

    omv 6.9.6-2 (Shaitan) on RPi CM4/4GB with 64bit Kernel 6.1.21-v8+

    2x 6TB 3.5'' HDDs (CMR) formatted with ext4 via 2port PCIe SATA card with ASM1061R chipset providing hardware supported RAID1


    omv 6.9.3-1 (Shaitan) on RPi4/4GB with 32bit Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 3TB 3.5'' HDDs (CMR) formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

  • mi-hol

    Hat das Label gelöst hinzugefügt.
    • Offizieller Beitrag

    I wonder if log files can be configured to limit their size.

    That is what logrotate does already. It rolls off logs on a time interval and/or max size to a gzip'd copy. It also deletes old copies after a specified number of archives. Look in /etc/logrotate.d/.


    The real root cause was SMB file access auditing was enabled and I performed massive file moves over the last 2 days.

    This will cause problems on most things and is always dangerous because of how verbose smb auditing is. There is almost no way to prevent this but it shouldn't be left on very long especially on tiny systems.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.6 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • Look in /etc/logrotate.d/.

    The config for syslog seems in /etc/logrotate.d/rsyslog.

    But there are only basic settings unrelated to size.

    Would you happen to know if adding a size 100M and change frequency from weekly to daily would prevent this issue?


    pi@nas64-6:/etc/logrotate.d $ cat rsyslog

    /var/log/syslog

    /var/log/mail.info

    /var/log/mail.warn

    /var/log/mail.err

    /var/log/mail.log

    /var/log/daemon.log

    /var/log/kern.log

    /var/log/auth.log

    /var/log/user.log

    /var/log/lpr.log

    /var/log/cron.log

    /var/log/debug

    /var/log/messages

    {

    rotate 4

    weekly

    missingok

    notifempty

    compress

    delaycompress

    sharedscripts

    postrotate

    /usr/lib/rsyslog/rsyslog-rotate

    endscript

    }

    pi@nas64-6:

    omv 6.9.6-2 (Shaitan) on RPi CM4/4GB with 64bit Kernel 6.1.21-v8+

    2x 6TB 3.5'' HDDs (CMR) formatted with ext4 via 2port PCIe SATA card with ASM1061R chipset providing hardware supported RAID1


    omv 6.9.3-1 (Shaitan) on RPi4/4GB with 32bit Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 3TB 3.5'' HDDs (CMR) formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

    • Offizieller Beitrag

    But there are only basic settings unrelated to size.

    Size is an option and not mandatory.

    Would you happen to know if adding a size 100M and change frequency from weekly to daily would prevent this issue?

    In your case, adding daily wouldn't change anything. Adding size would probably help but may roll off the logs too soon if you enabled auditing. If you are going to do a lot of logging, you shouldn't be using tmpfs for storage.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.6 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • If you are going to do a lot of logging, you shouldn't be using tmpfs for storage.

    Well, it wasn't intentional. I'd see it as a "side effect" of the install script or flash media plugin in combination with debugging a file share access issue and forgetting to reset SMB auditing after fixing this issue.
    It resulted in Web GUI access issues, hence I wonder if adding a size limit would "harden" OMV against this type of user error (that happens UNintentional BUT occasionally).

    What is your view?

    omv 6.9.6-2 (Shaitan) on RPi CM4/4GB with 64bit Kernel 6.1.21-v8+

    2x 6TB 3.5'' HDDs (CMR) formatted with ext4 via 2port PCIe SATA card with ASM1061R chipset providing hardware supported RAID1


    omv 6.9.3-1 (Shaitan) on RPi4/4GB with 32bit Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 3TB 3.5'' HDDs (CMR) formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

    • Offizieller Beitrag

    Well, it wasn't intentional. I'd see it as a "side effect" of the install script or flash media plugin in combination with debugging a file share access issue and forgetting to reset SMB auditing after fixing this issue.
    It resulted in Web GUI access issues, hence I wonder if adding a size limit would "harden" OMV against this type of user error (that happens UNintentional BUT occasionally).

    What is your view?

    The install script has a flag to not install the flash memory plugin.

    Yes, filling the root filesystem when /var/log in it will result in web interface issues.

    Adding size *might* harden your system against what *you* are doing. This isn't hardening as much as just throwing away logs which you purposely enabled. OMV does not maintain the syslog logrotate config. This is default for Debian. I do not think this should be changed. If you are going to enable high level smb logging known for using a lot of disk space, you need to know what you are doing. Imposing a low logrotate size to prevent your issue could potentially cause other users to lose important info. Also, the log viewer in the OMV web interface only shows logs from the current log file. If you rotated more often, people would have to use the command line to view older logs. If you want to change your system, go for it. Nothing in OMV will change it back.


    Also, many things can use tmpfs. If your system only had 95M of free space in tmpfs left, rotating logs at 100M would not help.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.6 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I suffer from the same problem that starts with an email

     cat: write error: No space left on device


    After investigation it turns out that the folder2ram /var/log hits 100% . After a couple of reboots the system boots without the folder2ram /var/log what allows me to do further checks.

    I started to make an inventory of /var/log which looks as follows:

    root@rpi:/var/log# du -s .

    736588 .

    root@rpi:/var/log# du

    4 ./chrony

    4 ./openmediavault

    8 ./proftpd

    4 ./runit/ssh

    8 ./runit

    12 ./minidlna

    16 ./cron-apt

    8 ./salt

    64 ./apt

    4 ./private

    4 ./samba/cores/smbd

    4 ./samba/cores/nmbd

    12 ./samba/cores

    8796 ./samba

    32 ./clamav

    11684 ./nginx

    524480 ./journal/428e68fb775e41d5904f1a673b4211e2

    524484 ./journal

    4 ./watchdog

    736604 . 


    1) The biggest chunck (10x number 2) is the journal directory . I have other RPI's but none has this journal directory , so where does it come from and what is the purpose?

    2) the rest together is still way over 100MB, too big to fit in to RAMdisk.
    So are there any knobs to empty the folder2ram disk devices faster or change the high watermark or something.


    Thank you , Yann

  • journal is the systemd log. The size of it is configured in /etc/systemd/journald.conf and it can be cleared by

    Code
    journalctl --vacuum-time=10d

    10d giving the amount of log messages to keep.


    But before deleting it, normally the log rotation is configured adequatly, so check the messages before deleting them.

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!