Web Frontend crashed

DJ600 · 20. Januar 2022

Hi everybody,

all of a sudden (and without actively doing something) the web frontend is no longer accessible. I'm running OMV6 on a Raspberry. I can access my samba shares, I do have access via SSH but the web frontend is not available. After reboot I get a bunch of emails (content listed below) telling me that something is wrong. However, I have no clue what this is all about and what to do now. Any help appreciated.

Andreas

Monitoring restart -- Does not exist php-fpm

Monitoring alert -- Execution failed php-fpm:

Code

Host:        raspberrypi
Date:        Thu, 20 Jan 2022 22:06:20
Service:     php-fpm
Event:       Execution failed
Description: failed to start (exit status 1) -- '/bin/systemctl start php7.4-fpm': Job for php7.4-fpm.service failed because a fatal signal was delivered to the control process.
See "systemctl status php7.4-fpm.service" and "journalctl -xe" for details.

Monitoring restart -- Does not exist nginx

Monitoring alert -- Execution failed nginx:

Code

Host:        raspberrypi
Date:        Thu, 20 Jan 2022 22:06:51
Service:     nginx
Event:       Execution failed
Description: failed to start (exit status 1) -- '/bin/systemctl start nginx': Job for nginx.service failed because a fatal signal was delivered to the control process.
See "systemctl status nginx.service" and "journalctl -xe" for details.

Monitoring restart -- Does not exist omv-engined

Monitoring alert -- Execution failed omv-engined:

Code

Host:        raspberrypi
Date:        Thu, 20 Jan 2022 22:07:21
Service:     omv-engined
Event:       Execution failed
Description: failed to start (exit status 1) -- '/bin/systemctl start openmediavault-engined': Job for openmediavault-engined.service failed because a fatal signal was delivered to the control process.
See "systemctl status openmediavault-engined.service" and "journalctl -xe" for details.

Zoki · 20. Januar 2022

Code

See "systemctl status php7.4-fpm.service" and "journalctl -xe" for details.
See "systemctl status nginx.service" and "journalctl -xe" for details.
See "systemctl status openmediavault-engined.service" and "journalctl -xe" for details.

Sounds like reasonable advice to me

DJ600 · 20. Januar 2022

Well, for a linux guy certainly true, for me.... just a mystery

Code

 php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Thu 2022-01-20 23:16:18 CET; 21s ago
       Docs: man:php-fpm7.4(8)
    Process: 2761 ExecStart=/usr/sbin/php-fpm7.4 --nodaemonize --fpm-config /etc/php/7.4/fpm/php-fpm.conf (code=killed, signal=SEGV)
    Process: 2762 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
   Main PID: 2761 (code=killed, signal=SEGV)
        CPU: 25ms

Code

 nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Thu 2022-01-20 23:18:49 CET; 1min 3s ago
       Docs: man:nginx(8)
    Process: 2812 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=killed, signal=SEGV)
        CPU: 48ms

Code

 openmediavault-engined.service - The OpenMediaVault engine daemon that processes the RPC request
     Loaded: loaded (/lib/systemd/system/openmediavault-engined.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Thu 2022-01-20 23:21:20 CET; 56s ago
    Process: 2866 ExecStart=/usr/sbin/omv-engined (code=killed, signal=SEGV)
        CPU: 30ms

Code

░░ The job identifier is 17.
Jan 20 23:07:17 raspberrypi systemd[2557]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 12.
Jan 20 23:07:17 raspberrypi systemd[2557]: Listening on GnuPG cryptographic agent and passphrase cache.
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 18.
Jan 20 23:07:17 raspberrypi systemd[2557]: Reached target Sockets.
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 11.
Jan 20 23:07:17 raspberrypi systemd[2557]: Reached target Basic System.
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 10.
Jan 20 23:07:17 raspberrypi systemd[2557]: Reached target Main User Target.
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 9.
Jan 20 23:07:17 raspberrypi systemd[2557]: Startup finished in 685ms.
░░ Subject: User manager start-up is now complete
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The user manager instance for user 1000 has been started. All services queued
░░ for starting have been started. Note that other services might still be starting
░░ up or be started at any later time.
░░
░░ Startup of the manager took 685940 microseconds.

Alles anzeigen

votdev · 21. Januar 2022

Execute omv-engined -d -f in the CLI and post the output if there is an error.

DJ600 · 21. Januar 2022

Thank you. Will do that tonight.

In addition to the email I aready posted above I received this message:

Code

/etc/cron.daily/openmediavault-check_locked_users:
Segmentation fault
Segmentation fault
/etc/cron.daily/openmediavault-flush-mailq:Segmentation fault
Segmentation fault
/etc/cron.daily/openmediavault-pending_config_changes:Segmentation fault
Segmentation fault

No clue what this tells me

votdev · 21. Januar 2022

Corrupt file system or incorrect platform image?

DJ600 · 21. Januar 2022

Zitat von votdev

Corrupt file system or incorrect platform image?

Hm. Platform image should be OK as OMV was running flawlessly after installation for a month or so... This is basically an "out of the box" installation without any further modifications/addons/dockers etc...

Corrupt file system might well be but everything is working so far but the web frontend...

I'll post the output from omv-engined -d -f later when I#m back at home.

votdev · 21. Januar 2022

Another idea: memory issues

DJ600 · 21. Januar 2022

All that omv-engined -d -f puts out is "segmentation fault".

Not very helpful I guess....

How could I find out whether it's a memory issue?

DJ600 · 23. Januar 2022

Anybody an idea how to solve this issue? I' m not the linux expert and I want to avoid to re- install the whole thing. 😣

KM0201 · 23. Januar 2022

Did you test the RAM as suggested?

(this is for an older Pi, but I would think it would work just the same.. but I don't mess w/ ARM stuff much)

https://forums.raspberrypi.com…php?f=63&t=103847#p717845

Soma · 23. Januar 2022

Run this commands and post the output please:

uname -a

dpkg -l | grep openm

df -h

free

dmesg | grep voltage

Zoki · 23. Januar 2022

A segfault occours, when a program writes to a memory address, it is not supposed to.

If it was an occasional failure, I might believe, it is the memory, but I fear, it is a corrupted library somewhere. So you better go for a fresh install.

Shell scripts, PHP or python can not be the source if this, so it is not OMV, but something else.

DJ600 · 23. Januar 2022

Zitat von Soma

Run this commands and post the output please:

uname -a
dpkg -l | grep openm
df -h
free
dmesg | grep voltage

Alles anzeigen

Thanks for picking this up. Here are the outputs of the commands:

uname -a

Code

Linux raspberrypi 5.10.63-v7l+ #1488 SMP Thu Nov 18 16:15:28 GMT 2021 armv7l GNU                                                                                                                     /Linux

dpkg -l | grep openm

Code

iU  openmediavault                       6.0.8-1                          all                                                                                                                               openmediavault - The open network attached storage solution
iU  openmediavault-flashmemory           6.1                              all                                                                                                                               folder2ram plugin for openmediavault
ii  openmediavault-keyring               1.0                              all                                                                                                                               GnuPG archive keys of the OpenMediaVault archive
ii  openmediavault-omvextrasorg          6.0.5                            all                                                                                                                               OMV-Extras.org Package Repositories for OpenMediaVault

df -h

Code

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  2.1G   26G   8% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           768M  2.6M  766M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G  8.0K  1.9G   1% /tmp
/dev/mmcblk0p1  253M   49M  204M  20% /boot
folder2ram      1.9G  195M  1.7G  11% /var/log
folder2ram      1.9G     0  1.9G   0% /var/tmp
folder2ram      1.9G  276K  1.9G   1% /var/lib/openmediavault/rrd
folder2ram      1.9G  976K  1.9G   1% /var/spool
folder2ram      1.9G   12M  1.9G   1% /var/lib/rrdcached
/dev/sda1       2.7T  1.1T  1.6T  41% /srv/dev-disk-by-uuid-f4f87c2d-0645-4ae9-8                                                                                                                     059-0100f04b5682
folder2ram      1.9G  4.0K  1.9G   1% /var/lib/monit
folder2ram      1.9G   32K  1.9G   1% /var/cache/samba
/dev/sdb1       3.6T  502G  3.1T  14% /srv/dev-disk-by-uuid-9890b049-49cb-4187-8                                                                                                                     214-05a6b0ea08bc
/dev/sdc1       3.6T  327G  3.3T   9% /srv/dev-disk-by-uuid-781a0fbe-18a8-4ad5-9                                                                                                                     355-61c3d02c61d8
tmpfs           384M     0  384M   0% /run/user/1000

Alles anzeigen

free

Code

               total        used        free      shared  buff/cache   available
Mem:         3930860       69344      166588      215148     3694928     3502944
Swap:         102396           0      102396

dmesg | grep voltage

Doesn't output anything.

I only can say that I didn't do anything to the system. I installed OMV 6 some four weeks ago without enabling any plugins or using docker etc... The problem occured all of a sudden from one dy to the other...

DJ600 · 24. Januar 2022

OK. Looks like nobody has an idea. I'll start over from scratch and hope that it doesn't happpen again (whatever caused this issue).

Thanks for reading anyway.

Andreas

Soma · 26. Januar 2022

Zitat von DJ600

OK. Looks like nobody has an idea. I'll start over from scratch and hope that it doesn't happpen again (whatever caused this issue).
Thanks for reading anyway.
Andreas

Did you reinstalled it?

Is everything normal or it still shows the same error?

DJ600 · 26. Januar 2022

No. Did not find the time yet...

Soma · 26. Januar 2022

If you can, do what KM0201 showed in #11

It's not unusual that it might have a damaged component.

Soma · 26. Januar 2022

Just FYI, doing the test on my 4Gb aarch64 Pi:

Code

pi@raspberrypi:~ $ sudo memtester 500 1
memtester version 4.5.0 (64-bit)
Copyright (C) 2001-2020 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 500MB (524288000 bytes)
got  500MB (524288000 bytes), trying mlock ...locked.
Loop 1/1:
  Stuck Address       : ok
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : ok
  Block Sequential    : ok
  Checkerboard        : ok
  Bit Spread          : ok
  Bit Flip            : ok
  Walking Ones        : ok
  Walking Zeroes      : ok
  8-bit Writes        : ok
  16-bit Writes       : ok

Done.

Alles anzeigen

If you run and it shows the same, you need to consider getting a new SD card.

I use the Sandisk Extreme 32Gb and so far, haven't got any issues with them.

DJ600 · 26. Januar 2022

This is my memest output. I ran 10 loops. Everything seems OK to me....

Code

pi@raspberrypi:~ $ sudo memtester 500 10
memtester version 4.5.0 (32-bit)
Copyright (C) 2001-2020 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffff000
want 500MB (524288000 bytes)
got  500MB (524288000 bytes), trying mlock ...locked.
Loop 1/10:
  Stuck Address       : ok
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : ok
  Block Sequential    : ok
  Checkerboard        : ok
  Bit Spread          : ok
  Bit Flip            : ok
  Walking Ones        : ok
  Walking Zeroes      : ok
  8-bit Writes        : ok
  16-bit Writes       : ok

Alles anzeigen

I also use a Sandisk Extreme 32Gb which I bought specifically for OMV three month ago...

Jetzt mitmachen!