[RESOLVED] My Openmediavault server has fallen over today

linux4thewin · 23. Mai 2014

First off, I am a complete Linux n00b - installing and (vaguely) maintaining the OMV build is the complete extent of my Linux knowledge.

I use Openmediavault (0.4.?) to run my media download and organisation tools (SABnzbd, Sickbeard, Couchpotato), and also run Greyhole for replication of a Backup_SMB share. This build has been running rock solid for the last 16 months or so. OS is installed on a 40GB 2.5" drive that I salvaged from an old laptop, and data is spread over 3 x 2Tb 3.5" drives and 1 x 250Gb 3.5" drive (which came with the HP N40L).

I watch media via an Xtreamer Prodigy over wireless (solid concrete slab between ground and first floor), which has also been working perfectly (high bitrate movies very occasionally stutter, but nothing I can't live with).

This evening I tried to play back a media file on the media centre, and hit problems straight away. The Xtreamer reacts to unknown network streaming problems by restarting, so I knew something was up. tried restarting the wireless router - no change. Tried restarting the server - no change. So I got the laptop out and opened up a putty session to see what I could see ...

Noticed straight away that SABnzbd and CouchPotato were both giving SQL errors - checked and found that mysql service wasn't running and would not start, hence why Greyhole was not running. Then checked my NTFS data drive mounts with

Code

mount

- they were all missing! Checked

Code

df -h

and they were also missing there - last check was

Code

fdisk -l | grep NTFS

, and finally saw them! Checked /etc/fstab, and noticed a conflicting mount entry for usb0, so commented that out. Also commented out my Box mount, in case that was causing some problems. Then another restart - NTFS drives are now mounting.

Code

df -h

now yields this ouput:

root@openmediavault:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 36G 18G 16G 53% /
tmpfs 943M 0 943M 0% /lib/init/rw
udev 937M 144K 937M 1% /dev
tmpfs 943M 0 943M 0% /dev/shm
tmpfs 943M 4.0K 943M 1% /tmp
/dev/sdb1 1.9T 1.8T 115G 94% /media/AE5CE52B5CE4EED1
/dev/sdc1 1.9T 1.6T 232G 88% /media/5626203F2620228B
/dev/sdd1 233G 170G 64G 73% /media/2673E493021BFF5D
/dev/sde1 1.9T 1.7T 215G 89% /media/034E5779491ACAB7
root@openmediavault:~#

I wanted to access the web interface to check shares etc, but it won't open - Apache service is not running and fails to start.

Please help!

tekkb · 23. Mai 2014

What does it say, if anything, when you try to start apache??

With df -h I was checking if your system drive was full but that is not the case:

/dev/sda1 36G 18G 16G 53% /

Show copy of your fstab...

Did you do anything when this started???

linux4thewin · 23. Mai 2014

Update: /var/log/syslog is almost completely full of lines like these:

May 23 22:54:32 openmediavault collectd[6232]: fopen (/var/run/collectd.pid): No space left on device

Which device is it referring to? All drives in the "df" seem to be quite healthy?

davidh2k · 23. Mai 2014

Check apache logs in /var/log for errors.

edit: too slow

Greetings
David

linux4thewin · 23. Mai 2014

root@openmediavault:~# /etc/init.d/apache2 start
Starting web server: apache2Action 'start' failed.
The Apache error log may have more information.
failed!

tekkb · 23. Mai 2014

With NTFS you should not go above 80 percent full for drive. With ext4 you should not go above 90 percent, but I think web-gui is blocked above 85 percent full. This is general rule. If these are NTFS drives they are very full. Please show your fstab.

/dev/sdb1 1.9T 1.8T 115G 94% /media/AE5CE52B5CE4EED1
/dev/sdc1 1.9T 1.6T 232G 88% /media/5626203F2620228B
/dev/sdd1 233G 170G 64G 73% /media/2673E493021BFF5D
/dev/sde1 1.9T 1.7T 215G 89% /media/034E5779491ACAB7

linux4thewin · 23. Mai 2014

Zitat

[Sun May 18 07:35:32 2014] [notice] Apache/2.2.16 (Debian) mod_fcgid/2.3.6 mod_ssl/2.2.16 OpenSSL/0.9.8o configured -- resuming normal operations
The low-level probing mode requires a device
The low-level probing mode requires a device
The low-level probing mode requires a device
The low-level probing mode requires a device
error: /dev/sdf1: No such file or directory
The low-level probing mode requires a device
The low-level probing mode requires a device
error: /dev/sdf1: No such file or directory
The low-level probing mode requires a device
error: /dev/sdf1: No such file or directory
The low-level probing mode requires a device
The low-level probing mode requires a device
error: /dev/sdf1: No such file or directory
The low-level probing mode requires a device
The low-level probing mode requires a device
The low-level probing mode requires a device
The low-level probing mode requires a device
The low-level probing mode requires a device
The low-level probing mode requires a device
The low-level probing mode requires a device
The low-level probing mode requires a device
The low-level probing mode requires a device
[Mon May 19 08:32:25 2014] [notice] SIGUSR1 received. Doing graceful restart
[Mon May 19 08:32:28 2014] [notice] Apache/2.2.16 (Debian) mod_fcgid/2.3.6 mod_ssl/2.2.16 OpenSSL/0.9.8o configured -- resuming normal operations
[Fri May 23 20:59:21 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.0
[Fri May 23 20:59:21 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:22 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.1
[Fri May 23 20:59:22 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:23 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.2
[Fri May 23 20:59:23 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:24 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.3
[Fri May 23 20:59:24 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:25 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.4
[Fri May 23 20:59:25 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:26 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.5
[Fri May 23 20:59:26 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:27 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.6
[Fri May 23 20:59:27 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:28 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.7
[Fri May 23 20:59:28 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:29 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.8
[Fri May 23 20:59:29 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:30 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.9
[Fri May 23 20:59:30 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error
[Fri May 23 20:59:31 2014] [error] (28)No space left on device: mod_fcgid: couldn't bind unix domain socket /var/lib/apache2/fcgid/sock/24097.10
[Fri May 23 20:59:31 2014] [warn] (28)No space left on device: mod_fcgid: spawn process /var/www/openmediavault/php-fcgi error

Alles anzeigen

linux4thewin · 23. Mai 2014

Zitat

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# / was on /dev/sda1 during installation
UUID=480ed085-f73c-42e6-ba0c-e1443d43ef6f / ext4 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=db9c40d5-b6cd-4e97-93a7-dea98d3ab9f3 none swap sw 0 0
#/dev/sdb1 /media/usb0 auto rw,user,noauto 0 0
tmpfs /tmp tmpfs defaults 0 0
# >>> [openmediavault]
/dev/disk/by-uuid/AE5CE52B5CE4EED1 /media/AE5CE52B5CE4EED1 ntfs defaults 0 2
/dev/disk/by-uuid/5626203F2620228B /media/5626203F2620228B ntfs defaults 0 2
/dev/disk/by-uuid/2673E493021BFF5D /media/2673E493021BFF5D ntfs defaults 0 2
/dev/disk/by-uuid/034E5779491ACAB7 /media/034E5779491ACAB7 ntfs defaults,nofail 0 2
# <<< [openmediavault]
#https://www.box.com/dav /box davfs rw,user,noauto 0 0

Alles anzeigen

davidh2k · 23. Mai 2014

Code

rm /var/run/collectd.pid
service collectd restart
service apache2 restart

edit: there seems to be more problems than just that.

Greetings
David

tekkb · 23. Mai 2014

Run this and restart.. Tell me what happens.

apt-get clean

tekkb · 23. Mai 2014

Does the nofail option work with NTFS ??

What drive has your mysql database on it?

This seems to be issues with the NTFS and the drives being so full is also an issue. You really should use ext4.

linux4thewin · 23. Mai 2014

Web interface, mysql and Greyhole back up and running - thanks very much!

The "nofail" option I think was from the Greyhole instructions, if I remember correctly.

I don't know which drive stores the database - how do I check?

Would you recommend freeing up more space - was that the cause of the problem do you think?

linux4thewin · 23. Mai 2014

I opted to retain the NTFS file system as an easier way of getting to the replicated data (pull out the drive and plug into Windows machine if necessary) - I will most definitely use ext4 as I replace the drives over time. Thanks a million for your help.

tekkb · 23. Mai 2014

NTFS is bad with space. They don't like to go above 80 percent full, period.

Honestly, if you have aother spare drives I would take the NTFS drives out. Then I would install .5 OMV. Connect one NTFS driive to a Windows machines and move the data to another drive. Once one drive is empty put it back in OMV. Format it ext4. The move data back to your OMV, from the drive you moved it too, through your LAN. This is big pain in the ass. You would have to redo your OMV. But NTFS is not best filesystem for OMV. You have more knowledge now. It is best route.

Or if you coudl take all NTFS drives out. Put the spare drive in and format ext4. Move data from an NTFS drive mounted on a Windows machine through your lan to the spare drive now on the OMV. Once data moved you take that NTFS drive and put it in OMV. Wipe it and format ext4.

Best advice ever. Get rid of NTFS on OMV. Sorry it is a lot of work.

tekkb · 23. Mai 2014

I understand how you want to be able to move drives back and forth. But NTFS just is not a good fit for linux. I think the ability to mount NTFS is just to make it easy to move data and not meant to be permanent on linux.

I would rebuild all the databases after an install of .5 OMV. .5 OMV is way better then .4 OMV. You might wait til OMV 1 is released. Your drives are full man. Do not add more data or you will have more issues.

PS- This is only drive where you should be able to add some data.

/dev/sdd1 233G 170G 64G 73% /media/2673E493021BFF5D

Jetzt mitmachen!