Everything down. Can't Login.

jkingaround · 9. April 2016

Hi all. I woke up this morning to two pushbullets telling me my NAS had restart at some point during the night but had been brought back up (plexpy said it was back up) but didn't have a chance to look. get home from class and everything is down. plex, plexpy etc. webui shows a blank screen and no login form. i did a bit of research and used

Code

omv-mkconf omvextrasorg aptclean

and the login screen shows up but when i log in it goes back to a blank screen. console shows

Code

Uncaught Error: Ext.Loader is not enabled, so dependencies cannot be resolved dynamically. Missing required class: OMV.window.Login

. I can ssh in (obviously) and my network shares show up but i can't access them. When I boot up it looks like I have a corrupted raid array. Any help would be appreciated.

ryecoaaron · 9. April 2016

Post the output of the commands from this page

jkingaround · 10. April 2016

Code

root@NAS:~# cat /proc/mdstat


Personalities : [raid6] [raid5] [raid4] 


md126 : inactive sde[0](S) sdf[9](S)


      5860531120 blocks super 1.2


       


md127 : inactive sdd[1] sda[10] sdc[6] sdi[5] sdg[4] sdh[8] sdj[2]


      20511858920 blocks super 1.2


       


unused devices: <none>






root@NAS:~# blkid


/dev/sdb1: UUID="2d594a47-498b-486b-b7d4-c93b6d0f570d" TYPE="ext4" 


/dev/sdb5: UUID="46930547-3a30-4c7a-92c7-c423ed4db09c" TYPE="swap" 


/dev/sda: UUID="756e6ee8-fd0b-44a0-e5cb-abc9f817dd56" UUID_SUB="cac3e0ac-06cb-2cf8-a662-3a367610dbdc" LABEL="NAS:RAID" TYPE="linux_raid_member" 


/dev/sdc: UUID="756e6ee8-fd0b-44a0-e5cb-abc9f817dd56" UUID_SUB="a998566f-4b1d-b415-58f2-6b79915d49b5" LABEL="NAS:RAID" TYPE="linux_raid_member" 


/dev/sdf: UUID="756e6ee8-fd0b-44a0-e5cb-abc9f817dd56" UUID_SUB="3dbfd396-c4e0-dd27-aec1-41420826a157" LABEL="NAS:RAID" TYPE="linux_raid_member" 


/dev/sde: UUID="756e6ee8-fd0b-44a0-e5cb-abc9f817dd56" UUID_SUB="d4db375c-47de-8112-5d70-4d26881aa46a" LABEL="NAS:RAID" TYPE="linux_raid_member" 


/dev/sdd: UUID="756e6ee8-fd0b-44a0-e5cb-abc9f817dd56" UUID_SUB="8cd9d4b5-37b7-50de-8ef5-570217e71b30" LABEL="NAS:RAID" TYPE="linux_raid_member" 


/dev/sdg: UUID="756e6ee8-fd0b-44a0-e5cb-abc9f817dd56" UUID_SUB="d66f06c3-ba77-e843-9d3e-eb400800a8d7" LABEL="NAS:RAID" TYPE="linux_raid_member" 


/dev/sdh: UUID="756e6ee8-fd0b-44a0-e5cb-abc9f817dd56" UUID_SUB="5c715eed-a17e-41f2-cc22-9f6f77ff9000" LABEL="NAS:RAID" TYPE="linux_raid_member" 


/dev/sdi: UUID="756e6ee8-fd0b-44a0-e5cb-abc9f817dd56" UUID_SUB="f306fb0d-c49e-1e9f-0326-946d3a54a868" LABEL="NAS:RAID" TYPE="linux_raid_member" 


/dev/sdj: UUID="756e6ee8-fd0b-44a0-e5cb-abc9f817dd56" UUID_SUB="85191201-6012-0f8b-77d4-1c2cfd5cb214" LABEL="NAS:RAID" TYPE="linux_raid_member" 






root@NAS:~# fdisk -l


WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 3000.6 GB, 3000592982016 bytes


256 heads, 63 sectors/track, 363376 cylinders, total 5860533168 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 4096 bytes


I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk identifier: 0x00000000


   Device Boot      Start         End      Blocks   Id  System


/dev/sda1               1  4294967295  2147483647+  ee  GPT


Partition 1 does not start on physical sector boundary.


Disk /dev/sdb: 320.1 GB, 320072933376 bytes


255 heads, 63 sectors/track, 38913 cylinders, total 625142448 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 512 bytes


I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk identifier: 0x0008a16e


   Device Boot      Start         End      Blocks   Id  System


/dev/sdb1   *        2048   599773183   299885568   83  Linux


/dev/sdb2       599775230   625141759    12683265    5  Extended


/dev/sdb5       599775232   625141759    12683264   82  Linux swap / Solaris


Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes


255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 4096 bytes


I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk identifier: 0x00000000


Disk /dev/sdd doesn't contain a valid partition table


Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes


255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 4096 bytes


I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk identifier: 0x00000000


Disk /dev/sdc doesn't contain a valid partition table


Disk /dev/sde: 3000.6 GB, 3000592982016 bytes


255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 4096 bytes


I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk identifier: 0x00000000


Disk /dev/sde doesn't contain a valid partition table


Disk /dev/sdf: 3000.6 GB, 3000592982016 bytes


255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 4096 bytes


I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk identifier: 0x00000000


Disk /dev/sdf doesn't contain a valid partition table


Disk /dev/sdg: 3000.6 GB, 3000592982016 bytes


255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 4096 bytes


I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk identifier: 0x00000000


Disk /dev/sdg doesn't contain a valid partition table


Disk /dev/sdh: 3000.6 GB, 3000592982016 bytes


255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 4096 bytes


I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk identifier: 0x00000000


Disk /dev/sdh doesn't contain a valid partition table


Disk /dev/sdi: 3000.6 GB, 3000592982016 bytes


255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 4096 bytes


I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk identifier: 0x00000000


Disk /dev/sdi doesn't contain a valid partition table


Disk /dev/sdj: 3000.6 GB, 3000592982016 bytes


255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors


Units = sectors of 1 * 512 = 512 bytes


Sector size (logical/physical): 512 bytes / 4096 bytes


I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk identifier: 0x00000000


Disk /dev/sdj doesn't contain a valid partition table






root@NAS:~# cat /etc/mdadm/mdadm.conf


# mdadm.conf


#


# Please refer to mdadm.conf(5) for information about this file.


#


# by default, scan all partitions (/proc/partitions) for MD superblocks.


# alternatively, specify devices to scan, using wildcards if desired.


# Note, if no DEVICE line is present, then "DEVICE partitions" is assumed.


# To avoid the auto-assembly of RAID devices a pattern that CAN'T match is


# used if no RAID devices are configured.


DEVICE partitions


# auto-create devices with Debian standard permissions


CREATE owner=root group=disk mode=0660 auto=yes


# automatically tag new arrays as belonging to the local system


HOMEHOST <system>


# definitions of existing MD arrays


ARRAY /dev/md/RAID metadata=1.2 name=NAS:RAID UUID=756e6ee8:fd0b44a0:e5cbabc9:f817dd56






root@NAS:~# mdadm --detail --scan --verbose


mdadm: cannot open /dev/md/RAID: No such file or directory


mdadm: cannot open /dev/md/RAID: No such file or directory

Alles anzeigen

ryecoaaron · 10. April 2016

A couple more questions...

Did you only have one array?
How many drives were in the array?

jkingaround · 10. April 2016

1 array. 8 or 9.

ryecoaaron · 10. April 2016

mdadm --stop /dev/md126
mdadm --stop /dev/md127
mdadm --assemble /dev/md127 /dev/sd[acdefghij] --force --verbose

Post the output. If it looks ok, post the output of cat /proc/mdstat

jkingaround · 10. April 2016

Code

root@NAS:~# mdadm --stop /dev/md126


mdadm: stopped /dev/md126


root@NAS:~# mdadm --stop /dev/md127


mdadm: stopped /dev/md127


root@NAS:~# mdadm --assemble /dev/md127 /dev/sd[acdefghij] --force --verbose


mdadm: looking for devices for /dev/md127


mdadm: /dev/sda is identified as a member of /dev/md127, slot 8.


mdadm: /dev/sdc is identified as a member of /dev/md127, slot 6.


mdadm: /dev/sdd is identified as a member of /dev/md127, slot 1.


mdadm: /dev/sde is identified as a member of /dev/md127, slot 0.


mdadm: /dev/sdf is identified as a member of /dev/md127, slot 7.


mdadm: /dev/sdg is identified as a member of /dev/md127, slot 4.


mdadm: /dev/sdh is identified as a member of /dev/md127, slot 3.


mdadm: /dev/sdi is identified as a member of /dev/md127, slot 5.


mdadm: /dev/sdj is identified as a member of /dev/md127, slot 2.


mdadm: Marking array /dev/md127 as 'clean'


mdadm: added /dev/sde to /dev/md127 as 0 (possibly out of date)


mdadm: added /dev/sdj to /dev/md127 as 2


mdadm: added /dev/sdh to /dev/md127 as 3


mdadm: added /dev/sdg to /dev/md127 as 4


mdadm: added /dev/sdi to /dev/md127 as 5


mdadm: added /dev/sdc to /dev/md127 as 6


mdadm: added /dev/sdf to /dev/md127 as 7 (possibly out of date)


mdadm: added /dev/sda to /dev/md127 as 8


mdadm: added /dev/sdd to /dev/md127 as 1


mdadm: /dev/md127 has been started with 7 drives (out of 9).


root@NAS:~# cat /proc/mdstat


Personalities : [raid6] [raid5] [raid4] 


md127 : active (auto-read-only) raid6 sdd[1] sda[10] sdc[6] sdi[5] sdg[4] sdh[8] sdj[2]


      20511855616 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/7] [_UUUUUU_U]


      


unused devices: <none>

Alles anzeigen

ryecoaaron · 10. April 2016

It started but is degraded. Looks like you have a raid 6 array? Two drives are having issues. I would:

update-initramfs -u
reboot
Post the output of cat /proc/mdstat again.

jkingaround · 10. April 2016

Code

root@NAS:~# cat /proc/mdstat


Personalities : [raid6] [raid5] [raid4] 


md126 : inactive sdf[9](S)


      2930265560 blocks super 1.2


       


md127 : active raid6 sdd[1] sda[10] sdc[6] sdi[5] sdg[4] sdh[8] sdj[2]


      20511855616 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/7] [_UUUUUU_U]


      


unused devices: <none>

Alles anzeigen

jkingaround · 10. April 2016

That seemed to fix plex but plexpy and the webui are still down

ryecoaaron · 10. April 2016

Your array is still having issues.

Try:

mdadm --stop /dev/md126
mdadm --zero-superblock /dev/sdf
mdadm --add /dev/md127 /dev/sdf

jkingaround · 10. April 2016

Code

root@NAS:~# mdadm --stop /dev/md126


mdadm: stopped /dev/md126


root@NAS:~# mdadm --zero-superblock /dev/sdf


root@NAS:~# mdadm --add /dev/md127 /dev/sdf


mdadm: added /dev/sdf






root@NAS:~# cat /proc/mdstat


Personalities : [raid6] [raid5] [raid4] 


md127 : active raid6 sde[9] sda[10] sdc[6] sdh[5] sdf[4] sdg[8] sdi[2] sdd[1]


      20511855616 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/7] [_UUUUUU_U]


      [>....................]  recovery =  0.3% (9948032/2930265088) finish=552.4min speed=88100K/sec


      


unused devices: <none>

Alles anzeigen

so should i keep checking this to see if it's done? also my login form for webUI is still missing

ryecoaaron · 10. April 2016

Your array is still missing two disks. You shouldn't worry about the webui yet. And yes, I would wait until it is done.

jkingaround · 10. April 2016

Code

root@NAS:~# cat /proc/mdstat


Personalities : [raid6] [raid5] [raid4] 


md127 : active raid6 sde[9] sda[10] sdc[6] sdh[5] sdf[4] sdg[8] sdi[2] sdd[1]


      20511855616 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/8] [UUUUUUU_U]


      


unused devices: <none>

Alles anzeigen

is this good?

ryecoaaron · 11. April 2016

Yes. Because it is raid 6, you have one drive to protect you. The other one must have failed or is failing.

To get the web interface working, post the output of:

df -h
service nginx status
service php5-fpm status

jkingaround · 11. April 2016

Code

root@NAS:~# df -h


Filesystem                                              Size  Used Avail Use% Mounted on


rootfs                                                  282G  277G     0 100% /


udev                                                     10M     0   10M   0% /dev


tmpfs                                                   771M  752K  771M   1% /run


/dev/disk/by-uuid/2d594a47-498b-486b-b7d4-c93b6d0f570d  282G  277G     0 100% /


tmpfs                                                   5.0M     0  5.0M   0% /run/lock


tmpfs                                                   4.0G  4.0K  4.0G   1% /run/shm


tmpfs                                                   3.8G     0  3.8G   0% /tmp


/dev/md127                                               20T   17T  2.9T  86% /media/dce7ba36-614c-4255-91c3-44630fdd8929


root@NAS:~# service nginx status


nginx is running.


root@NAS:~# service php5-fpm status


php5-fpm is running.

Alles anzeigen

ryecoaaron · 11. April 2016

Your root filesystem is full. Your rsync job wrote to something not mounted I am guessing. That is why you cannot login. nginx can't log so it won't go any further.

jkingaround · 11. April 2016

how do i delete stuff? and prevent it from happening again?

ryecoaaron · 11. April 2016

Maybe your logs are full. Not sure. Hard to tell you how to prevent when I don't know what happened. I would start with apt-get clean. That hopefully clears up enough space to let you login. You fill 277 gigs worth of drive. Are you downloading stuff using a torrent program or something?

jkingaround · 11. April 2016

Yes i'm using a lftp job to grab stuff from a remote server. Is there a way to check if something is mounted before running it?

apt-get clean didn't do anything

Jetzt mitmachen!