Help needed // OMV unable to boot // RPI4// Unable to access the server //

  • Hello all,


    I think I shot myself in the foot - not sure how to recover from this.

    I'm using a RPI 4 - ARGON case - booting off an M.2 drive - installed the latest OMV updates.


    Made the rookie mistake of not backing up the server - the last backup is months old - worst case I could give that a shot.



    The only major thing I could think of is I installed this script

    Code
     curl https://download.argon40.com/argon1.sh | bash 

    Just so I can control the in case fan - I also played around with docker and installed OMV updates.


    I went to reboot the server and nothing - green light turns on for a few moments then shuts off.

    Not sure what to do here - is there any way to recover or do I have to start from scratch


    I did have some duplicati backups but I think that was just for docker?


    Any advice is appreciated


    UPDATE;


    Was able to take a screenshot of the boot up


    https://imgur.com/a/IHUmoXp


    Thanks,

    SM

  • smvb64

    Added the Label OMV 5.x
  • Your bootloader looks OK.


    I'm assuming you only have the M.2 as boot && root. (NO SDcard on it)


    You can take out the M.2 from the argon and plug it to a PC.

    Your boot partition will be seen on the Explorer.

    See if the cmdline.txt && config.txt are readable and don't show wierd artifacts/symbols.



    Then, flash a fresh RaspiOS Lite to a SDcard and add an empty ssh file to the boot partition.


    Boot the Pi with only the SDcard on it.


    After it boots normally, while it's running, plug in the M.2 to the Pi (with that USB link that comes with the Argon)


    post the output of

    fdisk -l /dev/mmcblk0

    fdisk -l /dev/sda

  • Thank you for the response,


    I checked both cmdline.txt &config.txt - all readable text - nothing looked out of the ordinary.


    I was able to launch a fresh image of RaspiOS Lite and was able to SSH to it. The rules out any issue with the pi, phew.


    The output of fdisk -l /dev/mmcblk0 is



    and the output of fdisk -l /dev/sda is



    Thank you again for helping - It seems like it does detect the drive? The 232.6G is the M.2 Drive

  • All is OK on the drive side.

    Now you need to check for issues on the filesystem of the M.2


    Shutdown again the Pi and leave ONLY the new SDcard in it.


    Boot up the Pi and run sudo raspi-config to set up the bootloader to the latest and to confirm it's booting first from USB.


    You can now, test installing the argon script (on the SDcard without anything else) or just continue with the rest first.



    With the Bootloader taken care, while the Pi is running, plug again the M.2 drive and run:

    After these steps, Power down the Pi and remove the SDcard (keep it safe for now).

    Reboot the Pi with the M.2 drive only and see if the boot goes normal.

  • I finished with the bootloader steps and moved on to the 2nd part. I skipped the argon script for now


    fdisk -l /dev/sda - showed the M.2 Drive

    df -h | grep sda - nothing happened with this command - it just went to a new line


    fsck -N /dev/sda - outputted this


    root@raspberrypi:/home/pi# fsck -N /dev/sda

    fsck from util-linux 2.36.1

    [/usr/sbin/fsck.ext2 (1) -- /dev/sda] fsck.ext2 /dev/sda


    That's all

    I have a feeling it's unable to open/access sda?


    I tried rebooting the Pi with the M.2 drive only and unfortunately, it didn't boot


    I'll plug the sd card back in for now


    Thanks,

    SM

  • In that case, the only last thing to try is to mount the M.2 drive and see if there's any files there:

    With only the SDcard, boot the Pi.


    After it's running, plug the M.2 drive.


    Run this commands:

    sudo su -

    fdisk -l /dev/sda # <- only to confirm that it shows the sda1 && sda2 partitions

    mkdir -p /mnt/bootfsM2

    mkdir -p /mnt/rootfsM2

    mount /dev/sda1 /mnt/bootfsM2 # <- or use the mount -t vfat /dev/sda1 /mnt/bootfsM2

    df -h /mnt/bootfsM2 # <- This will show how much space is on the boot of the M2

    ls -al /mnt/bootfsM2 # <- Post the output in a code box

    umount /dev/sda1


    mount /dev/sda2 /mnt/rootfsM2 # <- or use the mount -t ext4 /dev/sda2 /mnt/rootfsM2

    df -h /mnt/rootfsM2 # <- This will show how much space is on the / root of the M2

    ls -al /mnt/rootfsM2 # <- Post the output in a code box

    umount /dev/sda2


    If you see the files/folders on the partitions, you can try to run the fdisk on the partition instead of the drive:

    fsck /dev/sda1

    fsck /dev/sda2


    If no files are shown, then it is possible that your M.2 is toasted, :(


    Had one PNY 64Gb USB stick that toasted in less then a week (but I was experimenting a lot of stuff on it)

    .But my M.2 is running for a long time now on the Argon without any issues (that I know of, at least)


    Hope you can salvage your's.

  • root@raspberrypi:~# df -h /mnt/bootfsM2

    Definitely good news! files were found - the M.2 is still kicking



    and ls -al /mnt/rootfsM2


    Running fsck did show results


    fsck /dev/sda1



    fsck /dev/sda2 looks good



    Code
    root@raspberrypi:~# fsck /dev/sda2
    
    fsck from util-linux 2.36.1
    e2fsck 1.46.2 (28-Feb-2021)
    rootfs: clean, 538256/15015168 files, 9756010/60983086 blocks
    root@raspberrypi:~#

    For fsck /dev/sda1 - should I copy the original to backup and remove dirty bit?


    Thanks again,

    SM

  • That looks good, ;)


    Run fsck -a /dev/sda1 to clear the dirty bit error but leave the bootloader as is.

    Then, power down the Pi and boot it with only the M.2


    If it (finally) works, great.


    If it doesn't, it's possible to make a hybrid-boot:

    Use the boot from the SDcard to mount the rootFS on the M.2


    But try the first option, ;)

  • Good stuff aha


    I ran fsck -a /dev/sda1 and it automatically fixed the dirty bit error and ran it again


    Quote

    root@raspberrypi:~# fsck /dev/sda1

    fsck from util-linux 2.36.1

    fsck.fat 4.2 (2021-01-31)

    /dev/sda1: 292 files, 98568/516190 clusters

    Looks like it fixed it but I'm not getting the bootloader question anymore?


    I tried booting it again with just the M.2 but unfortunately didn't want to boot.


    We may have to do the hybrid boot method - looks like just the boot is the issue now


    Thanks,

    SM

  • I tried booting it again with just the M.2 but unfortunately didn't want to boot.

    Damn, was really hoping this would fix it.


    Ok, for hybrid boot, you'll need to change the bootloader order to start first from SDcard:

    As I explained previously on #4, run sudo raspi-config and set the bootloader for SDcard.


    Power on the Pi with just the SDcard.


    Run this commands and post them all here in a code box:

    The last thing to do, is modify the old fstab on the M.2 to see the boot partition from the SDcard (still as root):


    Reboot the Pi with both SDcard and the M.2 plugged in:

    reboot now and keep fingers crossed, ;)


    If the M.2 filesystems is really OK, the Pi will read the SD /boot partition and the cmdline.txt will point to the M.2 sda2 ( / ) and load it.


    Good luck, :)

  • :thumbup: - I ran blkid and got this

    Quote

    /dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="C839-E506" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="815c7a01-01"

    /dev/mmcblk0p2: LABEL="rootfs" UUID="568caafd-bab1-46cb-921b-cd257b61f505" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="815c7a01-02"

    /dev/sda1: LABEL_FATBOOT="boot" LABEL="boot" UUID="EBBA-157F" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="4b9bc9e0-01"

    /dev/sda2: LABEL="rootfs" UUID="b3ce35cd-ade9-4755-a4bb-1571e37fc1b9" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="4b9bc9e0-02"

    and opened nano cmdline.txt and changed root=PARTUUID to 4b9bc9e0-02


    I changed nano fstab and made sure the boot pointed to PARTUUID to 4b9bc9e0-02


    I reboot the Pi and it got a little bit further this time but it didn't boot - it's refusing connections from Putty? it's upset with me aha


    Do you recommend just starting from the beginning? Is it possible an OMV update can cause this? Maybe I should back up before I update.

    I want to say thanks again though for all your help - learned a lot about Linux and troubleshooting boot issues

  • I changed nano fstab and made sure the boot pointed to PARTUUID to 4b9bc9e0-02

    Here you made a mistake:

    The /dev/sda2 fstab should point to /boot from the SDcard.

    In your case PARTUUID="815c7a01-01"


    Do you recommend just starting from the beginning?

    It will be the fastest route, if you want.

    Honestly, don't see how an update could mess anything on the OS unless you did something out of the gui.


  • Oops - I see the mistake - I flashed a new pi image (Unable to access SSH anymore) - made sure it was using the latest version of the Bootloader and booting from the SD card first


    Ran blkid got


    Code
    root@raspberrypi:~# cd /boot
    root@raspberrypi:/boot# cp -a cmdline.txt cmdline.old
    root@raspberrypi:/boot# blkid
    /dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="C839-E506" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="701962cf-01"
    /dev/mmcblk0p2: LABEL="rootfs" UUID="568caafd-bab1-46cb-921b-cd257b61f505" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="701962cf-02"
    /dev/sda1: LABEL_FATBOOT="boot" LABEL="boot" UUID="EBBA-157F" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="4b9bc9e0-01"
    /dev/sda2: LABEL="rootfs" UUID="b3ce35cd-ade9-4755-a4bb-1571e37fc1b9" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="4b9bc9e0-02"
    root@raspberrypi:/boot#


    Made the nano cmdline.txt change

    Code
    console=serial0,115200 console=tty1 root=PARTUUID=4b9bc9e0-02 rootfstype=ext4 fsck.repair=yes rootwait


    and the nano fstab change


    Code
    proc            /proc           proc    defaults          0       0
    PARTUUID=701962cf-01  /boot           vfat    defaults          0       2
    PARTUUID=4b9bc9e0-02            /       ext4    noatime,nodiratime,defaults     0 1
    # a swapfile is not a swap partition, no line here
    #   use  dphys-swapfile swap[on|off]  for that
    # >>> [openmediavault]
    /dev/disk/by-label/4TB          /srv/dev-disk-by-label-4TB      ext4    defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
    /dev/disk/by-label/PLEXRSYNC            /srv/dev-disk-by-label-PLEXRSYNC        ext4    defaults,nofail,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
    /dev/disk/by-uuid/695e0790-f896-4ef0-9c86-237df935f00f          /srv/dev-disk-by-uuid-695e0790-f896-4ef0-9c86-237df935f00f      ext4    defaults,nofail,user_xattr,usrjquota=aquota.user,grp>
    # <<< [openmediavault]

    Rebooted the Pi and unfortunately get connection refused after a few moments


    Quote

    Honestly, don't see how an update could mess anything on the OS unless you did something out of the gui.

    Ok good to know - the only change I made was running the argon1.sh code outside of the gui. - I normally only use the gui or docker

  • Rebooted the Pi and unfortunately get connection refused after a few moments

    Can you connect the Pi to a screen to see if it boots normal?

    Or if it shows any error?


    If it boots, you can at least, run sudo raspi-config and activate the SSH.

    If I remember properly, it's 3 Interface options

  • Can you connect the Pi to a screen to see if it boots normal?

    Or if it shows any error?


    If it boots, you can at least, run sudo raspi-config and activate the SSH.

    If I remember properly, it's 3 Interface options

    I just connected it to a monitor and we have life!


    Picture 1

    and after pressing enter

    Picture 2


    I am able to get access to my server using the web browser and access SSH again.

    Any other steps I should do? I think we got it!


    Thanks,

    SM

  • Nice, :thumbup::thumbup::thumbup:


    For now, just powerdown the Pi, plug back your external drives and boot up.


    See if everything is green on the OMV GUI and if docker containers are also OK.


    About the 1st pic, if the above has no issues, you can ignore it.


    I'll poke Zoki for confirmation on the next phrases:

    As long as you can get to root with sudo su -, this makes not much difference AFAIK.

    If not, run via SSH, sudo passwd root and it will ask for a password.

    You won't see nothing on the screen while you type.


    If all is good, change the thread to resolved, 😉

  • This error will occur if the home directory of root does not exist or root does not have a password set. Can you check that?


    If you can boot and get to an admin shell you are good for the moment, but in the long run, you should get this error resolved. If it is a missing home directory, you will not be able to log in via ssh as root.


    Past the output of:


    cat /etc/passwd | grep root

    ls -la /

    ls -la /root

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

  • I powered down the pi and booted back up - Picture 1 happened again - after pressing enter I was able to log back in.

    sudo su - did work and was able to access root@OMVSUPER:~#


    Docker containers are functional and were able to access my shared network files as well.


    When you refer to "see if everything is green on the OMV GUI"

    do you mean this screen?


    OMV


    These are the services I have enabled.

    The only issue is just the press enter to boot - everything else looks A-OK



    Yes - root does not have a password set - I can create one


    cat /etc/passwd | grep root

    Code
    root:x:0:0:root:/root:/bin/bash


    ls -la /


    ls -la /root


    Thanks again guys,

    SM

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!