BAD GATEWAY in WebUI and ZFS pool is blank

    • OMV 3.x
    • Resolved

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • BAD GATEWAY in WebUI and ZFS pool is blank

      I recently added another object to my ZFS pool. Rather than the mountpoint being /TPool/newObject, I accidentally set the mountpoint as /. Once this applied, I lost access to the WebUI, and I can't navigate the ZFS pool via SSH. How would I reverse this? Thanks.

      Edit: I actually can't SSH into the machine any more, either.
    • I think you need to boot a live distro (systemrescuecd) chroot into rootfs partition. From there disable zfs-import and zfs-mount systemd units. After that reboot see if system comes back again. Zfs pools won’t be there mounted anymore. Atm I don’t know how to change the mountpoint of a zfs people pool. Pretty sure you can info in google or another user can help you. Also if the new pool doesn’t contain any data you can wipe the disks and recreate them with a correct mountpoint .
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • Thanks @subzero79, I'd rather not erase the disks as they have a lot of data and my latest backup isn't so current. I'm assuming that when I have physical access to the terminal, I'll be able to change the mountpoint. I just don't quite know the command for that, yet. I was hoping for some feedback from this community in that regard.

      I wouldn't have any issue wiping out OMV if that would help me keep my ZFS pool's data intact, but I'd be worried that the same issue would be present once I tried to re-import the pool after a fresh install. I'm almost positive there is a simple fix regarding fixing the mountpoint, but I'm a bit wary to try something that could cause the system or my data to be irrecoverable. Is there a ZFS specialist I need to get in contact with on here?

      Edit: For clarity, the main pool is located properly as are several subsystems. The most recent one is the one that I messed up.
    • johndoe86x wrote:

      I accidentally set the mountpoint as /.
      @plugin developers: This shouldn’t be possible, please validate such user input before submitting the data to the backend.
      Absolutely no support through PM!

      I must not fear.
      Fear is the mind-killer.
      Fear is the little-death that brings total obliteration.
      I will face my fear.
      I will permit it to pass over me and through me.
      And when it has gone past I will turn the inner eye to see its path.
      Where the fear has gone there will be nothing.
      Only I will remain.

      Litany against fear by Bene Gesserit
    • johndoe86x wrote:

      I recently added another object to my ZFS pool. Rather than the mountpoint being /TPool/newObject, I accidentally set the mountpoint as /. Once this applied, I lost access to the WebUI, and I can't navigate the ZFS pool via SSH. How would I reverse this? Thanks.

      Edit: I actually can't SSH into the machine any more, either.
      I'm trying to recreate this in a VM.
      So that I have the scenario right - correct anything wrong in the following:

      First, I'm assuming you did the "object add" in the GUI.
      You started off with a pool named and mounted as /TPool? Was it 1 vdev? (A mirror?, RAID5?, number of disks?,etc.)


      What type of object were you adding? Better yet, if you remember, what did you enter into the fields, in this dialog?



      Did you have other objects previously attached to your pool? What were they?

      (While this will not help you recover your pool, it may help in other ways, with doc's, etc.)

      Thanks
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • flmaxey wrote:

      johndoe86x wrote:

      I recently added another object to my ZFS pool. Rather than the mountpoint being /TPool/newObject, I accidentally set the mountpoint as /. Once this applied, I lost access to the WebUI, and I can't navigate the ZFS pool via SSH. How would I reverse this? Thanks.

      Edit: I actually can't SSH into the machine any more, either.
      I'm trying to recreate this in a VM.
      So that I have the scenario right - correct anything wrong in the following:

      First, I'm assuming you did the "object add" in the GUI.
      You started off with a pool named and mounted as /TPool? Was it 1 vdev? (A mirror?, RAID5?, number of disks?,etc.)


      What type of object were you adding? Better yet, if you remember, what did you enter into the fields, in this dialog?



      Did you have other objects previously attached to your pool? What were they?

      (While this will not help you recover your pool, it may help in other ways, with doc's, etc.)

      Thanks
      Thank you for your detailed response. I have an 8 disk pool that is a stripe of mirrors or the ZFS equivalent of RAID10.

      Yes, you have assumed correctly. The main pool was called TPool. And I have several filesystem objects inside of it named "nas", "movies", "tv", etc. I added another filesystem object aptly named "virus" for the purpose of storing quarantined objects via clamav.

      So the fields entered in the GUI were:
      Object type: Filesystem
      Prefix: I don't recall this one
      Name: virus
      Mountpoint: /

      Please let me know what other information I can provide. I have accessed the physical machine after a reboot via the recovery option in grub and have root access.

      Sent from my Pixel 2 using Tapatalk

      Edit: I've got an external hard drive and am performing an rsync backup at the moment in case I need to obliterate everything. I was able to start the ZFS service and mount the external hard drive.

      The post was edited 1 time, last by johndoe86x ().

    • So, I'm not entirely sure that the problem was with the mountpoint in the ZFS plugin. Once I had all my data backed up as mentioned in previous post, I ran the #zfs destroy TPool/virus command and rebooted. Upon reboot, I was asked to apply the configuration changes in the WebUI. I did so and was greeted with the Bad Gateway error again. I rebooted once more (this time via hardware reset button) and then chose to "Revert" the changes rather than apply them. Now all seems to be working fine. I'm going to mark this as resolved.
    • johndoe86x wrote:

      So, I'm not entirely sure that the problem was with the mountpoint in the ZFS plugin. Once I had all my data backed up as mentioned in previous post, I ran the #zfs destroy TPool/virus command and rebooted. Upon reboot, I was asked to apply the configuration changes in the WebUI. I did so and was greeted with the Bad Gateway error again. I rebooted once more (this time via hardware reset button) and then chose to "Revert" the changes rather than apply them. Now all seems to be working fine. I'm going to mark this as resolved.
      Thanks for adding the final note on your observations and the fix - whether it was "revert" or the "destroy" command, but I suspect it was both.

      I don't think "revert" can undo ZFS operations done on the command line. Revert usually applies to changes made in the GUI. Since you were locked away from the GUI, the "revert" notice may have been waiting for confirmation, in reference to the initial add object "filesystem virus". (I've verified that a "revert" notice for a ZFS "add object" will survive a reboot.)
      While it's speculation, the command line "destroy" Tpool/virus would reconcile with OMV's "revert" back to before Tpool/virus was created. All would be as it was before, both with your pool and in OMV's /etc/openmediavault/config.xml.

      In any case, all's well that ends well.
      ________________________________

      (While I haven't had a chance to do it yet, I'm still going to attempt to recreate the issue.)
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • There is already PR waiting for review

      github.com/OpenMediaVault-Plug…penmediavault-zfs/pull/35

      Commit is to master, it can probably be cherry picked for 3.x.

      This will check only if the directory exists. If directory already exist it will fail to create a dataset or a pool
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • I created a pool ZFS1, with a 2 disk mirror
      - Then 3 more 2 disk mirrors were added, for a pool of 4 mirrors (8 disks)
      - To achieve some similarity I created a few files systems, tv, nas, movies and populated them with some data.

      I attempted to add a "virus" filesystem:

      When I plugged in @johndoe86xs variables, I got the following:



      I clicked OK on the error dialog and tried the "Save" button again. The second time, the object was added and the yellow Apply/Revert banner appeared:



      (("virus" Details showed that a ZFS filesystem object was created, with listed attributes, but with a mount point that didn't exist.))

      I refreshed the screen and got "502 Bad Gateway". Thereafter, I couldn't get back in the GUI and SSH wouldn't respond.
      **After the following reboots: the ZFS pool did not mount during startup, until the destroy command was used. **

      After a hard reboot; SSH and the WEB GUI was OK but the Apply/Revert banner were still there.
      Clicking "Apply" generated the following error dialog box:
      Failed to open file (filename=/tmp/bgstatus5TEgLM, mode=r).

      After this event, Bad Gateway dialogs disabled the GUI, and the SSH command line didn't respond correctly in a session already opened. (The shutdown command was interpreted as, "unknown file or directory".)

      After another hard reboot; SSH and the WEB GUI was OK but the Apply/Revert banner reappeared. I tried "Revert". This didn't have any noticeable effect, with the exception to removing the Apply/Revert banner.


      zfs destroy ZFS1/virus was the actual fix. After the command, it was immediately possible to navigate the pool on the command line which indicates that the pool mounted after the "virus" filesystem was removed.
      _________________________________________________

      @subzero79
      While I don't know what code would be required; the fix would seem to be forcing the parent mount point (as a minimum starting point), in "Add Object - Filesystem" dialogs.

      (But, I'm sure you already know this - the above is for the rest of us. :) )
      Good backup takes the "drama" out of computing
      ____________________________________
      OMV 3.0.99 Erasmus
      ThinkServer TS140, 12GB ECC / 32GB USB3.0
      4TB SG+4TB TS ZFS mirror/ 3TB TS

      OMV 3.0.99 Erasmus - Rsync'ed Backup
      R-PI 2 $29 / 16GB SD Card $8 / Real Time Clock $1.86
      4TB WD My Passport $119
    • That’s the default of path if left empty. If user inputs /usr or / or /var now with the patch a dialog will appear saying directory exists so it won’t go further. All zfs commands are executed at php backend so the revert button won’t do anything.
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • flmaxey wrote:

      I created a pool ZFS1, with a 2 disk mirror
      - Then 3 more 2 disk mirrors were added, for a pool of 4 mirrors (8 disks)
      - To achieve some similarity I created a few files systems, tv, nas, movies and populated them with some data.

      I attempted to add a "virus" filesystem:

      When I plugged in @johndoe86xs variables, I got the following:



      I clicked OK on the error dialog and tried the "Save" button again. The second time, the object was added and the yellow Apply/Revert banner appeared:



      (("virus" Details showed that a ZFS filesystem object was created, with listed attributes, but with a mount point that didn't exist.))

      I refreshed the screen and got "502 Bad Gateway". Thereafter, I couldn't get back in the GUI and SSH wouldn't respond.
      **After the following reboots: the ZFS pool did not mount during startup, until the destroy command was used. **

      After a hard reboot; SSH and the WEB GUI was OK but the Apply/Revert banner were still there.
      Clicking "Apply" generated the following error dialog box:
      Failed to open file (filename=/tmp/bgstatus5TEgLM, mode=r).

      After this event, Bad Gateway dialogs disabled the GUI, and the SSH command line didn't respond correctly in a session already opened. (The shutdown command was interpreted as, "unknown file or directory".)

      After another hard reboot; SSH and the WEB GUI was OK but the Apply/Revert banner reappeared. I tried "Revert". This didn't have any noticeable effect, with the exception to removing the Apply/Revert banner.


      zfs destroy ZFS1/virus was the actual fix. After the command, it was immediately possible to navigate the pool on the command line which indicates that the pool mounted after the "virus" filesystem was removed.
      _________________________________________________

      @subzero79
      While I don't know what code would be required; the fix would seem to be forcing the parent mount point (as a minimum starting point), in "Add Object - Filesystem" dialogs.

      (But, I'm sure you already know this - the above is for the rest of us. :) )
      It looks like you were able to replicate my experience, exactly. Something that I forgot to mention is that I received that error initially when I left the mountpoint blank. I assumed the / mountpoint was inside of the pool, and would place the virus filesystem inside the root of the pool instead of the root folder. It had been awhile since I had created a ZFS object. Thank you for being so diligent in this.