What is this crazy error? I'm stuck

1. offizieller Beitrag

    Some time last year I started getting this crazy error when I was making changes to to file systems. I reinstalled OMV and reconfigured everything from scratch, and everyone was happy. But now.... It's BACK! What is it?


    This is the first few lines of the error when I hit Apply. I can't find anything i the logfiles - Only on the screen.


    Code
    Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; omv-salt deploy run --no-color quota 2>&1' with exit code '1': omv.privat.lan: ---------- ID: quota_off_no_quotas_07486652-3d63-4830-9658-bc31a880af36 Function: cmd.run Name: quotaoff --group --user /dev/disk/by-uuid/07486652-3d63-4830-9658-bc31a880af36 || true Result: True Comment: Command "quotaoff --group --user /dev/disk/by-uuid/07486652-3d63-4830-9658-bc31a880af36 || true" run Started: 08:09:20.759265 Duration: 89.242 ms Changes: ---------- pid: 2782986 retcode: 0 stderr: stdout: ---------- ID: quota_check_no_quotas_07486652-3d63-4830-9658-bc31a880af36 Function: cmd.run Name: quotacheck --user --group --create-files --no-remount --verbose /dev/disk/by-uuid/07486652-3d63-4830-9658-bc31a880af36 Result: True Comment: Command "quotacheck --user --group --create-files --no-remount --verbose /dev/disk/by-uuid/07486652-3d63-4830-9658-bc31a880af36" run Started: 08:09:20.849234 Duration: 109.904 ms Changes: ---------- pid: 2782988 retcode: 0 stderr: quotacheck: Scanning /dev/mapper/wd30--raid5-media2 [/srv/dev-disk-by-uuid-07486652-3d63-4830-9658-bc31a880af36] quotacheck: Checked 1877 directories and 18759 files stdout:

    No, never - It just suddenly started to fail when adding a new disk. But it's not the new disk fail - it's like all the existing filesystems... I do not use Quota. It's the second time now in OMV5 - Can I active debugging somehow?


    Thanks

    • Offizieller Beitrag

    I've never seen or heard of this before, but you could try turning the quota service off.

    (sudo not needed for the following if SSH'ing in as root)


    sudo /etc/init.d/quota stop


    (Run the following for each data drive, substituting in the appropriate label for each drive, in place of DATA)


    sudo quotaoff --user --group /srv/dev-disk-by-label-DATA

    Sure, here is the results:

    Last one (/srv/dev-disk-by-label-basic) is the new filesystem I'm trying to create.

    I can create the filesystem, apparently also mount it, but I cannot Apply the configuration - it fails. But If I hit "Revert" it stays mounted


    However I think you are right about "some quota going on". There are some strange "quota" files on all filesystems :

    Code
    root@openmediavault:~# ls -l /srv/dev-disk-by-uuid-8221b09f-a0ac-4ae1-b216-47641cebe874/
total 20
-rw-------  1 root users 6144 Mar 26 07:10 aquota.group
-rw-------  1 root users    0 Mar 25 22:19 aquota.group.new
-rw-------  1 root users 7168 Mar 26 07:10 aquota.user
-rw-------  1 root users    0 Mar 25 22:19 aquota.user.new
drwxrws---+ 7 root users 4096 Mar 28 08:43 backup
    • Offizieller Beitrag
    Zitat von thk

    I can create the filesystem, apparently also mount it, but I cannot Apply the configuration - it fails. But If I hit "Revert" it stays mounted

    I can't explain this.
    ____________________________________________________

    I haven't seen aquota.group.new and aquota.user.new before. On the other hand, I haven't used the quota service.
    The others aquota.group and aquota.user seem to be a default on drives formatted EXT4.

    Is the error gone?

    The error is gone when I hit revert, but the situation is NOT comforting. The aquota files are pretty new, so something is using them.

    I'm using EXT4 filesystem on all filesystems, and I have one merger.fs. by the Union Filesystem plugin.


    Would it help to attach full output of the error, or is is possible to set the omv log to debug so it would be possible to see which module is failing the quota?

    • Offizieller Beitrag
    Zitat von thk

    The aquota files are pretty new, so something is using them

    There are plenty of possible reasons why quotas were activated that have nothing to do with OMV. Maybe it's a server addon, a dockers with root access, etc. I don't think I'd be able to tell you anything meaningful about how to find the exact cause. You might look for something that does file operations on data drive(s) or, if you have users with home directories on the boot drive, that might be a consideration.

    Zitat von thk

    I have one merger.fs. by the Union Filesystem plugin.

    Now this might be interesting.

    I looked at the aquota service on two machines, one with mergerfs (UnionFS) and one without. Both allowed the service to be turned off. However, the aquota.* files at the root of EXT4 disks could only be deleted on the machine without mergerfs. The machine with mergerfs denied deleting or accessing aquota files which usually means that they're locked and in use.

    I looked at /etc/fstab and eliminated the statements on the physical drives for the quota service. ,usrjquota=aquota.user,grpjquota=aquota.group
    After a reboot, no change. Then I looked at the merged drive line, in /etc/fstab, and found minfreespace=4G. Maybe mergerfs is looking at aquota.* files for this purpose?

    So are you, by chance, reformatting a drive that is a member of a merged drive? (Or a SNAPRAID protected drive?)

    I'm only trying to add a new disk. I do not touch any of the existing filesystems. If mergerfs is using aquota files it's using it on all filesystems - not only the two merged ones. However I can't find any documention on that.


    Also it seems that the issue is not there from the beginning. So it must be building up...

    • Offizieller Beitrag

    I'm out of ideas but, if you will, post the error you're getting that's forcing you to revert.

    __________________________________________________


    Things you try but I have doubts that they will help:

    You might run the command service quota status. The service should be dead. If it's active, run service quota stop and try the following.


    After backing the file up to /etc/fstab.bak2 , edit /etc/fstab removing ,usrjquota=aquota.user,grpjquota=aquota.group from drive mount lines.

    Reboot and see how that goes.

    Also, if you can purge these files from each drive that might break the log jam. (I doubt that you can. I couldn't with the install where mergerfs is in use.)

    aquota.group.new, aquota.user.new, aquota.group and aquota.user


    Lastly, you might make a minor change somewhere in the GUI and save it, to insure that something else is not going on with the confirmation banner and "revert".

    I managed to do all steps, but OMV still fails.

    • service quota status = Disabled
    • delete the ,usrjquota=aquota.user,grpjquota=aquota.group entries from all mountpoints in /etc/fstab = ok
    • reboot = ok
    • rm /srv/filessystems/aquota.* = succesfully
    • unmount new filesystem, hit Apply = Error
    • hit "revert" = OK (new filesystem unmounted though)

    OMV just puts ,usrjquota=aquota.user,grpjquota=aquota.group back into /etc/fstab entries, and also creates the aquota files on all filesystems again.


    So, I think we can agree that it is OMV that does the "quota thing"?


    However, the error is only appearing when I make changes to filesystems. Apparently other changes eg. sharing, is applied without error. But I'm still not happy about the filesystem error.

    Zitat von thk

    delete the ,usrjquota=aquota.user,grpjquota=aquota.group entries from all mountpoints in /etc/fstab = ok

    It doesn't work that way.


    The proper way to edit the fstab on OMV is (do this with carefull):


    sudo cp /etc/openmediavault/config.xml /etc/openmediavault/config.bak


    sudo nano /etc/openmediavault/config.xml


    Press Ctrl+W and write "usrjquota=aquota.user". The cursor will be in a <mntent> of the file.

    Double check that it is in the <mntent> that corrensponds to the UUID you want to edit.

    Move the cursor (with the arrows keys and don't press other keys blindly)(If you feel you did something wrong, just press Ctrl+X and say "No").


    Where you see this "<opts>....usrjquota=aquota.user....</opts>", delete it in all the disks you want/need.


    After those changes, press Ctrl+O and then Enter and then Ctrl+X



    Then, you can run the sudo omv-salt deploy run fstab and reboot

    • Offizieller Beitrag
    Zitat von thk

    So, I think we can agree that it is OMV that does the "quota thing"?

    This would only be an OMV thing if user quotes are in use. As mentioned prior, I believe this is an add-on issue of some kind.

    I've successfully stopped the service (it stays dead) , purged the aquota files, removed /etc/fstab entries on a system with mergerfs and SNAPRAID. On the other hand, I don't have a package that's trying to use / active the quota service.

    pi@R-PI4-WF:~ $ service quota status

    ● quota.service - Initial Check File System Quotas

    Loaded: loaded (/lib/systemd/system/quota.service; disabled; vendor preset: e

    Active: inactive (dead)

    Docs: man:quotacheck(8)


    ____________________________________________________________________


    FYI: My mergerfs (unionfs) minimum free space is 4G (default) and my storage policy is most free space (not a default).


    BTW: You will have to run through the drill outlined by mi-hol above. When you find the first instance of usrjquota=aquota.user, in openmediavault.xml, you can scroll through the drives and remove (carefully) the aquota entries. After running the omv-salt deploy command, /etc/fstab will be updated.
    ____________________________________________________________________


    What add-on's do you have installed (docker's, etc.) that manipulate files and folders on your data drives?

    Also, what version of OMV are you running?

    Zitat von crashtest

    This would only be an OMV thing if user quotes are in use.

    I've never assigned quotas (nor do I want them) and OMV automatically added it to the fstab on my external drive.

    Haven't got any problems though. Main reason I still haven't fixed it, FWIW


    Code
    # >>> [openmediavault]
/dev/disk/by-label/DATA         /srv/dev-disk-by-label-DATA     ext4    defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl  0 2
# <<< [openmediavault]


    Although, @OP seems to have more file inter-connections for quotas than just a "mount" issue with fstab (way out of my league)

    I have the same Error - i reinstalled OMV today on a new ssd. So i have a clean install without plugins.
    Now I have problems mounting my hardrive because of the quota error. Service is not running.

    Deleting or deactivating it not solves the problem - its very frustrating.

    Here the entries from config.xml:



    I don't know why in the opts in quota set :(


    here the system infos:



    Update:
    fixed it - the error message from the omv ui was a bit misleading :D
    i tried to start the quota service manually (systemctl start quota) => then following error message apperead
    quotacheck-something-weird-happened


    after some googling (https://www.golinuxhub.com/201…something-weird-happened/) i figured out, that maybe the file system is corrupt. So i did a fsck on the concerned disk and now the saving error disaperead



    Solved - kind of my own fault but... I totally agree with sakulthefirst - The UI error is very misleading. The OMV developers should definitely improve the ui error and do some kind of system readiness before Applying new config.

    Running the quota command manually gave me the very simple answer to the issue:


    >omv-salt deploy run quota 


    ID: quota_check_no_quotas_8221b09f-a0ac-4ae1-b216-47641cebe874

    Function: cmd.run

    Name: quotacheck --user --group --create-files --no-remount --verbose /dev/disk/by-uuid/8221b09f-a0ac-4ae1-b216-47641cebe874

    Result: False

    Comment: Command "quotacheck --user --group --create-files --no-remount --verbose /dev/disk/by-uuid/8221b09f-a0ac-4ae1-b216-47641cebe874" run

    Started: 20:15:17.191684

    Duration: 40.796 ms

    Changes:

    ----------

    pid:

    3665407

    retcode:

    2

    stderr:

    quotacheck: Scanning /dev/mapper/st2000lm007-backup [/srv/dev-disk-by-uuid-8221b09f-a0ac-4ae1-b216-47641cebe874] quotacheck: Cannot sta

    t old user quota file /srv/dev-disk-by-uuid-8221b09f-a0ac-4ae1-b216-47641cebe874/aquota.user: No such file or directory. Usage will not be subtracted.

    quotacheck: Cannot stat old group quota file /srv/dev-disk-by-uuid-8221b09f-a0ac-4ae1-b216-47641cebe874/aquota.group: No such file or d

    irectory. Usage will not be subtracted.

    quotacheck: Cannot stat old user quota file /srv/dev-disk-by-uuid-8221b09f-a0ac-4ae1-b216-47641cebe874/aquota.user: No such file or dir

    ectory. Usage will not be subtracted.

    quotacheck: Cannot stat old group quota file /srv/dev-disk-by-uuid-8221b09f-a0ac-4ae1-b216-47641cebe874/aquota.group: No such file or d

    irectory. Usage will not be subtracted.

    quotacheck: Checked 952 directories and 13716 files

    quotacheck: Old file not found.

    quotacheck: Cannot allocate new quota block (out of disk space).

    quotacheck: Cannot write quota (id 100): No space left on device

    stdout:

    done


    :rolleyes:


    Would be much nicer to give that output in the error UI instead - could have saved me a lot of hours.

    Why it's doing the quota check in the first place... I don't know - I'm not using any quota. Maybe the merger.fs...?

  • thk

    Hat das Label gelöst hinzugefügt.

    And the next time a new disk is added, it will have quotas enabled.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

    i also has that crazy error message when saving the configuration, after doing some minor messing with file systems.


    embedded in the error message (which is simply the output of that "omv-salt deply run quota" command) was very useful info, but hard to tease out in the weird long error message display.


    when i ran the command from the command line, it was much easier to read - and in my case i had some aquota.*.new files that were in the way, in some manner.


    i simply deleted those 2 .new files and everything seems OK. but the whole quota thing does seem like it arose out of the blue for some reason.


    Code
    root@optiplex-740:~# ls -l /srv/dev-disk-by-uuid-046f66df-7b6a-4ee7-98a2-14bd6e629b03/
total 44
drwxrwsr-x  7 root users  4096 Jul 10 21:34 appdata
-rw-------  1 root root   6144 Jul 13 13:20 aquota.group
-rw-------  1 root root      0 Jul  9 12:01 aquota.group.new
-rw-------  1 root root   7168 Jul 13 13:20 aquota.user
-rw-------  1 root root      0 Jul  9 12:01 aquota.user.new
drwxrwsr-x+ 3 root users  4096 Sep 23  2021 back-01
drwx------  2 root root  16384 Sep 22  2021 lost+found


    again, simply removing those 2 .new files manually did the trick for me.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden