OMV cleanup cron fails

MarcS · 30. Oktober 2023

Since a few days I am getting this error message every hour. I have not changed anything to my system in several weeks.

Has anyone seen this? Seems to be related to BTRFS cleanup tasks.

Any ideas how to fix this?

Code

/etc/cron.hourly/openmediavault-cleanup_sf_snapshots:
ERROR: Could not destroy subvolume/snapshot: Directory not empty
Traceback (most recent call last):
  File "/sbin/omv-sfsnapadm", line 244, in <module>
    sys.exit(main())
  File "/sbin/omv-sfsnapadm", line 240, in main
    cli()
  File "/usr/lib/python3/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/sbin/omv-sfsnapadm", line 235, in cleanup_cmd
    delete_snapshot(snapshot, simulate)
  File "/sbin/omv-sfsnapadm", line 107, in delete_snapshot
    openmediavault.procutils.check_call(
  File "/usr/lib/python3/dist-packages/openmediavault/procutils.py", line 58, in check_call
    return subprocess.check_call(*popenargs, **kwargs)
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['btrfs', 'subvolume', 'delete', '/srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d/.snapshots/OMVmoving@hourly_20231029T010001']' returned non-zero exit status 1.

Alles anzeigen

votdev · 30. Oktober 2023

Other question, what did you have done before and how can we reproduce this?

MarcS · 30. Oktober 2023

I have not done anything. The system has been fixed for at least 1 month. All I did was keeping it up to date with updates.

Sorry - I dont know how to reproduce it as the above cron job seems to be an OMV system job which I cannot even see.

MarcS · 30. Oktober 2023

I have the latest docker update (docker-ce 5:24.0.7-1~debian.11~bullseye) ready for installation.

Shall I go ahead?

votdev · 30. Oktober 2023

What is the output of

Bash

# btrfs subvol list OMVmoving

Does this shared folder contain other shared folders?

MarcS · 30. Oktober 2023

output:

Code

ID 263 gen 188709 top level 5 path composefiles
ID 264 gen 42135 top level 5 path NetBackup_IN
ID 278 gen 190179 top level 5 path LIVE180
ID 288 gen 190089 top level 5 path OMVmoving
ID 289 gen 190111 top level 5 path .snapshots
ID 290 gen 49329 top level 289 path .snapshots/LIVE180_20230826T151017
ID 1876 gen 185591 top level 289 path .snapshots/LIVE180@daily_20231029T000001
ID 1877 gen 185814 top level 289 path .snapshots/OMVmoving@hourly_20231029T0100                                           01
ID 1878 gen 185814 top level 1877 path .snapshots/OMVmoving@hourly_20231029T010                                           001/OMVmoving
ID 1901 gen 188340 top level 289 path .snapshots/LIVE180@daily_20231030T000001
ID 1915 gen 189742 top level 289 path .snapshots/OMVmoving@hourly_20231030T1300                                           02
ID 1916 gen 189853 top level 289 path .snapshots/OMVmoving@hourly_20231030T1400                                           01
ID 1917 gen 189964 top level 289 path .snapshots/OMVmoving@hourly_20231030T1500                                           01
ID 1918 gen 190078 top level 289 path .snapshots/OMVmoving@hourly_20231030T1600                                           01

Alles anzeigen

MarcS · 31. Oktober 2023

I also noticed that my mergerfs pool is not mounting. The pool has two disks

Pool1
/srv/dev-disk-by-uuid-01cc26c2-19ae-48f8-ab41-377f9028a8cf

/srv/dev-disk-by-uuid-52c56732-4fd7-4cc3-ac37-64760ac981ca

and this is the error message:

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; systemctl restart srv-mergerfs-Pool1.mount' with exit code '1': 

OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; systemctl restart srv-mergerfs-Pool1.mount' with exit code '1':  in /usr/share/php/openmediavault/system/process.inc:242
Stack trace:
#0 /usr/share/openmediavault/engined/rpc/mergerfs.inc(203): OMV\System\Process->execute(Array, 1)
#1 [internal function]: OMVRpcServiceMergerfs->restartPool(Array, Array)
#2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#3 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('restartPool', Array, Array)
#4 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Mergerfs', 'restartPool', Array, Array, 1)
#5 {main}

MarcS · 31. Oktober 2023

Any ideas about the

/etc/cron.hourly/openmediavault-cleanup_sf_snapshots:

error.

ERROR: Could not destroy subvolume/snapshot: Directory not empty

It still appears every hour, even after upgrading to latest OMV version.

votdev · 31. Oktober 2023

Zitat von MarcS

Any ideas about the
/etc/cron.hourly/openmediavault-cleanup_sf_snapshots:
error.
ERROR: Could not destroy subvolume/snapshot: Directory not empty

It still appears every hour, even after upgrading to latest OMV version.

Alles anzeigen

You still did not answer my question if this shared folder contains other shared folders.

MarcS · 1. November 2023

Sorry I didnt see your 2nd question.

-> No there are no other shared folders inside that folder. Below is the list of all Shared Folders:

MarcS · 2. November 2023

any more ideas how to stop this error?

votdev · 2. November 2023

Zitat von MarcS

any more ideas how to stop this error?

I can not reproduce this. If the subvolume does not contain any other subvolume, then i really have no idea what this error means.

votdev · 2. November 2023

What is the output of

Bash

# btrfs subvolume list -o /srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d/OMVmoving

MarcS · 5. November 2023

the out put is < > (nothing)

MarcS · 5. November 2023

btrfs subvolume list -o /srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d/.snapshots gives the following output:

Code

root@elite2:/srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d#  btrfs subvolume list -o /srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d/.snapshots
ID 290 gen 49329 top level 289 path .snapshots/LIVE180_20230826T151017
ID 1877 gen 205521 top level 289 path .snapshots/OMVmoving@hourly_20231029T010001
ID 2049 gen 202888 top level 289 path .snapshots/LIVE180@daily_20231104T160808
ID 2058 gen 203770 top level 289 path .snapshots/LIVE180@daily_20231105T000001
ID 2070 gen 205076 top level 289 path .snapshots/OMVmoving@hourly_20231105T120001
ID 2071 gen 205186 top level 289 path .snapshots/OMVmoving@hourly_20231105T130001
ID 2072 gen 205300 top level 289 path .snapshots/OMVmoving@hourly_20231105T140001
ID 2073 gen 205410 top level 289 path .snapshots/OMVmoving@hourly_20231105T150001

MarcS · 5. November 2023

Did some more digging and there seems to be a problem with a nested sub-volume.

There is a subvolume

ID 1878 gen 200649 top level 1877 path OMVmoving@hourly_20231029T010001/OMVmoving

I dont know how this subvolume got created but it becomes visible with "btrfs subvol list .snapshots"

Code

root@elite2:/srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d# btrfs subvol list .snapshots
ID 263 gen 204154 top level 5 path composefiles
ID 264 gen 205519 top level 5 path NetBackup_IN
ID 278 gen 205555 top level 5 path LIVE180
ID 288 gen 205533 top level 5 path OMVmoving
ID 289 gen 205556 top level 5 path .snapshots
ID 290 gen 49329 top level 289 path LIVE180_20230826T151017
ID 1877 gen 205521 top level 289 path OMVmoving@hourly_20231029T010001
ID 1878 gen 200649 top level 1877 path OMVmoving@hourly_20231029T010001/OMVmoving
ID 2049 gen 202888 top level 289 path LIVE180@daily_20231104T160808
ID 2058 gen 203770 top level 289 path LIVE180@daily_20231105T000001
ID 2071 gen 205186 top level 289 path OMVmoving@hourly_20231105T130001
ID 2072 gen 205300 top level 289 path OMVmoving@hourly_20231105T140001
ID 2073 gen 205410 top level 289 path OMVmoving@hourly_20231105T150001
ID 2074 gen 205523 top level 289 path OMVmoving@hourly_20231105T160001

Alles anzeigen

votdev · 5. November 2023

Zitat von MarcS

btrfs subvolume list -o /srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d/.snapshots gives the following output:

Code

root@elite2:/srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d#  btrfs subvolume list -o /srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d/.snapshots
ID 290 gen 49329 top level 289 path .snapshots/LIVE180_20230826T151017
ID 1877 gen 205521 top level 289 path .snapshots/OMVmoving@hourly_20231029T010001
ID 2049 gen 202888 top level 289 path .snapshots/LIVE180@daily_20231104T160808
ID 2058 gen 203770 top level 289 path .snapshots/LIVE180@daily_20231105T000001
ID 2070 gen 205076 top level 289 path .snapshots/OMVmoving@hourly_20231105T120001
ID 2071 gen 205186 top level 289 path .snapshots/OMVmoving@hourly_20231105T130001
ID 2072 gen 205300 top level 289 path .snapshots/OMVmoving@hourly_20231105T140001
ID 2073 gen 205410 top level 289 path .snapshots/OMVmoving@hourly_20231105T150001

You did not execute the command I requested. Please check my last post.

MarcS · 5. November 2023

you mean this?

# btrfs subvolume list -o /srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d/OMVmoving

votdev · 5. November 2023

Zitat von MarcS

you mean this?
# btrfs subvolume list -o /srv/dev-disk-by-uuid-786fbfe4-2782-4b41-b28a-9ec8130edb8d/OMVmoving

Yes

MarcS · 5. November 2023

Zitat von votdev

Yes

i did. see post 14

Jetzt mitmachen!