Posts by prtigger1

Hi folks

Does anybody knows a method to get rid of the automated mount option, for my data BTRFS Raid1 with three SSDs: discard=async?

I want to do the trim of the filesystem manually, because there may be a problem with the Samsung 870 QVOs.

/dev/sdb on /srv/dev-disk-by-uuid-a8a06053-0cc4-491d-adf2-01bb8a02fb44 type btrfs (rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)

I tried to set this and reboot, with no effect!:

omv-env set OMV_FSTAB_MNTOPS_BTRFS "nodiscard"
monit restart omv-engined
omv-salt stage run prepare
omv-salt stage run deploy

Any suggestions?

Best regards

prtigger

Quote from raulfg3

for future reference:

Code

omv-env set -- OMV_BTRFS_SCRUB_ENABLED NO

Hi

This is doing the same :

omv-env set OMV_BTRFS_SCRUB_ENABLED NO

omv-env set OMV_BTRFS_SCRUB_ENABLED "NO"

Best regards

prtigger

Krisbee

crashtest

Hi together

here are my research results (without having a shutdown hang, up to now):

-I/O-waits for writing 50GB of data to the btrfs Raid1 with three SSDs is from 2 to 5... That's good!

(Edited: With write cache disabled on all disks and writing 14GB of data, I/O-waits are between 0,1 to 7,9)

-'Synchronizing SCSI Cache/Stopping disk' is not the Problem... This always needs only a very short time during shutdown.

Not a noticeable difference, when write cache is disabled.

-'Stopping disk' is the part that could take time from very short (<0.5) to 6 seconds per disk, with my setup!

That's combined with the drive usage LED... No relation to SSD write cache usage....

I guess, stopping of the SSDs is not necessary because, no rotating disk...

No relation to btrfs driver, because at this time no filesystem is active anymore.

Activ is only the Sata Layer (AHCI driver in my setup).

Varying stopping times could be SSD firmware related or Linux kernel driver, managing the stopping of the drives...

It´s the same with 6.12.x Debian kernel and 6.14.x Proxmox kernel.

When i think back to the last shutdown hang:

'[FAILED] Failed unmounting srv-dev\x2dddisk\x2dby\x2duuid\x2...srv/dev-disk-by-uuid-<my-server-volume-uuid>'

may happen at this part of the shutdown... But can't confirm at this time.

I guess, there is nothing to unmount at this time of shutdown, because all file systems are already down!

Possible workaround:

Don't use shutdown! Use reboot (No disk stopping), go to Grub or BIOS and power off, by power button.

So far at this point...

Best regards

prtigger

crashtest

Thank's, that helped!

Best regards

prtigger

Krisbee

crashtest

Hi

I've got a question: I want to try, to disabling the write cache of the three Samsung SSDs (for Testing). The cache is enabled by default.

I made no disk specific settings in the OMV GUI! Is there an easy way to disable it?

Do you know any way to do a 'Synchronizing SCSI Cache' (Syncing on ahci driver layer) before shutdown server with mounted filesystem?

Best regards

prtigger

Krisbee

Ok, that's a wise answer!

I only can give it a try, if the shutdown-hang is coming back...

Best regards

prtigger

Krisbee

Thank's, i will keep an eye on the writing to the three SSDs with Btrfs Raid1, too...

At this time i'm using the Debian 6.12.43 bpo kernel. Do you think, that a newer Proxmox kernel has newer (better) btrfs kernel drivers?

Best regards

prtigger

Krisbee

Thank you very much for your information!

Do you think, that there is any way to optimize this behaviour (Shutdown process)?

My test (i posted here) is showing no I/O performance problem....

I will keep my eye on all the coming shutdowns!

Best regards

prtigger

Krisbee

Hi here the result of the I/O-waits test:

Had done this:

root@pr-srv-01:~# btrfs device stats --check /srv/dev-disk-by-uuid-a8a06053-0cc4-491d-adf2-01bb8a02fb44
[/dev/sdc].write_io_errs    0
[/dev/sdc].read_io_errs     0
[/dev/sdc].flush_io_errs    0
[/dev/sdc].corruption_errs  0
[/dev/sdc].generation_errs  0
[/dev/sdb].write_io_errs    0
[/dev/sdb].read_io_errs     0
[/dev/sdb].flush_io_errs    0
[/dev/sdb].corruption_errs  0
[/dev/sdb].generation_errs  0
[/dev/sdd].write_io_errs    0
[/dev/sdd].read_io_errs     0
[/dev/sdd].flush_io_errs    0
[/dev/sdd].corruption_errs  0
[/dev/sdd].generation_errs  0
root@pr-srv-01:~#
root@pr-srv-01:~# btrfs check --force --readonly /dev/disk/by-uuid/a8a06053-0cc4-491d-adf2-01bb8a02fb44
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/disk/by-uuid/a8a06053-0cc4-491d-adf2-01bb8a02fb44
UUID: a8a06053-0cc4-491d-adf2-01bb8a02fb44
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 729811038208 bytes used, no error found
total csum bytes: 711802588
total tree bytes: 925188096
total fs tree bytes: 47824896
total extent tree bytes: 9469952
btree space waste bytes: 155819394
file data blocks allocated: 728885850112
 referenced 728885850112
root@pr-srv-01:~#
root@pr-srv-01:~# btrfs scrub start -B -d /srv/dev-disk-by-uuid-a8a06053-0cc4-491d-adf2-01bb8a02fb44

Scrub device /dev/sdc (id 1) done
Scrub started:    Tue Sep 16 18:58:00 2025
Status:           finished
Duration:         0:33:40
Total to scrub:   454.06GiB
Rate:             229.54MiB/s
Error summary:    no errors found

Scrub device /dev/sdb (id 2) done
Scrub started:    Tue Sep 16 18:58:00 2025
Status:           finished
Duration:         0:33:04
Total to scrub:   455.03GiB
Rate:             233.89MiB/s
Error summary:    no errors found

Scrub device /dev/sdd (id 3) done
Scrub started:    Tue Sep 16 18:58:00 2025
Status:           finished
Duration:         0:34:02
Total to scrub:   455.03GiB
Rate:             227.37MiB/s
Error summary:    no errors found
root@pr-srv-01:~#

Running 'top' with 5 seconds delay time:

I/O-waits (wa) is most of the time between 1,4 - 2,3!

The maximum i had seen during scrub was 3,3

The maximum load average (1 min.) was 10,08

Best regards

prtigger

ryecoaaron

I guess it's an issue with the folder2ram script. But i don't care anymore, because i fixed it manually...

Thank's for your information!

Krisbee

Here attached a picture of a normal shutdown process ending, before my system is powering off!

Sometimes it's quick and sometimes 'Synchronizing SCSI Cache/Stopping disk' could take a long time!

In this case it took about 3-4 seconds for each disk of the Btrfs Raid1 (The three Samsung SSDs).

I don´s know what's happening at this time, because i can't find this part in the journal!

I guess, journal service is down already, at this time...

In this period of syncing and stopping the SSDs, the drive access led is on, permanently!

My conclusion at this time:

The 'Failed unmounting srv-dev\x2ddisk\.......' from picture of my post #12 is happening here!

disk access led will stay permanently on....

And this causes the server hang!

Waiting more than 10 minutes doesn't help!:

Post

RE: Shutdown hang by failed unmounting srv volume (btrfs raid1, 3 disks) with bpo kernel 6.12.12

@ryecoaaron

Hi Aaron
Today the problem is coming back with kernel 6.12.22. I can say, there is no relation to your flashmemory plugin.
Because it´s not installed anymore!
In this case today i was removing an older stable kernel with your plugin and after this i was initiating a shutdown from web-gui.
Same failed message and then heavy load on a disk (Access light is permanently on with very little flicker).
I guess , it may be the system disk.

Today, i wait more than 10 minutes... No change... No…

prtigger1

May 18th 2025

Best regards

prtigger

ryecoaaron

No, the folder2ram package was uninstalled correctly during the uninstall of the flashmemory plugin in the GUI!

I checked this after uninstall from GUI (apt list).

But, i got a connection lost during the uninstall process... (May be #502, can't remember correctly)...

Doesn't matter, disabling service fixes it.

Other solution may be, installing folder2ram package manual and purge it after that...

Best regards

prtigger

ryecoaaron

Thank's for your comments.

After uninstalling the flashmemory again from GUI, the temporary folder2ram_cleaner.service was still present and running... Disabling this service removed the link and the unit file is present!

Don't know why not before...

This cleaner service is not necessary anymore and can be removed, because folder2ram is not installed anymore...

Best regards

prtigger

ryecoaaron

Hi Aaron

The 'folder2ram_cleaner.service' seems to be installed by removing your flashmemory plugin.

I guess, it´s not nesessary anymore and could be removed...

Could you confirm this?

Best regards

prtigger

ryecoaaron

Hi Aaron

I found a 'folder2ram_cleaner.service' error during shutdown:

….
Sep 12 18:46:36 pr-srv-01 systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Sep 12 18:46:36 pr-srv-01 systemd[1]: Stopping cpufrequtils.service - LSB: set CPUFreq kernel parameters...
Sep 12 18:46:36 pr-srv-01 systemd[1]: Stopping cron.service - Regular background program processing daemon...
Sep 12 18:46:36 pr-srv-01 systemd[1]: Stopping folder2ram_cleaner.service - temporary folder2ram systemd service...
Sep 12 18:46:36 pr-srv-01 (_cleaner)[1711]: folder2ram_cleaner.service: Failed to execute /sbin/folder2ram_cleaner: Exec format erro
r
Sep 12 18:46:36 pr-srv-01 (_cleaner)[1711]: folder2ram_cleaner.service: Failed at step EXEC spawning /sbin/folder2ram_cleaner: Exec
format error
Sep 12 18:46:36 pr-srv-01 systemd[1]: Stopping getty@tty1.service - Getty on tty1...
Sep 12 18:46:36 pr-srv-01 systemd[1]: lm-sensors.service: Deactivated successfully.
Sep 12 18:46:36 pr-srv-01 systemd[1]: Stopped lm-sensors.service - Initialize hardware monitoring sensors.
Sep 12 18:46:36 pr-srv-01 systemd[1]: Stopping minidlna.service - MiniDLNA lightweight DLNA/UPnP-AV server...
….

Your flashmemory plugin was not installed!

I disabled the this service, to try cleaning this up:

root@pr-srv-01:~# systemctl disable folder2ram_cleaner.service
Removed "/etc/systemd/system/multi-user.target.wants/folder2ram_cleaner.service".
root@pr-srv-01:~#

This disabling removed the unit-file!

I reinstalled the your flashmemory plugin...

No 'folder2ram_cleaner.service' unit file was installed!! No such service is running!

My question:

Is the 'folder2ram_cleaner.service' outdated and not nesessary anymore??

If not, could you please post the file, directory and file rights here, so i could restore ist?

Best regards

prtigger

Krisbee

Thank's for your comment!

Yes, i'm on latest 6.12. Debian bpo kernel..

In normal case, i've got no problem with the shutdown of the server!

The failed unmount of the data drives is coming up randomly and sometimes, bringing system to the hang condition!

That's the problem...

Now, i permanently deleted the quiet boot option, to keep a look at the shutdown...

The astonishing thing with the last shutdown was, the long time the stopping of every data drive took!

Don't know what was going on, in this period of time.

That may be an indication for where the unmounting problem relates to...

Next boot i will save the journal.

After that, i will try the manual scrub...

Keeping an eye on the i/o waits....

Best regards

prtigger

Krisbee

Hi Krisbee

Would it be sensible to deactivate 'fstrim.timer' because of the mount option 'discard=async'? Because of doubled trim?

The long shutdown time for the three data drives, was an addition of long stop times, for each single SSD of der Btrfs Raid1...

Edit: I just found that discard=async option is not possible with Ext4... There is still my boot SSD with Ext4...

Best regards

prtigger

Krisbee

By the way:

After getting the informations for the post above, i shutdown the server at the console.

I found out, that the shutting down (stop, sync) of the data drives (Btrfs Raid1) took a long time!

That´s the heavy traffic shown with the server drive led!

In this case, there was no client access to the server.

Funny thing: Sometimes shutdown is faster, sometimes took a longer time...

Best regards

prtigger

Krisbee

Here it is:

Quote from Krisbee

Before you do anything else, please post the output of the following commands:

1. findmnt --real -t btrfs

2. btrfs fi us /srv/dev-disk-by-uuid-a8a06053-0cc4-491d-adf2-01bb8a02fb44 

3.btrfs sub li /srv/dev-disk-by-uuid-a8a06053-0cc4-491d-adf2-01bb8a02fb44

Display More

1.

root@pr-srv-01:~# findmnt --real -t btrfs
TARGET                                                     SOURCE                FSTYPE OPTIONS
/srv/dev-disk-by-uuid-a8a06053-0cc4-491d-adf2-01bb8a02fb44 /dev/sdc              btrfs  rw,relatime,ssd,discard=async,space_cache=v2
/export/Dream                                              /dev/sdc[/Data/Dream] btrfs  rw,relatime,ssd,discard=async,space_cache=v2
root@pr-srv-01:~#

2.

root@pr-srv-01:~# btrfs fi us /srv/dev-disk-by-uuid-a8a06053-0cc4-491d-adf2-01bb8a02fb44
Overall:
    Device size:                  10.92TiB
    Device allocated:              1.33TiB
    Device unallocated:            9.58TiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                          1.33TiB
    Free (estimated):              4.79TiB      (min: 4.79TiB)
    Free (statfs, df):             3.20TiB
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,RAID1: Size:681.00GiB, Used:678.83GiB (99.68%)
   /dev/sdc      454.00GiB
   /dev/sdb      454.00GiB
   /dev/sdd      454.00GiB

Metadata,RAID1: Size:1.00GiB, Used:882.22MiB (86.15%)
   /dev/sdb        1.00GiB
   /dev/sdd        1.00GiB

System,RAID1: Size:64.00MiB, Used:112.00KiB (0.17%)
   /dev/sdc       64.00MiB
   /dev/sdb       32.00MiB
   /dev/sdd       32.00MiB

Unallocated:
   /dev/sdc        3.19TiB
   /dev/sdb        3.19TiB
   /dev/sdd        3.19TiB
root@pr-srv-01:~#

3.

root@pr-srv-01:~# btrfs sub li /srv/dev-disk-by-uuid-a8a06053-0cc4-491d-adf2-01bb8a02fb44
ID 256 gen 3600 top level 5 path Data
ID 257 gen 3599 top level 256 path Data/Medien
ID 258 gen 3312 top level 256 path Data/Dream
root@pr-srv-01:~#

Quote from Krisbee

prtigger1 Could you add the output of systemctl list-timers  --no-pager to the previous list.

4.

root@pr-srv-01:~# systemctl list-timers  --no-pager
NEXT                       LEFT          LAST                       PASSED     UNIT                       ACTIVATES
Fri 2025-09-12 18:34:01 C… 17min left    Fri 2025-09-12 17:37:57 C… 38min ago  anacron.timer              anacron.service
Fri 2025-09-12 18:39:00 C… 22min left    Fri 2025-09-12 18:09:01 C… 7min ago   phpsessionclean.timer      phpsessionclean.service
Fri 2025-09-12 20:42:39 C… 2h 26min left Mon 2025-09-08 18:59:23 C… 3 days ago apt-daily.timer            apt-daily.service
Fri 2025-09-12 22:09:37 C… 3h 53min left Fri 2025-09-12 18:03:21 C… 13min ago  apt-daily-upgrade.timer    apt-daily-upgrade.service
Sat 2025-09-13 00:00:00 C… 5h 43min left -                          -          dpkg-db-backup.timer       dpkg-db-backup.service
Sat 2025-09-13 00:00:00 C… 5h 43min left Fri 2025-09-12 17:34:52 C… 41min ago  logrotate.timer            logrotate.service
Sat 2025-09-13 17:57:21 C… 23h left      Fri 2025-09-12 17:57:21 C… 19min ago  systemd-tmpfiles-clean.ti… systemd-tmpfiles-clean.se…
Sun 2025-09-14 03:10:13 C… 1 day 8h left Sun 2025-09-07 11:00:38 C… 5 days ago e2scrub_all.timer          e2scrub_all.service
Mon 2025-09-15 00:20:14 C… 2 days left   Mon 2025-09-08 18:56:37 C… 3 days ago fstrim.timer               fstrim.service

9 timers listed.
Pass --all to see loaded but inactive timers, too.
root@pr-srv-01:~#

Best regards

prtigger

Krisbee

Hi

As i wrote above, i'm still with Btrfs!

I'm not a person that's doing filesystem hopping!

My friend prefers ZFS... He is testing and my system is in production...

Just busy with the last memory test...

If i got time, you will get the outputs...

Best regards

prtigger

crashtest

Thank you for this interesting information! I´m only using the services you can see in my signature!

For media i installed the MiniDLNA plugin... Nothing else.

With the md-raid 5 and Btrfs single installed, i had a maximum storage capacity of around 7 TB...

My friend is just trying ZFS with the identical server hardware.

I'm still with Btrfs Raid1 over the three SSDs, with storage capacity around 5TB..

I still try to fix the last shutdown problem... It´s not coming up again, now... It's very rare and randomly!

Next step, i will follow the advice of Krisbee !

Best regards

prtigger