ZFS plugin missing ZSTD option

Scythe · 13. Februar 2024

I've noticed in the ZFS plugin, under - Storage>ZFS>Pools>Create - there is no ZSTD option for compression when enabling that checkbox.

I have proxmox 6.2 kernel installed, on a fully updated OMV7 fresh install.

OpenZFS is 2.1.14 which the documentation suggests does include ZSTD

As such, I can do this via CLI, but I thought it was worth pointing out it was missing in the GUI.

While on the ZFS plugin topic; a feature suggestion if thats not too bold.

I was thinking a checkbox for 'options' (passing -o to the zpool create) followed by a plain text field (marked 'advanced users only'?) would be brilliant.

There are other settings like 'noatime' that most home users would want to set. but can't from the GUI.

A pre emptive thank you for your amazing work. How such a small team manages to handle so much utility, I'll never know.

Been using OMV for near on a decade, and it only gets better (plugins too).

Krisbee · 13. Februar 2024

@Scythe Although ryecoaaron maintains what is a 3rd-party plugin, I don't know if you can expect new features. Properties like atime=on|off and what compression algorithm is used are not immutable and can be set at the pool level via the WebUI post zpool creation if you wish.

A plain text field sounds great in theory, but in practice would have to be parsed and validated in some way in the background if a proper zfs pool is to be created via the WebUI. I can't see that happening. Just imagine all the forum questions about any generic error message the WebUI might be able to generate.

If the savvy user wants to create a zfs pool at the CLI, they are responsible for shooting themselves in the foot and/or sorting any command errors, not an overly complex plugin.

One thing I'd really like to see changed is the ashift setting defaulting to 12 rather than relying on ashift=0 to "auto-detect using the kernel's block layer and a ZFS internal exception list."

ryecoaaron · 13. Februar 2024

Zitat von Scythe

I was thinking a checkbox for 'options' (passing -o to the zpool create) followed by a plain text field (marked 'advanced users only'?) would be brilliant.
There are other settings like 'noatime' that most home users would want to set. but can't from the GUI.

Are there any options that can't be set after creation in the Properties window in the plugin? I know it isn't as convenient but it is much less work and maintenance for me.

Zitat von Scythe

OpenZFS is 2.1.14 which the documentation suggests does include ZSTD
As such, I can do this via CLI, but I thought it was worth pointing out it was missing in the GUI.

I had looked at adding this but if someone is not using the proxmox kernel and doesn't have backports enabled, they will get zfs 2.1.11. I guess I could look at making the dropdown dynamically populated.

Zitat von Krisbee

One thing I'd really like to see changed is the ashift setting defaulting to 12 rather than relying on ashift=0 to "auto-detect using the kernel's block layer and a ZFS internal exception list."

set default ashift to 12 · OpenMediaVault-Plugin-Developers/openmediavault-zfs@2a250aa

github.com

Krisbee · 13. Februar 2024

Thanks.

Scythe · 14. Februar 2024

Thanks for the fast reply ryecoaaron; I spent a few days catching up on all the current forum topics and you really are a bit of a Ninja on these issues Colour me impressed.

Zitat von ryecoaaron

I had looked at adding this but if someone is not using the proxmox kernel and doesn't have backports enabled, they will get zfs 2.1.11.

That shouldn't be a problem

ZSTD has been available since 2.0, so 2.1.11 or earlier is perfectly acceptable.

Workload Tuning — OpenZFS documentation

Zitat von Krisbee

A plain text field sounds great in theory, but in practice would have to be parsed and validated in some way in the background if a proper zfs pool is to be created via the WebUI. I can't see that happening. Just imagine all the forum questions about any generic error message the WebUI might be able to generate.

I don't agree there. ZFS reacts in one of two ways;

If the option is legitimate but the value is bad, it's quite verbose on what went wrong.

Otherwise, it'll simply say nothing and ignore the -o parameter you tried to set.

In regards to forum questions, that's fair 'people can be stupid'. I've worked in IT for a few decades now, I'm well aware, haha

However, people can be taught if it's kept simple.

If you flagged the option as 'advanced' to begin with, with some scary 'This can break your array' subtext.

You could then make all -o errors explicitly link to the OpenZFS manual page on what -o's are available; rather than trying to explain what you did wrong within the plugin, then I think you'd mitigate a large chunk of your concern.

So, our OMV error would just say "Your Options did not work, refer to <URL> to check your formatting and spelling" or such.

I don't think that'd trigger notably more support calls than people needing to do it CLI now.

Zitat von ryecoaaron

Are there any options that can't be set after creation in the Properties window in the plugin? I know it isn't as convenient but it is much less work and maintenance for me.

Annoyingly, most of them, in a sense.

Datasets will pull a majority of their default settings from the pool; so what you set at pool creation is pretty important.

You can override it, but then users are still dropping to the CLI for dataset creation now, not just pools.

Most home users I've spoken to, helping out on Reddit don't actually use a secondary dataset anyway and just use the root dataset thats made during pool creation.

Krisbee · 14. Februar 2024

@Scythe I'm still not convinced. Perhaps you have some examples of the options you would add. You don't make the distinction between ZFS pool properties set with -o and ZFS pool filesystem properties set with -O .

I'd argue for these setting to be made automatically at pool creation time:

1. -o failmode=continue -o autoexpand=on ( currently default to failmode =wait and autoexpand=off)

2. -O atime=off -O acltype=posix -O xattr=sa

The later brings ZFS into line with all other native linux systems in OMV which are mounted with ACL and xattr suppport. There would be no more error messages when using the ACL tab in the WebUI on ZFS filesystems. ( The related properties of alcmode and aclinherit can be left at their defaults and advanced user can set the appropriate values on a per-dataset basis.)

If there was general agreement on this, I think this would be a simple change for ryecoaaron to make.

Zitat von Scythe

Datasets will pull a majority of their default settings from the pool; so what you set at pool creation is pretty important.
You can override it, but then users are still dropping to the CLI for dataset creation now, not just pools.
Most home users I've spoken to, helping out on Reddit don't actually use a secondary dataset anyway and just use the root dataset thats made during pool creation.

I don't know if you agree, but one reason for dropping to the CLI to act on datasets is that using the WebUI to change dataset props is tedious/cumbersome. Listing all the properties of a given dataset leads to having to scroll back and forth to find the properties to change.

I'd argue that you don't need to display the entire set of a dataset's filesystem properties. A much reduced set fitting on one page should suffice.

This is another change I'd really like to see, but could entail quite a bit of work for ryecoaaron .

Home users who do not create filesystem datasets on a pool are not using ZFS effectively. It there was tighter integration between ZFS and OMV, then like BTRFS, perhaps the creation of a shared folder should create a filesystem dataset and you would be blocked from certain actions the top level pool datasets.

ryecoaaron · 14. Februar 2024

Zitat von Scythe

Annoyingly, most of them, in a sense.
Datasets will pull a majority of their default settings from the pool; so what you set at pool creation is pretty important.
You can override it, but then users are still dropping to the CLI for dataset creation now, not just pools.

I guess I don't understand what people are doing well enough. You can't create the pool, set the properties, and then create the dataset(s) to inherit the properties?

My biggest problem with the zfs plugin is a large number of zfs users are fairly experienced zfs users or experts. And they all seem to want to do things differently. Maintaining the zfs plugin is difficult when I don't use zfs or the plugin. Adding new features is even harder because it is difficult to know if I am breaking things.

Zitat von Krisbee

I'd argue for these setting to be made automatically at pool creation time:

1. -o failmode=continue -o autoexpand=on ( currently default to failmode =wait and autoexpand=off)
2. -O atime=off -O acltype=posix -O xattr=sa

The later brings ZFS into line with all other native linux systems in OMV which are mounted with ACL and xattr suppport. There would be no more error messages when using the ACL tab in the WebUI on ZFS filesystems. ( The related properties of alcmode and aclinherit can be left at their defaults and advanced user can set the appropriate values on a per-dataset basis.)

If there was general agreement on this, I think this would be a simple change for ryecoaaron to make.

While I hate ACLs and would argue to leave them off, I get why it makes sense to avoid breaking the ACL tab. I can look at this change.

Zitat von Krisbee

I don't know if you agree, but one reason for dropping to the CLI to act on datasets is that using the WebUI to change dataset props is tedious/cumbersome. Listing all the properties of a given dataset leads to having to scroll back and forth to find the properties to change.

I'd argue that you don't need to display the entire set of a dataset's filesystem properties. A much reduced set fitting on one page should suffice.
This is another change I'd really like to see, but could entail quite a bit of work for

Are you saying the properties page in the plugin lists too many properties? I'm not sure how the plugin would decide which properties not to list. The plugin has always listed them all.

I really would love if someone could help maintain the plugin. I know enough about zfs to be dangerous but don't know how many changes would affect users of the plugin.

Zitat von Krisbee

It there was tighter integration between ZFS and OMV, then like BTRFS, perhaps the creation of a shared folder should create a filesystem dataset and you would be blocked from certain actions the top level pool datasets.

I don't see that happening since it would require core OMV changes and Volker has said he does not want to add zfs because of its license.

Krisbee · 14. Februar 2024

ryecoaaron

Zitat von ryecoaaron

Are you saying the properties page in the plugin lists too many properties? I'm not sure how the plugin would decide which properties not to list. The plugin has always listed them all.

Yes, depending on your physical screen, the list can stretch to 3 or more pages. Assuming the plugin is doing a zfs get all on the zfs filesystem in the background to fetch and populate the WebUI page object, filter the command instead.

Decide on a basic useful set properties and use that as the filter for the zfs get. The equivalent of something like:

basicpropset=(sync,recordsize,aclmode,alcinherit)

zfs get $basicpropset <filesystem path>

The question is what should the "basicpropset" be? Around 10-11 items should fit on a single screen without paging.

Elsewhere in OMV selecting an option indicated by WebUI page icon will offer a further menu of choices. When pointing to a zfs filesystem perhaps the "properties icon" could offer the choice of displaying a basic properties set or the full unfiltered set.

Obviously, tighter integration between ZFS and OMV will not happen. That was just me speculating about a what might have been.

ryecoaaron · 14. Februar 2024

Zitat von Krisbee

Yes, depending on your physical screen, the list can stretch to 3 or more pages. Assuming the plugin is doing a zfs get all on the zfs filesystem in the background to fetch and populate the WebUI page object, filter the command instead.

Decide on a basic useful set properties and use that as the filter for the zfs get. The equivalent of something like:

basicpropset=(sync,recordsize,aclmode,alcinherit)
zfs get $basicpropset <filesystem path>

The question is what should the "basicpropset" be? Around 10-11 items should fit on a single screen without paging.

Elsewhere in OMV selecting an option indicated by WebUI page icon will offer a further menu of choices. When pointing to a zfs filesystem perhaps the "properties icon" could offer the choice of displaying a basic properties set or the full unfiltered set.

Alles anzeigen

Having a filtered list would be another thing I would have to maintain that I am not the right one to decide. And it would break the ability to add custom properties because you most likely wouldn't see them after adding them. Why is pagination so bad? It isn't like you are changing properties all the time.

I guess I could offer a basic and full properties button but this seems like a lot of work for me for very little gain.

Krisbee · 14. Februar 2024

I appreciate it might seem like a lot or work for little gain. Point taken re: custom properties, hence a basic set and a full set. Pagination is bad as the current interface forces user to search through a long list to find each of the few typical properties that might be changed from their default, perhaps even having to go back and forth if they miss one. Most people are only going to want/need to work on a subset of filesystem properties.

TBH, I wouldn't use the WebUI to change filesystem properties, it's too cumbersome.

ryecoaaron · 14. Februar 2024

Zitat von Krisbee

Pagination is bad as the current interface forces user to search through a long list to find each of the few typical properties that might be changed from their default, perhaps even having to go back and forth if they miss one. Most people are only going to want/need to work on a subset of filesystem properties.

Isn't that argument the same for every datatable in OMV? I try to enable the filter box on every datatable in plugins I write but the datatable on a form doesn't have that option. I can ask Volker what it would take to add that.

Zitat von Krisbee

I wouldn't use the WebUI to change filesystem properties, it's too cumbersome.

I feel that way about a lot of GUIs but I can only do so much with a plugin. That is also why the plugin doesn't store a list of zfs in the database. It provides the OMV storage and filesystem backend which allows you to do everything from the command line but still use those filesystems in the web interface.

Krisbee · 14. Februar 2024

ryecoaaron Obviously it's just my opinion as to why I think the full and long list of zfs filesystem properties is cumbersome to use. A WebUI/GUI comes into it's own when a few clicks replaces more complex work at the CLI, but in this case it does not .

I can never remember where in the list a certain property appears. I have to find and select to edit each property I want to change. Then change it, and remember to save each change. At the CLI its "zfs set x y z <path filesystem> " and done. I just don't think you need to see the full list most of the time. But I appreciate this may just be a personal thing and no doubt there would be some debate about what basic list of properties should be displayed. So you can treat this as a "nice to have" suggestion.

The fact that ZFS is not linked to the OMV database as you describe is a major plus. I wouldn't want that to change.

Scythe · 16. Februar 2024

I mean, there is always the more simple solution for these options (in regards to your maintenence time).

Literally list all the flags that have binary options, with no further explaination.

Instead, link to the manual pages at the top of said list, and hide it behind a 'I know what I'm doing' button or menu.

Actually getting new flags in ZFS is a slow process, and anything that you might hear about (for example, FastDedupe was released on the 14th) will be considered 'testing' for quite a long time, so you're not going to corner yourself into weekly commits; It'd be the sort of thing you'd want to review perhaps yearly.

At the end of the day, we can do it all in CLI.

It just depends on your vision for OMV.

If you're imaging a WebUI where more novice users can configure their server, then with ZFS becoming the 'new hotness' (mostly thanks to extreme social media exposure recently) is a crucial aspect.

If you're more picturing a fun 'reference' web UI for more advanced users? Catching up to Cockpit and Webmin becomes a much more laborious task in my mind; an admirable one, but a huge undertaking.

ryecoaaron · 16. Februar 2024

Zitat von Scythe

It just depends on your vision for OMV.
If you're imaging a WebUI where more novice users can configure their server, then with ZFS becoming the 'new hotness' (mostly thanks to extreme social media exposure recently) is a crucial aspect.

If you're more picturing a fun 'reference' web UI for more advanced users? Catching up to Cockpit and Webmin becomes a much more laborious task in my mind; an admirable one, but a huge undertaking.

I don't really have a vision for the zfs plugin. I keep porting it to new versions of OMV because I know it is wanted. It is hard to have a vision for the plugin when I don't use zfs and want to spend minimal time on the plugin. New changes are hard to test because I'm not familiar with many use cases of zfs. Many new features people ask for would take a lot of work and expertise in zfs. If btrfs raid5/6 was better, I would really push people toward btrfs because its integration with OMV is much, much better.

ryecoaaron · 16. Februar 2024

Zitat von Krisbee

I can never remember where in the list a certain property appears.

The property form will now sort properties by name by default. The form has a search field and has sortable columns now as well. I didn't realize I could use these features on a form datatable or else I would've added them a long time ago. Nothing for me to maintain this way either.

add search field and make columns sortable on properties form · OpenMediaVault-Plugin-Developers/openmediavault-zfs@fc39de9

sort properties by name by default · OpenMediaVault-Plugin-Developers/openmediavault-zfs@4d01912

Signed-off-by: Aaron Murray <plugins@omv-extras.org>

github.com

ryecoaaron · 16. Februar 2024

The plugin will also have all of the supported zstd options as well
https://github.com/OpenMediaVa…9c6573eea88de91e22cfb1a98

Krisbee · 16. Februar 2024

ryecoaaron Thanks for those changes. Did you decide on default zfs pool filesystem options, -O atime=off -O acltype=posix -O xattr=sa?

ryecoaaron · 16. Februar 2024

Zitat von Krisbee

Did you decide on default zfs pool filesystem options, -O atime=off -O acltype=posix -O xattr=sa?

I was going to start looking at that next.

ryecoaaron · 16. Februar 2024

So, the plugin is only adding options/properties at the pool level and then the filesystem inherits those properties. I would like to keep it that way. Do you see a problem with that?

openmediavault-zfs/usr/share/omvzfs/Zpool.php at master · OpenMediaVault-Plugin-Developers/openmediavault-zfs

OpenMediaVault plugin for zfs. Contribute to OpenMediaVault-Plugin-Developers/openmediavault-zfs development by creating an account on GitHub.

github.com

openmediavault-zfs/usr/share/omvzfs/Filesystem.php at master · OpenMediaVault-Plugin-Developers/openmediavault-zfs

OpenMediaVault plugin for zfs. Contribute to OpenMediaVault-Plugin-Developers/openmediavault-zfs development by creating an account on GitHub.

github.com

Krisbee · 16. Februar 2024

ryecoaaron Coding's not my thing, especially OO, but as far as I can follow this then the current flow should stay as is. If zpool.php is the main code behind the "add pool" WebUI page then it seems able to distinguish between zfs pool properties set with -o and zfs pool filesystem properties set with -O and the create function ends up calling a filesystem function.

For example, creating a a raidz1 pool via the webUI generates this zpool history:

Code

root@omv7vm:~# zpool history 
History for 'Opool':
2024-02-16.16:34:02 zpool create -o ashift=12 Opool raidz1 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-2
2024-02-16.16:34:02 zfs set compression=lz4 Opool
2024-02-16.16:34:03 zfs set omvzfsplugin:uuid=59f0624b-7fb0-4dcf-9237-0581517f6991 Opool
2024-02-16.16:34:03 zfs set omvzfsplugin:uuid=59f0624b-7fb0-4dcf-9237-0581517f6991 Opool
2024-02-16.16:34:13 zfs set omvzfsplugin:uuid=59f0624b-7fb0-4dcf-9237-0581517f6991 Opool

root@omv7vm:~# zfs get all Opool | grep plugin
Opool  omvzfsplugin:uuid     59f0624b-7fb0-4dcf-9237-0581517f6991  local

Creating the same pool at the CLI generates this slightly different zpool history:

Code

root@omv7vm:~# zpool create -o ashift=12 -O compression=lz4 Opool raidz1 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-2
root@omv7vm:~# zpool history
History for 'Opool':
2024-02-16.16:58:17 zpool create -o ashift=12 -O compression=lz4 Opool raidz1 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-1 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-2
2024-02-16.16:58:19 zfs set omvzfsplugin:uuid=b171bfa9-4739-4509-8d11-b50d892b4cf4 Opool

root@omv7vm:~# zfs get compression Opool
NAME   PROPERTY     VALUE           SOURCE
Opool  compression  lz4             local

Currently unless an option is on the WebUI form, a pool is created with default properties. But you don't really want to put acltype=posix and xattr=sa on the WebUI form for adding a pool.

ZFS plugin missing ZSTD option

chente 13. Februar 2024

Jetzt mitmachen!

Tags