[HOWTO] Instal ZFS-Plugin & use ZFS on OMV

    • OMV 1.0
    • [HOWTO] Instal ZFS-Plugin & use ZFS on OMV

      Hi,

      what do you think about lz4 compression? Is it applicable to compress 1080p movies?

      What about the performance while streaming?

      Regards Hoppel
      ---------------------------------------------------------------------------------------------------------------
      frontend software - tvos | android tv | libreelec | win10 | kodi krypton
      frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
      -------------------------------------------
      backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
      backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
      ---------------------------------------------------------------------------------------------------------------------------------------

      The post was edited 1 time, last by hoppel118 ().

    • Hey guys,

      I need a little help with smb home directory and zfs snapshots. I want to use the windows shadow_copy but cant get it to work.
      Ive added a few lines to the extra options in the smb configuration (Screenshot 1) and added a line in the "30homes" config file of smb (Screenshot 2), but shadow copies dont show up in Windows. Any advise?
      Images
      • zfsconf.PNG

        52.46 kB, 1,465×721, viewed 211 times
      • zfsconf2.PNG

        44.98 kB, 661×866, viewed 191 times
    • k567890 wrote:

      By Volume do you mean "dataset"? What do you mean by Filesystem then? Is that "dataset"... Confusing.
      Also why do you need to specify size for volume... this shouldn't be required in zfs.


      Volumes in zfs are virtual block devices, like when you create loop device. YOu have to specify a size, same as when you create a loop device.
      Filesystems are just paths with in the pool but they have properties, can be "snapshoted" and probably can't transverse in between each other, someone might confirm this. For example winscp should not let you move a file in between two filesystems, same as happens in traditional linux fs
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • [HOWTO] Instal ZFS-Plugin & use ZFS on OMV

      ikogan wrote:

      I've been running with lz4...always, for years. I think that unless your server is a potato, you should be fine performance wise. In terms of compression ratio...I doubt you'll see much if the only thing on your filesystem is compressed video.


      I have an actual Xeon-CPU.

      What about streaming music or 1080p-movies from your zfs-filesystem? Do you recognize that the files must be decompressed before playback.
      ---------------------------------------------------------------------------------------------------------------
      frontend software - tvos | android tv | libreelec | win10 | kodi krypton
      frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
      -------------------------------------------
      backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
      backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
      ---------------------------------------------------------------------------------------------------------------------------------------
    • [HOWTO] Instal ZFS-Plugin & use ZFS on OMV

      Ok, perfekt. Thanks for the information. omv isn't doing it by default, so I manually have to configure it per command line.

      I read about transparent compression and the great performance of lz4, but the informations in the web about compression in combination with streaming are rare.
      ---------------------------------------------------------------------------------------------------------------
      frontend software - tvos | android tv | libreelec | win10 | kodi krypton
      frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
      -------------------------------------------
      backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
      backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
      ---------------------------------------------------------------------------------------------------------------------------------------
    • [HOWTO] Instal ZFS-Plugin & use ZFS on OMV

      Ok, how can I adjust the ratio?
      ---------------------------------------------------------------------------------------------------------------
      frontend software - tvos | android tv | libreelec | win10 | kodi krypton
      frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
      -------------------------------------------
      backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
      backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
      ---------------------------------------------------------------------------------------------------------------------------------------
    • You can't. zfs tries to compress everything. video and music really aren't compressible anymore than they currently are. The resulting size difference determines what ratio you are achieving.
      omv 4.1.13 arrakis | 64 bit | 4.15 proxmox kernel | omvextrasorg 4.1.13
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • hoppel118 wrote:

      what do you think about lz4 compression? Is it applicable to compress 1080p movies?


      You want to save some space? Get yourself a decent HEVC capable player and reencode your 1080p files from h.264 to h.265.

      Greetings
      David
      "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"

      Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.


      Upload Logfile via WebGUI/CLI
      #openmediavault on freenode IRC | German & English | GMT+1
      Absolutely no Support via PM!

      I host parts of the omv-extras.org Repository, the OpenMediaVault Live Demo and the pre-built PXE Images. If you want you can take part and help covering the costs by having a look at my profile page.
    • [HOWTO] Instal ZFS-Plugin & use ZFS on OMV

      Ok, so I don't need to activate the compression. 99% of my data is music and videos.

      Offtopic: Which h.264/h.265-encoder can you recommend?

      Thanks for all your answers. This really helps in setting up my first zfs.
      ---------------------------------------------------------------------------------------------------------------
      frontend software - tvos | android tv | libreelec | win10 | kodi krypton
      frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
      -------------------------------------------
      backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
      backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
      ---------------------------------------------------------------------------------------------------------------------------------------
    • hoppel118 wrote:

      Which h.264


      x264

      hoppel118 wrote:

      h.265-encoder can you recommend?



      x265 - x265.org/ - You can use handbrake to convert files.

      Greetings
      David
      "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"

      Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.


      Upload Logfile via WebGUI/CLI
      #openmediavault on freenode IRC | German & English | GMT+1
      Absolutely no Support via PM!

      I host parts of the omv-extras.org Repository, the OpenMediaVault Live Demo and the pre-built PXE Images. If you want you can take part and help covering the costs by having a look at my profile page.
    • davidh2k wrote:

      x265 - x265.org/ - You can use handbrake to convert files.


      Ok, thanks, I will check it. Never used handbrake before, but heard about it.

      For your information, maybe it's the wrong place to report bugs for beta software. While playing around with creating pools, I recognized that creating pools "by-id" doesn't work with the omv-webui. I had to create the pool with the command line, to get that to work. I am on omv 3.0.13, omvextrasorg 3.0.11 and use the stable "zfs" repository. Yeah, I know it's beta. I only want to inform you. Maybe it works as expected with the "zfs testing" repository.

      Greetings Hoppel
      ---------------------------------------------------------------------------------------------------------------
      frontend software - tvos | android tv | libreelec | win10 | kodi krypton
      frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
      -------------------------------------------
      backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
      backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
      ---------------------------------------------------------------------------------------------------------------------------------------
    • Hey guys,

      made some performance checks with lz4. Here are the results:

      I have got two empty filesystems. One (/mediatank/videosc) with lz4 compression enabled and one (/mediatank/videos) without compression.

      Source Code

      1. root@mediatank:/# zfs get compression mediatank/videosc
      2. NAME PROPERTY VALUE SOURCE
      3. mediatank/videosc compression lz4 local
      4. root@mediatank:/# zfs get compression mediatank/videos
      5. NAME PROPERTY VALUE SOURCE
      6. mediatank/videos compression off default


      First I built data with dd in the compressed filesystem:

      Source Code

      1. root@mediatank:/# dd if=/dev/zero of=disk1.img bs=1024000k count=10
      2. 10+0 Datensätze ein
      3. 10+0 Datensätze aus
      4. 10485760000 Bytes (10 GB) kopiert, 3,43668 s, 3,1 GB/s


      Now I do the same in the uncompressed filesystem:

      Source Code

      1. root@mediatank:/# dd if=/dev/zero of=disk1.img bs=1024000k count=10
      2. 10+0 Datensätze ein
      3. 10+0 Datensätze aus
      4. 10485760000 Bytes (10 GB) kopiert, 8,06394 s, 1,3 GB/s


      Building the data in the compressed filesystem is round about 5sec faster. =O

      Now I want to have a look at the size:

      Source Code

      1. root@mediatank:/# df -h /mediatank/videosc/
      2. Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
      3. mediatank/videosc 20T 256K 20T 1% /mediatank/videosc
      4. root@mediatank:/# df -h /mediatank/videos/
      5. Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
      6. mediatank/videos 20T 9,8G 20T 1% /mediatank/videos


      Source Code

      1. root@mediatank:/# df -k /mediatank/videosc/
      2. Dateisystem 1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
      3. mediatank/videosc 21435487232 256 21435486976 1% /mediatank/videosc
      4. root@mediatank:/# df -k /mediatank/videos/
      5. Dateisystem 1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
      6. mediatank/videos 21445735552 10248576 21435486976 1% /mediatank/videos


      The compression for this kind of data is really impressive. The size is nearly zero. 8o

      Now, I want to copy the same 1080p-movie (h.264) to both filesystems to compare the results:

      Source Code

      1. root@mediatank:/# df -h /mediatank/videosc/
      2. Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
      3. mediatank/videosc 20T 30G 20T 1% /mediatank/videosc
      4. root@mediatank:/# df -h /mediatank/videos/
      5. Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
      6. mediatank/videos 20T 31G 20T 1% /mediatank/videos


      Source Code

      1. root@mediatank:/# df -k /mediatank/videosc/
      2. Dateisystem 1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
      3. mediatank/videosc 21414228736 30552704 21383676032 1% /mediatank/videosc
      4. root@mediatank:/# df -k /mediatank/videos/
      5. Dateisystem 1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
      6. mediatank/videos 21415182976 31506944 21383676032 1% /mediatank/videos


      So, the compression decreases the file size nearly 1GB : 21415182976 - 21414228736 = 954240 That isn't half bad.

      A look at "top -u htpc" (htpc is my htpc-user) shows me a cpu usage by raound about 1% in both cases, streaming the compressed or streaming the uncompressed video over a samba share. So for the cpu it is not a problem.

      My compressionratio is also by about 1 %.

      Source Code

      1. root@mediatank:/# zfs get all mediatank/videosc | grep compressratio
      2. mediatank/videosc compressratio 1.02x -


      So, at the moment, for me it is an option to compress all my data with lz4.

      Can you recommend some other checks I can do, before copying my whole library to the zfs filesystem?

      Thanks and regards Hoppel
      ---------------------------------------------------------------------------------------------------------------
      frontend software - tvos | android tv | libreelec | win10 | kodi krypton
      frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
      -------------------------------------------
      backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
      backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
      ---------------------------------------------------------------------------------------------------------------------------------------

      The post was edited 2 times, last by hoppel118 ().

    • hoppel118 wrote:

      Building the data in the compressed filesystem is round about 5sec faster.


      Try with bigger, real, data. Not just a bunch of zeroes.

      hoppel118 wrote:

      The compression for this kind of data is really impressive. The size is nearly zero.


      Yeah. You just wrote a bunch of zeroes. Text and Numbers can achieve a really high amount of compression, unlike media files.

      Greetings
      David
      "Well... lately this forum has become support for everything except omv" [...] "And is like someone is banning Google from their browsers"

      Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.


      Upload Logfile via WebGUI/CLI
      #openmediavault on freenode IRC | German & English | GMT+1
      Absolutely no Support via PM!

      I host parts of the omv-extras.org Repository, the OpenMediaVault Live Demo and the pre-built PXE Images. If you want you can take part and help covering the costs by having a look at my profile page.
    • [HOWTO] Instal ZFS-Plugin & use ZFS on OMV

      Can you tell me a method to measure the throughput while copying files? With "cp" it is seemingly not possible.

      Regards
      ---------------------------------------------------------------------------------------------------------------
      frontend software - tvos | android tv | libreelec | win10 | kodi krypton
      frontend hardware - appletv 4k | nvidia shield tv | odroid c2 | yamaha rx-a1020 | quadral chromium style 5.1 | samsung le40-a789r2
      -------------------------------------------
      backend software - debian | openmediavault | latest backport kernel | zfs raid-z2 | docker | emby | unifi | vdr | tvheadend | fhem
      backend hardware - supermicro x11ssh-ctf | xeon E3-1240L-v5 | 64gb ecc | 8x10tb wd red | digital devices max s8
      ---------------------------------------------------------------------------------------------------------------------------------------