OMV docker compose, slow?

  • I recently installed OMV7 and decided to give a chance to docker-compose plugin, this should replace my bunch of directories and scripts for few services.

    Right now I managed to fire up two examples (uptime kuma and nextcloud), but so far failed with my own service, it needs some more work, fiddling with compose files and docker build, this is ok,

    But the thing that stops me now is the fact that each compose edit is terribly slow to save, I'm changing something that is not important, like adding #comment and whole operation takes about 25s. This instance is not much used yet, less than <5% of CPU on idle. When applying change htop show me 100% of one of cores for command "python3 /sbin/omv-salt deploy run --no-color compose", when I run this on ssh it took about 9s to do "nothing", but it also returns some actions and durations. The thing that catch my attention is function file.managed, it takes about 750ms for each to complete (and there is plenty of them). Comments are mostly like "File /etc/cron.d/omv-compose-backup is in the correct state"


    What can cause such issue? I already changed IP to static and turned off one ethernet interface,

    This is small, power efficient system, but probably it should be faster than this. It's very hard to edit those compose files and wait that long for each save and apply. Any suggestions?

    • Official Post

    the thing that stops me now is the fact that each compose edit is terribly slow to save, I'm changing something that is not important, like adding #comment and whole operation takes about 25s

    People complained about having to save then apply the changes. So, the plugin is doing both when saving. It is slow because it is doing a saltstack run. This makes sure the files on the filesystem match what is in the database.

    When applying change htop show me 100% of one of cores for command "python3 /sbin/omv-salt deploy run --no-color compose", when I run this on ssh it took about 9s to do "nothing", but it also returns some actions and durations. The thing that catch my attention is function file.managed, it takes about 750ms for each to complete (and there is plenty of them). Comments are mostly like "File /etc/cron.d/omv-compose-backup is in the correct state"

    Running a command through php is going to be slower (especially on a slow system). Saltstack is configuration management. It is ensuring that the files the code owns are in the proper state. All of OMV does this.


    It's very hard to edit those compose files and wait that long for each save and apply

    It is hard to make a plugin do everything people want. People complain no matter which way the plugin works. I will look to see if I can make the apply have to be manually done but the saltstack code has to be executed at some point to write the database changes to the filesystem.

    omv 7.4.10-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.14 | k8s 7.3.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.9


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • People complained about having to save then apply the changes. So, the plugin is doing both when saving. It is slow because it is doing a saltstack run. This makes sure the files on the filesystem match what is in the database.

    I'm trying to understand the process,

    I noticed that it's generating some variables, comments etc. but not apply those changes (there is another button for that)

    Also comparing project with DB should not take that long,

    What is saltstack? Can I somehow benchmark it?

    Running a command through php is going to be slower (especially on a slow system). Saltstack is configuration management. It is ensuring that the files the code owns are in the proper state. All of OMV does this.

    I know PHP would have some overhead, but I don't expect it will make that difference unless something is really wrong.

    What is input for saltstack? Just filename on filesystem to check? Is it comparing whole content with the one from DB?


    It is hard to make a plugin do everything people want. People complain no matter which way the plugin works. I will look to see if I can make the apply have to be manually done but the saltstack code has to be executed at some point to write the database changes to the filesystem.

    For now I think that I'm just editing one docker-compose file, maybe with some variables and some extra integration with omv. There can be of course something I don't know, but for now I see effect - about minute to make change. If You need any tests - let me know, I'm sure that something can be done faster here. I saw new update for plugin today, but it's something different.


    For saltstack - maybe I have something wrong on my instance? It's ARM SoC with 8G of RAM, 2.5G nic, quad A55 and SSDs on JMB585 via two pcie3 lanes. Nothing super fast, but it should not be that laggy on quite simple operation. Any suggestions what to check here?

  • The reason is simply that it is going via saltstack. It is never going to be as fast as simply creating a text file and done. So not the best choice to try, test, develop stacks with it. I‘m doing my stack development in the cli. Docker compose plug-in for production use only. (Or when I know there is no heavy docker editing involved)

  • The reason is simply that it is going via saltstack.

    @ryecoaaron mentioned that, he will also check if saving that files need all of that.

    I'm now reding about saltstack and need to figure out if everything works there correctly so it's not taking more time than needed.


    I‘m doing my stack development in the cli.

    Are You editing files that plugin use? (those with comment to not do that?) or doing that on different instance?

    I'm new in this plugin functionality, what extra stuff is hidden in this plugin compared to usual docker compose?

    I need to understand how to prepare those.

  • Saltstack making things slow due to massive overhead is well known and probably nothing I would deep dive into. Unless it takes reeeeally long, like 5 minutes or so.


    By developing in cli I mean to completely avoid OMV / Compose plugin / Saltstack. Just ssh into the machine, create a folder on a storage ssd named eg „docker-testing“. Create your text file docker-compose.yml where you put what to test. Then exec ‚docker-compose up‘. (That is basically what the plug-in is doing too, but as said, very slow because of saltstack.. And yes, you should NOT go into the plug-in’s files via ssh and edit them - because the files would not match anymore with the info in saltstack)


    When the yml file is complete and working I import it via the compose plugin. When you became more firm with docker and you know what you are doing you will not need the manual way that often anymore. But at the beginning when you’re editing a yml very often it’s just a way faster workflow.

    • Official Post

    I deal with configuration management (puppet, ansible, saltstack) all day long. None of them are fast. I also do a lot of dev where I have to commit to git and then wait for a CI runner to finish. So, I guess to me, I don't even notice the delay and I have done lots of compose file dev in the plugin. For how often most users do rapid changes in a compose file, it isn't worth completely rewriting the plugin to not use saltstack. I don't have time for that anyway.

    omv 7.4.10-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.14 | k8s 7.3.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.9


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • I mostly agree. I can remember when I was a Docker beginner who had to fiddle around a lot.. Or setting up a bit more complex network environment while not yet familiar with dockers networking concepts. U look into the docker docs and build your config step by step, test every step if sth does not work as expected… That’s probably stuff I would still do manually.


    But for daily use the plug-in is perfect and the delays totally acceptable.

    • Official Post

    On my primary server (older xeon e5-2697v3), I have 10 docker containers on it. editing and saving a file in the Files tab takes 5-6 secs.

    omv 7.4.10-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.14 | k8s 7.3.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.9


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Official Post

    Compared to 0. when doing it manually.

    Your 0 second save is doing far less. The saltstack is making sure every file in the database is written to file on the filesystem and has the correct content and permissions. It is making sure your docker install has the correct packages installed and config file written. It is making sure the cron job for backups is correct.


    The save & apply speed is the only part of the plugin that is slow. Everything else is faster than you can do from the command line or editor.

    omv 7.4.10-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.14 | k8s 7.3.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.9


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • For me it's just unclear why "simple edit" takes that long. "Stop" "Start" operations are far more complex and we got docker compose output immediately.


    On my quad A55 instance this takes about minute to complete. This makes it really uncomfortable to work with so I will rather not use OMV via GUI but just ssh to box and work on docker-compose there and be forced to sync changes via GUI later.


    ryecoaaron - Your main argument is that this is rare operation and it's faster on Your side. On mine it's too long to work on that. On ARM many external docker images are missing arm64 build and some of those that has them simply don't work or are not correctly maintained. It's not that easy to just fire up any compose file from network.

    he saltstack is making sure every file in the database is written to file on the filesystem and has the correct content and permissions. It is making sure your docker install has the correct packages installed and config file written. It is making sure the cron job for backups is correct.

    Do we really need that on each docker compose file save? Could we just get some "quick save" option without complex sync to service that has not changed between project edits?

    I mostly agree. I can remember when I was a Docker beginner who had to fiddle around a lot.. Or setting up a bit more complex network environment while not yet familiar with dockers networking concepts. U look into the docker docs and build your config step by step, test every step if sth does not work as expected… That’s probably stuff I would still do manually.

    I've been using docker for many years now, but docker on ARM with limited resources is quite new for me and usually projects are not going that well as I want. We we need to check docker-everything on each project save? This really forces me to do those manually and then sync back.

  • Your 0 second save is doing far less. The saltstack is making sure every file in the database is written to file on the filesystem and has the correct content and permissions. It is making sure your docker install has the correct packages installed and config file written. It is making sure the cron job for backups is correct.


    The save & apply speed is the only part of the plugin that is slow. Everything else is faster than you can do from the command line or editor.


    I know I know, that is basically what I’m trying to explain to OP the whole time lol 😅


    All I say is for some kinds of workflow and setup it is simply taking too long - so one may prefer doing it manually.

  • On my server (Intel J5005) there are about 20 containers. Saving a file via plugin takes around 15 seconds. Compared to 0. when doing it manually.

    I also have one of those with wyse5070, this one is running as k8s node so I don't need to downgrade there to docker-compose ;)

    Mine instance is quad A55, for now saving docker compose project file takes about minute, on command line it's about 45s. AS I mentioned x86 is much easier in this area to fire up such projects, but I decided for ARM because of power efficiency and it's great at that area.


    I think that many people choose OMV because there is no good NAS software alternative on ARM. This includes all raspberry pi, radxa, orange pi and many other great boards. All of them are much more difficult to set up and fire some docker stuff, therefore we eighter encourage users to use OMV plugin and this interface or force them to work via ssh and then sync. With limited resources it's just important to do whatever is needed versus everything that is possible, but takes too much time for simple tasks.

    • Official Post

    takes too much time for simple tasks.

    One task! Saving the file is the only task that takes longer. This is not something most people do every day. Literally every other task is faster. Even if you do 10 edits on 5 files and each save takes 20 secs, this is going to be around 15mins. This is an extreme amount of edits and unlikely that any user does this.. More like 5 edits on 2 files which is about 3 mins of their time. Forgive if I don't feel bad about because I have put hundreds of hours into this plugin. I am not going to completely rewrite the plugin taking dozens of hours to be very non-OMV like to save a handful of users 3 mins. These few extra minutes don't detract from OMV as an NAS on any board. People can write their own plugin if they can make things so much better.

    omv 7.4.10-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.14 | k8s 7.3.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.9


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • One task! Saving the file is the only task that takes longer. This is not something most people do every day. Literally every other task is faster.

    I'm not arguing with statistics - most probably will be happy with examples and don't need to develop their own custom services and build own images. It's true that this action is probably the only one that takes so much time in plugin also making it not that good for development.


    Even if you do 10 edits on 5 files and each save takes 20 secs, this is going to be around 15mins. This is an extreme amount of edits and unlikely that any user does this.. More like 5 edits on 2 files which is about 3 mins of their time.

    I told You that at best it takes 45s for each save on my instance. Maybe most users have much better resources and they don't care that much also having part of that duration, but still why not optimize this part or just split into two separate actions?


    Forgive if I don't feel bad about because I have put hundreds of hours into this plugin. I am not going to completely rewrite the plugin taking dozens of hours to be very non-OMV like to save a handful of users 3 mins. These few extra minutes don't detract from OMV as an NAS on any board. People can write their own plugin if they can make things so much better.

    I'm not blaming You in any way, just asking what's there and if it's really needed. Let's come back to this comment:

    The saltstack is making sure every file in the database is written to file on the filesystem and has the correct content and permissions. It is making sure your docker install has the correct packages installed and config file written. It is making sure the cron job for backups is correct.

    And You also said:

    it isn't worth completely rewriting the plugin to not use saltstack. I don't have time for that anyway.

    Why You concluded that saltstack is needed on such project save?

    Why it's OMV-like and why all those are that needed there?

    Can't saltstack simply save this particular file without checking docker over and over again on each save?


    What is the reason to not put yet another button that do exactly that and allow to simply fast-save project?


    Of course maybe I don't fully understand this process yet, why it needs to be like it is, please don't be offended by those questions. It's not just reasonable for me why here we need to get all compose changed applied at once. On Docker file section it's just saving quickly dockerfiles so at least its not consistent. Why not do the same on compose project files?

    For saltstack - I don't demanded at any stage to replace that. Rather I don't understand why button labelled "save" on project needs to do so much.

    • Official Post

    Why You concluded that saltstack is needed on such project save?

    When OMV writes files on the filesystem, it uses configuration management to write those files. Saltstack is the choice to do that. People just assume everything is created like the directory structure and that permissions are correct and other factors. Saltstack makes sure all of that is done so you don't have problems. Your editors or command line stuff is not doing that. You just are focus on compose file saves where the plugin and OMV tries to do everything the same way. Maybe saltstack isn't the fastest but it is the most correct.

    Can't saltstack simply save this particular file without checking docker over and over again on each save?

    That misses the point of configuration management. It is making sure everything is correct. Speed is not the primary concern of configuration management.


    What is the reason to not put yet another button that do exactly that and allow to simply fast-save project?

    I just made a change that will write the changes to a compose file directly to the file when EDITING the file not CREATING the file. Creating the file the first time will still use saltsack because it has to make sure many things are in place. When EDITING a file, the plugin also won't clear the cache that speeds up loading the entries in the datatable. This is as fast as I can make the plugin without removing information from the files tab. I don't like it since I have to maintain code in two places. It will be in next release. If it still isn't fast enough, well...

    On Docker file section it's just saving quickly dockerfiles so at least its not consistent. Why not do the same on compose project files?

    The Dockerfiles section does just save to the database BUT you still have to click the Apply button (which runs saltsack) to perform any docker command on those changes. The changes I just made are actually better than that since they are writing to both the database and file directly.

    omv 7.4.10-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.14 | k8s 7.3.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.9


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    • Official Post

    7.2.12 is in the repo.

    omv 7.4.10-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.14 | k8s 7.3.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.9


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Edited once, last by ryecoaaron ().

  • That misses the point of configuration management. It is making sure everything is correct. Speed is not the primary concern of configuration management.

    I completely agree - there is good reason to use system like this. But we need to use it wisely and choose between long setup and quick change. Both should be possible and clear to user why he wait so long (of course on my super power efficient system ;))


    --


    I just checked new version and and it's TOTALLY different experience than before.

    Now saving project file takes less than 1s and I can clearly see that changes are ready to use, i.e for quick port change etc.

    This changes everything on this instance and I don't need to find other methods like plain ssh to work on those now.


    The only thing that I think may be worth to add is some kind of check before save or some indication that project file may be broken. I made typo for test, clicked on save button, it changed status from "Stop" to "Down", of course I could click on "check configuration" and get right error message. For usability I think that it's much more usable on edit page. Syntax is colored, but obvious errors are not found. Having that on one page and knowing before edit finish would be just much more valuable.


    The Dockerfiles section does just save to the database BUT you still have to click the Apply button (which runs saltsack) to perform any docker command on those changes. The changes I just made are actually better than that since they are writing to both the database and file directly.

    I just checked it, modified one of docker files and change was reflected but there was no "apply" message.

    Played a bit with those and noticed what You are talking about and the need for saltstack stuff.


    Test scenario was:

    • edit already added dockerfile (test1) and build it, this was fast and easy, no apply dialog
    • added test2 docker file, save (it was quick), but notification about apply changes shows up
      • I wanted to see if I can build that file - there was error
      • I applied changes and built it - it was ok

    I assume that it will work the same as on compose files now.

    So saltstack is needed to setup file and all needed stuff, then everything should work until something unexpected happened.


    Now I better understand how and why it's working like that, but my suggestion is to make it bit more consistent and clear:

    1. I don't see big difference between compose and docker files (or any other objects here), I think that all of them should be similar in this aspect - all of them should display apply or not.
    2. Saving compose file now applies changes, while this of course may be needed (test scenario from docker file!) it would be much better for users to insert right information about that. "save" if confusing and it's different than later when file is edited. If this would be relabelled as "setup docker env and create" or just added information that it need to apply many changes on creation, then this will be much more intuitive.


    (of course when we don't have project yet, probably would be not possible to check syntax before that - my earlier note about edits)


    Please don't be offended byt those notes, I'm aware that those require time and development, the reason why I'm writing about those is to just make this plugin better (and of course to understand how and why it works like this). With last change it improved a lot and now those are just details that I believe would make it just better. If You would like to get more comments like this let me know, I can try to make my notes while working with it :) Thanks for quick change - it really helped a lot! :)

    • Official Post

    The only thing that I think may be worth to add is some kind of check before save or some indication that project file may be broken.

    I do not want to add that because to do that would require throwing an error if docker compose check failed. If an error is thrown, the user cannot leave the editor screen. And this would slow the save down. People can click the provided check button.


    modified one of docker files and change was reflected but there was no "apply" message.

    It is not needed when just modifying a Dockerfile because the change is already made.


    edit already added dockerfile (test1) and build it, this was fast and easy, no apply dialog

    As expected.


    added test2 docker file, save (it was quick), but notification about apply changes shows up

    I wanted to see if I can build that file - there was error
    I applied changes and built it - it was ok

    You have always had to apply changes after adding a new dockerfile. This has not changed. A new dockerifle does not do the instant write Perhaps I should for the user to apply but this isn't new behavior. In general, you should not try to run actions when the apply banner is showing.


    I assume that it will work the same as on compose files now.

    No. Compose files run the saltstack code when creating a new compose file. They only do the instant write when editing the file.

    I don't see big difference between compose and docker files (or any other objects here), I think that all of them should be similar in this aspect - all of them should display apply or not.

    I will make new Dockerfiles apply changes when creating new. Then they will be the same as compose.


    Saving compose file now applies changes, while this of course may be needed (test scenario from docker file!) it would be much better for users to insert right information about that. "save" if confusing and it's different than later when file is edited. If this would be relabelled as "setup docker env and create" or just added information that it need to apply many changes on creation, then this will be much more intuitive.

    Saving compose files has always applied changed. It is just being done from php instead of saltstack when editing files. New files has not changed at all. I don't understand the rest of your point. I am not putting a ton of documentation in the plugin. It belongs in the guide if it is relevant. But the behavior has not changed. It is just faster. 99% of OMV users don't care how it works. If I add the auto apply when creating a new dockerfile, there will be no scenario like you hit. And realize very few people use the Dockerfile tab.

    omv 7.4.10-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.14 | k8s 7.3.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.9


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!