Borgbackup 'Archives' tab should be named 'Jobs' and we need a new 'Archives' tab

    • OMV 4.x
    • Borgbackup 'Archives' tab should be named 'Jobs' and we need a new 'Archives' tab

      Hi all folks.

      Looking desperately for a mechanism to save the data handled by OMV in Hetzner Storage Boxes (tried everything I could think of including rsync scripts but as no SSH commands no easy way to handle retention policies) I ended installing and using BorgBackup plugin that, at least, is doing the job.

      But the plugin falls short under my point of view. Yes, we can configure repos and launch backup tasks, but we have no other kind of information about the repos. For example we do not know how much space we are using in the repos, nor the compressed and deduped size they are taking. And lot of stuff must be done from the command line (mounting a given archive not the full repo for example).

      Regarding the actual 'Archives' tab, I think this shouls be named Jobs or Tasks because it defines the jobs (or taks) to get done not the archives that are created. The actual archives are listed when you click the 'List' button not when you go the the archives tab.

      And there should exist a new 'Archives' tab that listed all the archives that we have in each repo with its original, compressed and deduped size (at the very least) as we get from the info borg query. This listing should be in a collapsable panel hanging over the corresponding repo entry (the repo they belong to). And it is here where we should have the 'mount' command to mount a given archive and not the whole repo (the full mount still can be useful nonetheless) .

      So I tried to achieve something of this by myself. I managed to rename the 'Archives' tab (and all the related code) and have a functional 'Jobs' tab. This was easy as you only have to go to all the related files that someone else's has already worked out (and they are only in two places) so nothing to be proud of.

      Now comes the difficult part that is to build a whole new tab with the actual archives info. And this is where I am stuck. I do not know enough ExtJS nor OMV programming model to know how to make this collapsable listing nor to make the borg calls nor to fill up the data structures to hold the data to be presented. Mostly this should be what the 'List' button does for a repo but presented in its own tab.

      So I request your help. I have seen that you folks are quite helpful and supportive so may be that it could be possible to improve this plugin and make it truly useful to make it easy for an admin to publish an archive contents to a SMB share without needing to go SSH.

      The attached image shows some of the former concepts I have been talking about.

      And if this is not possible or of any interest for anyone thanks anyway for your time and for having developed such a good product for all of us.


      Miguel.

    • merill wrote:

      No one?
      When I created the plugin (that I don't actually) use, I tried to use the same labels that the documentation used. If they are wrong, it doesn't affect how the plugin works. I still don't think it is wrong. A compressed file filled with other files is usually called an archive. If the plugin needs more features, someone else will have to do that after I finish porting it to OMV 5.x.
      omv 5.1.2 usul | 64 bit | 5.3 proxmox kernel | omvextrasorg 5.1.7
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • Hi Ryecoaaron. Thanks for your reply.

      I am not complaining about your job. At all. Full the contrary I am quite grateful that you did it (even when you are not using it).

      I also know that the specific naming used does not affect the way the plugin works. It is only that, though an achive can be a grouping of other archives, in this case what we are configuring at the 'Archives' tab are not archives themselves but the jobs to create the archives. Probably semantics only that would not go any further if we could handle the archives themselves and their contents in a more granular way somewhere else.

      Mounting a full repository can be a mess. You may have tens or hundreds of archives inside a given repo and you end up with a mess of folders at the mount point that you cannot point the final user to. This is why I suggest to have a way to mount a single archive and, for that task, an archive tab would be fine.

      As for the more features, that is why I told that I was trying with the code. But I am an absolute nerd both regarding the framework used (I use a different one) and the object model developed in OMV (I have read the docs I have been able to find but even that no way man. Probably I'm getting old...).

      That is why I humbly request your help. I am not requesting that anyone else (like you that even does not use it) take all the task over his or her shoulders but, at least, help me with it until I get enough knowledge to go alone. I am starting with the simplest action that I can imagine that is to get the full, compressed and deduped size of the repo and present it in the repos screen and I am unable to do even this easy task (I am NOT making the call to borg right know just filling up an array and using it later but the numbers in the array do not get their way to the screen so far :( ) so I really need your help and I humbly request it if you can, and want, to.

      And, in the meantime, as you are the one who are porting it, it would be fine if we could agree over the final naming and deployment now that it is being ported to OMV5 to build a better plugin for this service. I am actually using it and it would be a great releaf if we could handle the individual archives from the plugin and not from the command line.

      Anyway thanks again for your time and hard work.
    • merill wrote:

      Probably semantics
      The bane of my existence... I work with many people who like to spend more time on semantics than just getting the job done. So, forgive me when I am touchy on this one.

      merill wrote:

      Mounting a full repository can be a mess. You may have tens or hundreds of archives inside a given repo and you end up with a mess of folders at the mount point that you cannot point the final user to. This is why I suggest to have a way to mount a single archive and, for that task, an archive tab would be fine.
      When originally creating this plugin (had no feedback or help), everything seemed fine since I had very small samples. From a code perspective, it would make sense (to me) to have another mount dialog on the repos tab that would you to select an archive and shared folder.

      merill wrote:

      I am starting with the simplest action that I can imagine that is to get the full, compressed and deduped size of the repo and present it in the repos screen and I am unable to do even this easy task
      That is probably easier to do now with the --json flag. I don't think that existed a couple of years ago. It would be nice to put those in the repo grid but it maybe too slow. I will have to try it I guess.

      merill wrote:

      And, in the meantime, as you are the one who are porting it, it would be fine if we could agree over the final naming and deployment now that it is being ported to OMV5 to build a better plugin for this service. I am actually using it and it would be a great releaf if we could handle the individual archives from the plugin and not from the command line.
      Just to warn you that even though I am porting it, I didn't want to rewrite the whole thing. So, just to itemize the things to change:

      • rename archives tab to jobs
      • have way to show original, compressed, deduped sizes
      • have way to mount archive instead of full repo
      omv 5.1.2 usul | 64 bit | 5.3 proxmox kernel | omvextrasorg 5.1.7
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • Hello back ryecoaaron. Thanks for your reply and sorry for this being a looong one.

      ryecoaaron wrote:

      So, forgive me when I am touchy on this one.
      You are plenty forgiven. Believe me :)

      ryeccoaaron wrote:

      When originally creating this plugin (had no feedback or help), everything seemed fine since I had very small samples. From a code perspective, it would make sense (to me) to have another mount dialog on the repos tab that would you to select an archive and shared folder.
      And you are quite right. It had a lot of sense and more if you had no data where to get an idea about the working of the tool from. I did not even kow about this tool a few months ago but having the chance to have an efficient backup storage at the price that the Hetzner Storage Boxes are (2TB for less than 10€/month) made me focus on this. I have to admit that it may not be the best backup tool that may be around but it is available from that cloud storage provider so.... And I think that if we are able to take this tool to the next step it may be quite useful for many others.

      Regarding the option to mount a given archive I think that option should be in its own tab, one that shows a listing of the existing backups (much the same as the one you have right now in the repos tab) where the user can see all the archives that he/she already has in the repo with it's stats (size, compressed size and deduped size), can select one and mount/umount it. As I am doing it now (by hand) I mount the repo in a given mount point (that could be requested in the mount dialog or configured in the jobs tab near he job definition. More on this later)

      ryecoaaron wrote:

      That is probably easier to do now with the --json flag. I don't think that existed a couple of years ago. It would be nice to put those in the repo grid but it maybe too slow. I will have to try it I guess.
      I swear I have tried all that I have imagined to show this data without success. And it loos like it should be not difficult. I am adding

      objectv->add('reposize', 'integer', 100); (and similar lines for the other fields that have been defined in the corresponding file just below the 'mounted' field)

      in the getRepoList funcion inside the foreach loop. This should be enough to push the data into the field but I get columns with '0' inside or with '--' if I use a render function.

      i do not know if the --json flag could make any difference as I even do not know where to use it. I even have not found a way to debug (show variable's contents) nor in the client area nor in the console log area. And have not found any info about how to achieve this either, so my debugging capabilities are very limited rigt now. How do you do it, man?

      You are right that getting the actual data from the repo using borg info may be slow. If it could be an asynchronous call that would be great. If not it would have to be decided if it is better to know how many space you are using out of your repo even if you have to wait some time or not. Prehaps an 'info' button giving this data would be enough if it takes too long to get that data on-line.


      ryecoaaron wrote:

      Just to warn you that even though I am porting it, I didn't want to rewrite the whole thing
      Fully understandable. I have managed to make the changes for the Archives tab (and ALL related elements as the structure in the config.xml file, etc.) to be named Jobs. Now ALL references to archive (unless those that make sense like getArchiveListing) are changed to Job (or Jobs depending the number). If you want I can send you all the files (OMV 4.x) that make this setup.

      I have creted an skeleton for the Archive tab too and started to change the repo code to insert (as told before) the repo stats but I am stuck on both tasks.

      The Archive tab as I foresee it should have a list of the repos (the same as the repo tab but with only the name and encryption fields) and, for each repo, a collapsable listing of all the archives that it holds (this could be very similar to what the List button offer now). Thus you get a complete view of your saved data in a single pane. Selecting one of the archives in one of the repos will enable the mount button and you can mount it in a given mount point. It could be fine if we could add a delete button too to drop a ginven archive from the repo.

      Now, after mounting by hand an archive of a repo, I soft-link (ln -s) the inner folder of the mounted archive (borg has the bad use to create a full tree folder structure with the name of the device and many other things before getting to the saved data. You cannot give that to the user because he/she does not know anything about how your disk device is named and alike. So I soft link this to a share that I have created for this, that I then publish to the user that have to get access to the backup. Now he/she has an entry in his windows explorer with the data where it can be restored the needed files. Easy for them, difficult for us as system admins. The old story of our lives...

      So in short, your points are correct but they may need some fancy touch just to take some of the wierdness of the tool out of the way :)

      And thanks for honouring me with your reply and interest. :thumbup:

      Kind regards.
    • merill wrote:

      i do not know if the --json flag could make any difference as I even do not know where to use it.
      The json flag only makes the output parseable. When enumerating the repos, a second call to borgbackup info /path/to/repo --json will have to be called and parsed before the fields can be added to the object. The receiving grid in javascript will also have to have the fields added. Otherwise, an info button (or something) could be added to request this info for a selected repo.

      merill wrote:

      The Archive tab as I foresee it should have a list of the repos (the same as the repo tab but with only the name and encryption fields) and, for each repo, a collapsable listing of all the archives that it holds (this could be very similar to what the List button offer now). Thus you get a complete view of your saved data in a single pane.
      The performance of this idea would degrade quickly the larger the number of archives in a repo and the larger the number of repos. The zfs plugin uses this idea and I'm not really a fan of the look or the work required to code it. Is there a reason to show them all in the grid versus just in a dropdown when one wants to mount one?

      And as for code changes, I don't want to touch the 4.x plugin. The porting to 5.x is about 75% done (I think).
      omv 5.1.2 usul | 64 bit | 5.3 proxmox kernel | omvextrasorg 5.1.7
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      When enumerating the repos, a second call to borgbackup info /path/to/repo --json
      Oh. I understand now. I didn't catch that you were refering to the borg call. Yes receiving the reply in JSON format will make parsing it a lot easier. The info button may also be the best option depending of the mean time that a call to borg to get it will take (not the same for a local or a remote repo)

      ryecoaaron wrote:

      The performance of this idea would degrade quickly the larger the number of archives in a repo and the larger the number of repos
      Completely agree. There is no special reason to select this mechanism (a grid or table) except it is the traditional way that backup front-ends present the history of finished jobs. Anyway reading any of them just to fill the drop-down will take time too so I also do not see any gain of having them in that kind of control versus a collapsable grid/panel/table. The expected performace should be the same.

      I am not proposing to present all the panes for all the repos expanded. The idea is to only, initially, show the list of repos (that we have in the local config file) the same that it is done in the repos tab and read, fill and present the list of archives of a given repo only when requested by the user by means of the expand icon (much like the dropdown).

      We probably we are speaking about the same. In fact this could be marged with the repos tab but having its own tab makes things a bit more ordered. We already have it in the listing button. The only thing with this ist is that it is difficult to read. The proposed archives tab is a merge of the repos+list button with some fancy view.

      ryecoaaron wrote:

      And as for code changes, I don't want to touch the 4.x plugin
      I understand. I do not know if porting to next version is too hard or not. Taking into account that I am not able neither to display a single constant in the panel I doubt that I could be of any help with that but I will keep trying a bit more to make this work in 4.x if only for not saying that I surrendered too early. Though, for now, OMV is goaling me out by far. :(
    • merill wrote:

      Anyway reading any of them just to fill the drop-down will take time too so I also do not see any gain of having them in that kind of control versus a collapsable grid/panel/table. The expected performace should be the same.
      There is a huge code difference though.

      merill wrote:

      The proposed archives tab is a merge of the repos+list button with some fancy view.
      The current List button could be improved. I think I should finish the porting and see where it can go after that.
      omv 5.1.2 usul | 64 bit | 5.3 proxmox kernel | omvextrasorg 5.1.7
      omv-extras.org plugins source code and issue tracker - github

      Please read this before posting a question and this and this for docker questions.
      Please don't PM for support... Too many PMs!