RSync task RFE's

  • After having used OMV on two NAS systems for a few months now, I can say that I'm quite happy with how it's all worked out. Thank you for your effort. The life of two NAS systems that were no longer supported has been prolonged this way.


    One thing high on my list of things that could be improved a bit are RSync tasks. Which I'd like to address here now.


    1. known hosts handling is a bit cumbersome. When you run a task for the first time, it is likely to fail with a "Host key verification failed" error because the key of the system you connect to is not known yet. The same issue will occur when e.g. the IP address of a known host has changed. The only way to add new keys to the list of known host is to either add it manually to `/root/.ssh/known_hosts` or make an ssh connection through ssh manually. Either way, it can not be done through the web UI.
    2. A task run log would be highly appreciated. To be able to see when a task has run and if it ran correctly. At the moment, there is no way of knowing if all went well for automated tasks, nor what the issues were when things didn't go as planned.


    For the first point, ideally it should:

    • Perform a test connection when creating a new task or when altering an existing on. Or at least offer to perform a test of not done automatically.
    • Offer to add/accept new keys through the UI and store/replace them in the known_hosts file

    An alternative would be to accept every key automatically, but I'm less comfortable with that from a security point of view. It could be an optional service though.


    For the second point, a global log viewer could be introduced to include other automated task logs as well. I do would like to be able to access/filter the logs of a specific task from its definition though.


    Any thoughts?

    • Official Post

    A task run log would be highly appreciated. To be able to see when a task has run and if it ran correctly. At the moment, there is no way of knowing if all went well for automated tasks, nor what the issues were when things didn't go as planned.

    rsync runs are logged to /var/log/rsync.log and you can view that in the log viewer (Diagnostics -> System Logs -> Logs). You could always have it email you as well.

    For the second point, a global log viewer could be introduced to include other automated task logs as well. I do would like to be able to access/filter the logs of a specific task from its definition though.

    Something other than Diagnostics -> System Logs -> Logs?

    known hosts handling is a bit cumbersome. When you run a task for the first time, it is likely to fail with a "Host key verification failed" error because the key of the system you connect to is not known yet. The same issue will occur when e.g. the IP address of a known host has changed. The only way to add new keys to the list of known host is to either add it manually to `/root/.ssh/known_hosts` or make an ssh connection through ssh manually. Either way, it can not be done through the web UI.

    Please file an issue here - https://github.com/openmediavault/openmediavault/issues

    omv 7.1.1-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.1 | k8s 7.1.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.8


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • rsync runs are logged to /var/log/rsync.log and you can view that in the log viewer (Diagnostics -> System Logs -> Logs). You could always have it email you as well.

    Something other than Diagnostics -> System Logs -> Logs?

    Those are individual log messages. I'd like to see logs per run, all the messages of a single run grouped together, with some form of colour or icon for each to quickly see if that run passed or failed.

    Access to logs per run from where the tasks are defined I mean. Either as a window on that page or a link to a global log viewer that automatically applies a filter for a specific task definition.


    I also noticed lots of empty log entries in the existing log viewer after a task run:

    • Official Post

    Those are individual log messages. I'd like to see logs per run, all the messages of a single run grouped together, with some form of colour or icon for each to quickly see if that run passed or failed.

    Access to logs per run from where the tasks are defined I mean. Either as a window on that page or a link to a global log viewer that automatically applies a filter for a specific task definition.

    File an issue then. That would be a big change since all runs are stored in one log file and there is no component in OMV currently that would do the colorizing. Even things like Splunk show individual messages and don't show colorized log files. You can get the log of each run in email now though.

    I also noticed lots of empty log entries in the existing log viewer after a task run:

    rsync is being told to log to that file. Either rsync is logging a blank line OR the log line doesn't conform the viewer's regex (which is very open - https://github.com/openmediava…nc/90logfilespec.inc#L235).

    omv 7.1.1-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.1 | k8s 7.1.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.8


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

  • rsync is being told to log to that file. Either rsync is logging a blank line OR the log line doesn't conform the viewer's regex (which is very open - https://github.com/openmediava…nc/90logfilespec.inc#L235).

    There's an issue with log messages starting with the '<' character. From /var/log/rsync.log :


    Code
    ...
    2024/04/23 14:57:11 [615691] <f..tp..... fjord/fjord/06. Let's Degenerate.mp3
    2024/04/23 14:57:11 [615691] <f..tp..... fjord/fjord/07. Sign Up.mp3
    2024/04/23 14:57:11 [615691] .d..tp..... fjord/fjord/artwork/
    2024/04/23 14:57:11 [615691] <f..tp..... fjord/fjord/artwork/Front.jpg
    2024/04/23 14:57:12 [615691] sent 310,437,640 bytes  received 333,749,287 bytes  126,646.40 bytes/sec
    2024/04/23 14:57:12 [615691] total size is 153,038,554,884  speedup is 237.57
    2024/04/23 14:57:12 [615691] sent 310437645 bytes  received 333749292 bytes  total size 153038554884
    2024/04/23 14:57:12 [615691] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]


    The regex is fine, it correctly extracts the message. But it looks like the starting character is preventing the message from being processed correctly later on.


    The 'span' element in the UI where the text should be is left empty:


    Not that those truncted bits before each file in the messages provide much info though.

    • Official Post

    But it looks like the starting character is preventing the message from being processed correctly later on.


    The 'span' element in the UI where the text should be is left empty:

    Interesting. Probably another issue for that.

    omv 7.1.1-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.14 | compose 7.2.1 | k8s 7.1.1-1 | cputemp 7.0.2 | mergerfs 7.0.5 | scripts 7.0.8


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!