Posts by Zoki

    Interestingly enough the order in which apt removes the packages is strange:


    Let's see how far we can get: Execute manually, what the postrm file would have done, remove the postrm file and restart dist-upgrade


    dpkg-trigger update-fixperms

    rm /var/lib/dpkg/info/openmediavault-backup.postrm

    and this all in one step:

    Code
    apt-get --yes --force-yes --fix-missing --auto-remove --allow-unauthenticated \
    --show-upgraded --option Dpkg::Options::="--force-confdef" \
    --option DPkg::Options::="--force-confold" --no-install-recommends \
    dist-upgrade

    Thats right, looks very similair. There are to many layers between the disk writes and the processes initiating the writes.

    You will need to take a different route to find the culprit.


    My understanding is, som process is writing some data using async write), zfs collects the write requests and then write them to disk. First to the log and then to the other devices.


    I give up.

    -d expects a device string like 8.0, ...


    use lsblk to find it:


    In my case the device string for sdb is 8.16

    Execute this as root, one line at a time. If one line gives an error, report back and do not continue.

    Code
    cd
    rm /usr/share/openmediavault/release-upgrade/pre.d/05-install-usrmerge
    rm /usr/share/openmediavault/release-upgrade/pre.d/10-*
    rm /usr/share/openmediavault/release-upgrade/pre.d/20-apt-update-source-list
    sed -i -e '/apt-get update/s/^/#/' $(which omv-release-upgrade)
    set -x
    omv-release-upgrade | tee -a omv-release-upgrade.log


    The last thing it does is ask you to reboot and you will have a log file in /root

    Found it. Tried it, but this is insane, I don't know what am I suppose to figure out from this mess?!


    At least you now by now that txg_sync is not writing at all. (only Read).


    run iosnoop -st -d <device id of the divice with the sound> 10


    This will give you the IO on the device by program within 10 sec.

    It looks like it faild after 20-apt-update-source-list


    Do you have a backup of the os disk? If not, it is time to create one now, before we proceed.


    If someone knowlegable is reading here, please comment on this:

    • Current state is, omv-release-upgrade bailed out after changing sources.list (pre.d/20-apt-update-source-list)
    • System has been rebooted since that
    • Proposal is to
      • delete all run pre.d scripts 05-*, 10-*, 20-apt-update-source-list
      • delete line apt-get update in omv-release-upgrade before the runparts pre.d
      • execute omv-release-upgrae | tee upgrade.log
    • Hope for the best

    Post the result of ls -l /usr/share/openmediavault/release-upgrade/, ls -l /usr/share/openmediavault/release-upgrade/pre.d and ls -l /usr/share/openmediavault/release-upgrade/post.d to see if the upgrade scripts are still there.


    Do not reboot, because the omv-release-upgrade script keeps a copy of these scripts in /tmp., just in case we need them

    With the volumes only having the "right side" (missing colon) you created anonymous volumes inside <docker-root>/volumes.

    Shoul should clean up the volumes.


    In portainer go to volumes and delete the ones you do not need, most probably the ones with the long numbers.

    In swarm you need remote volumes too.


    Here are the docs: https://docs.docker.com/storag…#create-cifssamba-volumes


    it is as simple as this (create a volume names cif-volume using the share //uxxxxx.your-server.de/backup)

    Code
    docker volume create \
    --driver local \
    --opt type=cifs \
    --opt device=//uxxxxx.your-server.de/backup \
    --opt o=addr=uxxxxx.your-server.de,username=uxxxxxxx,password=*****,file_mode=0777,dir_mode=0777 \
    --name cif-volume

    Test it with:


    docker run --rm -it -v cif-volume:/mnt ubuntu ls -l /mnt

    How to find that out? I found out 4 dockers which make those syncs every 5 seconds, but have no idea what they are doing. For example, Handbrake docker is not suppose to do anything at all until I give it some video to compress. But it makes those syncs to show up.

    Did you check if the containers write logs or have a healthcheck in the container?


    Disk I/O is caused by a system call, so you can [tt]strace[/tt ]the programs running in the containers: Here is a howto: https://rothgar.medium.com/how…te-container-983f11740dc6


    Basically, you build a container with strace installed, run it witin the same namespaces als the container to debugg and strace -a to the progroam yo uare going to analyze.


    The simpler way would be to install strace inside he containers (if they have a shell and a package manager installed)