Help requested - Docker/portainer issue - possibly permissions issue? Error 400

1st Official Post

    Hello, thanks in advance for reading and I have done a search of forums/google to see if I can solve this on my own but haven't had any luck.


    OMV Version: 6.3.10-2 (Shaitan)

    Kernel: Linux 5.19.17-2-pve

    Poratainer version: 2.18.1 (community)

    Docker version: 5:23.0.4-1~debian.11~bullseye


    It all started when I decided to install a handbrake container to try and automatically transcode new Plex DVR recordings. I installed the container and started playing around with it. *NOTE: I forgot to add the normal PUID/PGID I use for other containers*. I found that handbrake was automatically transcoding all kinds of other video files rather than the ones I wanted. I'm sure this is due to the settings and the watch folder being my entire TV library. I'm dumb. I quickly shut down the handbrake container, and figured I'd do some research and come back to it at some point in the future. That's when issues started cropping up...


    The first sign of trouble, was when I restarted my Plex container, it showed no TV shows, only movies/music/etc. I thought this must be related to the handbrake issue, as handbrake had access to the TV folder, but not the movies. I figured this could have been a permissions issue. Perhaps handbrake changed the permissions to the TV folder and now the Plex container couldn't access it? I used sudo chmod 777 on the tv folder which seemed to fix the issue. Plex could once again see my shows and started rebuilding my library.


    I also used the Reset Perms plugin for OMV to try and fix things on a larger scale. I fear this may have made things worse. I can't say for sure, but this may have also changed permissions on my docker config folders.


    Where I am now is that docker and portainer are the only things running. Any time I try to restart/redeploy and of my other containers in Portainer, I get an error 400. From what I've read, error 400 can be caused by a multitude of issues.


    I've tried uninstalling and re-installing both docker and portainer. They are still the only things that will run. Any additional containers I've added are giving error 400 (I think I also saw an error 500 at some point).


    Aside from a permissions issue, the only other thing I can think is some kind of networking issue? But I haven't fiddled with anything there.


    Attached below is a brief log of transmission that might shed some light. I've noticed a similar line in logs from other containers "s6-svscan: warning: unable to spawn .s6-svscan/SIGTERM: Permission denied"



    My Build:

    HP Z2 G4 - C246 Chipset w/ ECC Support

    Intel Xeon E-2124G Quad Core 3.4 GHz with QuickSync

    64GB ECC DDR4-2666 (Unregistered)

    500GB WD Blue NVMe for OMV install and separate partition for Docker containers

    3x 12TB Toshiba MG07 in RAIDZ1 ZFS

    OMV 7.4.x PVE Kernel 6.8.x

  • KM0201

    Approved the thread.
    • Official Post

    Try resetting your folder permissions again with resetperms. This time do it so that they are left with 775. Owner root, group users. Make sure the remove ACL button is checked.

    Make sure the user running docker is in the users group.

    Quote from chente

    Try resetting your folder permissions again with resetperms. This time do it so that they are left with 775. Owner root, group users. Make sure the remove ACL button is checked.

    Make sure the user running docker is in the users group.

    Thank you, I think this is the solution, but what folder do I need to reset? I tried this on my container config files and still no luck. I rebooted the system and portainer now won't start either, but docker is running. I can get portainer to work by uninstalling and re-installing, But I think I need to do this at every reboot.


    Was this caused by giving the handbrake container admin/root priveleges by not setting the PGID/PUID? I have a specific user set up for docker called "dockeruser". with PUID 1001.


    Here's another error code from when I try to start a container via CLI:

    Code
    Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/init": permission denied: unknown
Error: failed to start containers: Sickchill


    At this point, I'm afraid to make things worse =O

    My Build:

    HP Z2 G4 - C246 Chipset w/ ECC Support

    Intel Xeon E-2124G Quad Core 3.4 GHz with QuickSync

    64GB ECC DDR4-2666 (Unregistered)

    500GB WD Blue NVMe for OMV install and separate partition for Docker containers

    3x 12TB Toshiba MG07 in RAIDZ1 ZFS

    OMV 7.4.x PVE Kernel 6.8.x

    • Official Post
    Quote from corzocone

    At this point, I'm afraid to make things worse

    I think you have a little trouble set up. If your containers are implemented using compose (or portainer) with a stack (yml configuration file) the easiest thing to do would be to start from scratch. Launching the containers is child's play.


    So I would do the following.


    Uninstall docker and portainer. Delete the docker working files, NOT the container configuration volumes, but the files in the folder where docker is installed. If you didn't change it, it will be /var/lib/docker. If you changed it, it will be another. Make sure it's empty.


    Reset permissions of container configuration volumes to standard root user 775 permissions. Generally this should not be a problem, although there are containers that do not support this.


    Make sure that in all your stacks you have defined the user that runs docker. Make sure that user has permission to access all volumes defined in the stacks. And that this user belongs to the docker group.


    Install docker and portainer (or openmediavault-compose) again and launch the containers.

    I tried this. Still no luck with my containers. That said, Portainer is now starting on boot along with Docker, so there is some improvement. I assume Portainer has root access, as it's installed via omv-extras, not using the PUID/PGID for my personal containers.


    I've changed the permissions on all the docker config folders. I've also done this to the main docker install folder, but every time I reboot the system, it restores the main docker folder permissions. See attached images

    • Official Post
    Quote from corzocone

    I've also done this to the main docker install folder

    Why did you do that?

    • Official Post
    Quote from corzocone

    I tried this.

    Maybe you should describe the steps you took. With details. There may be some error.

    Quote from chente

    Maybe you should describe the steps you took. With details. There may be some error.

    1. Uninstall Portainer via omv-extras
    2. Uninstall Docker via omv-extras
    3. Reboot
    4. Install Docker via omv-extras
    5. Install Portainer via omv-exras
    6. Change permissions on config files

    I realize now I originally missed one of your steps: Delete the docker working files.


    I have now done this this, and when I re-installed docker + portainer, portainer had essentially done a "factory reset". All portainer config data must have been deleted. Thankfully, I made a backup of the full docker file before I deleted, so I can go back. Can you please be more specific as to which files I could delete? Attached is a screenshot of all the subfolders in the docker install folder. It seems I deleted all the portainer data, which I do not want to do. Is that correct?


    Again, thanks so much for your help, and sorry if I'm slow with these things. I'm not an experienced user.

    Images

    My Build:

    HP Z2 G4 - C246 Chipset w/ ECC Support

    Intel Xeon E-2124G Quad Core 3.4 GHz with QuickSync

    64GB ECC DDR4-2666 (Unregistered)

    500GB WD Blue NVMe for OMV install and separate partition for Docker containers

    3x 12TB Toshiba MG07 in RAIDZ1 ZFS

    OMV 7.4.x PVE Kernel 6.8.x

    • Official Post
    Quote from corzocone

    I realize now I originally missed one of your steps: Delete the docker working files.

    You should repeat the process without skipping that step.

    Quote from corzocone

    I have now done this this, and when I re-installed docker + portainer, portainer had essentially done a "factory reset". All portainer config data must have been deleted. Thankfully, I made a backup of the full docker file before I deleted, so I can go back. Can you please be more specific as to which files I could delete? Attached is a screenshot of all the subfolders in the docker install folder. It seems I deleted all the portainer data, which I do not want to do. Is that correct?

    Everyone in that folder, including portainer. If it restores, then fantastic, it is not a problem. There is nothing in Portainer that you need.

    Portainer is just another container. Why don't you want to remove it?

    Quote from corzocone

    Again, thanks so much for your help, and sorry if I'm slow with these things. I'm not an experienced user.

    Don't worry. If you were an experienced user I wouldn't be trying to help you. Maybe it would be the opposite. ;)

    • Official Post
    Quote from chente

    If your containers are implemented using compose (or portainer) with a stack (yml configuration file)

    Before continuing, please confirm if this is true.

    Quote from chente

    You should repeat the process without skipping that step.

    I did so.


    Quote from chente

    Everyone in that folder, including portainer. If it restores, then fantastic, it is not a problem. There is nothing in Portainer that you need.

    Portainer is just another container. Why don't you want to remove it?

    Maybe I'm mistaken, but if I am starting over in Portainer, won't I need to rebuild all of my containers from scratch? I spent a lot of time setting up the containers, and would prefer if I could keep those settings. Is there a way to backup and restore?

    My Build:

    HP Z2 G4 - C246 Chipset w/ ECC Support

    Intel Xeon E-2124G Quad Core 3.4 GHz with QuickSync

    64GB ECC DDR4-2666 (Unregistered)

    500GB WD Blue NVMe for OMV install and separate partition for Docker containers

    3x 12TB Toshiba MG07 in RAIDZ1 ZFS

    OMV 7.4.x PVE Kernel 6.8.x

    • Official Post
    Quote from corzocone

    Maybe I'm mistaken, but if I am starting over in Portainer, won't I need to rebuild all of my containers from scratch? I spent a lot of time setting up the containers, and would prefer if I could keep those settings. Is there a way to backup and restore?

    That is what i want to know. How have you configured the containers? Do you have a yml configuration file?

    • Official Post

    There is one aspect on Linux that I think is important to know about. In linux, modifying permissions can break the operation of the system, to the point of having to reinstall. It's one thing to play with your file permissions. With that you can do whatever you want. But modifying the permissions of the system files should never be done. You should not touch the files in /var/lib/docker or change their permissions. That's why I insist that you delete them and start from scratch. I don't know what you've done with them.

    Quote from chente

    There is one aspect on Linux that I think is important to know about. In linux, modifying permissions can break the operation of the system, to the point of having to reinstall. It's one thing to play with your file permissions. With that you can do whatever you want. But modifying the permissions of the system files should never be done. You should not touch the files in /var/lib/docker or change their permissions. That's why I insist that you delete them and start from scratch. I don't know what you've done with them.

    Yes, this makes sense. And I don't want a half-fixed solution that will have issues in the future.


    I was able to go back into the old portainer and save docker compose instructions for all of my containers. I'm now starting from scratch using those, and I have a few containers up and running so far. I'll report back if I have any more issues, but I hope I'm nearing the finish line!


    Again, thank you so much. I appreciate you!

    My Build:

    HP Z2 G4 - C246 Chipset w/ ECC Support

    Intel Xeon E-2124G Quad Core 3.4 GHz with QuickSync

    64GB ECC DDR4-2666 (Unregistered)

    500GB WD Blue NVMe for OMV install and separate partition for Docker containers

    3x 12TB Toshiba MG07 in RAIDZ1 ZFS

    OMV 7.4.x PVE Kernel 6.8.x

    • Official Post
    Quote from corzocone

    I was able to go back into the old portainer and save docker compose instructions for all of my containers.

    You can also take advantage of the situation and use openmediavault-compose instead of portainer. Portainer is overkill if it's only used to launch stack-powered containers. You will also have your batteries saved. Look at the last two sentences of point 4 and point 8 of this guide. [How to] Docker in OMV

    Quote from chente

    You can also take advantage of the situation and use openmediavault-compose instead of portainer. Portainer is overkill if it's only used to launch stack-powered containers. You will also have your batteries saved. Look at the last two sentences of point 4 and point 8 of this guide. [How to] Docker in OMV

    Thanks again. You not only helped me get the issue fixed, but helped me to learn more about OMV/Docker/Portainer.

    My Build:

    HP Z2 G4 - C246 Chipset w/ ECC Support

    Intel Xeon E-2124G Quad Core 3.4 GHz with QuickSync

    64GB ECC DDR4-2666 (Unregistered)

    500GB WD Blue NVMe for OMV install and separate partition for Docker containers

    3x 12TB Toshiba MG07 in RAIDZ1 ZFS

    OMV 7.4.x PVE Kernel 6.8.x

    • Official Post
    Quote from corzocone

    Thanks again. You not only helped me get the issue fixed, but helped me to learn more about OMV/Docker/Portainer.

    You are welcome :thumbup:

Participate now!

Don’t have an account yet? Register yourself now and be a part of our community!

Register Yourself Login