docker failing to restart with nvidia runtime

aaronb · Feb 12th 2024

Any ideas on this?

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; docker restart  9d3b6a1d5921439bc60c7e7508a2078d2509319f18aa11313f12682ec12a9af2 2>&1': Error response from daemon: Cannot restart container 9d3b6a1d5921439bc60c7e7508a2078d2509319f18aa11313f12682ec12a9af2: unknown or invalid runtime name: nvidia

OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; docker restart  9d3b6a1d5921439bc60c7e7508a2078d2509319f18aa11313f12682ec12a9af2 2>&1': Error response from daemon: Cannot restart container 9d3b6a1d5921439bc60c7e7508a2078d2509319f18aa11313f12682ec12a9af2: unknown or invalid runtime name: nvidia in /usr/share/openmediavault/engined/rpc/compose.inc:1009
Stack trace:
#0 /usr/share/php/openmediavault/rpc/serviceabstract.inc(620): OMVRpcServiceCompose->{closure}('/tmp/bgstatusVG...', '/tmp/bgoutput64...')
#1 /usr/share/openmediavault/engined/rpc/compose.inc(1012): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#2 [internal function]: OMVRpcServiceCompose->doContainerCommand(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('doContainerComm...', Array, Array)
#5 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Compose', 'doContainerComm...', Array, Array, 1)
#6 {main}

Display More

Followed the video made by ryecoaaron linked in the #1 post of this thread.

Portainer is running fine however both Plex and and Tautulli Remote containers have exited status.

ryecoaaron · Feb 12th 2024

Did you configure nvidia drivers/etc on your system? What does your daemon.json look like?

aaronb · Feb 13th 2024

yes, GPU was transcoding Plex content for years. It broke upgrading to omv6/deb11. Installed the latest drivers to fix the broken upgrade. Transcoding worked again until I started this shift to the new compose, as per that video. Haven't touched anything since.

Code

# cat /etc/docker/daemon.json
{
  "data-root": "/var/lib/docker"
}

Specified /var/lib/docker in the config as per the video. Don't know what the issue is.

Driver shows up with nvidia-smi

Code

# nvidia-smi
Tue Feb 13 18:02:06 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.223.02   Driver Version: 470.223.02   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   43C    P0    27W / 130W |      0MiB /  4043MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Display More

aaronb · Feb 13th 2024

Seems fixed, well at least as far as the containers start now.

You were on the right track with daemon.json. Had to add back the runtime detail for nvidia.

Found this:

Docker Error - Unknown or Invalid Runtime Name: Nvidia

Let’s take a look at the Visual SLAM package above in the Isaac Sim environment. This package works in ROS2 environment. Edit /etc/docker/daemon.json: I…

forums.developer.nvidia.com

Code

# nvidia-ctk runtime configure --runtime=docker
INFO[0000] Loading docker config from /etc/docker/daemon.json
INFO[0000] Successfully loaded config
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that the docker daemon be restarted.
# systemctl restart docker

Code

# cat /etc/docker/daemon.json
{
    "data-root": "/var/lib/docker",
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }

Will test, don't expect issues

ryecoaaron · Feb 13th 2024

Just make sure to leave the Docker storage field blank since you are customizing it.

bowser4 · Feb 20th 2024

Thank you for posting this solution - I had the same issue once updating. Very grateful, thanks again.

aaronb · Feb 23rd 2024

Discovered that (hw) transcoding had stopped working. After going through the above, I've found that the plex container runtime defaulted back to "runc".

Using portainer, I used duplicated/edit the container to change the runtime to "nvidia". Deployed container and (hw) transcoding once again is working.

aaronb · Sep 26th 2024

Just an update on this Nvidia runtime in Docker.

Something killed the hw transcoding over the last couple of months and I've only just realised.

Still on OMV 6.x which this thread upgrade is based upon.

Had to add this to the compose file for it to start working again.

Code

deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Hope this helps anyone in the same situation

ryecoaaron · Sep 26th 2024

Quote from aaronb

Something killed the hw transcoding over the last couple of months

Probably a change in docker itself. OMV 6 and 7 basically use the same version of docker since they are using the official docker repos. I recommend upgrade to OMV7 because OMV6 is EOL

Participate now!