docker failing to restart with nvidia runtime

aaronb · 12. Februar 2024

Any ideas on this?

Code

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; docker restart  9d3b6a1d5921439bc60c7e7508a2078d2509319f18aa11313f12682ec12a9af2 2>&1': Error response from daemon: Cannot restart container 9d3b6a1d5921439bc60c7e7508a2078d2509319f18aa11313f12682ec12a9af2: unknown or invalid runtime name: nvidia

OMV\ExecException: Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C.UTF-8; export LANGUAGE=; docker restart  9d3b6a1d5921439bc60c7e7508a2078d2509319f18aa11313f12682ec12a9af2 2>&1': Error response from daemon: Cannot restart container 9d3b6a1d5921439bc60c7e7508a2078d2509319f18aa11313f12682ec12a9af2: unknown or invalid runtime name: nvidia in /usr/share/openmediavault/engined/rpc/compose.inc:1009
Stack trace:
#0 /usr/share/php/openmediavault/rpc/serviceabstract.inc(620): OMVRpcServiceCompose->{closure}('/tmp/bgstatusVG...', '/tmp/bgoutput64...')
#1 /usr/share/openmediavault/engined/rpc/compose.inc(1012): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))
#2 [internal function]: OMVRpcServiceCompose->doContainerCommand(Array, Array)
#3 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)
#4 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('doContainerComm...', Array, Array)
#5 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('Compose', 'doContainerComm...', Array, Array, 1)
#6 {main}

Alles anzeigen

Followed the video made by ryecoaaron linked in the #1 post of this thread.

Portainer is running fine however both Plex and and Tautulli Remote containers have exited status.

ryecoaaron · 12. Februar 2024

Did you configure nvidia drivers/etc on your system? What does your daemon.json look like?

aaronb · 13. Februar 2024

yes, GPU was transcoding Plex content for years. It broke upgrading to omv6/deb11. Installed the latest drivers to fix the broken upgrade. Transcoding worked again until I started this shift to the new compose, as per that video. Haven't touched anything since.

Code

# cat /etc/docker/daemon.json
{
  "data-root": "/var/lib/docker"
}

Specified /var/lib/docker in the config as per the video. Don't know what the issue is.

Driver shows up with nvidia-smi

Code

# nvidia-smi
Tue Feb 13 18:02:06 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.223.02   Driver Version: 470.223.02   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   43C    P0    27W / 130W |      0MiB /  4043MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Alles anzeigen

aaronb · 13. Februar 2024

Seems fixed, well at least as far as the containers start now.

You were on the right track with daemon.json. Had to add back the runtime detail for nvidia.

Found this:

Docker Error - Unknown or Invalid Runtime Name: Nvidia

Let’s take a look at the Visual SLAM package above in the Isaac Sim environment. This package works in ROS2 environment. Edit /etc/docker/daemon.json: I…

forums.developer.nvidia.com

Code

# nvidia-ctk runtime configure --runtime=docker
INFO[0000] Loading docker config from /etc/docker/daemon.json
INFO[0000] Successfully loaded config
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that the docker daemon be restarted.
# systemctl restart docker

Code

# cat /etc/docker/daemon.json
{
    "data-root": "/var/lib/docker",
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }

Will test, don't expect issues

ryecoaaron · 13. Februar 2024

Just make sure to leave the Docker storage field blank since you are customizing it.

bowser4 · 20. Februar 2024

Thank you for posting this solution - I had the same issue once updating. Very grateful, thanks again.

aaronb · 23. Februar 2024

Discovered that (hw) transcoding had stopped working. After going through the above, I've found that the plex container runtime defaulted back to "runc".

Using portainer, I used duplicated/edit the container to change the runtime to "nvidia". Deployed container and (hw) transcoding once again is working.

Jetzt mitmachen!