Skip to main content
WebsiteGitHub last commitGitHub commit activityGitHub IssuesDocker PullsDiscordLocalized

Docker Troubleshooting

Failed to create network: iptables No chain/target/match by that name​

Error​

failed to create network <network_name>: Error response from daemon: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i <bridge> -j RETURN: iptables: No chain/target/match by that name.
(exit status 1))

Cause​

Docker sets up its own chains in iptables (such as the DOCKER chain in the nat table) when it starts. If a firewall service (UFW, firewalld) is restarted, or iptables is flushed after Docker has already started, those chains are removed. When Docker then tries to create a new network, it attempts to insert a rule into a chain that no longer exists, resulting in this error.

Resolution​

  1. Stop your containers:
docker compose down
  1. Restart Docker. This causes Docker to rebuild all of its iptables chains:
sudo systemctl restart docker
  1. Bring your containers back up:
docker compose up -d

Out of Memory (OOM): Container Killed by Kernel​

Error​

Out of memory: Killed process 2426654 (uvicorn) total-vm:8242728kB, anon-rss:2725272kB

Visible via sudo dmesg after the system exhausts both RAM and SWAP. Check current memory state with:

free -h

Cause​

Docker containers have no memory limits by default — they can consume all available RAM and SWAP on the host. Resource-intensive tasks (such as Jellyfin generating trickplay images, Tdarr transcoding, etc.) can exhaust system memory, causing the Linux OOM killer to terminate processes.

Resolution​

1. Add memory limits to containers in docker-compose.yaml​

For each resource-heavy container, add a mem_limit under deploy.resources.limits:

docker-compose.yaml
services:
jellyfin:
image: jellyfin/jellyfin:latest
# ... other config ...
deploy:
resources:
limits:
memory: 12G # Adjust to fit your available RAM
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]

tdarr:
image: ghcr.io/haveagitgat/tdarr:latest
# ... other config ...
deploy:
resources:
limits:
memory: 4G

tdarr-node:
image: ghcr.io/haveagitgat/tdarr_node:latest
# ... other config ...
deploy:
resources:
limits:
memory: 4G

You can also set memswap_limit and mem_swappiness at the top level of the service (not under deploy) to cap swap usage per container and reduce how aggressively it swaps:

docker-compose.yaml
services:
jellyfin:
image: jellyfin/jellyfin:latest
# ... other config ...
# top-level service options (not under deploy)
memswap_limit: 6G # Total RAM + SWAP the container may use; set equal to mem_limit to disable swap entirely
mem_swappiness: 0 # 0 = never swap; 100 = swap aggressively
deploy:
resources:
limits:
memory: 12G
note

memswap_limit and mem_swappiness require the --compatibility flag or Docker Swarm mode to be respected in some older Compose versions. In modern Compose V2 they are applied directly.

Apply the changes:

sudo docker compose up -d

2. Flush SWAP to reclaim memory immediately​

sudo swapoff -a && sudo swapon -a

3. Reduce vm.swappiness to prevent aggressive SWAP usage​

Check the current value (default is usually 60):

cat /proc/sys/vm/swappiness

Set it to 10 so the kernel prefers RAM over SWAP:

sudo sysctl vm.swappiness=10 && echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf

4. Verify memory is recovered​

free -h

Before:

total used free shared buff/cache available
Mem: 31Gi 28Gi 555Mi 880Mi 3.3Gi 2.5Gi
Swap: 8.0Gi 8.0Gi 0B

After:

total used free shared buff/cache available
Mem: 31Gi 15Gi 388Mi 551Mi 16Gi 15Gi
Swap: 8.0Gi 0B 8.0Gi
Identify memory hogs

Use docker stats sorted by memory percentage to find which containers are consuming the most RAM:

sudo docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}" | sort -k3 -rh

Tdarr (or any NVIDIA container) won't start after driver update​

Error​

Error response from daemon: failed to create task for container: failed to create shim task:
OCI runtime create failed: runc create failed: unable to start container process:
error during container init: failed to fulfil mount request:
open /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.580.159.04: no such file or directory

The library version in the error (580.159.04) is the old driver — not the one you just installed.

Cause​

Docker uses CDI (Container Device Interface) to mount NVIDIA libraries into containers. The CDI spec file (/etc/cdi/nvidia.yaml) is generated at install time and caches the exact library paths for the driver version present at that moment. When the driver is updated, the CDI spec is not automatically regenerated, so Docker still tries to mount libraries from the previous version that no longer exist on disk.

Resolution​

1. Restart the CDI refresh service and check the current CDI device list:

sudo systemctl restart nvidia-cdi-refresh.service
sudo nvidia-ctk --debug cdi list

2. Check whether the old driver version is still referenced in the CDI config:

# Replace 580.159.04 with the OLD driver version shown in the error
sudo grep -R "580.159.04" /etc/cdi /var/run/cdi 2>/dev/null

If this returns output, the CDI spec still contains stale paths and must be regenerated.

3. Regenerate the CDI spec:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

4. Recreate the container:

docker compose up -d --force-recreate tdarr

5. If the container still fails, restart Docker to clear its cached device configuration:

sudo systemctl restart docker
docker compose up -d tdarr
Avoid unnecessary capabilities

Remove NVIDIA_DRIVER_CAPABILITIES=all from the Compose file if you have it. Tdarr already defaults to the capabilities it needs (compute,video,utility). Requesting all unnecessarily includes graphics/EGL libraries — which are also what triggers this error when they go stale after a driver update.


NVIDIA container fails: nvidia-persistenced socket missing + NVML version mismatch​

Error​

Error response from daemon: failed to create task for container: failed to create shim task:
OCI runtime create failed: runc create failed: unable to start container process:
error during container init: failed to fulfil mount request:
open /run/nvidia-persistenced/socket: no such file or directory
$ sudo nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 580.178

Cause​

This is a different failure mode than the stale CDI library paths above. NVML library version mismatch means the userspace NVIDIA libraries/tools were upgraded (typically via apt upgrade pulling in a newer nvidia-driver-*/libnvidia-* package) but the kernel module (nvidia.ko) currently loaded in memory is still the old version, because nothing has triggered a module reload yet.

Since the kernel module is out of sync, the nvidia-persistenced service can't talk to it and exits, which removes its socket at /run/nvidia-persistenced/socket. Docker's CDI spec mounts that socket into the container, so container creation fails when the socket file doesn't exist.

To confirm this was caused by a driver package upgrade, check the APT history. A plain grep -i nvidia only prints matching lines and won't include the Start-Date: timestamp, so pair each match with the timestamp of the transaction it belongs to:

awk '/^Start-Date:/{d=$0} /nvidia-driver/{print d; print; print ""}' /var/log/apt/history.log

Look specifically for an Upgrade: line containing nvidia-driver-<version> (not just any nvidia-* package — most are unrelated firmware/kernel-module builds that get pulled in automatically). Then compare that Start-Date against how long the system has been running:

uptime -s

If the driver package was upgraded after the system last booted, the running kernel module predates it — this confirms the mismatch, and a reboot (or manual module reload) is required.

Resolution​

A reboot guarantees the kernel module and userspace driver come back in sync, with no risk of leaving the system in a half-reloaded state:

sudo reboot

After reboot, confirm everything matches before starting containers:

sudo nvidia-smi
systemctl status nvidia-persistenced
docker compose up -d jellyfin

2. If you can't reboot right now, try reloading the kernel module​

Only use this if something on the host can't be interrupted and a reboot isn't an option. Unlike a reboot, this can fail partway through and leave NVIDIA in a broken state until you do reboot anyway.

Stop everything currently holding the NVIDIA modules open:

sudo docker compose down
sudo systemctl stop nvidia-persistenced

Check what's still using the modules:

sudo lsof /dev/nvidia* 2>/dev/null
sudo fuser -v /dev/nvidia*

If nothing is using them, unload and reload:

sudo modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia
sudo modprobe nvidia
sudo systemctl start nvidia-persistenced

Verify the socket exists again and the version mismatch is gone:

ls -l /run/nvidia-persistenced/socket
sudo nvidia-smi

If modprobe -r fails with in use (desktop environments, other containers, or lingering processes often keep the modules loaded), stop deferring and reboot — it's the only reliable fix at that point.

note

This is distinct from the CDI stale-path error above: here nvidia-smi itself fails with a version mismatch, whereas the CDI issue only breaks container mounts while nvidia-smi on the host works fine. If nvidia-smi works normally on the host after reboot/reload but containers still fail, regenerate the CDI spec as described in the section above.

Buy me a beer

Changelog

docs(docker): add NVML version mismatch / nvidia-persistenced socket troubleshooting entry
docs(docker): add NVIDIA CDI stale driver troubleshooting entry
docs(docker): Add memswap_limit and mem_swappiness top-level service options to OOM troubleshooting
docs: add OOM troubleshooting and memory limits to Jellyfin/Docker docs
docs(docker): Add iptables DOCKER chain missing error troubleshooting entry


💬 Discord Community Chat

Join the conversation! Comments here sync with our Discord community.

💬 Recent Comments

Loading comments...