Docker Troubleshooting
Failed to create network: iptables No chain/target/match by that name
Error
failed to create network <network_name>: Error response from daemon: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i <bridge> -j RETURN: iptables: No chain/target/match by that name.
(exit status 1))
Cause
Docker sets up its own chains in iptables (such as the DOCKER chain in the nat table) when it starts. If a firewall service (UFW, firewalld) is restarted, or iptables is flushed after Docker has already started, those chains are removed. When Docker then tries to create a new network, it attempts to insert a rule into a chain that no longer exists, resulting in this error.
Resolution
- Stop your containers:
docker compose down
- Restart Docker. This causes Docker to rebuild all of its iptables chains:
sudo systemctl restart docker
- Bring your containers back up:
docker compose up -d
Out of Memory (OOM): Container Killed by Kernel
Error
Out of memory: Killed process 2426654 (uvicorn) total-vm:8242728kB, anon-rss:2725272kB
Visible via sudo dmesg after the system exhausts both RAM and SWAP. Check current memory state with:
free -h
Cause
Docker containers have no memory limits by default — they can consume all available RAM and SWAP on the host. Resource-intensive tasks (such as Jellyfin generating trickplay images, Tdarr transcoding, etc.) can exhaust system memory, causing the Linux OOM killer to terminate processes.
Resolution
1. Add memory limits to containers in docker-compose.yaml
For each resource-heavy container, add a mem_limit under deploy.resources.limits:
services:
jellyfin:
image: jellyfin/jellyfin:latest
# ... other config ...
deploy:
resources:
limits:
memory: 12G # Adjust to fit your available RAM
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
tdarr:
image: ghcr.io/haveagitgat/tdarr:latest
# ... other config ...
deploy:
resources:
limits:
memory: 4G
tdarr-node:
image: ghcr.io/haveagitgat/tdarr_node:latest
# ... other config ...
deploy:
resources:
limits:
memory: 4G
You can also set memswap_limit and mem_swappiness at the top level of the service (not under deploy) to cap swap usage per container and reduce how aggressively it swaps:
services:
jellyfin:
image: jellyfin/jellyfin:latest
# ... other config ...
# top-level service options (not under deploy)
memswap_limit: 6G # Total RAM + SWAP the container may use; set equal to mem_limit to disable swap entirely
mem_swappiness: 0 # 0 = never swap; 100 = swap aggressively
deploy:
resources:
limits:
memory: 12G
memswap_limit and mem_swappiness require the --compatibility flag or Docker Swarm mode to be respected in some older Compose versions. In modern Compose V2 they are applied directly.
Apply the changes:
sudo docker compose up -d
2. Flush SWAP to reclaim memory immediately
sudo swapoff -a && sudo swapon -a
3. Reduce vm.swappiness to prevent aggressive SWAP usage
Check the current value (default is usually 60):
cat /proc/sys/vm/swappiness
Set it to 10 so the kernel prefers RAM over SWAP:
sudo sysctl vm.swappiness=10 && echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
4. Verify memory is recovered
free -h
Before:
total used free shared buff/cache available
Mem: 31Gi 28Gi 555Mi 880Mi 3.3Gi 2.5Gi
Swap: 8.0Gi 8.0Gi 0B
After:
total used free shared buff/cache available
Mem: 31Gi 15Gi 388Mi 551Mi 16Gi 15Gi
Swap: 8.0Gi 0B 8.0Gi
Use docker stats sorted by memory percentage to find which containers are consuming the most RAM:
sudo docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}" | sort -k3 -rh
Tdarr (or any NVIDIA container) won't start after driver update
Error
Error response from daemon: failed to create task for container: failed to create shim task:
OCI runtime create failed: runc create failed: unable to start container process:
error during container init: failed to fulfil mount request:
open /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.580.159.04: no such file or directory
The library version in the error (580.159.04) is the old driver — not the one you just installed.
Cause
Docker uses CDI (Container Device Interface) to mount NVIDIA libraries into containers. The CDI spec file (/etc/cdi/nvidia.yaml) is generated at install time and caches the exact library paths for the driver version present at that moment. When the driver is updated, the CDI spec is not automatically regenerated, so Docker still tries to mount libraries from the previous version that no longer exist on disk.
Resolution
1. Restart the CDI refresh service and check the current CDI device list:
sudo systemctl restart nvidia-cdi-refresh.service
sudo nvidia-ctk --debug cdi list
2. Check whether the old driver version is still referenced in the CDI config:
# Replace 580.159.04 with the OLD driver version shown in the error
sudo grep -R "580.159.04" /etc/cdi /var/run/cdi 2>/dev/null
If this returns output, the CDI spec still contains stale paths and must be regenerated.
3. Regenerate the CDI spec:
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
4. Recreate the container:
docker compose up -d --force-recreate tdarr
5. If the container still fails, restart Docker to clear its cached device configuration:
sudo systemctl restart docker
docker compose up -d tdarr
Remove NVIDIA_DRIVER_CAPABILITIES=all from the Compose file if you have it. Tdarr already defaults to the capabilities it needs (compute,video,utility). Requesting all unnecessarily includes graphics/EGL libraries — which are also what triggers this error when they go stale after a driver update.
NVIDIA container fails: nvidia-persistenced socket missing + NVML version mismatch
Error
Error response from daemon: failed to create task for container: failed to create shim task:
OCI runtime create failed: runc create failed: unable to start container process:
error during container init: failed to fulfil mount request:
open /run/nvidia-persistenced/socket: no such file or directory
$ sudo nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 580.178
Cause
This is a different failure mode than the stale CDI library paths above. NVML library version mismatch means the userspace NVIDIA libraries/tools were upgraded (typically via apt upgrade pulling in a newer nvidia-driver-*/libnvidia-* package) but the kernel module (nvidia.ko) currently loaded in memory is still the old version, because nothing has triggered a module reload yet.
Since the kernel module is out of sync, the nvidia-persistenced service can't talk to it and exits, which removes its socket at /run/nvidia-persistenced/socket. Docker's CDI spec mounts that socket into the container, so container creation fails when the socket file doesn't exist.
To confirm this was caused by a driver package upgrade, check the APT history. A plain grep -i nvidia only prints matching lines and won't include the Start-Date: timestamp, so pair each match with the timestamp of the transaction it belongs to:
awk '/^Start-Date:/{d=$0} /nvidia-driver/{print d; print; print ""}' /var/log/apt/history.log
Look specifically for an Upgrade: line containing nvidia-driver-<version> (not just any nvidia-* package — most are unrelated firmware/kernel-module builds that get pulled in automatically). Then compare that Start-Date against how long the system has been running:
uptime -s
If the driver package was upgraded after the system last booted, the running kernel module predates it — this confirms the mismatch, and a reboot (or manual module reload) is required.
Resolution
1. Reboot (recommended)
A reboot guarantees the kernel module and userspace driver come back in sync, with no risk of leaving the system in a half-reloaded state:
sudo reboot
After reboot, confirm everything matches before starting containers:
sudo nvidia-smi
systemctl status nvidia-persistenced
docker compose up -d jellyfin
2. If you can't reboot right now, try reloading the kernel module
Only use this if something on the host can't be interrupted and a reboot isn't an option. Unlike a reboot, this can fail partway through and leave NVIDIA in a broken state until you do reboot anyway.
Stop everything currently holding the NVIDIA modules open:
sudo docker compose down
sudo systemctl stop nvidia-persistenced
Check what's still using the modules:
sudo lsof /dev/nvidia* 2>/dev/null
sudo fuser -v /dev/nvidia*
If nothing is using them, unload and reload:
sudo modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia
sudo modprobe nvidia
sudo systemctl start nvidia-persistenced
Verify the socket exists again and the version mismatch is gone:
ls -l /run/nvidia-persistenced/socket
sudo nvidia-smi
If modprobe -r fails with in use (desktop environments, other containers, or lingering processes often keep the modules loaded), stop deferring and reboot — it's the only reliable fix at that point.
This is distinct from the CDI stale-path error above: here nvidia-smi itself fails with a version mismatch, whereas the CDI issue only breaks container mounts while nvidia-smi on the host works fine. If nvidia-smi works normally on the host after reboot/reload but containers still fail, regenerate the CDI spec as described in the section above.
💬 Recent Comments