Check that /etc/docker/daemon.json is valid JSON. Run sudo python3 -m json.tool /etc/docker/daemon.json. A trailing comma or a stray quote in that file is the single most common cause, and the daemon exits before writing anything useful anywhere else.
sudo python3 -m json.tool /etc/docker/daemon.json sudo journalctl -xeu docker.service --no-pager | tail -40
If the file does not exist at all, that is fine and not your problem: Docker runs on defaults without it.
Run the daemon in the foreground
Docker has the best diagnostic of any service on this list, and almost nobody uses it. Running dockerd directly prints the failure straight to your terminal instead of burying it:
sudo systemctl stop docker.socket docker.service sudo dockerd --debug
The reason appears in the first few lines. Stop it with Ctrl+C and start the service normally once fixed. Stop docker.socket as well as docker.service, or socket activation will restart the daemon underneath you and the port will appear to be in use.
The causes worth checking, in order
- Invalid or contradictory daemon.json. Beyond malformed JSON, the other trap is setting a key that also appears in the unit’s
ExecStartflags. The classic ishosts: ifdaemon.jsonsets it and the unit also passes-H fd://, the daemon refuses to start withunable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: hosts. Remove it from one place, normally the JSON. /var/run/docker.sockexists as a directory. This happens when something tried to start the daemon while it was mid-shutdown, or a bind mount created the path. Docker then cannot create its own socket. Remove the directory and start again.- A runtime named in daemon.json is not installed. Adding
"default-runtime": "nvidia"without the NVIDIA container toolkit present kills the daemon on startup. The journal names the missing runtime. Install the toolkit or remove the key. - Storage driver mismatch. Changing
storage-driverwhen/var/lib/dockeralready holds data from a different driver stops the daemon rather than risk the existing images. Either revert the setting, or move the old directory aside and accept losing local images and containers. - The disk is full. Check
df -handdf -iboth. Image layers create enormous numbers of small files, so Docker hosts exhaust inodes well before they exhaust bytes, anddf -halone looks fine while everything fails. - iptables or nftables is unavailable. The daemon sets up its own chains at startup and exits if it cannot. The journal mentions
iptablesorfailed to start daemon. Common inside minimal containers and on hosts where nftables replaced iptables without the compatibility shim.
Reading the exit code
status=1/FAILURE means dockerd started and rejected its own configuration, which covers causes 1, 3, 4 and 6. status=137 is SIGKILL and usually the OOM killer. A code of 203/EXEC is worth knowing here specifically: it means the dockerd binary named in the unit does not exist or is not executable, which happens after a partial package upgrade or when a manually installed binary was removed. The full table is in the nginx version of this error.
How to stop it happening again
Validate the JSON before every restart. The file has no schema check at write time, so a typo sits there silently until the next restart, which may be weeks later during an unrelated reboot:
sudo python3 -m json.tool /etc/docker/daemon.json && sudo systemctl restart docker
That is the whole discipline. Most Docker startup failures are a configuration file edited months ago and never tested, discovered at the worst possible moment because a reboot finally exercised it.
Related
- Job for nginx.service failed: the same systemd error, plus the full exit-code table.
- Job for apache2.service failed: the same failure pattern on Apache.
- Docker Compose ports generator: build port mappings in short or long syntax without guessing at the YAML.
Why does Docker restart by itself after I stop it?
Socket activation. docker.socket listens on the Unix socket and starts docker.service the moment anything touches it, including your own docker ps. To genuinely stop the daemon you must stop both: sudo systemctl stop docker.socket docker.service. This is also why foreground debugging appears to fail with the address already in use if you only stopped the service.
The journal shows nothing at all
Then the daemon died before logging was initialised, which points at the unit itself rather than at Docker. Check systemctl cat docker to see the unit plus every drop-in applied to it: a drop-in with a bad ExecStart path produces 203/EXEC and no daemon logs whatsoever, because the binary never ran.
Leave a Reply