Read the cluster log, not the systemd message. On Debian and Ubuntu that is /var/log/postgresql/postgresql-16-main.log, on RHEL it is /var/lib/pgsql/data/log/. PostgreSQL writes the real reason there and exits, and systemd only ever sees the exit code.
sudo tail -40 /var/log/postgresql/postgresql-16-main.log sudo journalctl -xeu postgresql@16-main.service --no-pager | tail -40
Check the unit name first
This is the step that wastes the most time on Debian and Ubuntu, and it is specific to PostgreSQL. There are two units. postgresql.service is an empty wrapper that starts nothing itself; the real work happens in a per-cluster unit named postgresql@16-main.service. Running systemctl status postgresql frequently shows a green, happy wrapper while the actual cluster is dead.
# list every cluster and its real state pg_lsclusters # then target the cluster unit, not the wrapper sudo systemctl status postgresql@16-main.service
pg_lsclusters is the fastest diagnostic on this platform: it shows version, port, status and log path for every cluster in one line each. RHEL family systems use a single postgresql-16.service and do not have this split.
The causes, in order of how often they appear
- Data directory permissions. PostgreSQL refuses to start if the data directory is group or world readable. This catches people constantly after a restore or a manual copy:
sudo chown -R postgres:postgres /var/lib/postgresql/16/main sudo chmod 700 /var/lib/postgresql/16/main
The log says
data directory has invalid permissionsandpermissions should be u=rwx. - A stale postmaster.pid. After an unclean shutdown or a host that lost power, the lock file survives and PostgreSQL will not start while it believes another instance is running. The log line is
lock file "postmaster.pid" already exists. Confirm no postgres process is actually running withps aux | grep postgres, then remove the file. - A syntax error in postgresql.conf or pg_hba.conf. The log names the file and line. Note that a bad
pg_hba.confblocks startup entirely, not just individual connections, which surprises people who expect it to only affect authentication. - Port 5432 already bound. Nearly always a second cluster, very often an older major version left behind by an upgrade.
pg_lsclustersshows both and their ports. Check withsudo ss -tlnp | grep ':5432 '. - The disk is full. PostgreSQL stops rather than risk corruption. Check
df -handdf -iboth, since exhausted inodes present exactly like exhausted bytes. A runaway WAL directory is the usual reason, often because a replication slot was left behind and is pinning WAL that can never be recycled.
Reading the exit code
status=1/FAILURE means the postmaster started and rejected something, so go to the cluster log. status=137 means SIGKILL and normally the OOM killer, which for PostgreSQL usually means shared_buffers or work_mem multiplied by concurrent connections exceeded available memory. Confirm with dmesg -T | grep -i oom. The full table, which applies to any systemd unit, is in the nginx version of this error.
How to stop it happening again
PostgreSQL can check its own configuration without starting, which catches cause 3 before it costs you a restart:
sudo -u postgres /usr/lib/postgresql/16/bin/postgres -C shared_buffers -D /var/lib/postgresql/16/main
If the configuration is invalid that command fails and prints the reason; if it is valid it prints the setting and exits without starting a server. For pg_hba.conf changes specifically, prefer sudo systemctl reload postgresql@16-main over a restart: a reload re-reads authentication rules without dropping connections, and a bad file leaves the running server untouched instead of refusing to come back.
Related
- Job for nginx.service failed: the same systemd error, plus the full exit-code table.
- Job for mysql.service failed: the same failure on MySQL, where the diagnostics differ.
- PostgreSQL connection string builder: once the cluster is up, build a correct URI with sslmode and percent-encoded passwords.
systemctl status postgresql says active but I cannot connect
That is the wrapper unit reporting success while the cluster behind it is down. On Debian and Ubuntu, postgresql.service starts nothing by itself. Run pg_lsclusters to see the true state, then act on postgresql@16-main.service instead. This is the single most misleading thing about PostgreSQL on these distributions.
Can I delete postmaster.pid safely?
Only after confirming no postgres process is running. Check with ps aux | grep postgres. If a postmaster is genuinely alive and you remove its lock file, you risk a second instance starting against the same data directory, which can corrupt the cluster. If no process exists, the file is a leftover from an unclean shutdown and removing it is the correct fix.
Leave a Reply