Read the MySQL error log, not the systemd message. Run sudo tail -40 /var/log/mysql/error.log (or /var/log/mysqld.log on RHEL). MySQL writes the real reason there and then exits, and systemd only sees the exit code.
sudo tail -40 /var/log/mysql/error.log sudo journalctl -xeu mysql.service --no-pager | tail -40
Unlike nginx or Apache, MySQL has no configuration test command that reliably catches startup failures, so the error log is the only place the cause appears.
The five causes that account for almost all of these
- Permissions on the data directory. Easily the most common after a manual file copy, a restore, or a disk move. Every file under the data directory must be owned by the mysql user:
sudo chown -R mysql:mysql /var/lib/mysql sudo chmod 750 /var/lib/mysql
The log line is usually
Can't create/write to fileorPermission denied. - The disk is full. InnoDB cannot start if it cannot extend its files, and the error can be misleadingly generic. Check before anything else with
df -handdf -i. Running out of inodes produces the same symptom as running out of bytes, anddf -halone will not show it. - An unknown directive in my.cnf. MySQL aborts on options it does not recognise, which bites after a major version upgrade removes one. The log names it:
unknown variable 'query_cache_size=64M'. That particular variable was removed in MySQL 8.0 and is a very common upgrade casualty. Comment the line out and start again. - InnoDB log file size changed. If
innodb_log_file_sizewas edited while the server was stopped uncleanly, InnoDB refuses to start because the existing log files no longer match the configured size. The log says so explicitly. Either restore the original value, or shut down cleanly and remove the oldib_logfile*files so they are recreated. - Port 3306 is already bound. Usually a second MySQL or MariaDB instance, or a stale process that survived a bad shutdown:
sudo ss -tlnp | grep ':3306 '
When the error log is empty
An empty or missing error log is itself a diagnosis. It means MySQL failed before it could open the log, which narrows things to three possibilities: the log path in my.cnf points somewhere that does not exist or is not writable, the data directory is missing entirely, or AppArmor is blocking access.
On Ubuntu, AppArmor is the one people miss. If the data directory was moved to a non-standard path, the profile still only permits the old one, and MySQL is denied before it writes a single line. Confirm it with:
sudo dmesg | grep -i apparmor | tail -20
A denial naming mysqld and your new path confirms it. Add the path to /etc/apparmor.d/usr.sbin.mysqld and reload the profile. On RHEL the equivalent is SELinux, and sudo ausearch -m avc -ts recent shows the same thing.
Reading the exit code
systemctl status mysql prints a status number worth checking first. status=1/FAILURE means mysqld started and rejected something, so go to the error log. status=137 means SIGKILL, which for MySQL almost always means the OOM killer: confirm with dmesg -T | grep -i oom, because a MySQL instance killed for memory will keep dying until you lower innodb_buffer_pool_size or add RAM. The full exit-code table is in the nginx version of this error.
How to stop it happening again
Validate configuration changes before restarting. MySQL will at least parse its options without starting the server:
sudo mysqld --validate-config
That catches unknown and malformed variables, which is the one cause on this page you can detect in advance. It will not catch permissions, disk space or log-size mismatches, so keep the error log open in a second terminal whenever you restart after a change.
Related
- Job for nginx.service failed: the same systemd error, plus the full exit-code table.
- Job for postgresql.service failed: the same failure on PostgreSQL, where the diagnostics differ.
- logrotate config generator: keep the MySQL error log from filling the disk and causing cause 2 above.
Is MariaDB the same?
Close enough that this page applies. The unit is mariadb.service, the error log is usually /var/log/mysql/error.log on Debian family systems, and the data directory rules are identical. The main divergence is configuration variables: MariaDB kept several options MySQL 8.0 removed, so the unknown-variable failure in cause 3 tends to appear when migrating between the two rather than when upgrading within one.
Can I just delete ib_logfile0 and ib_logfile1?
Only after a clean shutdown. If MySQL stopped cleanly the log files hold no unflushed data and InnoDB will recreate them at the new size. If it crashed, those files contain the redo log needed for recovery, and deleting them can lose committed transactions. Check the error log for a clean shutdown message before touching them, and take a filesystem copy of the data directory first.
Leave a Reply