Job for grafana-server.service failed because the control process exited with error code

Read /var/log/grafana/grafana.log, then check the status number. Grafana logs the real reason there before it exits. If systemctl status grafana-server shows status=217/USER, the grafana user does not exist and Grafana never ran at all.

sudo tail -40 /var/log/grafana/grafana.log
systemctl status grafana-server | grep -E 'Process|status=|Result'

The causes, in order of how often they appear

1. Ownership of the data directory. Grafana runs as the grafana user and needs to write its SQLite database, sessions and plugins under /var/lib/grafana. The classic trigger is installing a plugin with sudo grafana-cli, which leaves a root-owned plugin directory behind. Grafana then runs fine by hand as root and fails under systemd:

   sudo chown -R grafana:grafana /var/lib/grafana /var/log/grafana
   sudo systemctl restart grafana-server

2. A bad edit in grafana.ini. A malformed line, usually in the [smtp] or [auth.ldap] section after someone configured alerts or login, stops the server on startup. The log names the section. A ; at the start of a line comments it out, which catches people who uncomment one line and leave the next commented.
3. Could not find config defaults. Grafana was started with the wrong home path, so it cannot find conf/defaults.ini. It appears when a custom unit or an override drops the --homepath=/usr/share/grafana argument. Compare your unit against the packaged one with systemctl cat grafana-server.
4. A provisioning file that fails to parse. YAML under /etc/grafana/provisioning/ for datasources or dashboards is read at startup, and one bad indent or a tab character stops the whole server. The log names the file.
5. The SQLite database is locked or damaged. After a full disk or an unclean stop, grafana.db can be left locked. Check df -h first, then look for database is locked in the log. Copy the file before touching it; it holds every dashboard and user.
6. Port 3000 already bound. Another dashboard tool or a development server often claims it: sudo ss -tlnp | grep ':3000 '. Change http_port in grafana.ini or stop the other process.

If the status says start-limit-hit

Result: start-limit-hit is not a cause; it means Grafana crashed several times quickly and systemd stopped retrying. Fix the real failure from the log first, then clear the counter, or the next start is refused even though nothing is wrong any more:

sudo systemctl reset-failed grafana-server
sudo systemctl start grafana-server

Reading the exit code

status=1/FAILURE means Grafana started and rejected something, so read its log. status=217/USER means the user in the unit does not exist, common when configuration management installs Grafana from a tarball rather than the package. status=203/EXEC means the binary path in the unit is wrong. The full table is in the nginx version of this error.

How to stop it happening again

Install plugins as the grafana user, never as root, and keep a copy of grafana.ini before each edit:

sudo -u grafana grafana-cli --pluginsDir /var/lib/grafana/plugins plugins install <plugin-id>
sudo systemctl restart grafana-server

Back up grafana.db on a schedule. It is the single file holding your dashboards, users and alert rules, and the failure modes above that touch it are the only ones on this page that can lose work.

Related

• Job for elasticsearch.service failed: a common Grafana data source that fails on host settings.
• Job for mongod.service failed: another service whose YAML configuration rejects tabs.
• Job for nginx.service failed: the reverse proxy usually in front of Grafana, plus the full exit-code table.