MariaDB on Trellis Failed to Start – /usr/bin/mysqld_safe: No such file or directory

February 24, 2018 • By Jasper Frumau

Post Digital Ocean restart to deal with security patches for vulnerabilities (Spectre and Meltdown Mitigation )MariaDB on Trellis failed to start. So the site failed to connect to the database and went down.

MariaDB Down

MariaDB was down and restarting it failed suggesting us to check journalctl -xe for more information. In the journalctl -xe logs we had:

Feb 23 02:26:07 domain mysql[1963]:  * Starting MariaDB database server mysqld
Feb 23 02:26:07 domain /etc/init.d/mysql[1993]: /etc/init.d/mysql: line 114: /usr/bin/mysqld_safe: No such file or directory
Feb 23 02:26:38 domain /etc/init.d/mysql[2275]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' re
Feb 23 02:26:38 domain /etc/init.d/mysql[2275]: [61B blob data]
Feb 23 02:26:38 domain /etc/init.d/mysql[2275]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' 
Feb 23 02:26:38 domain /etc/init.d/mysql[2275]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Feb 23 02:26:38 domain /etc/init.d/mysql[2275]: 
Feb 23 02:26:38 domain mysql[1963]:    ...fail!
Feb 23 02:26:38 domain systemd[1]: mysql.service: Control process exited, code=exited status=1
Feb 23 02:26:38 domain systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.

Mysqld_safe MIA

Main line that stood out was:

/etc/init.d/mysql: line 114: /usr/bin/mysqld_safe: No such file or directory

The mysql Safe was missing in action. When I finally Googled this issue I bumped into many threads- example – recommending me to reinstall MySQL.

MySQL.Sock Restart

With a:

mysqld --datadir=/var/lib/mysql --user=mysql --socket=/var/run/mysqld/mysqld.sock

I was able to get MySQL up and running again. Earlier commands like

service mysql start,
service mysql restart,
/etc/init.d/mysql start

failed.

Then I still had the issue so I go looking, and mysqld_safe doesn’t exist on this system as stated before.

MariaDB Purge & Installation

So as suggested by Swalkinshaw I did a:

apt-get purge mariadb-client mariadb-server

and I got:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package 'mariadb-client' is not installed, so not removed
Package 'mariadb-server' is not installed, so not removed

So it said there was nothing to remove!

Trellis Provisioning Failure

When I re-provisioned (as I did not notice the fact the removal had not worked) I got a failure in the provisioning. Here is the part of the output on the failure (shortened somewhat with dots as replacement of some parts):

'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options
::=--force-confold"     install 'mariadb-server'' failed: No apport report
written because the error message indicates its a followup error from a
previous failure.
E: Sub-process /usr/bin/dpkg returned an error code (1)

No apport report written because the error message indicates its a followup
error from a previous failure.
E: Sub-process /usr/bin/dpkg returned an error code (1)

fatal: [xxx.131.166.212]: FAILED! => {"cache_update_time": 1519360682, "cache_updated": false, "changed": false, "failed": true, "rc": 100, "stderr_lines": ["No apport report written because the error message indicates its a followup error from a previous failure.", "E: Sub-process /usr/bin/dpkg returned an error code (1)"], "stdout": "Reading package lists...\nBuilding dependency ......... mariadb-server mariadb-server-10.2\n  mariadb-server-core-10.2 socat\n0 upgraded, 18 newly installed, 0 to remove and 98 not upgraded.\nNeed to get 18.3 MB of archives.\nAfter this operation, 147 MB of additional disk space will be used.\nGet:1 http://ftp.osuosl.org/pub/mariadb/repo/10.2/ubuntu xenial/main amd64 galera-3 amd64 25.3.23-xenial [8015 kB]\nGet:2 ..................http://nyc2.mirrors.digitalocean.com/ubuntu xenial/main amd64 libhttp-message-perl all 6.11-1 [74.3 kB]\nPreconfiguring packages ...\nFetched 18.3 MB in 1s (14.8 MB/s)\nSelecting previously unselected package galera-3.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 97853 files and directories currently installed.)\r\nPreparing to unpack .../galera-3_25.3.23-xenial_amd64.deb ...\r\nUnpacking galera-3 (25.3.23-xenial) ...\r\nSelecting previously unselected package libaio1:amd64.\r\nPreparing to unpack .../libaio1_0.3.110-2_amd64.deb ...\r\nUnpacking libaio1:amd64 (0.3.110-2) ...\r\nSelecting previously unselected package mariadb-server-core-10.2.\r\nPreparing to unpack .../mariadb-server-core-10.2_10.2.13+maria~xenial_amd64.deb ...\r\nUnpacking mariadb-server-core-10.2 (10.2.13+maria~xenial) ...\r\nSelecting previously unselected package socat.\r\nPreparing to unpack .../socat_1.7.3.1-1_amd64.deb ...\r\nUnpacking socat (1.7.3.1-1) ...\r\nSelecting previously unselected package mariadb-server-10.2.\r\nPreparing to unpack .../mariadb-server-10.2_10.2.13+maria~xenial_amd64.deb ...\r\n/var/lib/mysql: found previous version 10.0\r\nUnpacking mariadb-server-10.2 (10.2.13+maria~xenial) ...\r\nSelecting previously unselected package libhtml-tagset-perl.\r\nPreparing to unpack .../libhtml-tagset-perl_3.20-2_all.deb ...\r\nUnpacking libhtml-tagset-perl (3.20-2) ...\r\nSelecting previously unselected package liburi-perl.\r\nPreparing to unpack .../liburi-perl_1.71-1_all.deb ...\r\nUnpacking liburi-perl (1.71-1) ...\r\nSelecting previously unselected package libhtml-parser-perl.\r\nPreparing to unpack .../libhtml-parser-perl_3.72-1_amd64.deb ...\r\nUnpacking libhtml-parser-perl (3.72-1) ...\r\nSelecting previously unselected package libcgi-...
 ....................... 
"Get:9 http://nyc2.mirrors.digitalocean.com/ubuntu xenial/main amd64 libfcgi-perl amd64 0.77-1build1 [32.3 kB]", "Get:10 http://ftp.osuosl.org/pub/mariadb/repo/10.2/ubuntu xenial/main amd64 mariadb-server-10.2 amd64 10.2.13+maria~xenial [3897 kB]", "Get:11 http://nyc2.mirrors.digitalocean.com/ubuntu xenial/main amd64 libcgi-fast-perl all 1:2.10-1 [10.2 kB]", "Get:12 http://ftp.osuosl.org/pub/mariadb/repo/10.2/ubuntu xenial/main amd64 mariadb-server all 10.2.13+maria~xenial [2962 B]", "Get:13 http://nyc2.mirrors.digitalocean.com/ubuntu xenial/main amd64 libencode-locale-perl all 1.05-1 [12.3 kB]", "Get:14 .......................", "Fetched 18.3 MB in 1s (14.8 MB/s)", "Selecting previously unselected package galera-3.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 97853 files and directories currently installed.)", "Preparing to unpack .../galera-3_25.3.23-xenial_amd64.deb ...", "Unpacking galera-3 (25.3.23-xenial) ...", "Selecting previously unselected package libaio1:amd64.", "Preparing to unpack .../libaio1_0.3.110-2_amd64.deb ...", "Unpacking libaio1:amd64 (0.3.110-2) ...", "Selecting previously unselected package mariadb-server-core-10.2.", "Preparing to unpack .../mariadb-server-core-10.2_10.2.13+maria~xenial_amd64.deb ...", "Unpacking mariadb-server-core-10.2 (10.2.13+maria~xenial) ...", "Selecting previously unselected package socat.", "Preparing to unpack .../socat_1.7.3.1-1_amd64.deb ...", "Unpacking socat (1.7.3.1-1) ...", "Selecting previously unselected package mariadb-server-10.2.", "Preparing to unpack .../mariadb-server-10.2_10.2.13+maria~xenial_amd64.deb ...", "/var/lib/mysql: found previous version 10.0", "Unpacking mariadb-server-10.2 (10.2.13+maria~xenial) ...", "Selecting previously unselected package libhtml-tagset-perl.", "Preparing to unpack .../libhtml-tagset-perl_3.20-2_all.deb ...", "Unpacking libhtml-tagset-perl .......
"Feb 23 04:50:20 publiqly systemd[1]: \u001b[0;1;31mFailed to start MariaDB 10.2.13 databa...r.\u001b[0m", "Feb 23 04:50:20 publiqly systemd[1]: \u001b[0;1;39mmariadb.service: Unit entered failed state.\u001b[0m", "Feb 23 04:50:20 publiqly systemd[1]: \u001b[0;1;39mmariadb.service: Failed with result 'e...'.\u001b[0m", "Hint: Some lines were ellipsized, use -l to show in full.", "dpkg: error processing package mariadb-server-10.2 (--configure):", " subprocess installed post-installation script returned error exit status 1", "Setting up libhtml-tagset-perl 
......
"dpkg: error processing package mariadb-server (--configure):", " dependency problems - leaving unconfigured", "Processing triggers for libc-bin (2.23-0ubuntu10) ...", "Processing triggers for systemd (229-4ubuntu21) ...", "Processing triggers for ureadahead (0.100.0-19) ...", "Errors were encountered while processing:", " mariadb-server-10.2", " mariadb-server"]}

However, when I checked whether the database was up and running it was and the databases were there as well.. Weird.

DPKG Dependency Problems

However when I read

dpkg: dependency problems prevent configuration of mariadb-server:", " mariadb-server depends on mariadb-server-10.2 (>= 10.2.13+maria~xenial); however:", " Package mariadb-server-10.2 is not configured yet.", "", "dpkg: error processing package mariadb-server (--configure):", " dependency problems - leaving unconfigured

I saw there was a dependency problem.

DPKG Lock Issues

I also had issues running provisioning as it stated apt was busy:

fatal: [104.131.166.212]: FAILED! => {"changed": false, "rc": 100, "stdout": "/usr/bin/python\r\nE: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)\r\nE: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?\r\n", "stdout_lines": ["/usr/bin/python", "E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)", "E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?"]}

So I did a

sudo rm /var/lib/dpkg/lock

Once I did that I ran into

non-zero return code
Shared connection to xxx.xxx.xxx.xxx closed.
fatal: [104.131.166.212]: FAILED! => {"changed": false, "rc": 100, "stdout": "/usr/bin/python\r\nE: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. \r\n", "stdout_lines": ["/usr/bin/python", "E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. "]}

DPKG Config issues

So I ran a

sudo dpkg --configure -a

but then I got into this error

Setting up mariadb-server-10.2 (10.2.13+maria~xenial) ...
debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process: Resource temporarily unavailable
dpkg: error processing package mariadb-server-10.2 (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mariadb-server:
mariadb-server depends on mariadb-server-10.2 (>= 10.2.13+maria~xenial); however:
Package mariadb-server-10.2 is not configured yet.
dpkg: error processing package mariadb-server (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
mariadb-server-10.2
mariadb-server

This I think was because another MariaDB version is running, but one ignored by the Trellis MariaDB playbook.
Then I decided to try and reconfigure the MariaDB Server:

dpkg-reconfigure mariadb-server-10.2
debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process: Resource temporarily unavailable

So I checked what user was using the process and it was root. Then I killed the PID:

sudo fuser -v /var/cache/debconf/config.dat
kill PID

And that way I was able to start upgrading PHP, but MariaDB part of

ansible-playbook server.yml -e env=production --tags=php

failed anyways..

"Setting up mariadb-server-10.2 (10.2.13+maria~xenial) ...", "Job for mariadb.service failed because the control process exited with error code. See \"systemctl status mariadb.service\" and \"journalctl -xe\" for details.", "invoke-rc.d: initscript mysql, action \"start\" failed.", "\u001b[0;1;31m*\u001b[0m mariadb.service - MariaDB 10.2.13 database server", "   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)", "  Drop-In: /etc/systemd/system/mariadb.service.d", "           `-migrated-from-my.cnf-settings.conf", "   Active: \u001b[0;1;31mfailed\u001b[0m (Result: exit-code) since Mon 2018-04-23 08:34:24 UTC; 8ms ago", "     Docs: man:mysqld(8)", "           https://mariadb.com/kb/en/library/systemd/", "  Process: 9403 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION \u001b[0;1;31m(code=exited, status=1/FAILURE)\u001b[0m", "  Process: 9303 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)", "  Process: 9296 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)", "  Process: 9291 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)", " Main PID: 9403 (code=exited, status=1/FAILURE)", "   Status: \"MariaDB server is down\"", "", "Apr 23 08:32:12 publiqly systemd[1]: Starting MariaDB 10.2.13 database server...", "Apr 23 08:32:12 publiqly mysqld[9403]: \u001b[0;1;31m2018-04-23  8:32:12 140550644848832 .....\u001b[0m", "Apr 23 08:33:42 publiqly systemd[1]: \u001b[0;1;39mmariadb.service: Start operation timed...g.\u001b[0m", "Apr 23 08:34:24 publiqly systemd[1]: \u001b[0;1;39mmariadb.service: Main process exited, ...RE\u001b[0m", "Apr 23 08:34:24 publiqly systemd[1]: \u001b[0;1;31mFailed to start MariaDB 10.2.13 databa...r.\u001b[0m", "Apr 23 08:34:24 publiqly systemd[1]: \u001b[0;1;39mmariadb.service: Unit entered failed state.\u001b[0m", "Apr 23 08:34:24 publiqly systemd[1]: \u001b[0;1;39mmariadb.service: Failed with result 'e...'.\u001b[0m", "Hint: Some lines were ellipsized, use -l to show in full.", "dpkg: error processing package mariadb-server-10.2 (--configure):", " subprocess installed post-installation script returned error exit status 1", "dpkg: dependency problems prevent configuration of mariadb-server:", " mariadb-server depends on mariadb-server-10.2 (>= 10.2.13+maria~xenial); however:", "  Package mariadb-server-10.2 is not configured yet.", "", "dpkg: error processing package mariadb-server (--configure):", " dependency problems - leaving unconfigured",

Decided to check what MariaDB stuff I have installed:

apt list --installed  |grep mariadb

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libmariadb3/now 10.2.12+maria~xenial amd64 [installed,upgradable to: 10.2.14+maria~xenial]
libmariadbclient18/now 10.2.12+maria~xenial amd64 [installed,upgradable to: 10.2.14+maria~xenial]
mariadb-client/now 10.2.13+maria~xenial all [installed,upgradable to: 10.2.14+maria~xenial]
mariadb-client-10.2/now 10.2.13+maria~xenial amd64 [installed,upgradable to: 10.2.14+maria~xenial]
mariadb-client-core-10.2/now 10.2.13+maria~xenial amd64 [installed,upgradable to: 10.2.14+maria~xenial]
mariadb-common/now 10.2.13+maria~xenial all [installed,upgradable to: 10.2.14+maria~xenial]
mariadb-server/now 10.2.13+maria~xenial all [installed,upgradable to: 10.2.14+maria~xenial]
mariadb-server-10.2/now 10.2.13+maria~xenial amd64 [installed,upgradable to: 10.2.14+maria~xenial]
mariadb-server-core-10.2/now 10.2.13+maria~xenial amd64 [installed,upgradable to: 10.2.14+maria~xenial]

Installation from Scratch

Most SO threads including on on MariaDB recommends me something like:

1) sudo apt-get remove --purge mariadb-server mariadb-client 
2) sudo apt-get autoremove
3) sudo apt-get autoclean
4) sudo apt-get install mariadb-server mariadb-client

This is similar to what Swalkinshaw stated, but now remove and autoremove are added as well as autoclean.

Dry Run

I decided to do a dry run

sudo apt-get purge mariadb-server* mariadb-client* --dry-run

This is the gist of what was stated:

The following packages were automatically installed and are no longer required:
 galera-3 libaio1 libcgi-fast-perl libcgi-pm-perl libdbd-mysql-perl libdbi-perl libencode-locale-perl libfcgi-perl libhtml-parser-perl
 libhtml-tagset-perl libhtml-template-perl libhttp-date-perl libhttp-message-perl libio-html-perl liblwp-mediatypes-perl libmariadb3
 libmariadbclient18 liburi-perl socat
 Use 'sudo apt autoremove' to remove them.
 The following packages will be REMOVED:
 mariadb-client* mariadb-client-10.0* mariadb-client-10.2* mariadb-client-core-10.2* mariadb-server* mariadb-server-10.0* mariadb-server-10.2*
 mariadb-server-core-10.2*
 0 upgraded, 0 newly installed, 8 to remove and 161 not upgraded.
 2 not fully installed or removed.
 Purg mariadb-server [10.2.13+maria~xenial]
 Purg mariadb-server-10.2 [10.2.13+maria~xenial]
 Purg mariadb-client [10.2.13+maria~xenial]
 Purg mariadb-client-10.0
 Purg mariadb-client-10.2 [10.2.13+maria~xenial]
 Purg mariadb-client-core-10.2 [10.2.13+maria~xenial]
 Purg mariadb-server-10.0
 Purg mariadb-server-core-10.2 [10.2.13+maria~xenial]

Comments are closed for this post.