MySQL crashing InnoDB: mmap(bytes) failed; errno 12

Jim
6 min readApr 5, 2018

--

Sometimes there are things that you keep coming back to, this is one of them. Often this occurs when I launch a WordPress site on a new server, so here be the process to take:

“Error establishing a Database Connection”

If this happens, your database is not accessible to the web server.

If you’re running MySQL (likely, if you’re running WordPress) then you should check your MySQL log files (or contact your server admin/hosting company). You’re also likely using a Linux operating system, and your MySQL configuration file is likely in this place:

/etc/mysql/my.cnf

So you need to use a user account that is allowed to edit this file (likely root/ su/ sudo). Assumes you are using that user, and you have vi installed.

vi /etc/mysql/my.cnf

Now look for the location of your log:

/log_error

It’s probably here:

log_error = /var/log/mysql/error.log

Go through your log file and look for the time your MySQL server crashed.

This is what’s in mine:

2018–03–29 01:03:45 8946 [Note] Plugin ‘FEDERATED’ is disabled.
2018–03–29 01:03:45 8946 [ERROR] Function ‘innodb’ already exists
2018–03–29 01:03:45 8946 [Warning] Couldn’t load plugin named ‘innodb’ with soname ‘ha_innodb.so’.
2018–03–29 01:03:45 8946 [ERROR] Function ‘federated’ already exists
2018–03–29 01:03:45 8946 [Warning] Couldn’t load plugin named ‘federated’ with soname ‘ha_federated.so’.
2018–03–29 01:03:45 8946 [ERROR] Function ‘blackhole’ already exists
2018–03–29 01:03:45 8946 [Warning] Couldn’t load plugin named ‘blackhole’ with soname ‘ha_blackhole.so’.
2018–03–29 01:03:45 8946 [ERROR] Function ‘archive’ already exists
2018–03–29 01:03:45 8946 [Warning] Couldn’t load plugin named ‘archive’ with soname ‘ha_archive.so’.
2018–03–29 01:03:45 8946 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018–03–29 01:03:45 8946 [Note] InnoDB: The InnoDB memory heap is disabled
2018–03–29 01:03:45 8946 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018–03–29 01:03:45 8946 [Note] InnoDB: Memory barrier is not used
2018–03–29 01:03:45 8946 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018–03–29 01:03:45 8946 [Note] InnoDB: Using Linux native AIO
2018–03–29 01:03:45 8946 [Note] InnoDB: Using CPU crc32 instructions
2018–03–29 01:03:45 8946 [Note] InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
2018–03–29 01:03:45 8946 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2018–03–29 01:03:45 8946 [ERROR] Plugin ‘InnoDB’ init function returned error.
2018–03–29 01:03:45 8946 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.
2018–03–29 01:03:45 8946 [ERROR] Unknown/unsupported storage engine: InnoDB
2018–03–29 01:03:45 8946 [ERROR] Aborting
2018–03–29 01:03:46 8979 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
2018–03–29 01:03:46 8979 [Note] Plugin ‘FEDERATED’ is disabled.
2018–03–29 01:03:46 8979 [ERROR] Function ‘innodb’ already exists
2018–03–29 01:03:46 8979 [Warning] Couldn’t load plugin named ‘innodb’ with soname ‘ha_innodb.so’.
2018–03–29 01:03:46 8979 [ERROR] Function ‘federated’ already exists
2018–03–29 01:03:46 8979 [Warning] Couldn’t load plugin named ‘federated’ with soname ‘ha_federated.so’.
2018–03–29 01:03:46 8979 [ERROR] Function ‘blackhole’ already exists
2018–03–29 01:03:46 8979 [Warning] Couldn’t load plugin named ‘blackhole’ with soname ‘ha_blackhole.so’.
2018–03–29 01:03:46 8979 [ERROR] Function ‘archive’ already exists
2018–03–29 01:03:46 8979 [Warning] Couldn’t load plugin named ‘archive’ with soname ‘ha_archive.so’.
2018–03–29 01:03:46 8979 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018–03–29 01:03:46 8979 [Note] InnoDB: The InnoDB memory heap is disabled
2018–03–29 01:03:46 8979 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018–03–29 01:03:46 8979 [Note] InnoDB: Memory barrier is not used
2018–03–29 01:03:46 8979 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018–03–29 01:03:46 8979 [Note] InnoDB: Using Linux native AIO
2018–03–29 01:03:46 8979 [Note] InnoDB: Using CPU crc32 instructions
2018–03–29 01:03:46 8979 [Note] InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
2018–03–29 01:03:46 8979 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2018–03–29 01:03:46 8979 [ERROR] Plugin ‘InnoDB’ init function returned error.
2018–03–29 01:03:46 8979 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.
2018–03–29 01:03:46 8979 [ERROR] Unknown/unsupported storage engine: InnoDB
2018–03–29 01:03:46 8979 [ERROR] Aborting
2018–03–29 01:03:46 8979 [Note] Binlog end
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘partition’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_SYS_DATAFILES’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_SYS_TABLESPACES’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN_COLS’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_SYS_FOREIGN’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_SYS_FIELDS’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_SYS_COLUMNS’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_SYS_INDEXES’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_SYS_TABLESTATS’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_SYS_TABLES’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_FT_INDEX_TABLE’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_FT_INDEX_CACHE’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_FT_CONFIG’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_FT_BEING_DELETED’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_FT_DELETED’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_FT_DEFAULT_STOPWORD’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_METRICS’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_BUFFER_POOL_STATS’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE_LRU’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_BUFFER_PAGE’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX_RESET’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_CMP_PER_INDEX’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_CMPMEM_RESET’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_CMPMEM’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_CMP_RESET’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_CMP’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_LOCK_WAITS’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_LOCKS’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘INNODB_TRX’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘ARCHIVE’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘PERFORMANCE_SCHEMA’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘BLACKHOLE’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘MEMORY’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘MRG_MYISAM’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘MyISAM’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘CSV’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘sha256_password’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘mysql_old_password’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘mysql_native_password’
2018–03–29 01:03:46 8979 [Note] Shutting down plugin ‘binlog’
2018–03–29 01:03:46 8979 [Note] /usr/sbin/mysqld: Shutdown complete

This is what stands out (as the preceding errors are saying “already exists”):

InnoDB: mmap(137363456 bytes) failed; errno 12
2018–03–29 01:03:45 8946 [ERROR] InnoDB: Cannot allocate memory for the buffer pool

Which looks like a memory issue, so let’s check the syslog for mentions of MySQL (with logs, if you have lots of messages or you’ve got a lot of time to backtrack you might find you need to look in your gzipped archived logs)

grep “mysql” /var/log/syslog

Which displayed this:

Apr 5 01:53:52 Server-name kernel: [1255339.940320] [ 4117] 106 4117 327829 111643 270 0 0 mysqld
Apr 5 01:53:52 Server-name kernel: [1255339.940358] Out of memory: Kill process 4117 (mysqld) score 439 or sacrifice child
Apr 5 01:53:52 Server-name kernel: [1255339.940383] Killed process 4117 (mysqld) total-vm:1311316kB, anon-rss:446572kB, file-rss:0kB
Apr 5 01:53:52 Server-name kernel: [1255339.946702] init: mysql main process (4117) killed by KILL signal
Apr 5 01:53:52 Server-name kernel: [1255339.946717] init: mysql main process ended, respawning
Apr 5 01:53:52 Server-name kernel: [1255339.986959] type=1400 audit(1522907632.659:36): apparmor=”STATUS” operation=”profile_replace” profile=”unconfined” name=”/usr/sbin/mysqld” pid=20312 comm=”apparmor_parser”
Apr 5 01:53:53 Server-name kernel: [1255340.442943] init: mysql main process (20323) terminated with status 1
Apr 5 01:53:53 Server-name kernel: [1255340.442955] init: mysql main process ended, respawning
Apr 5 01:53:53 Server-name kernel: [1255341.061413] init: mysql post-start process (20324) terminated with status 1
Apr 5 01:53:53 Server-name kernel: [1255341.074043] type=1400 audit(1522907633.747:37): apparmor=”STATUS” operation=”profile_replace” profile=”unconfined” name=”/usr/sbin/mysqld” pid=20345 comm=”apparmor_parser”
Apr 5 01:53:54 Server-name kernel: [1255341.353654] init: mysql main process (20356) terminated with status 1
Apr 5 01:53:54 Server-name kernel: [1255341.353665] init: mysql respawning too fast, stopped
Apr 5 04:23:51 Server-name kernel: [1264339.250198] type=1400 audit(1522916631.923:38): apparmor=”STATUS” operation=”profile_replace” profile=”unconfined” name=”/usr/sbin/mysqld” pid=21604 comm=”apparmor_parser”
Apr 5 04:23:52 Server-name /etc/mysql/debian-start[21651]: Upgrading MySQL tables if necessary.
Apr 5 04:23:53 Server-name /etc/mysql/debian-start[21654]: Looking for ‘mysql’ as: /usr/bin/mysql
Apr 5 04:23:53 Server-name /etc/mysql/debian-start[21654]: Looking for ‘mysqlcheck’ as: /usr/bin/mysqlcheck
Apr 5 04:23:53 Server-name /etc/mysql/debian-start[21654]: This installation of MySQL is already upgraded to 5.6.33, use — force if you still need to run mysql_upgrade
Apr 5 04:23:53 Server-name /etc/mysql/debian-start[21661]: Checking for insecure root accounts.
Apr 5 04:23:53 Server-name /etc/mysql/debian-start[21665]: Triggering myisam-recover for all MyISAM tables

This is the standout part of that:

Out of memory: Kill process 4117 (mysqld) score 439 or sacrifice child

So let’s check that you have ‘spare memory’ in the form of swap which allows us to extend the available physical memory (RAM) by using space on the disk drive:

free -m

Which displays this:

total used free shared buffers cached
Mem: 993 890 103 64 19 159
-/+ buffers/cache: 711 282
Swap: 0 0 0

The…

Swap: 0 0 0

… means we need to setup swap space.

The WordPress site I just launched I did so on DigitalOcean plus their guides are great (as long as you make sure you’re reading for the appropriate operating system) so I defer you to the following:

--

--

Jim
Jim

Written by Jim

A developer in web and beyond; a huge original-trilogy Mass Effect fan; for anything else, just ask :)