Detected table cache mutex contention at instance 1: 60% waits. Additional table cache instance activated. Number of instances after activation: 2.

Scenario:

All three nodes of the Galera cluster use 98 percent of memory and swapping heavily, no DML could be performed.

Investigation:

From the error log on node 3 the below error has been notified:

Detected table cache mutex contention at instance 1: 60% waits. Additional table cache instance activated. Number of instances after activation: 2.

Querying the global status for the variable table_open_cache_misses and table_open_cache_overflows shows high cache misses and overflows.

The table_open_cache variable current value is 600 during the bottleneck.

Solution:

The type of operation performed by the application will employ many concurrent sessions and batch-like inserts which will require a high table cache, hence increasing the open_table_cache to a value of 2000 solved the issue.