监控告警：CYDB CPU 使用率超过 90%

登录主机查看 CPU 使用情况：

top - 15:53:49 up 3195 days, 23:36,  3 users,  load average: 4.03, 4.03, 4.00
Tasks: 254 total,   5 running, 247 sleeping,   2 stopped,   0 zombie
Cpu0  : 38.3%us, 61.7%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 37.1%us, 62.9%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 38.5%us, 61.5%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  : 39.3%us, 60.7%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16336264k total, 10029964k used,  6306300k free,   147380k buffers
Swap: 16777212k total,  2404828k used, 14372384k free,  8406040k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29599 root      20   0  122m 1636 1584 R 100.0  0.0 173211:38 mysql
26024 root      20   0  123m 1076 1024 R 99.7  0.0  47938,11 mysql
31105 root      20   0  124m 1092 1040 R 99.7  0.0  45476,12 mysql
13401 root      20   0  122m 2412 1724 R 99.0  0.0 279:07.95 mysql

检查数据库活动：

mysql> show processlist;
+----------+------+-------------------+------+-------------+-----------+-----------------------------------------------------------------------+------------------+
| Id       | User | Host              | db   | Command     | Time      | State                                                                 | Info             |
+----------+------+-------------------+------+-------------+-----------+-----------------------------------------------------------------------+------------------+
|       46 | sync | 113.3.8.127:17810 | NULL | Binlog Dump | 147733995 | Master has sent all binlog to slave; waiting for binlog to be updated | NULL             |
| 18333887 | root | localhost         | NULL | Sleep       |     18882 |                                                                       | NULL             |
| 18335620 | root | localhost         | NULL | Query       |         0 | init                                                                  | show processlist |
+----------+------+-------------------+------+-------------+-----------+-----------------------------------------------------------------------+------------------+
3 rows in set (0.00 sec)

数据库没有活动。

仔细观察发现，使用CPU的进程是 mysql，并不是 mysqld，看一下 mysql 相关进程：

[root@xuwdb ~]# ps aux | grep -i mysql
root      2035  0.0  0.0 106100   692 ?        S     2021   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/xuwdb.pid
mysql     2433  0.1  1.5 2194260 258368 ?      Sl    2021 3718:30 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/xuwdb.err --pid-file=/var/lib/mysql/xuwdb.pid
root      5033  0.0  0.0 103256   868 pts/9    S+   15:55   0:00 grep -i mysql
root     13401 89.0  0.0 125936  2412 ?        R    10:40 280:54 mysql -uroot -px xxxxxx
root     26024 99.7  0.0 126376  1076 ?        R     2020 2876293:06 mysql -uroot
root     29599 99.5  0.0 125936  1636 ?        R    Jan28 173213:25 mysql
root     31105 99.7  0.0 127620  1092 ?        R     2021 2728573:54 mysql -uroot -p

将上面4个客户端进程 kill 掉以后，恢复正常。