Any ideas on why the load would be through the roof, top shows each CPU user-space utilization at 85% or higher, yet no process is tagged with any CPU utilization?
First time I've ever seen this. We've got a server that has a load average of 20, and the break down on each CPU is extremely high (>85%), yet none of the processes in the list have any CPU % utilization. Here's a sample, and yes, this is with the processes sorted from highest to lowest CPU utilization - every once in a blue moon we'll actually see a process that shows "1" as the CPU %.
This is a mysql server box.
top - 16:08:48 up 185 days, 15:27, 5 users, load average: 20.60, 17.20, 13.17
Tasks: 221 total, 2 running, 219 sleeping, 0 stopped, 0 zombie
Cpu0 : 90.3%us, 5.8%sy, 0.0%ni, 3.2%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 89.1%us, 5.8%sy, 0.0%ni, 4.5%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 86.4%us, 5.0%sy, 0.9%ni, 6.9%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 88.1%us, 5.8%sy, 1.3%ni, 4.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 89.8%us, 7.2%sy, 0.0%ni, 1.3%id, 0.0%wa, 0.0%hi, 1.6%si, 0.0%st
Cpu5 : 91.0%us, 2.6%sy, 0.0%ni, 5.4%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 91.8%us, 4.6%sy, 0.3%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 78.8%us, 14.4%sy, 1.3%ni, 5.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 12326280k total, 12147144k used, 179136k free, 158512k buffers
Swap: 8420344k total, 0k used, 8420344k free, 10731260k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22691 root 20 0 19348 1536 1068 R 1 0.0 0:00.29 top