x
login Signup

hung_task_timeouts and underperforming agents in ubuntu 10.4 and RedHat 5.6 and later

If your recent Ubuntu or RedHat (possibly Suse as well) system with kernel versions 2.6.31,32,33,34 is experiencing erratic slowdowns, it might be due to a known bug in the linux kernel. A smoking gun for this problem is the following message (shown by "dmesg"). I've seen evidence of slowdowns (due to apparent i/o deadlocks) that did not last 120 seconds, and therefore did not log the following but were still affecting performance. Often, these slowdowns will mark the agent red for a bit of time in the CM list of agents.

{code} [16208.621077] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[16208.642946] ecagent D 0000000000000000 0 5302 1 0x00000000
[16208.642959] ffff880475531d98 0000000000000082 0000000000015bc0 0000000000015bc0
[16208.642977] ffff8804755a83c0 ffff880475531fd8 0000000000015bc0 ffff8804755a8000
[16208.642996] 0000000000015bc0 ffff880475531fd8 0000000000015bc0 ffff8804755a83c0
[16208.643014] Call Trace:
[16208.643031] [] ? bdi_sched_wait+0x0/0x20 [16208.643040] [] bdi_sched_wait+0xe/0x20 [16208.643051] [] _ wait_on_bit+0x5f/0x90 [16208.643064] [] ? bdi_sched_wait+0x0/0x20 [16208.643073] [] out_of_line_wait_on_bit+0x78/0x90 [16208.643084] [] ? wake_bit_function+0x0/0x40 [16208.643092] [] ? bdi_queue_work+0xa4/0xe0 [16208.643101] [] bdi_sync_writeback+0x6f/0x80 [16208.643109] [] sync_inodes_sb+0x20/0x30 [16208.643119] [] _sync_filesystem+0x82/0x90 [16208.643127] [] sync_filesystems+0xd9/0x130 [16208.643137] [] sys_umount+0xb1/0xd0 [16208.643149] [] system_call_fastpath+0x16/0x1b {code}

avatar image By peterg 2k asked Mar 30, 2012 at 10:41 AM
more ▼
(comments are locked)
10|750 characters needed characters left

1 answer: sort voted first
avatar image By peterg 2k answered Mar 30, 2012 at 10:47 AM
more ▼
(comments are locked)
10|750 characters needed characters left
Your answer
toggle preview:

Up to 8 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.