OProfile, Kernel Images and InnoDB (oh my!)

Kristian Köhntopp has a wonderful article about using oprofile to track down problems in running programs. I thought I’d add a few thoughts.

If you need to get a vmlinux kernel on redhat, apparently you just need to install kernel-debuginfo, which will provide a vmlinux image you can profile against.

If you are using debian, unfortunately there is no package I could find to allow you to get a vmlinux. so what I did was:
($kver isn’t a real variable – it’s your kernel version. tab completion probably comes in handy at some point)

1. Install linux-tree-$kver – which gets you the debian kernel sources
2. Unpack the tar.bz2 file that is now in /usr/src
3. Copy /boot/config-$kver to /usr/src/linux-$kver/.config
4. cd /usr/src/linux-$kver
5. make oldconfig
6. make prepare
7. make vmlinux
8. cp vmlinux /boot

And now I get things that look like this:

322451    5.7214  /usr/local/mysql-debug-4.1.22-unknown-linux-gnu-x86_64-glibc23/bin/mysqld  cmp_dtuple_rec_with_match
304963    5.4111  /lib/libpthread-0.60.so                                                                            __pthread_rwlock_rdlock_internal
275748    4.8927  /usr/local/mysql-debug-4.1.22-unknown-linux-gnu-x86_64-glibc23/bin/mysqld  my_lengthsp_8bit
275503    4.8884  /boot/vmlinux                                                                                       try_to_wake_up
221124    3.9235  /lib/libpthread-0.60.so                                                                            __pthread_rwlock_wrlock_internal
188777    3.3496  /boot/vmlinux                                                                                       futex_wait
170249    3.0208  /usr/local/mysql-debug-4.1.22-unknown-linux-gnu-x86_64-glibc23/bin/mysqld  btr_search_guess_on_hash
144030    2.5556  /lib/libc-2.3.2.so                                                                                   _wordcopy_bwd_aligned
131726    2.3373  /usr/local/mysql-debug-4.1.22-unknown-linux-gnu-x86_64-glibc23/bin/mysqld  rec_get_nth_field
125462    2.2261  /lib/libpthread-0.60.so                                                                            __pthread_mutex_unlock_internal
122389    2.1716  /usr/local/mysql-debug-4.1.22-unknown-linux-gnu-x86_64-glibc23/bin/mysqld  safe_mutex_lock
116276    2.0631  /boot/vmlinux                                                                                       futex_wake
115654    2.0521  /usr/local/mysql-debug-4.1.22-unknown-linux-gnu-x86_64-glibc23/bin/mysqld  safe_mutex_unlock

Which – btw – shows what your system looks like when it’s experiencing the InnoDB thread concurrency bug.