Facing up to a kernel panic (II)

From the previous article, facing up to a kernel panic (I), let's take advantage of being in rescue mode to fix the init file (the installation CD must be inserted into the system).

bash-4.1# mkdir /mnt/cdrw ; mount /dev/cdrw /mnt/cdrw

bash-4.1# rpm -Uvh --force --root=/mnt/sysimage /mnt/cdrw/Packages/upstart-0.6.5-10.el6.i686.rpm

Remember to identify the package associated with a certain file, you can run the following query.

[root@localhost ~]# rpm -qf /sbin/init
upstart-0.6.5-10.el6.i686

Another handy query is to list the files included inside a package.

[root@localhost ~]# rpm -ql httpd
/etc/httpd
/etc/httpd/conf
...

Continuing with other useful techniques to troubleshoot systems, we have to take into account possible failures related to the initial RAM disk, a temporary filesystem used during the boot process and uncompressed within the RAM memory, which includes kernel modules and user space programs needed to mount actual filesystems and run the first initialization programs. For example, so as to be able to mount and handle an ext4 filesystem, it is necessary that the ext4 module is previously loaded in the RAM disk.

A typical problem about this issue is for instance that some kind of module had not been inserted or properly updated inside the RAM disk. In order to take a look at the file structure of a RAM disk, the following command can be run.

[root@localhost ~]# lsinitrd /boot/initramfs-`uname -r`.img | more
/boot/initramfs-2.6.32-220.7.1.el6.i686.img:
========================================================================
dracut-004-256.el6_2.1
========================================================================
drwxr-xr-x  23 root     root            0 Mar 31 16:45 .
drwxr-xr-x   2 root     root            0 Mar 31 16:45 initqueue-timeout
drwxr-xr-x   2 root     root            0 Mar 31 16:45 sbin
-rwxr-xr-x   1 root     root        10404 Mar  7 19:08 sbin/rmmod
...

If you need to add a concrete module, you can use the dracut tool. This application, aside from managing RAM disks, takes care of creating initial RAM disks with preloaded modules.

[root@localhost ~]# dracut --add-drivers my_module initramfs-`uname -r`.img `uname -r`

After troubleshooting the kernal panic and not finding a solution, another interesting option is to debug the kernel crash. I will go on with this part in the next article.