Apr 29, 2012

Ubuntu 12.04 LTS (Precise Pangolin)

Last Thursday April 26 was released the last version of Ubuntu (12.04), also known as Precise Pangolin. I have been using Kubuntu instead of Ubuntu from a long time, but last October I switched to Ubuntu, and I am very satisfied with the change. In my personal opinion, I see more professional Ubuntu than Kubuntu, and on many occasions, even more practical.

I have not had any problem during the upgrade. After the process, I had the new Ubuntu release ready to be used. As it has been already happening from many Ubuntu versions above, everything worked correctly after the installation. The stability and performance of this current version are insuperable.

What is new?

LTS (Long Term Support)

This release is LTS, that stands for Long Term Support. In other words, it will be supported throughout five years, both server and desktop version. So this is a perfect distribution to be used as a server whether we are thinking of installing new services.

HUD (Head-Up Display)

Rather than clicking on menu items to give them instructions, now you can press the 'Alt' key in order to bring up the HUD and type what you want to do. Then you will have a list of available functions (applications, files, Internet history, etc.) without needing to know in which menu they are located.

System Applications

By default, Ubuntu 12.04 provides Gnome 3.4.1, Linux kernel 3.2, Upstart 1.5, LibreOffice 3.5.2, Nautilus 3.3.5, Thunderbird 11, and so on. The default music player has been switched to Rhythmbox. Remmina with FreeRDP is the new standard remote desktop client, replacing Vinagre and rdesktop; and the new kernel introduces meaningful improvements in performance and power management.


Hibernate has been disabled by default and resolvconf is now used to manage /etc/resolv.conf on all Ubuntu systems.


From now on, Ubuntu allows to decide what user data (recently used files and applications which has been opened) should be stored in its history. Apart from this, it also allows to send error reports to Canonical.

Apr 22, 2012

Facing up to a kernel panic (II)

From the previous article, facing up to a kernel panic (I), let's take advantage of being in rescue mode to fix the init file (the installation CD must be inserted into the system).

bash-4.1# mkdir /mnt/cdrw ; mount /dev/cdrw /mnt/cdrw

bash-4.1# rpm -Uvh --force --root=/mnt/sysimage /mnt/cdrw/Packages/upstart-0.6.5-10.el6.i686.rpm

Remember to identify the package associated with a certain file, you can run the following query.

[root@localhost ~]# rpm -qf /sbin/init

Another handy query is to list the files included inside a package.

[root@localhost ~]# rpm -ql httpd

Continuing with other useful techniques to troubleshoot systems, we have to take into account possible failures related to the initial RAM disk, a temporary filesystem used during the boot process and uncompressed within the RAM memory, which includes kernel modules and user space programs needed to mount actual filesystems and run the first initialization programs. For example, so as to be able to mount and handle an ext4 filesystem, it is necessary that the ext4 module is previously loaded in the RAM disk.

A typical problem about this issue is for instance that some kind of module had not been inserted or properly updated inside the RAM disk. In order to take a look at the file structure of a RAM disk, the following command can be run.

[root@localhost ~]# lsinitrd /boot/initramfs-`uname -r`.img | more
drwxr-xr-x  23 root     root            0 Mar 31 16:45 .
drwxr-xr-x   2 root     root            0 Mar 31 16:45 initqueue-timeout
drwxr-xr-x   2 root     root            0 Mar 31 16:45 sbin
-rwxr-xr-x   1 root     root        10404 Mar  7 19:08 sbin/rmmod

If you need to add a concrete module, you can use the dracut tool. This application, aside from managing RAM disks, takes care of creating initial RAM disks with preloaded modules.

[root@localhost ~]# dracut --add-drivers my_module initramfs-`uname -r`.img `uname -r`

After troubleshooting the kernal panic and not finding a solution, another interesting option is to debug the kernel crash. I will go on with this part in the next article.

Apr 15, 2012

Facing up to a kernel panic (I)

Is there any Linux Engineer who has not run into a kernel panic throughout his career?

Let's get started by explaining what is a kernel panic. A kernel panic is a kind of error generated when the kernel receives an instruction in an unexpected or unknown format and cannot process it, and this situation leads in general to a system crash. A kernel panic can show up when the operating system is not able to recover from another preceding error, attempts to access invalid memory segments, a damage, incompatible or unsupported software or hardware, etc.

How must we face up to a kernel panic? First of all, by keeping calm and applying common sense.

I am going to simulate a typical kernel panic by corrupting the content of the /sbin/init file. If you try to do it when the operating system is running in any of the existing runlevels, you will get an error as follows (also mention that for this article, I will use a CentOS 6.2 distribution).

[root@localhost ~]# >/sbin/init
-bash: /sbin/init: Text file busy

The above message is normal. There is no way that init dies or is killed when is running.

[root@localhost ~]# pgrep init ; kill -SIGKILL 1 ; pgrep init

So as to harm the /sbin/init file, you can boot the system in rescue mode and then, perform it.

bash-4.1# chroot /mnt/sysimage/

sh-4.1# >/sbin/init ; exit

bash-4.1# sync ; reboot

Now if you restart the system, you will come across the following sequence of messages.

dracut: Mounted root filesystem /dev/mapper/VolGroup-lv_root
dracut: Loading SELinux policy
dracut: Switching root
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: switch_root Not tainted 2.6.32-220.7.1.el6.i686 #1

In the previous kernel panic, the system is trying to run a sync call in order to push all data into the hard drive before going down. And the second part of the message is the text string passed to the panic function. When a kernel panic is raised, Linux calls the panic routine defined in kernel/exit.c.

In this situation, the quickest action is to pick up the error line, go to Google and try to find out more information about the kernel panic by looking up related cases. If  this process does not work, then we will have to  read up on the system by starting it in rescue mode, so as to go over the different log files and try to obtain more information.

Specifically in this case, the booting in rescue mode will not be useful, since the filesystem will not have been able to be mounted in read/write mode, and consequently, the logs will not have been able to be dumped into disk. Perhaps this is the worst case, when the init process acts up. Otherwise, you would have been able to investigate files such as messages, dmesg, boot, etc.

Apr 9, 2012

Apache performance tuning: benchmarking

Let's continue with the series of articles about Apache performance tuning. Remember that in the previous ones, I approached the topics about dynamic modules (I and II) and directives (I and II).

So after reviewing the directives which allow to fit the global configuration of Apache, I am going to focus on a couple of tools used to adjust the values of those directives, based on the available hardware and the kind of service which wants to be offered.

The first application to be presented is ab (Apache benchmark), normally utilized in order to get hold of the performance values which the web server will be able to achieve. It is designed to give an impression of how your current Apache server works, specially by showing how many requests per second which is able to serve.

This tool puts forward multiple options, but the normal operation format so as to carry out performance tests will be as follows (there are other many useful options that you can check by going to the man):

[root@localhost ~]# ab -c concurrency -n request [-k] [http[s]://]hostname[:port]/path

Through the '-c' option is specified the number of multiple requests to perform at once, that is to say, it stands for the concurrence whereby the web server will be requested. For example, '-c 100' means that one hundred users will be launching requests simultaneously.

By means of the '-n' option, you may establish the number of requests to perform for the benchmarking session, that is, the total number of requests which will be run. For instance, '-c 20 -n 1000' means that 50 groups of 20 requests per group will be executed. And with the '-k' option, the HTTP keepalive feature is enabled. In this way, it is possible to handle multiple requests within one HTTP session.

[root@localhost ~]# ab -c 200 -n 1000 http://localhost/index.html
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:        Apache/2.2.15
Server Hostname:        localhost
Server Port:            80

Document Path:          /index.html
Document Length:        1656 bytes

Concurrency Level:      200
Time taken for tests:   1.696 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      1934625 bytes
HTML transferred:       1664280 bytes
Requests per second:    589.67 [#/sec] (mean)
Time per request:       339.175 [ms] (mean)
Time per request:       1.696 [ms] (mean, across all concurrent requests)
Transfer rate:          1114.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    8  10.0      4      38
Processing:    14  163 353.1     85    1645
Waiting:       11  160 353.2     81    1638
Total:         51  171 354.7     89    1660

Percentage of the requests served within a certain time (ms)
  50%     89
  66%     92
  75%     94
  80%     95
  90%    102
  95%   1637
  98%   1645
  99%   1651
 100%   1660 (longest request)

The most important values output by ab are Failed requests, Requests per second and Time per request. As you can distinguish in the above example, all requests have been completed successfully, with an average of 589 requests per second and an average time of 1.696 ms for each request.

Another interesting tool used to assess the performance of a web server is curl, which takes care of finding out the response time for an unique request of a single element. Curl transfers data from or to a server, by using one of the supported protocols (HTTP, HTTPS, FTP, etc.).

[root@localhost ~]# curl -o /dev/null -s -w %{time_connect}:%{time_starttransfer}:%{time_total} http://localhost/index.html

In the preceding example, the time taken to process the request and start sending data has been 0.002 - 0.001 = 0.001 sg, and the time for sending 0.004 - 0.003 = 0.001 sg.

Apr 1, 2012

Setting up BIND to secure DNS with DNSSEC (III)

In the last article about Setting up BIND to secure DNS with DNSSEC, we completed the configuration of DNSSEC in order to sign a zone file. When you sign a zone, you have to take into account that the dnssec-signzone command generates signatures which are valid only for 30 days, unless you indicate another date by means of the option -e end-time.

If your signature expires, you will have to resign the zone and use the new signed file. You will also have to resign your zone if you make some change (add, modify or remove some record) in the zone file. At any rate, it is highly recommended to regenerate the KSK key at least every year and ZSK every three months.

After setting up DNSSEC, the next step is to provide our zone's DS (Delegation Signer) record to our parent, whenever the parent is also using DNSSEC. We can utilize for this purpose the following command, by passing as argument the ZSK file.

[root@dns test.local]# dnssec-dsfromkey Ktest.local.+005+05775.key 
test.local. IN DS 5775 5 1 01FC6E8961D76D6438CAC77260EFF2E28AC0EFB0
test.local. IN DS 5775 5 2 175D8A898137886AADE03886FD69F9EA0D3EF1230B803AAB7EF23F9B 409079E1

Let's move on by using another CentOS 6.2 client in order to check the DNSSEC configuration. This new computer will have as main DNS, the server that we have just set up, and it will request DNSSEC records (+dnssec) through a query.

[root@localhost ~] yum install bind-utils    

[root@localhost ~]# dig @dns.test.local test.local +dnssec
test.local.        86400    IN    SOA    dns.test.local. admin.test.local. 1 21600 3600 604800 86400
test.local.        86400    IN    RRSIG    SOA 5 2 86400 20120410195007 20120311195007 5775 test.local. KozjX5R/1gTw4Q4xYzVnkpOnyRwviJVbqrrWzH95qwuWVSaMqOrlchvn giTYs94B1esCdD1xHouPkqIQ+BmIin5OEd3+G0038b1GGv7QY7ylzJr5 LO+TFF6M7toZLAmTMrQigYvV/1IDVvqxKvI1zjJahU49e6Sl+Hyfl85B vCE=
test.local.        86400    IN    NSEC    dns.test.local. NS SOA RRSIG NSEC DNSKEY
test.local.        86400    IN    RRSIG    NSEC 5 2 86400 20120410195007 20120311195007 5775 test.local. Fxo9aBIpTgd52p+sEseqA7CWPLmRLhCzv2d4gZhdgbvk+whoSOpu6ZMf GotfqkocX6+nafHeFBJvknJY0R25yiYP75nSaZoxAdDnBWv7VgE0Ywzi lges8wZyGX0VUMVTUWrvJYdZr+aThJqE0Shf4TuVKjhQqlwxsio+YXeS WhU=

As you can appreciate above, we can get the corresponding RRSIG (Resource Record Signature) for each record.

And finally, also point out a quick method to validate if our DNS server is using DNSSEC. We can perform a DNS lookup by asking for the next name: www.dnssec-failed.org. If you come across a "not found" error as shown below, that means you are using a DNSSEC-validating DNS server.

[root@localhost ~]# host www.dnssec-failed.org
Host www.dnssec-failed.org not found: 3(NXDOMAIN)