May 31, 2012

Apache performance tuning: security (I)

Let's get started by remembering the series of articles published about Apache performance tuning:

  • Apache performance tuning: dynamic modules (I and II).
  • Apache performance tuning: directives (I and II).
  • Apache performance tuning: benchmarking (I)

In this post, I am going to talk about the points related to security, which you have to take into account when you are setting up an Apache installation.

Restrictions for the Apache user

The Apache user must not be able to log into the system. If you take a look at both passwd and shadow files, you will be able to appreciate that no shell is assigned to him (/sbin/nologin), and the field reserved for the password will contain "!!". That means that the Apache user will not be able to log on the system (he is blocked).

[root@localhost ~]# cat /etc/passwd | grep apache

[root@localhost ~]# cat /etc/shadow | grep apache

Restrictions for the system root

You have to prevent that the system root (/) is accessible through the web server. It is also better to disable all options on the root directory (Options none)  and control what directives can be used in the .htaccess file by means of the AllowOverride directive.

[root@localhost ~]# cat /etc/httpd/conf/httpd.conf
<Directory />
    Order deny,allow
    Deny from all
    Options none
    AllowOverride none

If you define the root directory with these characteristics, then you will have to add to each directory the allowed options.

Hiding a directory or a file

Perhaps you can have a directory completely indexed and in turn, it contains different subdirectories, but you do not want to make visible a concrete directory (hidden) and you desire that it is reachable only when you type its URL. For this purpose, you have to use the IndexIgnore option.

[root@localhost ~]# cat /etc/httpd/conf/httpd.conf
<Directory "/var/www/html/data">
    Options Indexes
    IndexIgnore status
    IndexIgnore *.bpm

In the previous example, Apache will keep hidden the status directory and all files with bmp extension included in the /var/www/html/data directory.

May 21, 2012

Maintaining packages on Debian/Ubuntu (II)

Let's end up the last part of the article about Maintaining packages on Debian/Ubuntu. In the previous writing and by means of the dh_make utility, I set up the necessary structure to create the final package later.

The most important file of that structure is control, which provides information about the package. Also pay attention to the postinst.ex and preinst.ex files, which can include shell scripts run after or before installing the package, and postrm.ex and prerm.ex, executed after or before uninstalling the application.

root@ubuntu-server:/tmp/nano/nano-2.2.6# cat debian/control 
Source: nano
Section: unknown
Priority: extra
Maintainer: root <root@ubuntu-server.local>
Build-Depends: debhelper (>= 8.0.0), autotools-dev
Standards-Version: 3.9.2
Homepage: <insert the upstream URL, if relevant>
#Vcs-Git: git://

Package: nano
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: <insert up to 60 chars description>
 <insert long description, indented with spaces>

And lastly, we just have to develop the deb package through dpkg-buildpackage, a program which automates the process of making a Debian package. First up, it prepares the build context by setting different environment variables. Then, it checks that the build dependencies and conflicts are satisfied, and finally, generates the deb package.

root@ubuntu-server:/tmp/nano/nano-2.2.6# dpkg-buildpackage
dpkg-deb: building package `nano' in `../nano_2.2.6-1_i386.deb'.
 dpkg-genchanges  >../nano_2.2.6-1_i386.changes
dpkg-genchanges: including full source code in upload
 dpkg-source --after-build nano-2.2.6
dpkg-buildpackage: full upload (original source is included)

Below you can see the package which has been turned out.

root@ubuntu-server:/tmp/nano/nano-2.2.6# file ../nano_2.2.6-1_i386.deb 
../nano_2.2.6-1_i386.deb: Debian binary package (format 2.0)

If you want to verify that this package has been correctly created, you can use the lintian tool, which dissects Debian packages and reports bugs and policy violations.

root@ubuntu-server:/tmp/nano/nano-2.2.6# lintian -i ../nano_2.2.6-1_i386.deb

Now you have to take into account that when you distribute this package on the target machine, you have to hold the package available in the official repository, that is, cancel any active installation, upgrade, or removal, and prevent in this way that this package is automatically updated in the future.

root@target:~# aptitude hold nano

root@target:~# aptitude show nano
Package: nano                            
State: installed [held]

May 15, 2012

Maintaining packages on Debian/Ubuntu (I)

When you have to take care of a production environment, it is normal to use distributions such as Debian, Ubuntu Server, RHEL or CentOS, because they are supported for a long time. In this way, for example, if your infrastructure is made up by Ubuntu Server LTS (Long Term Support) machines, they will be supported during five years. What does it mean? Throughout this period of time, your servers will be able to receive security updates and bug fixes.

What happens with this model? You will have to use the same version for most applications during that time. For instance, the official version of PHP that will be supported by Ubuntu Server 12.04 LTS in that term will be 5.3.10. That is to say, during this period of support, Canonical only will provide security updates and bug fixes for that package. Sometimes, some versions are updated or new features are introduced to the release, but it is not the normal behaviour.

In these cases, you will have to maintain the package versions that are not supported by the official distribution. In this article, I am going to talk about the particular situation of Ubuntu or Debian. For my tests, I will use an Ubuntu Server 12.04 virtual machine and the application chosen as an example will be the text editor nano.

So the idea of manually maintaining a package is to grab its source code, drop it off in the machine where we want to compile it (for example a 32-bit server), install the dependencies for that application, modify its configure file and finally, build the deb package. Afterwards, you will be able to distribute the package to the rest of servers by using tools such as Puppet.

First of all, let's get started by installing the needed applications to automatically make packages. Then, the source code of nano will be downloaded and unpacked.

root@ubuntu-server:~# aptitude install build-essential dh-make devscripts autoconf autotools-dev lintian

root@ubuntu-server:~# mkdir /tmp/nano ; cd /tmp/nano

root@ubuntu-server:/tmp/nano# wget

root@ubuntu-server:/tmp/nano# tar xvzf nano-2.2.6.tar.gz ; cd nano-2.2.6

Then, you have to install the dependencies required to compile the application: the developer's libraries for ncurses and the texinfo package (documentation system for on-line information and printed output).

root@ubuntu-server:/tmp/nano/nano-2.2.6# aptitude install libncurses-dev texinfo

At this time, you are ready to prepare Debian packaging from an original source archive, all this by means of dh_make, a tool which allows to convert a regular source code package into one formatted, according to the requirements of the Debian policy. It must be invoked inside a directory containing the source code, which has to be named <packagename>-<version>, where <packagename> must be all lowercase, digits and dashes.

root@ubuntu-server:/tmp/nano/nano-2.2.6# dh_make -e root@ubuntu-server.local -f ../nano-2.2.6.tar.gz 

Type of package: single binary, indep binary, multiple binary, library, kernel module, kernel patch?
 [s/i/m/l/k/n] s

Maintainer name  : root
Email-Address    : root@ubuntu-server.local 
Date             : Sat, 12 May 2012 18:56:39 +0200
Package Name     : nano
Version          : 2.2.6
License          : blank
Type of Package  : Single
Hit <enter> to confirm: 
Done. Please edit the files in the debian/ subdirectory now. nano
uses a configure script, so you probably don't have to edit the Makefiles.

After that, dh_make proceeds to generate a Debian subdirectory and the necessary control files in the program source directory. Those control files are customized with the package name and version extracted from the directory name. In addition, a new tgz file with the original program source code and packed with the Debian standards is created.

root@ubuntu-server:/tmp/nano/nano-2.2.6# ls -lh ../nano_2.2.6.orig.tar.gz 
-rw-r--r-- 1 root root 1.5M Nov 22  2010 ../nano_2.2.6.orig.tar.gz

root@ubuntu-server:/tmp/nano/nano-2.2.6# tree debian
├── changelog
├── compat
├── control
├── copyright
├── docs
├── emacsen-install.ex
├── emacsen-remove.ex
├── emacsen-startup.ex
├── info
├── init.d.ex
├── manpage.1.ex
├── manpage.sgml.ex
├── manpage.xml.ex
├── menu.ex
├── nano.cron.d.ex
├── nano.default.ex
├── nano.doc-base.EX
├── postinst.ex
├── postrm.ex
├── preinst.ex
├── prerm.ex
├── README.Debian
├── README.source
├── rules
├── source
│   └── format
└── watch.ex

1 directory, 26 files

May 6, 2012

Facing up to a kernel panic (III)

Let's learn how to debug a kernel panic. This is the continuation of the two previous articles about facing up to a kernel panic (I, II). This method can be really useful when after have been looking into a kernel panic, you have not been able to find out anything about it.

First of all, you need to enable the debug repository in order to install the kernel-debuginfo package, which provides debug information for the kernel.

Aside from this package, you also have to install kexec-tools, which contains the kexec binary. This application allows to load and boot into another kernel from the currently running kernel, by performing the function of the boot loader from within the own kernel. That is to say, the first kernel reserves a small size of memory used by the second kernel afterwards. In this way, the kernel panic will be caught from the context of a second booted kernel and not from the context of the crashed kernel.

And finally, crash is a tool used to analyze the state of the system while it is running, or as in our example, after a kernel crash has come out and a core dump has been generated, in this case by kdump, utility also provided by the kexec-tools package.

[root@localhost ~]# cat /etc/yum.repos.d/CentOS-Debuginfo.repo

[root@localhost ~]# yum install crash kexec-tools kernel-debuginfo

Now you just need to edit the grub.conf file and add the crashkernel parameter to the line of your current kernel. This option is used to allocate enough memory for the second kernel, for example 128 MB. If you do not want to modify this file, another choice is to aggregate this parameter in real time when you start your system and come across with the grub menu, by editing the corresponding stanza of your kernel. In addition, you will have to enable the automatic startup of kdump.

[root@localhost ~]# cat /etc/grub.conf
title CentOS (2.6.32-220.13.1.el6.i686)
    root (hd0,0)
    kernel /vmlinuz-2.6.32-220.13.1.el6.i686 ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8  KEYBOARDTYPE=pc KEYTABLE=es rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 rhgb crashkernel=128M quiet rd_LVM_LV=VolGroup/lv_root rd_NO_DM

[root@localhost ~]# chkconfig kdump on

Now you have to reboot your system in order to load the second kernel. If you take a look at the messages log file, you will see that a memory allocation has been carried out.

[root@localhost ~]# less /var/log/messages | grep crash
Apr 22 21:54:40 localhost kernel: Reserving 128MB of memory at 16MB for crashkernel (System RAM: 512MB)
Apr 22 21:54:40 localhost kernel: Kernel command line: ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8  KEYBOARDTYPE=pc KEYTABLE=es rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 rhgb crashkernel=128M quiet rd_LVM_LV=VolGroup/lv_root rd_NO_DM
Apr 22 21:54:40 localhost kernel: crash memory driver: version 1.1

From now on, if you run into a crash, a core dump will be generated. So as to try out this functionality, let's simulate a kernel panic by giving off the 'c' SysRq key to the kernel. A SysRq signal allows to hit the kernel, which will respond immediately regardless of whatever else it is doing at that moment. This key performs a system crash without first trying to unmount file systems or syncing disks attached to the system.

[root@localhost ~]# echo c > /proc/sysrq-trigger

After the core dump is captured, the system will be rebooted. Be patient because this operation can take up a long time, depending on the size of memory used by the system. You have to be aware of that the entire state of the system must be saved into disk. When the system has started over, a core file will have been dumped into the /var/crash directory.

[root@localhost ~]# file /var/crash/\:45\:26/vmcore 
/var/crash/ data

Now we are ready to research the core dump by means of crash. This application, similar to gdb, consists of common kernel core analysis tools such as kernel stack back traces of all processes, source code disassembly, formatted kernel structure and variable displays, virtual memory data, dumps of linked-lists, and so on.

To use this tool, you have to pass through the command line three parameters. First of all, the file (symbol table used by the kernel) of the original kernel which was running when the system crashed. Secondly, an uncompressed kernel image which has been compiled with the '-g' option. And finally, the kernel core dump created in this case by kdump.

[root@localhost ~]# uname -r

[root@localhost ~]# crash /boot/ /usr/lib/debug/lib/modules/ /var/crash/\:45\:26/vmcore
  SYSTEM MAP: /boot/                
DEBUG KERNEL: /usr/lib/debug/lib/modules/ (
    DUMPFILE: /var/crash/  [PARTIAL DUMP]
        CPUS: 1
        DATE: Sun Apr 22 22:45:16 2012
      UPTIME: 00:02:31
LOAD AVERAGE: 0.14, 0.16, 0.07
       TASKS: 85
    NODENAME: localhost.localdomain
     RELEASE: 2.6.32-220.13.1.el6.i686
     VERSION: #1 SMP Tue Apr 17 22:09:08 BST 2012
     MACHINE: i686  (1396 Mhz)
      MEMORY: 511.5 MB
       PANIC: "Oops: 0002 [#1] SMP " (check log for details)
         PID: 2065
     COMMAND: "bash"
        TASK: dfaed030  [THREAD_INFO: df8d0000]
         CPU: 0


Rather than dumping the core into the /var/crash directory, you can configure kdump to copy the core into a remote server, by mounting a partition through NFS or even directly copying the file with scp. These options can be set in the kdump.conf file. This configuration file also includes another helpful directives which allow to do a series of tasks when a kernel crash has happened and the kdump kernel has been loaded.