Redes Privadas Virtuales

Inventorying your system with cfg2html

2012-01-30T23:08:00.001+01:00

Cfg2html is a powerful tool available for Linux which performs an exhaustive inventory of the entire system, by creating a HTML or plain ASCII file which reflects all the features (both hardware and software) of the system, such as applications, kernel, modules and libraries, networking, filesystems and so on.

I am used to utilizating this tool each time that I install a Linux system, in order to write down a full record of it, and besides, I usually program a task to repeat this action periodically.

I am going to try out this tool (version 2.37) in this article on a CentOS 6.2 distribution (it is valid for any Linux system too), by downloading its source code and using the script included in the package. You also have the option of installing those scripts into the operating system (by means of the rpm package), but I am think that it is not necessary due to you can use directly the corresponding script.

[root@centos tmp]# wget http://www.cfg2html.com/cfg2html-linux-2.37-20111229_all.zip

[root@centos tmp]# unzip cfg2html-linux-2.37-20111229_all.zip

[root@centos tmp]# tar xvzf cfg2html-linux_2.37-1.tar.gz ; cd cfg2html-linux-2.37

After unpacking it, we may run the cfg2html-linux script to carry out the inventory.

[root@centos cfg2html-linux-2.37]# ./cfg2html-linux

--=[ http://come.to/cfg2html ]=-----------------------------------------------
Starting          cfg2html-linux version 2.37-2011-12-28
Path to Cfg2Html  ./cfg2html-linux
HTML Output File  ./centos.local.html
Text Output File  ./centos.local.txt
Partitions        ./centos.local.partitions.save
Errors logged to  ./centos.local.err
Started at        2012-01-30 21:31:19
WARNING           USE AT YOUR OWN RISK!!! :-))           <<<<<
--=[ http://come.to/cfg2html ]=-----------------------------------------------

Collecting:  Linux System CentOS release 6.2 (Final)  .................................
Collecting:  Cron and At  .....
Collecting:  Hardware  .................
Collecting:  Software  .......
Collecting:  Filesystems, Dump- and Swapconfiguration  ..........
Collecting:  Multipath Configuration  ........
Collecting:  LVM  ............
Collecting:  Network Settings  ................................
Collecting:  Kernel, Modules and Libraries  ...............
Collecting:  System Enhancements  .
Collecting:  Applications and Subsystems  .....

--=[ http://come.to/cfg2html ]=-----------------------------------------------

At the end of the collecting process, you will have an HTML and txt file with the result of the audit. Any problem occured during the gathering process, will be warn into the err file.

[root@centos cfg2html-linux-2.37]# ls -l centos.local.*
-rw-r--r--. 1 root root   8410 Jan 30 21:31 centos.local.err
-rw-r--r--. 1 root root 213406 Jan 30 21:31 centos.local.html
-rw-r--r--. 1 root root    259 Jan 30 21:31 centos.local.partitions.save
-rw-r--r--. 1 root root 182348 Jan 30 21:31 centos.local.txt

For example, next figure shows the HTML output.

Finally, also say that it is a good idea to create a crontab task so as to get these data for instance weekly and back up them.

Secure remote access to home through OpenVPN (III)

2012-01-22T19:02:00.003+01:00

Let's end up the series of articles about my secure remote access to home through OpenVPN. In the first part, I had to get over the issue about the dynamic IP address used by my ADSL service. I overcame it by using a free dynamic DNS service: DNSdynamic. In the second one, I relied on easy-rsa in order to generate the suitable digital certificates.

Now, we are ready to set OpenVPN up in both sides of the connection: the client and server. First up, let's begin with the server by installing OpenVPN directly from the Ubuntu repositories. Then, we have to copy the appropiate certificates made up by easy-rsa into the openvpn directory, and finally, edit the OpenVPN configuration file for the server.

root@javi-pc:/home/javi/tmp/2.0# aptitude install openvpn

root@javi-pc:/home/javi/tmp/2.0/keys# cp ca.crt server.crt server.key dh1024.pem /etc/openvpn/

root@javi-pc:/home/javi# cat /etc/openvpn/server.conf
# Use a dynamic TUN device
dev tun

# Set virtual point-to-point IP addresses
ifconfig 10.0.0.1 10.0.0.2

# Use TCP for communicating with client
proto tcp-server

# Enable TLS and assume server role during TLS handshake
tls-server

# File containing Diffie Hellman parameters
dh /etc/openvpn/dh1024.pem

# Certificate authority (CA) file
ca /etc/openvpn/ca.crt

# Local peer's signed certificate
cert /etc/openvpn/server.crt

# Local peer's private key
key /etc/openvpn/server.key

# Use fast LZO compression
comp-lzo

# Ping remote every 10sg and restart after 60sg passed without sign of life from remote
keepalive 10 60

# Output logging messages to openvpn.log file
log /var/log/openvpn.log

# Set output verbosity to normal usage range
verb 3

Now we only have to start the OpenVPN daemon and afterwards, we will be able to appreciate that the service is running on TCP port 1194. A final task will be to open that port on the router and redirect all that traffic to the server.

root@javi-pc:/home/javi# /etc/init.d/openvpn start

root@javi-pc:/home/javi# netstat -natp | grep openvpn
tcp        0      0 0.0.0.0:1194            0.0.0.0:*               LISTEN    19781/openvpn

Let's undertake now the other side of the tunnel: the client. It will be necessary as well to install OpenVPN from the Ubuntu repositories and move into the openvpn directory the adequate digital certificates.

root@javi-laptop:~# aptitude install openvpn

root@javi-laptop:~# cat /etc/openvpn/client.conf
# Use a dynamic TUN device
dev tun

# Connect to server
remote test.dnsdynamic.com

# Set virtual point-to-point IP addresses
ifconfig 10.0.0.2 10.0.0.1

# Use TCP for communicating with server
proto tcp-client

# Enable TLS and assume client role during TLS handshake
tls-client

# Certificate designed as a server-only certificate
remote-cert-tls server

# Certificate authority (CA) file
ca /etc/openvpn/ca.crt

# Local peer's signed certificate
cert /etc/openvpn/client.crt

# Local peer's private key
key /etc/openvpn/client.key

# Use fast LZO compression
comp-lzo

# Ping remote every 10sg and restart after 60sg passed without sign of life from remote
keepalive 10 60

# Output logging messages to openvpn.log file
log /var/log/openvpn.log

# Set output verbosity to normal usage range
verb 3

Lastly, we must remove any link in the runlevel directory for the OpenVPN script, so as to launch it manually whenever we want.

root@javi-laptop:~# update-rc.d -f openvpn remove

root@javi-laptop:~# /etc/init.d/openvpn start

Shutting out ARP poisoning and spoofing with ArpON

2012-01-14T00:33:00.004+01:00

Based on the series of articles that I wrote about ARP poisoning (I, II and III), I would like to put forward a great tool, ArpON (Arp handler inspectiON), aimed at protecting us against ARP poisoning, spoofing and routing, by preventing attacks as the Man in the Middle (MitM). It also avoids from derived attacks such as DNS and WEB spoofing, and session and SSL/TLS hijacking.

This is the typical program that I always install on any Linux computer, since it is essential in order to shut out any type of attack commented above. And furthermore, it is really meaningful when you get around and have to connect your laptop to some untrusted network, such as inside a library, pub, airport and so on.

ArpON uses two kinds of methods: DARPI (Dynamic Arp Inspection) and SARPI (Static Arp Inspection). With the second technique, you have to register into a configuration file, the MAC and IP address of each computer which you rely. This may be a hard task when you have got lots of devices in your network. In return, DARPI follows up all incoming and outgoing ARP packets.

In this article, I am going to set up DARPI on Ubuntu 11.10. I will install ArpON from the official repository (version 2.0). It is a pity because this version came out last year and I cannot understand why it has not been updated in the last release of Ubuntu. The current version which you can download from the ArpON web site is 2.7.

In PCs or laptops, I prefer to install it from the Ubuntu repositories, due to it will be automatically upgraded (in theory) with each new release of Ubuntu. Instead, on production servers, it pays off to compile it from its source code so as to have the latest version.

Ok, so we are going to install ArpON and put it into DARPI mode. In addition, ArpON will be automatically started during the boot.

root@victim:~# aptitude install arpon

root@victim:~# cat /etc/default/arpon
...
# For DARPI uncomment the following line
DAEMON_OPTS="-q -f /var/log/arpon/arpon.log -g -d"

# Modify to RUN="yes" when you are ready
RUN="yes"

root@victim:~# /etc/init.d/arpon start

First up, we are going to take a look at the ARP table of the victim (remember the involved computers were presented in the first article about ARP poisoning). As you can pick out, the dependable addresses are tagged as PERM (permanent).

root@victim:~# arp -a
? (192.168.1.150) at 00:80:5a:54:32:67 [ether] PERM on eth0
? (192.168.1.1) at 00:60:b3:50:ab:45 [ether] PERM on eth0
? (192.168.1.11) at 00:0c:29:18:36:e6 [ether] PERM on eth0

If we observe the log turned out by ArpON at the beginning, it first of all cleans up the ARP cache by removing all entries, in order to avoid that the table is poisoned.

root@victim:~# tail -f /var/log/arpon/arpon.log
  17:55:00 - Wait link connection on eth0...
  17:55:12 - DARPI on dev(eth0) inet(192.168.1.10) hw(0:c:29:69:81:47)
  17:55:12 - Deletes these Arp Cache entries:
  17:55:12 - 1)   192.168.1.150 ->  0:80:5a:54:32:67
  17:55:12 - 2)     192.168.1.1 ->  0:60:b3:50:ab:45
  17:55:12 - 3)    192.168.1.11 ->   0:c:29:18:36:e6
  17:55:12 - Cache entry timeout: 500 milliseconds.
  17:55:12 - Realtime Protect actived!
  17:55:46 - Request >> Add entry 192.168.1.150
  17:55:46 - Reply   << Refresh entry 192.168.1.150 -> 0:80:5a:54:32:67
  17:55:47 - Request >> Add entry 192.168.1.1
  17:55:47 - Reply   << Refresh entry 192.168.1.1 -> 0:60:b3:50:ab:45
  17:55:58 - Request << Delete entry 192.168.1.150 -> 0:80:5a:54:32:67
  17:55:58 - Reply   >> Send to 192.168.1.150 -> 0:80:5a:54:32:67
  17:55:58 - Request >> Add entry 192.168.1.150
  17:55:58 - Reply   << Refresh entry 192.168.1.150 -> 0:80:5a:54:32:67
...

To sum up the running of ArpON into DARPI mode, first point out that ArpON handles its own ARP table called DARPI cache, by applying several rules to different kinds of packets.

ARP request

For the outbound traffic (packets generated by us), ArpON lets them pass, by adding an entry with the target into the DARPI cache. For the inbound traffic (packets which come to us from the network), ArpON refuses the packet, by deleting the entry of the source address written down into the ARP cache, because that packet could be poisoned. Later, the kernel will send out an ARP request so as to make sure the origin.

ARP reply

For the outgoing traffic, ArpON just lets them pass. For the incoming traffic, ArpON verifies whether the source address matches an entry in the DARPI cache. If so, it lets the packet get in, by adding an entry into the ARP cache. Otherwise, it denies the packet, by removing the entry from the ARP cache.

To begin with the test, we are going to run a MitM attack between the router and the victim.

root@attacker:~# ettercap -TqM arp:remote /192.168.1.1/ /192.168.1.10/

If we review the ArpON log again, we can see that the poisoning attempts from the attacker are correctly rejected.

root@victim:~# tail -f /var/log/arpon/arpon.log
...
192.168.1.1 -> 0:c:29:20:9f:9b
  18:13:16 - Reply   << Delete entry
192.168.1.1 -> 0:c:29:20:9f:9b
  18:13:17 - Reply   << Delete entry
192.168.1.1 -> 0:c:29:20:9f:9b
  18:13:18 - Reply   << Delete entry
...

You may likewise check out this situation by activating the chk_poison plugin through the same ettercap.

root@attacker:~# ettercap -TqM arp:remote /192.168.1.1/ /192.168.1.10/
...
Plugin name (0 to quit): chk_poison
Activating chk_poison plugin...

chk_poison: Checking poisoning status...
chk_poison: No poisoning between 192.168.1.10 -> 192.168.1.1

Another way is to print the ARP cache again. As you can distinguish, a new entry relative to the attacker has been added, and the other ones keep in the same state.

root@victim:~# arp -a
? (192.168.1.11) at 00:0c:29:18:36:e6 [ether] PERM on eth0
? (192.168.1.20) at 00:0c:29:20:9f:9b [ether] PERM on eth0
? (192.168.1.1) at 00:60:b3:50:ab:45 [ether] PERM on eth0
? (192.168.1.150) at 00:80:5a:54:32:67 [ether] PERM on eth0

Secure remote access to home through OpenVPN (II)

2012-01-06T18:03:00.002+01:00

Let's get started by running the vars script, in order to set the parameters (openssl.cnf file, size, country, city, email, etc.) used by the other scripts. In addition, we must execute the clean-all script as well, which takes care of preparing and initializing the keys directory, place where new certificates, requests, private keys, etc. are stored.

root@javi-pc:/home/javi/tmp/2.0# . ./vars

root@javi-pc:/home/javi/tmp/2.0# . ./clean-all

After getting ready the environment, the next step will be to create a CA (Certification Authority), that is to say, a root certificate and private key whereby we will be able to make and sign certificates later.

root@javi-pc:/home/javi/tmp/2.0# . ./build-ca
Generating a 1024 bit RSA private key
.++++++
...................................++++++
writing new private key to 'ca.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:SP
State or Province Name (full name) [CA]:Madrid
Locality Name (eg, city) [SanFrancisco]:Madrid
Organization Name (eg, company) [Fort-Funston]:openvpn
Organizational Unit Name (eg, section) [changeme]:
Common Name (eg, your name or your server's hostname) [changeme]:ca
Name [changeme]:
Email Address [mail@host.domain]:

Afterwards we have to generate the Diffie-Hellman parameters. This file is used in the server side for SSL/TLS connections.

root@javi-pc:/home/javi/tmp/2.0# . ./build-dh

Now we have the necessary infraestructure to be able to issue digital certificates. So let's get going by building the server certificate first of all. As in the case of the CA certificate, you will be asked for certain information which will be aggregated into the certificate (country, state, location, common name, email, etc.).

In order to avoid Man in the Middle attacks (MitM) where an authorized client tries to connect to another client by impersonating the server, we must make the server certificate through the build-key-server script and not build-key. This operation will designate the certificate as a server-only certificate, by setting the right attributes (nsCertType=server). This will cut off clients from connecting to any server which lacks the nsCertType=server ownership in its certificate, even if the certificate has been signed by a valid CA.

root@javi-pc:/home/javi/tmp/2.0# . ./build-key-server server
Generating a 1024 bit RSA private key
.....++++++
...............++++++
writing new private key to 'server.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:SP
State or Province Name (full name) [CA]:Madrid
Locality Name (eg, city) [SanFrancisco]:Madrid
Organization Name (eg, company) [Fort-Funston]:openvpn
Organizational Unit Name (eg, section) [changeme]:
Common Name (eg, your name or your server's hostname) [server]:      
Name [changeme]:
Email Address [mail@host.domain]:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Using configuration from /home/javi/tmp/2.0/openssl-1.0.0.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'SP'
stateOrProvinceName   :PRINTABLE:'Madrid'
localityName          :PRINTABLE:'Madrid'
organizationName      :PRINTABLE:'openvpn'
organizationalUnitName:PRINTABLE:'changeme'
commonName            :PRINTABLE:'server'
name                  :PRINTABLE:'changeme'
emailAddress          :IA5STRING:'mail@host.domain'
Certificate is to be certified until Jan  3 00:27:23 2022 GMT (3650 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

In the case of the client certificate, we will take the same previous steps but now, by using the build-key script.

root@javi-pc:/home/javi/tmp/2.0# . ./build-key client

And finally, let's take a look at all files created by means of this procedure.

root@javi-pc:/home/javi/tmp/2.0# tree keys
keys
├── 01.pem
├── 02.pem
├── ca.crt
├── ca.key
├── client.crt
├── client.csr
├── client.key
├── dh1024.pem
├── index.txt
├── index.txt.attr
├── index.txt.attr.old
├── index.txt.old
├── serial
├── serial.old
├── server.crt
├── server.csr
└── server.key

Secure remote access to home through OpenVPN (I)

2011-12-31T12:28:00.002+01:00

I have prepared a secure access so that when I am living in London, I can connect to my home network securely. I have set up a VPN (Virtual Private Network) by means of OpenVPN.

Why have I preferred a VPN instead of a typical access such as SSH, VNC, etc.? Because in this way, I will be able to accomplish an encrypted tunnel between my laptop and home network, and over that secure line, to establish other types of connections later. Furthermore, I will be able to connect from any kind of insecure networks.

Why have I chosen OpenVPN? Because this application allows you to quickly build SSL/TLS channels, and this sort of VPN is really handy and straightforward to configure. OpenVPN is an open source software which easily implements VPNs over a public network, such as Internet. One of the main advantages of OpenVPN is that it just needs a single TCP or UDP port for transmissions and runs in userspace, rather than requiring IP stack operations, as for instance IPSec or PPTP.

Bellow you can observe a detailed outline of my infraestructure. It is a point to point link between my laptop and a PC connected inside the local network. The PC acts in the server role (takes care of listening for possible connection requests) and the laptop is the client (initiates the connection). Once I am connected to the PC via OpenVPN, I will be able to jump safely to any device located in the network. Both computers run Ubuntu 11.10.

One of the first things that I had to face up to is the issue of the dynamic IP address used by my ADSL service. Every time that I turn on the router, a temporary public IP address is assigned by the ADSL provider. To overcome it, I have signed up for a free dynamic DNS service: DNSdynamic. The registration process is pretty simple.

In this manner, I have obtained a subdomain which points to my router. To that end, I have installed ddclient on the PC, an address updating utility which keeps up to date the current public IP of the router. In order to show you my configuration, I will use a fictitious subdomain called test.dnsdynamic.com.

root@javi-pc:~# aptitude install ddclient

root@javi-pc:~# cat /etc/ddclient.conf
# Log messages to syslog
syslog=yes              

# Support SSL updates               
ssl=yes

# Obtain IP address from provider's IP by checking page                               
use=web, web=myip.dnsdynamic.com

# Update DNS information from server
server=www.dnsdynamic.org

# Login and password for server
login=test@gmail.com
password='xxxxxx'

# Update protocol used              
protocol=dyndns2

# Subdomain                        
test.dnsdynamic.com

root@javi-pc:~# cat /etc/default/ddclient 
...
# ddclient runs in daemon mode
run_daemon="true"

# Time interval between the updates of the dynamic DNS name (in seconds)
daemon_interval="3600"

root@javi-pc:~# /etc/init.d/ddclient start

The SSL/TLS connection configured by me is authenticated through digital certificates. So I have needed to make a couple of certificates, one for each end of the VPN tunnel. In addition, I have also had to create a CA (Certification Authority) in order to validate both certificates. OpenVPN allows peers to authenticate each other by using username/password, a pre-shared secret key or digital certificates. I have picked out the last option due to it is the most robust system.

So as to manage digital certificates, I am used to treating with easy-rsa, a small RSA key management package which contains a series of openssl scripts aimed at handling PKIs (Public Key Infrastructures). This tool is included within the OpenVPN source file.

javi@javi-pc:~/tmp$ wget http://swupdate.openvpn.org/community/releases/openvpn-2.2.2.tar.gz

javi@javi-pc:~/tmp$ tar xvzf openvpn-2.2.2.tar.gz

javi@javi-pc:~/tmp$ mv openvpn-2.2.2/easy-rsa/2.0/ . ; rm -rf openvpn-2.2.2*

Apache performance tuning: dynamic modules (II)

2011-12-22T20:11:00.001+01:00

Let's continue with the second part of the article titled Apache Performance tuning: dynamic modules (I). Remember that this paper is aimed at reviewing the different modules belonging to Apache, so as to determine whether they are useful for our requirements. To that end, we will be able to fit the amount of memory used by Apache processes.

The most important point is to be aware of that one only process consumes little memory, but if our Apache installation requires lots of processes, the total memory grabbed by Apache will be huge. So if we get hold of turning down the initial memory with which a process is created, afterwards it will run lighter and besides, we will have that free memory available in order to be allocated for other things.

mod_ext_filter

Forwards the response body to an external program before sending it out to the client.

# LoadModule ext_filter_module modules/mod_ext_filter.so

mod_include

Filters files before delivering them to the client.

# LoadModule include_module modules/mod_include.so

mod_info

Provides a comprehensive overview of the web server configuration.

# LoadModule info_module modules/mod_info.so

mod_ldap

Improves the performance of websites by pooling LDAP connections and caching responses.

# LoadModule ldap_module modules/mod_ldap.so

mod_logio

Logs the input and output number of bytes received/sent per request.

# LoadModule logio_module modules/mod_logio.so

mod_proxy

Puts into action a a proxy/gateway.

# LoadModule proxy_module modules/mod_proxy.so
# LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
# LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
# LoadModule proxy_http_module modules/mod_proxy_http.so
# LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
# LoadModule proxy_connect_module modules/mod_proxy_connect.so

mod speling

Tries to correct erroneous URLs that users could have typed by overlooking capitalization and allowing up to one misspelling.

# LoadModule speling_module modules/mod_speling.so

mod_status

Provides statistics about the activity and performance of the web server.

# LoadModule status_module modules/mod_status.so

mod_suexec

Allows CGI scripts to run as a concrete user and group.

# LoadModule suexec_module modules/mod_suexec.so

mod_userdir

Allows user directories can be accessed through the web server.

# LoadModule userdir_module modules/mod_userdir.so

mod_usertrack

Logs user activity.

# LoadModule usertrack_module modules/mod_usertrack.so

After disabling these modules, the memory used by one Apache process (and owned by apache user) went from 2.02 to 1.46 MB, that is to say, we have gained around 0.6 MB. If you take into account that a large number of processes can be running on the system at any given time, the saved memory might be appreciable. In addition, you have to consider that from now on, a process is much lighter, with what its startup and performance will be much better.

Apache performance tuning: dynamic modules (I)

2011-12-14T22:29:00.000+01:00

Apache is a cross-platform, modular and open source web server, widely used around the world for its quality, robustness and stability. But like most of the applications, it is installed with a default configuration which is not the most adequate. And I am going to say more: I have never seen an Apache installation where the administrator has set it up correctly later.

During several articles, you are going to learn how to properly optimize Apache, in order to achieve the best performance. The tests will be carried out on CentOS 6.2 (32 bits) with Apache 2.2.15. I am going to break up this first article relative to dynamic modules in two separate parts.

Apache has got two main operating modes, also known as multi-processing modules (MPMs):

Prefork: an unique Apache process (httpd) launchs child processes which take care of listening for potential connections and serving them. Apache keeps several idle processes ready to attend incoming requests. Thereby, a client does not need to wait for new children are forked. Another advantage of this operation mode is that if there is a problem in any process, this will not affect other processes (each child is independent of the rest).

Worker: as in the previous case, an only control process creates several child processes, and in turn, each child process handles a listener thread which passes the inbound connections to other server threads managed as well by the same child process. This mode is faster and more scalable, but in contrast, it is more fault tolerant (several threads share the same memory area, and if there is any problem in the parent, it will involve the rest).

You can install Apache either by compiling it from its source code or by getting directly the binary file from a repository. I for one prefer this second option, because in this way, any kind of update (security or bugfix) will be able to be applied without compiling it again.

A typical installation of Apache via yum comes with the following pre-compiled modules. As you may appreciate, prefork will be the default operating mode (you can change this by modifying the /etc/sysconfig/httpd file).

[root@centos ~]# httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

It is basic to know the funcionality of each module so as to figure out if it can be left out. Then we are going to put forward what modules can be ruled out in the most of the cases. Also point out that all directives showed below, are included into the Apache configuration file (httpd.conf). In many cases, the related modules will be also disabled, aside from the principal one.

mod_actions

Allows the execution of CGI scripts based on the MIME content type and the request method.

# LoadModule actions_module modules/mod_actions.so

mod_auth_basic

Limits access to certain users by using HTTP Basic Authentication. I usually disable its dependencies.

LoadModule auth_basic_module modules/mod_auth_basic.so
# LoadModule authn_file_module modules/mod_authn_file.so
# LoadModule authn_alias_module modules/mod_authn_alias.so
# LoadModule authn_anon_module modules/mod_authn_anon.so
# LoadModule authn_dbm_module modules/mod_authn_dbm.so
# LoadModule authn_default_module modules/mod_authn_default.so
# LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
# LoadModule authn_dbd_module modules/mod_authn_dbd.so

mod_auth_digest

Limits access to certain users by using MD5 Digest Authentication.

# LoadModule auth_digest_module modules/mod_auth_digest.so

mod_authz_*

Limits access to certain groups based on different origins (DBM or plaintext files, hostname or IP address, etc.). I get used to remove all less mod_authz_host.

LoadModule authz_host_module modules/mod_authz_host.so
# LoadModule authz_user_module modules/mod_authz_user.so
# LoadModule authz_owner_module modules/mod_authz_owner.so
# LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
# LoadModule authz_dbm_module modules/mod_authz_dbm.so
# LoadModule authz_default_module modules/mod_authz_default.so

mod_cache

Manages the content cache.

# LoadModule cache_module modules/mod_cache.so
# LoadModule disk_cache_module modules/mod_disk_cache.so

mod_cgi

Allows the execution of CGI scripts.

# LoadModule cgi_module modules/mod_cgi.so

mod_dav

Implements the WebDAV (Web-based Distributed Authoring and Versioning) funcionality.

# LoadModule dav_module modules/mod_dav.so
# LoadModule dav_fs_module modules/mod_dav_fs.so

mod_env

Controls the internal environment variables which are sent out to CGI scripts and SSI pages.

# LoadModule env_module modules/mod_env.so

I head off to London

2011-12-05T14:00:00.001+01:00

Last year, when I was in London, I already knew that it was not going to be the last time, and indeed, I was not mistaken. Today, I have handed in my resignation and I have given up my current job, where I will be bound until the end of this month. My flight to the United Kingdom, the next stage of my life, will take off on the 9th of January.

This idea was going around in my head from a long time. And the question was: why not? Why not work in another country, run away from the daily monotony, learn from other cultures, break the political correctness and in short, squeeze the life.

Here in Spain we have a big problem and its name is PSOE (political party). Whenever they have governed, they have finished messing up the country, and nowadays, in contradistinction to 1996 (the previous time that they ruined us), we do not dispose of the European cohesion funds and the crown jewels (the most important public companies) to be sold, in order to be able to get ahead. I could write another book about the misdeeds of these political figures...

Spain has to face up to a hard situation throughout next years, and this is other of the reasons because I think that now is a good moment to go abroad. Unlike fifteen years ago, The Bank of Spain cannot devalue the currency at present and we will have to resort to other financial instruments so as to get over this critical condition, such as reducing salaries, increasing taxes, improving the productivity, optimizing public resources, etc., and in this way, to be more competitive and efficient.

Regarding the IT world, I have always said that Spain is not a good place for engineers, due to this is a country of services. We do not have IT industry and in the most of the cases, you can only aspire to cover the needs or requirements of a client. And why do I say client and not company? Because over time, the business model has totally changed and at the present time, it is no longer possible (or at least very complicated) to belong to a final company.

Between the client and you, there will always be an intermediate company that we call "cárnica" or "charcutera" (butcher shop in English). Thereby, in general, this intermediary takes care of obtaining a final client for you, offering you like a bit of meat, and paying your salary. Practically without lifting a finger and taking advantage of your work, it will grab a part of the money that you make every month.

What happens with this system? You will never be or feel part of a company; today you can be working in a certain place and tomorrow, in another one, and on top of all that, there is no way to develop a career inside an enterprise.

I recently read the article titled "Las ilusiones perdidas" (the lost illusions in English), which reflects perfectly the situation of thousand of Spanish who have had to leave our country because of multiple reasons, but mainly due to a lack of future. This is a great issue, because during the next years, we are going to lose the best generation of young people better prepared in history. This paradox is also known as brain drain.

As I mentioned before, my case is totally different. I am not in need to look for a job far away from home; I have a permanent job here and I have dropped it off voluntarily. Furthermore, I am aware of that I might have switched to another work at any moment. Simply, I am just in the mood for taking this step.

I am a person who likes to work out everything in detail, and in this manner, I have mapped out a complete roadmap for my first weeks in London. I have to read up some points before ending up my plan, but mostly, I am going to boost my English at the beginning, by enrolling in some language school, at least during the first three months. I know that I have a good English level, but I also realize that it is turned into lower-intermediate when you arrive there.

After that initial period of time, I will search for a job. I consider that it is better to build the house starting with the floor rather than the roof. For that reason and as I pointed out before, first of all I will be enhancing my English and in turn, I will have free time to get used to those new lands, aside from to accomplish other typical tasks such as opening a bank account, getting the NIN (National Insurance Number) and a GP (General Practitioner), registering at the embassy and so on.

Perhaps, this is the most important decision that I have had to take on throughout my life, and I hope not to slip up. I am aware of that it will not be straightforward, but at any rate, I am really looking forward to it!

The tmp directory and tmpwatch daemon

2011-11-29T11:56:00.001+01:00

The tmp directory is normally used on Linux systems so that users or applications can store temporary information within it.

On Debian or Ubuntu distributions, the system cleans out all user data with each startup. On RHEL or CentOS 6, no operation is performed on that directory. But in version 5 of RHEL or CentOS, there was a great tool installed on the system by default, and utilized to periodically check the contents of the tmp directory: tmpwatch.

Tmpwatch is a cron job which takes care of removing files which have not been accessed for a period of time, or any file or folder that you configure. This operation is carried out based on guidelines which will be exposed later. The equivalent program on Debian/Ubuntu is tmpreaper, although you can compile tmpwatch perfectly for the aforementioned operating systems.

For the development of the present article, I am going to use CentOS 6.0 (32 bits).

[root@centos ~]# yum install tmpwatch

[root@centos ~]# cat /etc/cron.daily/tmpwatch 
#! /bin/sh
flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
        -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
        -X '/tmp/hsperfdata_*' 10d /tmp
/usr/sbin/tmpwatch "$flags" 30d /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
    if [ -d "$d" ]; then
        /usr/sbin/tmpwatch "$flags" -f 30d "$d"
    fi
done

By taking a look at the script launched daily by the system, we may observe that tmpwatch acts on a series of directories (/tmp, /var/tmp, /var/local, etc.) by clearing out their contents. This task is accomplished based on certain events which have taken place throughout the last 10 or 30 days.

-u (--atime): the decision to delete a file depends on its atime (access time).

-m (--mtime): the decision to delete a file depends on its mtime (modification time).

-c (--ctime): the decision to delete a file depends on its ctime (inode change time).

-f (--force): removes files even whether root does not have write access.

By means of the '-x' option, we can leave out a specific file or directory that matches the pattern.

TrueCrypt under the command line

2011-11-22T18:10:00.001+01:00

I have an external hard drive (LG XD3, 500 GB) broken up into a couple of partitions, 450 and 50 GB respectively. The first partition is public and formatted with NTFS. The second one is formatted with ext4 and encrypted by means of TrueCrypt, and it is where I store my private data.

So far, I used TrueCrypt into graphical mode, but over time, I realize that it is more comfortable to handle the command line version (aside from I tend to rule out any kind of graphical tool whenever possible).

TrueCrypt is a powerful program which may cypher partitions, logical volumes, whole hard drives or even installed operating systems. The encryption is transparently and automatically carried out, and on top of all that, on real time (that is to say, on the fly). Another plus is the option to hide volumes and its performance, which is excellent.

One practical detail of TrueCrypt is that is not necessary to install it on the system. To that end, you have to download the Console-only-32-bit file (in my case, the 32-bit version), decompress the included binary and run it. Then, you will have to choose the second option: Extract package file truecrypt_7.1_console_i386.tar.gz and place it to /tmp. Within this tgz file is located the executable file of TrueCrypt.

I get used to drop off this binary file into the public partition of the external hard drive. Thereby, when I have to use it, I just have to get it from there.

javi@javi-ubuntu:/tmp$ cp /media/public/truecrypt/truecrypt . ; chmod +x truecrypt

javi@javi-ubuntu:/tmp$ ./truecrypt --version
TrueCrypt 7.1

First of all, I had to encrypt the partition. This is a long process and depends on the size of your partition. Below you may appreciate that the average speed was 26 MB/s.

In the next output, you can see that in order to create the cyphered partition (sdb2), I followed the text wizard provided by TrueCrypt. Other choice would have been to pass the parameters through the command line (--encryption, --size, etc.).

javi@javi-ubuntu:/tmp$ sudo ./truecrypt -c
Volume type:
 1) Normal
 2) Hidden
Select [1]: 1

Enter volume path: /dev/sdb2

Encryption algorithm:
 1) AES
 2) Serpent
 3) Twofish
 4) AES-Twofish
 5) AES-Twofish-Serpent
 6) Serpent-AES
 7) Serpent-Twofish-AES
 8) Twofish-Serpent
Select [1]: 1

Hash algorithm:
 1) RIPEMD-160
 2) SHA-512
 3) Whirlpool
Select [1]: 1

Filesystem:
 1) None
 2) FAT
 3) Linux Ext2
 4) Linux Ext3
 5) Linux Ext4
Select [2]: 5

Enter password: 
Re-enter password: 

Enter keyfile path [none]: 

Please type at least 320 randomly chosen characters and then press Enter:


Done: 100.000%  Speed:   26 MB/s  Left: 0 s                

The TrueCrypt volume has been successfully created.

Once you have created the encrypted partition (remember that my example is based on a partition, but you can also cypher a file or logical volume), the procedure is pretty easy. When you want to work with that safe area, you only have to mount it by means of TrueCrypt.

javi@javi-ubuntu:/tmp$ mkdir /mnt/truecrypt

javi@javi-ubuntu:/tmp$ sudo ./truecrypt /dev/sdb2 /mnt/truecrypt
Enter password for /dev/sdb2: 
Enter keyfile [none]: 
Protect hidden volume (if any)? (y=Yes/n=No) [No]:

javi@javi-ubuntu:/tmp$ ./truecrypt --list
1: /dev/sdb2 /dev/mapper/truecrypt1 /mnt/truecrypt

By running the following command, you may collect more details about a mounted volume.

javi@javi-ubuntu:/tmp$ ./truecrypt --volume-properties /dev/sdb2
Slot: 1
Volume: /dev/sdb2
Virtual Device: /dev/mapper/truecrypt1
Mount Directory: /mnt/truecrypt
Size: 50.0 GB
Type: Normal
Read-Only: No
Hidden Volume Protected: No
Encryption Algorithm: AES
Primary Key Size: 256 bits
Secondary Key Size (XTS Mode): 256 bits
Block Size: 128 bits
Mode of Operation: XTS
PKCS-5 PRF: HMAC-RIPEMD-160
Volume Format Version: 2
Embedded Backup Header: Yes

You can dismount it by executing the next order.

javi@javi-ubuntu:/tmp$ sudo ./truecrypt --dismount /mnt/truecrypt

TrueCrypt has got many more options through the command line. I invite you to take a look at them by checking its help.

And finally, I would like to conclude this article by writing down the order (based on rsync) that I usually run to back up my data into the private partiton.

javi@javi-ubuntu:~$ rsync -altgvb --delete /data /mnt/truecrypt/

ARP poisoning (III)

2011-11-15T18:30:00.001+01:00

During the first article about ARP poisoning (I), we learnt the danger of connecting to a service by using a non-secure protocol, such as HTTP, FTP, SMTP and so on. The username and password are passed down in clear, and anyone could sniff them.

Ok, that's right, so we have to use safe protocols (HTTPS, SSH, FTPS, etc.). But what occurs whether the digital certificate utilized to authenticate and encrypt the communication is changed on the fly? That is just what we studied in the second article about ARP poisoning (II). The bottom line was that we always have to pay attention when we load a webpage and, we must only accept a trusted certificate.

What would happen if one day we are a little bit asleep and we do not realize that we are using HTTP rather than HTTPS? What? How is it possible that I am logging in to my bank account and that access is not provided through HTTPS? Well you should believe it.

Bellow you can look into the normal login in the Oracle website, both Firefox and Google Chrome. You may observe that both accesses are correctly served by means of HTTPS.

Imagine for a moment that an intruder carries out a poisoning attack between you and the router, in order to intercept all data transmitted. Then, he sets up a tool like sslstrip to establish two TCP communications. On the one hand, a first HTTPS connection between him and the Oracle web, by using the real certificate offered by Oracle, and on the other, a second HTTP connection between him and you. This is the target of sslstrip, to take advantage of a Man in the Middle attack (MitM) for tapping SSL/TLS conversations.

root@attacker:~# aptitude install sslstrip

root@attacker:~# iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 10000

root@attacker:~# sslstrip -w victim.log
sslstrip 0.9 by Moxie Marlinspike running...

root@attacker:~# ettercap -TqM arp:remote /192.168.1.1/ /192.168.1.10/

After running ettercap and forwarding all HTTP traffic to port 10000 (default port used by sslstrip), if the victim tries to open the aforementioned HTTPS Oracle web page, it will turn up the HTTP version of the site (sslstrip takes care of transforming the preceding content sent out by Oracle and serves it to the victim through a HTTP session).

The following figures show the manipulated web page created by sslstrip.

If the victim attempts to sign in, the credentials will be catched by the attacker.

root@attacker:~# cat victim.log
...
2011-11-05 19:51:47,876 POST Data (login.oracle.com):
...
AD91DC75E382F4E9ACDC66D839F095558488AA1754EB29D4513F832B83CB31BF05DB93ACCC18255184E5296825625A56EA6&amp;locale=&amp;ssousername=test%40mytest.com&amp;password=test2

Ok, perfect, so to get out of this kind of attack, first of all, we must have a good cup of coffe every morning, ;-), and second, to be very careful when we surf the Internet. At any rate, as commented in the first post, the end of this series of articles is to present later a great tool which will help us to shut out this sort of problems.

Carrying on with sslstrip, it still holds a last trick: to be able to draw a padlock icon in the navigation bar.

root@attacker:~# sslstrip -f -w victim.log
sslstrip 0.9 by Moxie Marlinspike running...

You can take a look at it in both browsers.

It is very important to underline the risks of this type of attack. You could check it out with hundreds of websites (banks, e-commerce, sports betting, etc.) and in the most of them, you could be spoofed. But I have also seen that there are other webs such as PayPal, where the altered web page does not work out very well.

Ubuntu Server instead of CentOS?

2011-11-08T13:03:00.001+01:00

Although both are outstanding Linux distributions, nowadays I choose Ubuntu Server. For a long time, I have prefered CentOS rather than Ubuntu Server, but today, I always install Ubuntu Server unless there is some requirement which forces me to do the opposite (for instance, when some application just is supported for CentOS/RHEL).

I am not going to focus on certain details such as the performance, architecture, support and so on. I only want to talk about those simple things that, when I finish the installation of an operating system, I usually say: I like it!

For my tests, I am going to use two similar versions: Ubuntu Server 10.04 LTS and CentOS 6.0, both 32 bits. After the initial installation (and their corresponding upgrades), here you are a typical view of the system status. As you can distinguish, Ubuntu Server grabs little memory, since the most of it is cached. In respect of the number of active processes, it also has got fewer than CentOS.

root@ubuntu-server:~# top
top - 12:17:54 up 13 min,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:  78 total,   1 running,  77 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2061260k total,   126644k used,  1934616k free,    17088k buffers
Swap:   565240k total,        0k used,   565240k free,    87796k cached
...

[root@centos ~]# top
top - 12:17:49 up 13 min,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:  84 total,   1 running,  83 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2071620k total,    99020k used,  1972600k free,     5272k buffers
Swap:  4161528k total,        0k used,  4161528k free,    29488k cached
...

What about the initial space taken up for the installation? (In order to get a more accurate result, I have cleaned the package cache). As you can see, CentOS occupies around 225 MB less than Ubuntu Server. I have to highlight this point, because this aspect has improved a lot on CentOS 6.0, since we have now a version of minimal installation. With CentOS 5, the final size was bigger.

root@ubuntu-server:~# aptitude clean

root@ubuntu-server:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--server-root
                       12G  888M  9.6G   9% /
none                 1002M  172K 1002M   1% /dev
none                 1007M     0 1007M   0% /dev/shm
none                 1007M   32K 1007M   1% /var/run
none                 1007M     0 1007M   0% /var/lock
none                 1007M     0 1007M   0% /lib/init/rw
/dev/sda1             228M   31M  185M  15% /boot

[root@centos ~]# yum clean all

[root@centos ~]# df -h
S.ficheros            Size  Used Avail Use% Montado en
/dev/mapper/vg_centos-lv_root
                      7,5G  664M  6,4G  10% /
tmpfs                1012M     0 1012M   0% /dev/shm
/dev/sda1             485M   56M  404M  13% /boot

This situation is reflected as well when we take a look at the number of packages installed on the system.

root@ubuntu-server:~# dpkg -l | grep ii | wc -l
358

[root@centos ~]# yum list installed | wc -l
234

Let's move on to the services which are listening on the system at the beginning. You may appreciate that the picture of Ubuntu Server is impeccable. There is no process bound to any port (aside from SSH). But what happens with CentOS? There are different applications which have already been started up (TCP and UDP). This is a waste of time for me, because at the end of each CentOS installation, I have to remove them.

root@ubuntu-server:~# netstat -nltup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      810/sshd        
tcp6       0      0 :::22                   :::*                    LISTEN      810/sshd 

[root@centos ~]# netstat -nltup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1071/rpcbind        
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1191/sshd           
tcp        0      0 0.0.0.0:44568               0.0.0.0:*                   LISTEN      1089/rpc.statd      
tcp        0      0 :::111                      :::*                        LISTEN      1071/rpcbind        
tcp        0      0 :::55445                    :::*                        LISTEN      1089/rpc.statd      
tcp        0      0 :::22                       :::*                        LISTEN      1191/sshd           
udp        0      0 0.0.0.0:822                 0.0.0.0:*                               1071/rpcbind        
udp        0      0 0.0.0.0:841                 0.0.0.0:*                               1089/rpc.statd      
udp        0      0 0.0.0.0:45143               0.0.0.0:*                               1089/rpc.statd      
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1071/rpcbind        
udp        0      0 :::822                      :::*                                    1071/rpcbind        
udp        0      0 :::43338                    :::*                                    1089/rpc.statd      
udp        0      0 :::111                      :::*                                    1071/rpcbind

Regarding the repositories provided for each distribution, Ubuntu Server supplies a larger number of packages than CentOS, and this is another plus. Although you can add excelent additional repositories such as EPEL, those extra packages are not officialy supported.

root@ubuntu-server:~# apt-cache stats | grep Normal
  Normal packages: 30299

[root@centos ~]# yum list all | wc -l
4595

Also point out the life cycle of each distribution. On Ubuntu Server you have got a LTS (Long Term Support) version each three years. In contrast, on CentOS, the first release of the branch 5 was shipped in March 2007 and CentOS 6.0, in July 2011 (more than four years after). What goes on with this? Over time, you have to use an operating system where the most of the packages, although still supported, are obsoleted.

And finally, I have metered the time spent in order to reboot the system (both use Upstart). This parameter is really important, mainly in production environments. I have obtained 20 seconds on Ubuntu Server and 40 on CentOS.

ARP poisoning (II)

2011-11-02T12:45:00.001+01:00

During the last article, ARP poisoning (I), you were able to learn the risks of using non-secure protocols inside an unreliable network. At any moment, your connection credentials can be captured by any intruder and you will not be aware of that. Note that this situation can be very common when you surf the Internet and go to HTTP websites, or for example, when you log into your MSN account.

So what happens with secure protocols such as HTTPS? That is to say, for instance when you access your online bank account, PayPal, Gmail, LinkedIn and so on. Are you safe? In most cases, that will depend on you.

Let's go over the normal behavior of a secure site like Facebook. If you click with the left mouse button on facebook.com (in the web browser bar, once you have opened the site), you will be able to appreciate that the connection to the web is encrypted and verified by DigiCert Inc (certification authority).

By pressing on the More Information button, you may take a look at the features of the digital certificate offered by Facebook. As you can pick out in the first screen, the certificate has been issued by DigiCert Inc to Facebook, and in the second one, it is made up by a valid Certificate Hierarchy.

Now we are going to use another audit tool: Ettercap (NG-0.7.3). This program is aimed at sniffing switched LANs, by supporting active and passive analysis of many protocols (HTTP, FTP, POP, IMAP, NFS, etc.), even ciphered ones.

In addition, it includes many options for network and host inspection, data injection in an established connection, lots of loadable modules at runtime, also known as plugins (arp_cop - report suspicious ARP activity -, dos_attack - run a DoS against a victim -, finger - fingerprint a remote host -, etc.), several MitM attacks (ARP poisoning, ICMP redirection, DHCP spoofing and port stealing) and so on.

The victim computer is going to open Facebook (HTTPS) through a web browser (Firefox). Therefore, the victim will go out across the router so as to reach Facebook via Internet.

Ettercap will be utilized in order to poison both elements, victim and router, to sniff all traffic between them. So how can the attacker capture the password, whether this one is sent out through the secure channel previously set up? First up, the traffic between the victim and Facebook is not going directly to the router, but that it will pass through the attacker, which will be picking up all data.

Thereby, on the one hand the attacker will establish an HTTPS connection between itself and Facebook by using the correct certificate issued by Facebook, and on the other, another HTTPS connection between itself and the victim, but this time, by means of a fake certificate created on the fly and which will have all fields filled according to the real certificate presented by Facebook.

Let's get started by editing the configuration file of Ettercap, in order to enable the iptables command to allow the TCP redirection at kernel level, so as to be able to handle SSL dissection.

root@attacker:~# aptitude install ettercap

root@attacker:~# cat /etc/etter.conf
...
[privs]
ec_uid = 0                # nobody is the default
ec_gid = 0                # nobody is the default
...
   redir_command_on = "iptables -t nat -A PREROUTING -i %iface -p tcp --dport %port -j REDIRECT --to-port %rport"
   redir_command_off = "iptables -t nat -D PREROUTING -i %iface -p tcp --dport %port -j REDIRECT --to-port %rport"
...

Now we are ready to run Ettercap by spoofing both targets and activating the ARP poisoning MitM attack. The 'remote' parameter is set in order to capture the connections which pass through the router, otherwise just the connections between them would be catched.

root@attacker:~# ettercap -TqM arp:remote /192.168.1.1/ /192.168.1.10/

ettercap NG-0.7.3 copyright 2001-2004 ALoR & NaGA

Listening on eth0... (Ethernet)

  eth0 ->       00:0C:29:20:9F:9B      192.168.1.20     255.255.255.0

Privileges dropped to UID 0 GID 0...

  28 plugins
  39 protocol dissectors
  53 ports monitored
7587 mac vendor fingerprint
1698 tcp OS fingerprint
2183 known services

Scanning for merged targets (2 hosts)...

* |==================================================>| 100.00 %

2 hosts added to the hosts list...

ARP poisoning victims:

 GROUP 1 : 192.168.1.1 00:60:B3:50:AB:45

 GROUP 2 : 192.168.1.10 00:0C:29:69:81:47
Starting Unified sniffing...


Text only Interface activated...
Hit 'h' for inline help

At this moment, if you open Facebook again, Firefox will warn you that it cannot confirm that the connection is secure. Normally, when you try to connect securely, sites such as banks, stores, public organisms, etc., present trusted identifications to prove that you are going to the right place.

If you confirm the security exception and accept the digital certificate, you will have fallen into the trap of the attacker. Let's review the characteristics of this invalid certificate, so as to be able to compare it with the real certificate (second figure).

As you can make out in the general features of the fake certificate, only the fingerprints are modified, because of the attacker has signed it with him private key. Besides, the undependable certificate does not present a correct hierarchy.

Now if you attempt to login into Facebook, your credentials will be catched by the attacker.

root@attacker:~# ettercap -TqM arp:remote /192.168.1.1/ /192.168.1.10/
...

Text only Interface activated...
Hit 'h' for inline help

HTTP : 69.171.224.39:443 -> USER: test@mytest.com  PASS: test1  INFO: https://www.facebook.com/

ARP poisoning (I)

2011-10-26T22:53:00.000+02:00

From a long time, I wanted to write some article about this issue. I think that people are not aware of the potential risks when they connect to a public network, such as inside an airport, library, pub and so on, even the own office network.

Many times I have heard: it is not not such a big deal, you know what? I have a good antivirus which protects my computer! And on top of all that, the Windows firewall is activated! At that moment is when I put poker face...

Most of the administrators think that by having a well-configured firewall, an IDS, an antivirus, etc., is enough to shield the network from external threats, but it turns out that around 70 or 80 percent of all attacks come from the own internal network.

Please, note that the things which I am going to explain throughout these articles, can be a cause of crime, so you will be the last responsible if you put them into action with bad intentions. The reason because I want to tell this is, on the one hand, due to it is good that you know the danger of connecting to an unreliable network, and on the other, because I will take advantage of this in order to show you how to avoid it.

To begin with, let's get started by saying how ARP works (Address Resolution Protocol). Basically, this protocol is used to associate MAC and IP addresses.

For example, one computer wants to know the MAC address of a router. In this case, that computer gives off a message to the network by asking who has the IP address of that router (ARP request). Then, only the router responds to the computer with its MAC address (ARP reply).

Hereafter, the computer stores into its MAC table (temporary) the IP and MAC address of the router. ARP poisoning, as its name suggests, is to manipulate the MAC table of the victim by injecting fake ARP packets.

What kind of attacks can derive from this situation? For instance, the well-known Man in the Middle attack (MitM).

Below you can see the environment which I will hold for my tests. Victim and attacker are an Ubuntu 10.11, and ubuntu-server is an Ubuntu Server 11.10 release.

In my first case, I am going to put the attacker computer intercepting all communications between ubuntu-server and victim. To be more precise, the victim will connect to a FTP service installed on ubuntu-server and the attacker will try to capture the password. Remember this sort of protocol, also such as HTTP, SMTP, POP3, etc., the credentials are passed down in clear.

So that the attacker node can work as a tranparent bridge, the IP forwarding must be enabled on it. Furthermore, we have to install the dsniff package which contains the arpspoof tool, program that will be used to poison both computers (client and server).

root@attacker:~# echo 1 > /proc/sys/net/ipv4/ip_forward

root@attacker:~# aptitude install dsniff

Let's take a look at their ARP tables before modifying them. As you may appreciate, both computers have registered the correct MAC addresses.

javi@ubuntu-server:~$ arp -a
? (192.168.1.1) at 00:60:b3:50:ab:45 [ether] on eth0
? (192.168.1.10) at 00:0c:29:69:81:47 [ether] on eth0

javi@victim:~$ arp -a
? (192.168.1.1) at 00:60:b3:50:ab:45 [ether] on eth0
? (192.168.1.11) at 00:0c:29:18:36:e6 [ether] on eth0

Next step is to alter those tables by transmitting fake ARP frames.

root@attacker:~# arpspoof -i eth0 -t 192.168.1.10 192.168.1.11
0:c:29:20:9f:9b 0:c:29:69:81:47 0806 42: arp reply 192.168.1.11 is-at 0:c:29:20:9f:9b
...

root@attacker:~# arpspoof -i eth0 -t 192.168.1.11 192.168.1.10
0:c:29:20:9f:9b 0:c:29:18:36:e6 0806 42: arp reply 192.168.1.10 is-at 0:c:29:20:9f:9b
...

If we output the ARP tables again, we can see that the entries have been changed.

javi@ubuntu-server:~$ arp -a
? (192.168.1.20) at 00:0c:29:20:9f:9b [ether] on eth0
? (192.168.1.1) at 00:60:b3:50:ab:45 [ether] on eth0
? (192.168.1.10) at 00:0c:29:20:9f:9b [ether] on eth0

javi@victim:~$ arp -a
? (192.168.1.11) at 00:0c:29:20:9f:9b [ether] on eth0
? (192.168.1.1) at 00:60:b3:50:ab:45 [ether] on eth0
? (192.168.1.20) at 00:0c:29:20:9f:9b [ether] on eth0

At this point, the attacker is ready to sniff all traffic between the implicated nodes. To simplify the test, just the FTP data will be picked up. In this case, I am dumping all FTP packets within a text file with tcpdump, so as to be able to analyze them before with Wireshark. I could also use Wireshark directly by means of a filter.

root@attacker:~# tcpdump -ni eth0 port 21 -s0 -w ftp.pcap

Last step is to establish a FTP session between victim and ubuntu-server.

javi@victim:~$ ftp 192.168.1.11
Connected to 192.168.1.11.
220 (vsFTPd 2.3.2)
Name (192.168.1.11:javi): javi
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

Now we are going to open the captured file through Wireshark. As you can distinguish, the password has been catched.

In addition, if you follow the TCP stream, you will be able to find out that there are several retransmissions. That occurs because the attacker has to forward the TCP/IP packets. This sequence would come out as well if you run tcpdump on ubuntu-server.

And finally, also mention that if IP forwarding was not activated, we would be causing a Denial of Service attack (DoS), due to the communication would be cut out.

Access Control Lists (II)

2011-10-19T17:34:00.001+02:00

In the preceding article about Access Control List, we saw how to grant permissions either on a file or directory for a particular user, and in addition, how to set those ones for new elements by default.

Now, we are going to give permissions to the nobody group and other users. Note that when you are applying ACLs for other users, it is like when you are handling the chmod command.

[root@centos logs]# setfacl -m g:nobody:-w- 002.log

[root@centos logs]# setfacl -m o:rw- 002.log

[root@centos logs]# getfacl 002.log 
# file: 002.log
# owner: root
# group: root
user::rw-
user:nobody:r-x
group::---
group:nobody:-w-
mask::rwx
other::rw-

In order to remove ACLs, we may delete them for a specific user, clear all entries or only get rid of the default ACLs.

[root@centos logs]# setfacl -x g:nobody 002.log

[root@centos /]# setfacl -R -b /logs

[root@centos /]# setfacl -k /logs

Other handy option for ACLs is to associate a mask with an ACL, that is to say, to establish real or effective permissions on a file or directory. In this case, we are limiting the permissions available on a file or directory. For instance, in the following case we are setting read, write and execution permissions for nobody user, but afterwards, we are also applying a mask of just execution.

[root@centos logs]# setfacl -m u:nobody:rwx 002.log

[root@centos logs]# setfacl -m m:--x 002.log

[root@centos logs]# getfacl 002.log
# file: 002.log
# owner: root
# group: root
user::rw-
user:nobody:rwx                 #effective:--x
group::---
mask::--x
other::rw-

Extra Packages for Enterprise Linux (EPEL)

2011-10-13T13:48:00.001+02:00

Today I am going to fill you in on EPEL, a repository maintained by a Fedora group (made up by Red Hat engineers, volunteer community members, etc.) which offers a set of additional packages for Enterprise Linux distributions, such as RHEL, CentOS, Scientific Linux and so on.

For instance, when you purchase a license for RHEL, Red Hat guarantees you support for a series of packages included within its repositories, but other many applications are not provided through them.

Thereby, you have got several options to install packages not located into the official repositories, such as grabbing them from RPM PBone. But another smart option is to set up on your machine, an EPEL repository, whereby you will have high quality add-on packages which will complement your system.

EPEL packages are built from the equivalent ones in Fedora project and they are updated as far as the corresponding RHEL release is supported.

There are EPEL repositories for RHEL4, RHEL5 and RHEL6 (they are too valid for their derived). For my tests, I am going to use a CentOS 6.0 distro where I will install the appropiate 'epel-release' package. By default, only the stable EPEL repository is enabled. Later, you might enable testing and not yet considered stable repositories (I don't recommend it).

[root@centos ~]# rpm -ivh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

[root@centos ~]# ls -l /etc/yum.repos.d/epel*
-rw-r--r--. 1 root root  957 oct 12  2010 /etc/yum.repos.d/epel.repo
-rw-r--r--. 1 root root 1056 oct 12  2010 /etc/yum.repos.d/epel-testing.repo

How can we check out if a package comes from EPEL?

[root@centos ~]# yum install keychecker

[root@centos ~]# keychecker httpd
CentOS-6 Key (CentOS 6 Official Signing Key)
--------------------------------------------
httpd-2.2.15-5.el6.centos.i686

[root@centos ~]# keychecker keychecker
EPEL (6)
--------
keychecker-0.2-2.el6.noarch

Other way is by using yum.

[root@centos ~]# yum info htop
...
Repo       : epel
Summary    : Interactive process viewer
...

Access Control Lists (I)

2011-10-04T18:21:00.001+02:00

One month ago I had to publish the log files of one application at work. The log directory had to be accessible by the development team (they use Windows). Also say that the application runs on CentOS 6.0

No problem. I shared the directory through Samba and granted access to the guest user (on Linux, this is translated to the nobody user).

[root@centos ~]# cat /etc/samba/smb.conf 
[global]
        security      = user
        map to guest  = bad user
        guest account = nobody

[logs]
        path     = /logs
        readonly = yes
        guest ok = yes

Later I was warned that certain files could not be read. By taking a look at it, I could see that some files were been created with wrong permissions.

[root@centos ~]# ls -l /logs/
total 6148
-rw-------. 1 root root 4730880 oct  4 11:37 001.log
-rw-------. 1 root root 1564672 oct  4 11:37 002.log

As you can appreciate, the files just could be read by the owner, in this case root. This was the second problem: the application ran as root and of course, I could not allow access by means of this user.

We opened a ticket to the support center, in order to find out if it were possible to force the program to create the log files with other permissions. The response was fantastic: set up a cron task so as to change them periodically. As I usually say... a real botched.

Fortunately, Linux is a great operating system which if you know it in depth, you will be able to solve problems in different ways.

I sized up the situation and I decided that the best option was to set an ACL (Access Control List). With ACLs, you can give selected users, read, write and execute permissions on a specific file or directory.

First up, you need to have configured the target filesystem with the acl option.

[root@centos ~]# mount -o remount,acl /

[root@centos ~]# cat /etc/fstab
/dev/mapper/vg_centos-lv_root   /       ext4    defaults,acl    1 1
...

Then, you must grant the nobody user, read and execute permissions on all elements of the directory and besides, new files or directories created within it, will also have this ACL by default.

[root@centos ~]# setfacl -R -m u:nobody:r-x /logs

[root@centos ~]# setfacl -d -R -m u:nobody:r-x /logs

In this manner, when a user logs on via Samba (guest user), will be able to read the files. Let's get now the full permissions from any of the files included into the logs directory.

[root@centos ~]# getfacl /logs/001.log
# file: logs/001.log
# owner: root
# group: root
user::rw-
user:nobody:r-x
group::---
mask::r-x
other::---

As you can see above, apart from root, the nobody user can also read the file.

It may seem incredible but ACLs are not well known. I have seen throughout my professional life, authentic disasters by applying permissions on files, mainly due to ignorance of the administrators.

And as you have been able to learn, ACLs are an elegant way to handle the file permissions. Next week I will end up this article with other stuff that you can perform with ACLs.

Zabbix client installation on Ubuntu

2011-09-28T21:35:00.002+02:00

Through this article, I wanted to write down how to set up the Zabbix client from its source code on Ubuntu distributions. Some time ago I posted a similar article but utilizing a CentOS host. For this case, I am going to accomplish the same task but choosing an Ubuntu Server 11.04 and Zabbix 1.8.7.

First of all, we need to download the source code from the Zabbix web site and decompress it inside the server. We must have installed too the build-essential package, so as to be able to compile the Zabbix client.

root@ubuntu-server:~# aptitude install build-essential

root@ubuntu-server:~/zabbix-1.8.7# ./configure --enable-agent

root@ubuntu-server:~/zabbix-1.8.7# make ; make install

Once we have correctly compiled and installed the Zabbix agent, next step is to create the appropiate directories, copy the configuration files and add a new user to the system called zabbix.

root@ubuntu-server:~/zabbix-1.8.7# mkdir -p /etc/zabbix/alert.d /var/log/zabbix /var/run/zabbix

root@ubuntu-server:~/zabbix-1.8.7# cp -a misc/conf/zabbix_agentd.conf /etc/zabbix/

root@ubuntu-server:~/zabbix-1.8.7# cp misc/init.d/ubuntu/zabbix-agent.conf /etc/init/

root@ubuntu-server:~/zabbix-1.8.7# useradd -r -d /var/run/zabbix -s /sbin/nologin zabbix

root@ubuntu-server:~/zabbix-1.8.7# chown zabbix:zabbix /var/run/zabbix /var/log/zabbix

Afterwards, we must edit the minimum information required for the Zabbix agent configuration file and in addition, it is also neccesary to establish an Upstart file for starting up and stopping the Zabbix agent service.

root@ubuntu-server:~# cat /etc/zabbix/zabbix_agentd.conf
...
# Zabbix client PID file
PidFile=/var/run/zabbix/zabbix_agentd.pid

# Zabbix client log file
LogFile=/var/log/zabbix/zabbix_agentd.log

# Allow remote commands from zabbix server
EnableRemoteCommands=1

# Maximum time for processing
Timeout=10

# System hostname
Hostname=ubuntu

# Zabbix server IP
Server=192.168.1.100


root@ubuntu-server:~# cat /etc/init/zabbix-agent.conf
# Start zabbix agent

pre-start script
   if [ ! -d /var/run/zabbix ]; then
           mkdir -p /var/run/zabbix
           chown zabbix:zabbix /var/run/zabbix
   fi
end script

start on filesystem
stop on starting shutdown
respawn
expect daemon
exec /usr/local/sbin/zabbix_agentd

The last point is to register the ports used by Zabbix into the services file and run the agent.

root@ubuntu-server:~# echo "zabbix-agent    10050/tcp  Zabbix Agent"   >> /etc/services
root@ubuntu-server:~# echo "zabbix-agent    10050/udp  Zabbix Agent"   >> /etc/services
root@ubuntu-server:~# echo "zabbix-trapper  10051/tcp  Zabbix Trapper" >> /etc/services
root@ubuntu-server:~# echo "zabbix-trapper  10051/udp  Zabbix Trapper" >> /etc/services


root@ubuntu-server:~# start zabbix-agent

Avira AntiVir Personal on Linux (IV)

2011-09-21T12:36:00.001+02:00

With this post, I am going to end up the series of articles about Avira Antivir Personal on Linux. So, let's take a look at one of its more important modules: AntiVir Guard.

AntiVir Guard takes care of scanning and protecting a filesystem on real-time, that is to say, a virus will be detected before accessing on it. How does it work? All directories which we want to protect by AntiVir Guard, will be mounted through DazukoFS module, previously compiled and inserted into the kernel.

[root@centos ~]# cat /etc/fstab
...
/home    /home    dazukofs

AntiVir Guard (avguard) can be handled either by means of the avguard command or as an init daemon. In this article, I am going to focus on the second option, since it's most useful and handy.

Thereby, we have to set it up by editing its configuration file (/etc/avira/avguard.conf). Below I am going to note the most important features.

[root@centos ~]# vi /etc/avira/avguard.conf
...
# It will try to delete the problem from the infected file (by default is disabled).
# If the repair fails, the AlertAction is carried out.
RepairConcerningFiles

# Once a virus is detected, the access to the file is blocked and the action is logged.
# This allows you to specify an additional action to be followed for the concerning file.
# none or ignore: no further action (by default).
# rename or ren: rename the file by adding the .XXX extension.
# delete or del: delete the concerning file.
# quarantine: move the concerning file into quarantine.
AlertAction delete

# If quarantine option is selected, the infected files are moved into it.
QuarantineDirectory /home/quarantine

# Types of files to be scanned.
# extlist: scan only files with certain extensions.
# smart: scan files based on both their name and content.
# all: scan all files (by default).
ScanMode all

# File where all important operations are logged.
LogFile /var/log/avguard.log

# Detection of harmful or unwanted software (dial-up programs, jokes, faked emails, etc.).
# With the 'alltypes' option, all supported malware types will be detected.
DetectPrefixes adspy=yes appl=no bdc=yes dial=yes game=no joke=no pck=no phish=yes spr=no

# Activate the heuristics for macro virus in office documents.
# [yes (by default) | no].
HeuristicsMacro yes

# Set the level of heuristic detection in all types of files.
# Available values are 0 (off), 1 (low - by default), 2 (medium) and 3 (high).
HeuristicsLevel 2


[root@centos home]# /etc/init.d/avguard restart

To check it out, we are going to download the EICAR file into the /home directory and try out to dump it.

[root@centos home]# wget https://secure.eicar.org/eicar.com.txt

[root@centos home]# cat eicar.com.txt 
cat: eicar.com.txt: Operation not supported

[root@centos home]# tail -f /var/log/avguard.log 
2011-09-18 18:52:48 centos.local avguard.bin[1396]: AVGU: ALERT AntiVir ALERT for file "/home/eicar.com.txt": Details:        Eicar-Test-Signature ; virus ; Contains code of the Eicar-Test-Signature virus
2011-09-18 18:52:48 centos.local avguard.bin[1396]: AVGU: INFO The concerning file /home/eicar.com.txt has been removed from disk.
2011-09-18 18:52:48 centos.local avguard.bin[1396]: AVGU: INFO Info: the alert in file /home/eicar.com.txt was handled. Action(s) taken: access denied, condition logged, file deleted

As you have been able to appreciate, the infected file has been removed when we have tried to read it. So imagine the amount of possibilities which turn out from this module, such as to analyze on-real time a file uploaded to a FTP or HTTP (WebDAV) server, or for instance, you might use tools like swatch in order to send an alert or execute a task.

Monitoring logs with swatch

2011-09-12T16:47:00.000+02:00

Swatch is a GPL tool programmed in Perl which allows monitoring logs on real-time, and it is aimed to be able to execute an action when a certain situation takes place.

An application can register an event into a file as a result of an error, warning, etc., and at that moment, it may be interesting to restart the involved service or for instance, to send an email reporting the alarm, all automatically.

Here is where swatch turns up. You have got two ways to install it: either by means of the package which each distribution keeps in its repositories or directly by compiling the source code.

In the case of Ubuntu, the installation is really simple: aptitude install swatch. But in RHEL or CentOS, the package is not available in the official repositories of such distributions.

Therefore, in the present article I am going to develop the installation of swatch (3.2.3) on CentOS 6.0 (32 bits, minimal installation) by downloading and installing the suitable packages from RPM PBone Search.

[root@centos tmp]# rpm -i perl-Carp-Clan-6.03-2.el6.noarch.rpm
[root@centos tmp]# rpm -i perl-Bit-Vector-7.1-2.el6.i686.rpm
[root@centos tmp]# rpm -i perl-Date-Calc-6.3-2.el6.noarch.rpm
[root@centos tmp]# rpm -i perl-Date-Manip-5.54-4.el6.noarch.rpm 
[root@centos tmp]# rpm -i perl-TimeDate-1.16-11.1.el6.noarch.rpm
[root@centos tmp]# rpm -i perl-Time-HiRes-1.9721-115.el6.i686.rpm
[root@centos tmp]# rpm -i perl-File-Tail-0.99.3-8.el6.noarch.rpm
[root@centos tmp]# rpm -i perl-Mail-Sendmail-0.79-12.el6.noarch.rpm

[root@centos tmp]# rpm -i swatch-3.2.3-2.el6.noarch.rpm

So that swatch can send alarms by email, you have to install some kind of MTA (Mail Transfer Agent) on your system, such as Postfix.

[root@centos ~]# yum install postfix

[root@centos ~]# cat /etc/postfix/main.cf
...
# Internet hostname
myhostname = centos.local

# Local Internet domain name
mydomain = local

# Domain that locally-posted mail appears to come from
myorigin = $myhostname

# Network interface addresses to receive mail
inet_interfaces = all

# List of domains to consider itself the final destination
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
...

[root@centos ~]# service postfix restart

[root@centos ~]# chkconfig postfix on

Through the following example, we will control the /var/log/secure file in order to detect the login of the user javi (we must look for the string "Accepted password for javi").

First of all, we have to create a directory to drop off the configuration files of swatch. Afterwards, we must set up a file with the needed instructions to log the access for the user javi.

[root@centos ~]# mkdir /etc/swatch

[root@centos ~]# cat /etc/swatch/swatch.conf
watchfor /Accepted password for javi/
        mail addresses=root\@centos.local,subject="Session opened by javi"

With the previous line, swatch will monitor the content of a concrete file which will be later given with the target of matching the requested string. When the coincidental text is found, an email will be passed down.

So as to start swatch, we must run the next command ('-t' option comes from the traditional 'tail -f'). If instead of using '-t' parameter, you add '-f', swatch would execute the defined configuration once and then, close the file. In this manner, the file is not open as in the case of a typical 'tail -f'.

[root@centos ~]# swatch -c /etc/swatch/swatch.conf -t /var/log/secure

Swatch has got other many options for its configuration file, such as outputting the matched pattern, sending a bell, executing commands and so on. The following example watches for a couple of strings.

[root@centos ~]# cat /etc/swatch/swatch.conf
watchfor /Accepted password for javi|Accepted password for pepe/
    echo=red

Avira AntiVir Personal on Linux (III)

2011-09-06T21:34:00.002+02:00

Now we have installed Avira AntiVir Personal on Linux (II), in this article I am going to treat one of its main modules: AntiVir Command Line Scanner (avcan).

This component is launched from the command prompt (on-demand), and it takes care of analyzing files in order to look for possible malware infections. Avscan can delete, repair, isolate or simply warn.

One of the most powerful advantages of this kind of tool is which can be integrated with scripts. In this way, you may use it for example with a web service, where the files are uploaded and it can be neccesary to scan them before storing the files into the hard drive.

Avcan can be configured by means of its own configuration file (/etc/avira/avscan.conf). In this manner, when you run the scanner, this will utilize the options established into the file (by default).

But indeed, the most interesting possibility is to be able to set the scanning options when you execute it (on real-time), because for instance, you might have various scanning tasks with different types of analysis.

Then let's take a look at the principal features of avcan. For this purpose, I will download the EICAR test file (harmless virus used to try out the behaviour of an antivirus).

[root@centos ~]# wget https://secure.eicar.org/eicar.com

[root@centos ~]# avscan -h
syntax: avscan [option ...] [directory] [filename] ...
...

When a virus is detected, you may choose between several actions: ignore the alert (none or ignore), remove the file (delete or del), change the name of the file (rename or ren) or move the file into the quarantine area (quarantine). You can also add the '-e' parameter so that the infected file is repaired whenever possible.

[root@centos ~]# avscan --batch --alert-action=quarantine eicar.com

[root@centos ~]# avscan --batch --alert-action=delete -e eicar.com

By adding the '--batch' parameter, we are avoiding to be asked by avscan during the analysis, and all decisions are performed based on the configuration file and command-line settings.

Other option is to detect certain categories of software which are not considered malware, such as jokes programs (joke), files compressed with an unusual tool (pck), dial-up programs (dial) and so on. With the 'alltypes' option, all available types will be treated.

[root@centos ~]# avscan --batch --alert-action=delete --detect-prefixes="joke=yes phish=yes" eicar.com

[root@centos ~]# avscan --batch --alert-action=delete --detect-prefixes=alltypes eicar.com

Regarding the virus analysis, other important option is to enable the heuristic scanning. Avcan is able to use heuristics to conclude if a certain file is malicious. This allows that new or unknown code can be detected before an update. The level of heuristics increases the intensity of the scanning: 0 (off), 1 (low, by default), 2 (medium) and 3 (high).

[root@centos ~]# avscan --batch --alert-action=delete --heur-level=3 eicar.com

By default, avscan decides what files must be scanned from their name or content (smart). You can force it to scan files according to their filename extensions (extlist) or analyze all files regardless of their name or content (all).

[root@centos ~]# avscan --batch --alert-action=delete --scan-mode=all dir/

With respect to the directories, if you want to enable the recursive scanning of all subdirectories within a specific path, you will have to add the '-s' parameter.

And finally, also point out that avscan returns a code after ending the analysis, and it can be really useful to be managed through scripts.

[root@centos ~]# avscan --help
...
list of return codes:
   0: Normal program termination, nothing found, no error
   1: Found concerning file
   3: Suspicious file found
   4: Warnings were issued
 255: Internal error
 254: Configuration error (invalid parameter in command-line
      or configuration file)
 253: Error while preparing on-demand scan
 252: The avguard daemon is not running
 251: The avguard daemon is not accessible
 250: Cannot initialize scan process
 249: Scan process not completed
 248: No valid license found
 211: Program aborted, because the self check failed

[root@centos ~]# avscan --batch --alert-action=delete eicar.com

[root@centos ~]# echo $?
1

If you want to review the rest of options, you can check the avscan.conf file or run the '--help' parameter.

Avira AntiVir Personal on Linux (II)

2011-08-30T15:58:00.030+02:00

Once we have installed DazukoFS on the system - Avira AntiVir Personal on Linux (I) -, we are going ahead with the installation of Avira AntiVir 3.1.3.5.

[root@centos tmp]# wget http://dlpe.antivir.com/package/wks_avira/unix/en/pers/antivir_workstation-pers.tar.gz

[root@centos tmp]# tar xvzf antivir_workstation-pers.tar.gz ; cd antivir-workstation-pers-3.1.3.5-0

The installation process is carried out by means of a bash script. After agreeing the license, the installer asks if we want to create a link for avupdate-guard.

[root@centos antivir-workstation-pers-3.1.3.5-0]# ./install
...
Would you like to create a link in /usr/sbin for avupdate-guard ? [y]
linking /usr/sbin/avupdate-guard to /usr/lib/AntiVir/guard/avupdate-guard ... done

Then the script can establish a cron task (/etc/cron.d/avira_updater) for automatic updates.

Would you like to setup Scanner update as cron task ? [y]
...
What time should updates be done [00:15]?
creating Scanner update cronjob ... done

The previous task checks if there is any update related to the scanner, engine or vdf files. On the contrary, if you accept the next request, the Guard module will be also updated periodically.

Would you like to check for Guard updates once a week ? [n]

setup internet updater complete

Next step takes care of installing DazukoFS. Due to this operation was previously accomplished, it will not be necessary to repeat it.

Preinstalled dazukofs module found on your system.

Would you like to reinstall dazukofs now ? [y] n
Dazukofs module is loaded

Through the following question, you can specify what directories must be protected by AntiVir Guard. I have selected the default option. Later, you may change this choice or add more directories by editing the fstab file.

Watch out with this selection, because regardless of the antivirus used, when you set up an on-access daemon, you have to avoid certain directories such as /sys, /proc, /root or directly /.

Guard will automatically protect all directories which are mounted upon dazukofs filesystem.

Please specify at least one directory to be protected by Guard to add in /etc/fstab : [/home]
The following directories will be protected by Guard:
/home

Then the installer verifies if the quarantine directory exists. This directory is used to isolate a suspect or infected file, so as to be able to repair it later.

Would you like to create /home/quarantine ? [y]
creating /home/quarantine ... done

Afterwards, you are asked if you want to make a link to AntiVir Guard and whether it should be automatically activated at system start.

Would you like to create a link in /usr/sbin for avguard ? [y]
linking /usr/sbin/avguard to /usr/lib/AntiVir/guard/avguard ... done

Please specify if boot scripts should be set up.
Set up boot scripts ? [y]

With the last step, we run AntiVir Guard.

Would you like to start AVIRA Guard now? [y]
Starting AVIRA AntiVir Workstation Personal ...
Starting: avguard.bin

After ending up the installation, it is highly recommended to perform a complete update of the application.

[root@centos ~]# avupdate-guard --product=Guard

Avira AntiVir Personal on Linux (I)

2011-08-22T15:56:00.015+02:00

I have always said over and over that the myth about there are no viruses for Linux is absolutely false. Occurs that there are fewer viruses on Linux because it is an open operating system, so many people may contribute quickly to fix its fails. In addition, it is more robust and less used than Windows, thereby hackers have got less interest to break it.

But figure for a moment when you are surfing the net, for instance with Firefox, and it turns out that your browser contains any critical vulnerability, or for example, the web page which you are visiting utilizes Java or Flash, and the versions that you have installed on your Linux system are vulnerable... your computer would be exposed to any attack or malware infection.

Now it is clear that many times, Linux is not really the guilty, but the responsability comes from third-party software. For that reason, I think that it is necessary to have installed a good antivirus on our Linux systems, regardless of the kind of distribution.

There are several good and free antivirus for Linux, such as avast, ClamAV, AVG, but my favourite option is Avira AntiVir. Its main features are:

Easy installation through a script.

Command Line Scanner: configurable on-demand searches for all malware types (viruses, horms, backdoors, trojans, etc.).

Resident guard: configurable on-access actions (block, delete, repair, move and rename) when malware is detected.

Heuristic detection.

Automatic update for product, scan engine and virus signature file.

The most important characteristic of Avira with regard to other solutions is the AntiVir Guard module (ClamAV has got it too), which runs as a daemon process and it is permanently monitoring all the accesses to the system (on-access) and saving it against possible viruses.

In addition, the AV-Comparatives organization published in April 2011 the last review about On-demand Detection of Malicious Software, and Avira AntiVir reached excellent results.

Other modules belonging to Avira are AntiVir Command Line Scanner (allows to scan files in search of viruses or suspicious elements, and it can be integrated with scripts) and AntiVir Updater (downloads current updates from the Avira web servers, manually or automatically).

Also say that AntiVir Guard is based on DazukoFS, an open source software that provides a kernel module which lets execute online file access control, by intercepting memory and disk calls and passing the information to an user space application, in this case Avira AntiVir. Other applications are also based on Dazuko, such as ClamAV, Panda Security for Linux, F-Secure, etc.

The AntiVir installation package supplies a DazukoFS version which is automatically configured and installed (in theory). DazukoFS depends on the kernel version; for this purpose it is better to install manually this module.

Therefore let's get going to download the version 3.1.2 of Dazuko (this number of version works fine with a 2.6.32 kernel) and install it on our testing system, CentOS 6.0 (32 bits). To begin with, I will also get some necessary packages.

[root@centos ~]# yum install gcc make kernel-devel file

[root@centos tmp]# wget http://dazuko.dnsalias.org/files/dazukofs-3.1.2.tar.gz

[root@centos tmp]# tar xvzf dazukofs-3.1.2.tar.gz ; cd dazukofs-3.1.2

Now we are ready to compile and install DazukoFS as a module into our Linux system.

[root@centos dazukofs-3.1.2]# make ; make dazukofs_install

[root@centos dazukofs-3.1.2]# modprobe dazukofs

[root@centos dazukofs-3.1.2]# echo "modprobe dazukofs" >> /etc/rc.modules

[root@centos dazukofs-3.1.2]# chmod +x /etc/rc.modules

Lastly, also point out that the license of this antivirus allows you to install it for a personal use, for instance on your own PC or your home server. Note that if you use AntiVir Guard via DazukoFS, you will need to compile this module when you change the kernel. For production environments I always suggest ClamAV.

Adding a KVM hypervisor to OpenNebula (II)

2011-08-15T14:35:00.004+02:00

Once I have finished to configure the KVM computing node, Adding a KVM hypervisor to OpenNebula (I), today I am going to conclude this series of technical articles about OpenNebula by setting a new instance up on kvm01.

First of all, I am going to use for my testing, a ttylinux image, downloaded directly from the OpenNebula website. This sort of Linux distribution is designed to consume fewer resources than a typical operating system such as Debian or CentOS.

oneadmin@frontend01:/tmp$ wget http://dev.opennebula.org/attachments/download/170/ttylinux.tar.gz

oneadmin@frontend01:/tmp$ tar xvzf ttylinux.tar.gz ; cd ~/templates

Next step is to define an image template so as to register it into OpenNebula.

oneadmin@frontend01:~/templates$ cat ttylinux.img
NAME        = "ttylinux"
PATH        = /tmp/ttylinux.img
DESCRIPTION = "Very small Linux distribution based on a 2.6 kernel"

oneadmin@frontend01:~/templates$ oneimage register ttylinux.img

oneadmin@frontend01:~/templates$ oneimage list
ID     USER                 NAME TYPE              REGTIME PUB PER STAT  #VMS
 0 oneadmin   Ubuntu Server 8.04   OS   Jul 02, 2011 10:34  No  No  rdy     0
 1 oneadmin             ttylinux   OS   Aug 07, 2011 18:30  No  No  rdy     0

Now we have a virtual image ready to be used on our KVM nodes, in this case kvm01.

oneadmin@frontend01:~/templates$ ls -lh ../var/images/8625d68b699fd30e64360471eb2c38fed47fcfb6
-rw-rw---- 1 oneadmin cloud 40M 2011-08-07 20:30 var/images/8625d68b699fd30e64360471eb2c38fed47fcfb6

oneadmin@frontend01:~/templates$ file ../var/images/8625d68b699fd30e64360471eb2c38fed47fcfb6
var/images/8625d68b699fd30e64360471eb2c38fed47fcfb6: x86 boot sector, LInux i386 boot LOader; partition 1: ID=0x83, starthead 1, startsector 63, 81585 sectors, code offset 0xeb

Then we have to make up a virtual network which will be utilized by all virtual machines built on our KVM computing node. Note that the key of this network is the bridge created in the previous article.

oneadmin@frontend01:~/templates$ cat kvm.net
NAME            = "KVM Network"
TYPE            = RANGED
PUBLIC          = NO
BRIDGE          = br0
NETWORK_ADDRESS = 192.168.1.160
NETWORK_SIZE    = 16
NETMASK         = 255.255.255.0
GATEWAY         = 192.168.1.1
DNS             = 194.30.0.1

oneadmin@frontend01:~/templates$ onevnet create kvm.net

oneadmin@frontend01:~/templates$ onevnet list
ID USER     NAME              TYPE BRIDGE P #LEASES
 0 oneadmin KVM Network     Ranged    br0 N       0

And lastly, we just have to set an instance template up where we outline the characteristics of our virtual machine and thus, to be able to run it over kvm01.

oneadmin@frontend01:~/templates$ cat ttylinux01.vm
NAME   = ttylinux01
CPU    = 1
MEMORY = 128

DISK   = [ SOURCE = "/srv/cloud/one/var/images/8625d68b699fd30e64360471eb2c38fed47fcfb6",
           TARGET = "hda" ]

NIC    = [ NETWORK = "KVM Network" ]

oneadmin@frontend01:~/templates$ onevm create ttylinux01.vm

oneadmin@frontend01:~/templates$ onevm list
ID     USER     NAME STAT CPU     MEM        HOSTNAME        TIME
 0 oneadmin ttylinux runn   0      0K           kvm01 00 00:01:03

Adding a KVM hypervisor to OpenNebula (I)

2011-08-09T12:56:00.006+02:00

After ending up how to add a VMware ESXi hypervisor to OpenNebula, now it is turn to configure a KVM node into our cloud infraestructure with OpenNebula.

To begin with, we are going to make up a network bridge on kvm01. For this purpose, we must put the NIC into manual mode and associate it to the bridge (br0). Remember that this new interface has also to have an IP address belonging to the own subnetwork.

root@kvm01:~# cat /etc/network/interfaces
...
auto eth0
   iface eth0 inet manual

auto br0
   iface br0 inet static
   address 192.168.1.12
   netmask 255.255.255.0
   network 192.168.1.0
   broadcast 192.168.1.255
   gateway 192.168.1.1
   dns-nameservers 194.30.0.1
   dns-search opennebula.local
   bridge_ports eth0
   bridge_fd 9
   bridge_hello 2
   bridge_maxage 12
   bridge_stp off

root@kvm01:~# /etc/init.d/networking restart

The reason for creating a bridge is clear: to be able to address the virtual machines built in this node. Otherwise, we would never link them.

Then we have to install the corresponding packages to be able to virtualize machines through KVM. The ruby package will be used to manage the node from OpenNebula and nfs-common to mount the shared area exported by storage01. As you can see, the libvirtd daemon must be put into listening mode without authentication.

root@kvm01:~# aptitude install kvm libvirt-bin ruby nfs-common

root@kvm01:~# cat /etc/libvirt/libvirtd.conf
...
listen_tls = 0
listen_tcp = 1
auth_tcp   = "none"

root@kvm01:~# cat /etc/libvirt/qemu.conf
...
dynamic_ownership = 0

root@kvm01:~# cat /etc/init/libvirt-bin.conf
...
env libvirtd_opts="-d -l"

root@kvm01:~# restart libvirt-bin

Besides, it is necessary to uncomment the line which says "dynamic_ownership = 1" (libvirt should dynamically change file ownership to match the configured user/group) and modify it to 0. Otherwise, you would get an error as follows.

neadmin@frontend01:~/templates$ tail -f ../var/oned.log
...
Sat Aug 13 20:32:11 2011 [TM][D]: Message received: TRANSFER SUCCESS 1 -
Sat Aug 13 20:32:12 2011 [VMM][D]: Message received: LOG - 1 Command execution fail: 'if [ -x "/var/tmp/one/vmm/kvm/deploy" ]; then /var/tmp/one/vmm/kvm/deploy /srv/cloud/one/var//1/images/deployment.0; else                              exit 42; fi'
Sat Aug 13 20:32:12 2011 [VMM][D]: Message received: LOG - 1 STDERR follows.
Sat Aug 13 20:32:12 2011 [VMM][D]: Message received: LOG - 1 error: Failed to create domain from /srv/cloud/one/var//1/images/deployment.0
Sat Aug 13 20:32:12 2011 [VMM][D]: Message received: LOG - 1 error: unable to set user and group to '104:112' on '/srv/cloud/one/var//1/images/disk.0': Invalid argument
Sat Aug 13 20:32:12 2011 [VMM][D]: Message received: LOG - 1 ExitCode: 255

Next step is to add a new user called oneadmin (with ID 1001, the same that in the rest of computers). I prefer to set a password up for this user because later, you have to copy the frontend01's public key in this machine.

root@kvm01:~# mkdir -p /srv/cloud/one/var

root@kvm01:~# groupadd --gid 1001 cloud

root@kvm01:~# useradd --uid 1001 -s /bin/bash -d /srv/cloud/one -g cloud -G kvm,libvirtd oneadmin

root@kvm01:~# passwd oneadmin

root@kvm01:~# chown -R oneadmin:cloud /srv/cloud

root@kvm01:~# id oneadmin
uid=1001(oneadmin) gid=1001(cloud) groups=1001(cloud),112(kvm),113(libvirtd)

root@kvm01:~# cat /etc/fstab
...
storage01:/srv/cloud/one/var /srv/cloud/one/var      nfs4    _netdev,auto    0       0

root@kvm01:~# mount -a

In addition, the node must be synchronized with all the machines of the cluster.

root@kvm01:~# crontab -e
...
0 * * * * ntpdate pool.ntp.org

And finally, we have to copy the public key from frontend01, so that this computer can be remotely handled by OpenNebula.

oneadmin@frontend01:~$ ssh-copy-id -i .ssh/id_rsa.pub oneadmin@kvm01

So as to check the installation, we can execute the next order from frontend01.

oneadmin@frontend01:~$ lib/remotes/im/run_probes kvm kvm01
ARCH=x86_64 MODELNAME="Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz"

Now we are ready to use the new KVM node in our cloud computing architecture.

oneadmin@frontend01:~$ onehost create kvm01 im_kvm vmm_kvm tm_nfs

oneadmin@frontend01:~$ onehost list
ID NAME              CLUSTER  RVM   TCPU   FCPU   ACPU    TMEM    FMEM STAT
 0 kvm01             default    0    100    100    100      2G    1.9G   on

Tuning Zabbix to improve its performance (II)

2011-08-01T11:31:00.010+02:00

Let's continue with the last article about tuning Zabbix to improve its performance. First of all, I am going to set the suitable kernel parameters into the sysctl.conf file.

root@zbx01:~# cat /etc/sysctl.conf
...
# Maximum percentage of physical memory usage before going to swap
vm.swappiness = 10

# Number of open files for all processes
fs.file-max = 407020

# Minimum, default and maximum size of the send/receive buffer used by each TCP socket
net.ipv4.tcp_wmem = 8192        87380   16777216
net.ipv4.tcp_rmem = 8192        87380   16777216

# Maximum number of queued connection requests which have still not received an ACK (three-way handshake)
net.ipv4.tcp_max_syn_backlog = 2048

# Number of seconds to wait for a final FIN packet before the socket is forcibly closed
net.ipv4.tcp_fin_timeout = 25

# Number of seconds a connection needs to be idle before TCP begins sending out keep-alive probes
net.ipv4.tcp_keepalive_time = 1200

# Maximum TCP send window
net.core.wmem_max = 16777216

# Maximum TCP receive window
net.core.rmem_max = 16777216

# Maximum size in bytes of a message queue
kernel.msgmnb = 65536

# Maximum size for a message text
kernel.msgmax = 65536

# Maximum size in bytes for a shared memory segment
kernel.shmmax = 68719476736

# System wide maximum of shared memory pages
kernel.shmall = 4294967296

Then I am going to fit the values of MySQL by means of its configuration file. This part is really important if you want to achieve a good performance.

In order to adjust them, I have been following the status of the database throughout several weeks, by using tuning tools such as MySQL Performance Tuning Primer Script or MySQLTuner.

root@zbx01:~# cat /etc/mysql/my.cnf
...
# Size of the buffer used for index blocks
key_buffer = 16M

# Maximum size of one packet or any generated/intermediate string
max_allowed_packet = 16M

# Number of threads the server should cache for reuse
thread_cache_size = 64

# Maximum allowed number of simultaneous client connections
max_connections = 256

# Number of open tables for all threads
table_cache = 1024

# Number of table definitions that can be stored in the definition cache
table_definition_cache = 1024

# Do not cache results that are larger than this number of bytes
query_cache_limit = 16M

# Amount of memory allocated for caching query results
query_cache_size = 1024M

# Minimum size (in bytes) for blocks allocated by the query cache
query_cache_min_res_unit = 512

# 0: do not cache
# 1: cache all cacheable query results except for those that begin with SELECT SQL_NO_CACHE
# 2: cache results only for cacheable queries that begin with SELECT SQL_CACHE
query_cache_type = 1

# Slow queries are logged
log_slow_queries = /var/log/mysql/mysql-slow.log

# If a query takes longer than this value (seconds), the server logs the query
long_query_time = 5

# Queries that are expected to retrieve all rows are logged
log-queries-not-using-indexes

# Size in bytes of the memory buffer that InnoDB uses to cache data and indexes of its tables
innodb_buffer_pool_size = 4096M

With respect to MySQL, stand out that it is also important to defragment the query cache to enhance its utilization, by carrying out a "flush query cache" on the database. In my installation, I have seen that the optimum period is every hour.

root@zbx01:~# crontab -e
...
0 */1 * * * mysql -u root -pxxxxxx -e "flush query cache"

And finally, I have changed certain parameters from the Zabbix configuration file. The most important variable is related to the pre-forked pollers.

If this number is not enough, your Zabbix server will not be able to save all monitored data and you will find lack of many values. This is due to if the server runs out of sufficient processes to attend the requests, they will be ruled out and not registered.

root@zbx01:~# cat /etc/zabbix/zabbix_server.conf
...
# Number of pre-forked instances of pollers
StartPollers=96

# Shared memory size for storing hosts and items data
CacheSize=64M

# Shared memory size for storing history data
HistoryCacheSize=8M

# Shared memory size for storing trends data.
TrendCacheSize=8M

# Shared memory size for storing character, text or log history data
HistoryTextCacheSize=8M

Regarding Housekeeping, I have not modified any default parameter. In this way, the housekeeping procedure runs every hour and deletes all unnecessary values into the database.

If you note that your server does not work properly because it is using up lots of resources (CPU, memory, I/O) in this task, you will have to fit these options.

root@zbx01:~# cat /etc/zabbix/zabbix_server.conf
...
# Housekeeping is removing unnecessary information from history, alert, and alarm tables
# HousekeepingFrequency=1

# No more than MaxHousekeeperDelete rows will be deleted per one task in one housekeeping cycle
# MaxHousekeeperDelete=500

# Enable/disable housekeeping
# DisableHousekeeping=0

Tuning Zabbix to improve its performance (I)

2011-07-25T12:42:00.015+02:00

I am really looking forward to this article. I think it is going to be really useful for Zabbix administrators.

When you have to control a small group of machines, it is enough to install Zabbix (either from the repositories or the source code) and not modify any parameter. But when the number of monitored machines or items is very large, it is necessary to fit some values related to the operating system, the database and the Zabbix itself. Otherwise it is possible that your system acts up or the performance is not expected.

Bellow you can see the status of my Zabbix server at work (Zabbix 1.8.5 with MySQL 5.1, on Ubuntu 11.04 - 64 bits). I am monitoring around 430 devices, between servers and switches, and you can distinguish that the requeried server performance (new values per second) is really huge: 1687.

This configuration would not be possible with a Zabbix base installation. Also point out the hardware features of the server: 4 vCPUs (2.66 GHz), 8 GB RAM and 254 GB of storage.

First of all, we are going to take a look at several graphics of the server. Let's get started with the memory consumption during a typical day. The figure shows that the average available memory is around 1.73 GB and the system is not swapping.

Regarding the CPU, I have chosen a period of 6 hours so as to explain the concept of Housekeeping in Zabbix. As you can make out in the next chart, the normal use of CPU is about 20-25%, but each hour, there is a strong increment. This situation coincides with a rise of the Input/Output operations.

The Housekeeping is a task run by Zabbix which takes care of removing the unnecessary data of the history, alerts and alarms tables. Taking a look at the zabbix log, you can find out how many records are deleted from the database.

root@zbx01:~# egrep 'housekeeper|Deleted' /var/log/zabbix/zabbix_server.log
1599:20110719:230307.692 Executing housekeeper
1599:20110719:231127.392 Deleted 1522478 records from history and trends
1599:20110720:001127.393 Executing housekeeper
1599:20110720:001927.742 Deleted 1480673 records from history and trends
...

This procedure is configured by means of different parameters into the zabbix_server.conf file.

Through the load average graph, we can also appreciate this issue, where the load average (1 min) reaches maximum increases of 1.30.

And finally, the following graphic represents the status of the Zabbix cache during a week. Its values are rightly suited too.

In the next article, I will teach how to set up correctly the parameters related to the Linux kernel, MySQL and Zabbix.

Adding a VMware ESXi hypervisor to OpenNebula (III)

2011-07-17T19:54:00.025+02:00

This is the last article about Adding a VMware ESXi hypervisor to OpenNebula. In this posting, we are going to build a virtual machine up on the esxi01 node.

To begin with, I am going to download a virtual image from the Virtual Appliances Marketplace, specifically an Ubuntu Server 8.04 LTS distribution (I will drop it off into the tmp directory).

oneadmin@frontend01:/tmp/ubuntu-server-8.04.1-i386$ ls -lh
total 529M
-rw-rw-r-- 1 oneadmin cloud  269 2008-07-05 17:59 README-vmware-image.txt
-rw------- 1 oneadmin cloud 8.5K 2008-07-05 17:59 ubuntu-server-8.04.1-i386.nvram
-rw------- 1 oneadmin cloud 144M 2008-07-05 17:59 ubuntu-server-8.04.1-i386-s001.vmdk
-rw------- 1 oneadmin cloud 207M 2008-07-05 17:59 ubuntu-server-8.04.1-i386-s002.vmdk
-rw------- 1 oneadmin cloud 177M 2008-07-05 17:59 ubuntu-server-8.04.1-i386-s003.vmdk
-rw------- 1 oneadmin cloud 1.8M 2008-07-05 17:59 ubuntu-server-8.04.1-i386-s004.vmdk
-rw------- 1 oneadmin cloud  64K 2008-07-05 17:59 ubuntu-server-8.04.1-i386-s005.vmdk
-rw------- 1 oneadmin cloud  592 2008-07-05 17:59 ubuntu-server-8.04.1-i386.vmdk
-rw------- 1 oneadmin cloud    0 2008-07-05 17:59 ubuntu-server-8.04.1-i386.vmsd
-rwxr-xr-x 1 oneadmin cloud 1.1K 2008-07-05 17:59 ubuntu-server-8.04.1-i386.vmx

Then I am going to register the image in OpenNebula. For this purpose, it is necessary to define an image template.

oneadmin@frontend01:/tmp/ubuntu-server-8.04.1-i386$ cat ubuntu-server-8.04.img
NAME        = "Ubuntu Server 8.04"
DESCRIPTION = "Ubuntu Server 8.04 LTS (32 bits)"

oneadmin@frontend01:/tmp/ubuntu-server-8.04.1-i386$ onevmware register --disk-vmdk ubuntu-server-8.04.1-i386.vmdk --disk-flat ubuntu-server-8.04.1-i386-s001.vmdk,ubuntu-server-8.04.1-i386-s002.vmdk,ubuntu-server-8.04.1-i386-s003.vmdk,ubuntu-server-8.04.1-i386-s004.vmdk,ubuntu-server-8.04.1-i386-s005.vmdk ubuntu-server-8.04.img

What happens now? We have a virtual image ready to be used. This image has been stored into the images directory.

oneadmin@frontend01:~$ oneimage list
ID     USER                 NAME TYPE              REGTIME PUB PER STAT  #VMS
 0 oneadmin   Ubuntu Server 8.04   OS   Jul 02, 2011 10:34  No  No  rdy     0

oneadmin@frontend01:~$ ls -l var/images/0ffb8867916a29e279e5ac2374833faa84fe5193/
total 540740
-rw-rw---- 1 oneadmin cloud       592 2011-07-02 12:34 disk.vmdk
-rw-rw---- 1 oneadmin cloud 150339584 2011-07-02 12:34 ubuntu-server-8.04.1-i386-s001.vmdk
-rw-rw---- 1 oneadmin cloud 216268800 2011-07-02 12:35 ubuntu-server-8.04.1-i386-s002.vmdk
-rw-rw---- 1 oneadmin cloud 185204736 2011-07-02 12:36 ubuntu-server-8.04.1-i386-s003.vmdk
-rw-rw---- 1 oneadmin cloud   1835008 2011-07-02 12:37 ubuntu-server-8.04.1-i386-s004.vmdk
-rw-rw---- 1 oneadmin cloud     65536 2011-07-02 12:37 ubuntu-server-8.04.1-i386-s005.vmdk

What is the next step? Easy, to make up a virtual network so as to be able to connect our future virtual machine on it. In my example, I have created a simple ranged network called ESXi Network. You can review the different parameters by taking a look at the exposed link in this paragraph.

oneadmin@frontend01:~$ mkdir templates ; cd templates

oneadmin@frontend01:~/templates$ cat esxi.net
NAME            = "ESXi Network"
TYPE            = RANGED
PUBLIC          = NO
BRIDGE          = "VM Network"
NETWORK_ADDRESS = 192.168.1.160
NETWORK_SIZE    = 16
NETMASK         = 255.255.255.0
GATEWAY         = 192.168.1.1
DNS             = 194.30.0.1

oneadmin@frontend01:~/templates$ onevnet create esxi.net

oneadmin@frontend01:~/templates$ onevnet list
ID USER     NAME              TYPE BRIDGE P #LEASES
 0 oneadmin ESXi Network    Ranged VM Net N       0

And finally, we must just set up an instance template in order to declare the features of our virtual machine. Also point out that a virtual machine is known as instance as well.

oneadmin@frontend01:~/templates$ cat ubuntu-server01.vm
NAME   = "UbuntuServer-01"
CPU    = 1
MEMORY = 512

DISK   = [ IMAGE  = "Ubuntu Server 8.04",
           TARGET = hda ]

NIC    = [ NETWORK = "ESXi Network" ]

oneadmin@frontend01:~/templates$ onevm create ubuntu-server01.vm

oneadmin@frontend01:~/templates$ onevm list
ID     USER     NAME STAT CPU     MEM        HOSTNAME        TIME
 0 oneadmin UbuntuSe pend   0      0K                 00 00:00:07

We can see in the output of the preceding onevm list command that, the state of the instance is pend (pending), that is to say, it is waiting to be deployed on a hypervisor, in my case, esxi01. So let's go.

oneadmin@frontend01:~/templates$ onevm deploy 0 0

oneadmin@frontend01:~/templates$ onevm list
ID     USER     NAME STAT CPU     MEM        HOSTNAME        TIME
 0 oneadmin UbuntuSe prol   0      0K          esxi01 00 00:00:36

oneadmin@frontend01:~/templates$ onevm list
ID     USER     NAME STAT CPU     MEM        HOSTNAME        TIME
 0 oneadmin UbuntuSe boot   0      0K          esxi01 00 00:00:50

oneadmin@frontend01:~/templates$ onevm list
ID     USER     NAME STAT CPU     MEM        HOSTNAME        TIME
 0 oneadmin UbuntuSe runn   0      0K          esxi01 00 00:01:07

When we treat to deploy the virtual machine on the node, its first state is prol (prolog), then it reaches a boot state (booting), and lastly, runn (running), once the instance has been started up.

Adding a VMware ESXi hypervisor to OpenNebula (II)

2011-07-09T16:03:00.008+02:00

Let's go ahead with the development of the cloud computing infraestructure based on OpenNebula by adding a VMware ESXi hypervisor. In the previous article, we saw how to configure a VMware vSphere node and now, we are going to set up the needed part on frontend01.

In order to manage the VMware node from OpenNebula, it is necessary to install libvirt on frontend01, and on top of all that, this software must be compiled with the ESX support. In this way, you cannot use the corresponding package located in the Ubuntu repositories. I am going to utilize the 0.9.2 version of libvirt.

root@frontend01:~# aptitude install libgnutls-dev libdevmapper-dev libcurl4-gnutls-dev python-dev libnl-dev libapparmor-dev

root@frontend01:/tmp# wget http://libvirt.org/sources/libvirt-0.9.2.tar.gz

root@frontend01:/tmp# tar xvzf libvirt-0.9.2.tar.gz ; cd libvirt-0.9.2

root@frontend01:/tmp/libvirt-0.9.2# ./configure --with-esx --with-apparmor --sysconfdir=/etc --libdir=/usr/lib --sbindir=/usr/sbin --datarootdir=/usr/share --localstatedir=/var --libexecdir=/usr/lib/libvirt

root@frontend01:/tmp/libvirt-0.9.2# make ; make install

I have also compiled the source code with AppArmor support, so I have had to move the required files.

root@frontend01:/tmp/libvirt-0.9.2# mkdir -p /etc/apparmor.d/libvirt

root@frontend01:/tmp/libvirt-0.9.2# cp -a examples/apparmor/usr.* /etc/apparmor.d/
root@frontend01:/tmp/libvirt-0.9.2# cp -a examples/apparmor/TEMPLATE /etc/apparmor.d/libvirt/
root@frontend01:/tmp/libvirt-0.9.2# cp -a examples/apparmor/libvirt-qemu /etc/apparmor.d/abstractions/

root@frontend01:/tmp/libvirt-0.9.2# cat /etc/apparmor.d/usr.sbin.libvirtd
...
owner /srv/cloud/one/var/** rw,
}

root@frontend01:/tmp/libvirt-0.9.2# /etc/init.d/apparmor restart

Next step is to download and install the VMware Drivers Addon (version 2.2.0 in my case). This wrapper enables the communication between OpenNebula and VMware ESXi through libvirt. This operation must be effected by means of the oneadmin user.

When you run the install.sh script and if everything went well, you will get a message about a specific code that you have to add into the oned.conf file.

oneadmin@frontend01:/tmp$ wget http://dev.opennebula.org/attachments/download/350/vmware-2.2.0.tar.gz

oneadmin@frontend01:/tmp$ tar xvzf vmware-2.2.0.tar.gz ; cd vmware-2.2.0

oneadmin@frontend01:/tmp/vmware-2.2.0$ ./install.sh
VMWare Drivers Addon successfully installed

# After the installation, please add the following to your oned.conf file
# and restart OpenNebula to activate the VMware Drivers Addon

#-------------------------------------------------------------------------------
#  VMware Driver Addon Virtualization Driver Manager Configuration
#-------------------------------------------------------------------------------
VM_MAD = [
name       = "vmm_vmware",
executable = "one_vmm_sh",
arguments  = "vmware",
default    = "vmm_sh/vmm_sh_vmware.conf",
type       = "vmware" ]
#-------------------------------------------------------------------------------

#-------------------------------------------------------------------------------
#  VMware Driver Addon Information Driver Manager Configuration
#-------------------------------------------------------------------------------
IM_MAD = [
name       = "im_vmware",
executable = "one_im_sh",
arguments  = "vmware" ]
#-------------------------------------------------------------------------------

#-------------------------------------------------------------------------------
# VMware Driver Addon Transfer Manager Driver Configuration
#-------------------------------------------------------------------------------
TM_MAD = [
name       = "tm_vmware",
executable = "one_tm",
arguments  = "tm_vmware/tm_vmware.conf" ]
#-------------------------------------------------------------------------------

Due to there is a mistake in the install.sh script, you must execute manually the following two orders.

oneadmin@frontend01:/tmp/vmware-2.2.0$ mkdir -p $ONE_LOCATION/var/remotes/im/vmware.d && cp -r im/remotes/* $ONE_LOCATION/var/remotes/im/vmware.d

oneadmin@frontend01:/tmp/vmware-2.2.0$ mkdir -p $ONE_LOCATION/var/remotes/vmm/vmware && cp -r vmm/remotes/* $ONE_LOCATION/var/remotes/vmm/vmware

This problem causes that you cannot start the VMware node up and the log file will show the next error.

oneadmin@frontend01:~$ cat var/oned.log
...
[ONE][E]: syntax error, unexpected $end, expecting VARIABLE at line 2, columns 1:2

Before restarting OpenNebula, you must type the user and password used to access to esxi01 and include a line into the sudoers file, so that OpenNebula may properly set some permissions.

oneadmin@frontend01:~$ cat etc/vmwarerc
...
USERNAME      = "oneadmin"
PASSWORD      = "xxxxxx"

root@frontend01:~# cat /etc/sudoers
...
oneadmin ALL=NOPASSWD:/srv/cloud/one/share/hooks/fix_owner_perms.sh ""

root@frontend01:~# /etc/init.d/opennebula restart

So as to check the correct installation, run the following command. Thereby, you will be able to obtain the physical resources of the managed node.

oneadmin@frontend01:~$ lib/remotes/im/run_probes vmware esxi01
/srv/cloud/one/lib/ruby/vmwarelib.rb:26: warning: already initialized constant ONE_LOCATION
/srv/cloud/one/lib/ruby/vmwarelib.rb:32: warning: already initialized constant RUBY_LIB_LOCATION
HYPERVISOR=vmware TOTALCPU=100 CPUSPEED=3001 TOTALMEMORY=2096460 FREEMEMORY=1677168

Now if you want to add the VMware vSphere node configured to OpenNebula, you can execute the next order.

oneadmin@frontend01:~$ onehost create esxi01 im_vmware vmm_vmware tm_vmware

oneadmin@frontend01:~$ onehost list
ID NAME              CLUSTER  RVM   TCPU   FCPU   ACPU    TMEM    FMEM STAT
0  esxi01            default    0    100    100    100      2G    1.6G   on

Adding a VMware ESXi hypervisor to OpenNebula (I)

2011-07-02T14:56:00.040+02:00

This is the first article about how to add a VMware vSphere hypervisor to OpenNebula. In the two previous entries (I and II), I carried out the installation of OpenNebula on Ubuntu.

First of all, it is very important to underline that OpenNebula can only work with VMware vSphere hypervisor if it has got an evaluation license (60 days) or a standard, advanced, enterprise or enterprise plus license. If you opt for a free license, this mode lacks many remote commands and just supports the read-only API.

For instance, if you try to remotely define a virtual machine, you will get an error as the follows.

oneadmin@frontend01:~$ virsh -c esx://esxi01/?no_verify=1
Enter username for esxi01 [root]: oneadmin
Enter oneadmin's password for esxi01:
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
     'quit' to quit

virsh #  define /srv/cloud/one/var/0/deployment.0
error: Failed to define domain from /srv/cloud/one/var/0/deployment.0
error: internal error HTTP response code 500 for call to 'RegisterVM_Task'. Fault: ServerFaultCode - fault.RestrictedVersion.summary

After this explanation, let's get started by configuring the ntp daemon so that this hypervisor will be synchronized with the rest of nodes which make up the cloud computing infraestructure. Remember that the VMware vSphere ESXi version which I am going to use for my tests is the 4.1.

So as to set up a NTP server (in my case, a public NTP server such as pool.ntp.org), we must go to Configuration, Time Configuration and press the Properties link.

Next step is to aggregate a new group named cloud, with ID 1001, by clicking with the right button of the mouse on Local Users & Groups, Groups tab and pressing the Add command.

Then you have to create a new user named oneadmin, by making the same operation, but this time on the Users tab section. This user have to belong to the cloud group and also have the 1001 ID.

In addition, this new user must have full privilegies on the node. Therefore, we have to click with the right button of the mouse over our hypervisor, choose the Add Permission option and select the fields that you may see in the following figure.

And finally, we have to mount the shared storage exported by the storage01 machine. In order to realize this task, you must go to Configuration, Storage and press on the Add Storage link. In this way, a wizard will be popped up which will allow us to mount the remote storage.

In the first screen, you must pick out the Network File System storage type and then, fulfill the fields showed in the preceding image. Note that we are just importing the /srv/cloud/one/var directory, because in this folder will be stored the virtual machines. Also say that I have called the datastore images.

Resolving and installing dependences with gdebi

2011-06-28T17:07:00.007+02:00

It may seem incredible but after many years using Linux, I have found out a great application in order to install deb packages, and automatically, their dependences: gdebi.

I have always had to install something, by general I have made it either from the corresponding repository or from the source code (by manually working out the dependences in this last case).

But what happens if we want to install directly a deb package? Typically we are going to employ the dpkg command.

javi@kubuntu:~$ sudo dpkg -i package.deb

If the package to be installed needs some dependence, the dpkg output shows it and we have to manually set it up. But so as to work around this task, gdebi can carry out all these necessary actions: to resolve the dependences and install the final package.

javi@kubuntu:~$ sudo gdebi package.deb

And finally, also say that this tool is available for Debian systems and their derivatives (Ubuntu, Kubuntu, Knoppix, etc.).

OpenNebula installation on Ubuntu (II)

2011-06-19T13:34:00.010+02:00

Once we have made up the shared storage, the database and the requeried users, we are going to carry on the article about OpenNebula installation on Ubuntu by installing the dependences of OpenNebula.

root@frontend01:~# aptitude install build-essential ruby libxmlrpc-c3-dev scons libopenssl-ruby libssl-dev flex bison ruby-dev rake rubygems libxml-parser-ruby libxslt1-dev libnokogiri-ruby libsqlite3-dev

Now we are ready to download the source code of OpenNebula, compile it (with the MySQL option activated) and install it into the /srv/cloud/one directory.

root@frontend01:~# su - oneadmin ; cd /tmp

oneadmin@frontend01:/tmp$ wget http://dev.opennebula.org/attachments/download/395/opennebula-2.2.1.tar.gz

oneadmin@frontend01:/tmp$ tar xvzf opennebula-2.2.1.tar.gz ; cd opennebula-2.2.1

oneadmin@frontend01:/tmp/opennebula-2.2.1$ cat src/vmm/LibVirtDriverVMware.cc
...
        if ( emulator != "vmware" )
        {
                file << "\t\t\t<driver name='";

                if ( !driver.empty() )
                {
                        file << driver << "'/>" << endl;
                }
                else
                {
                        file << default_driver << "'/>" << endl;
                }
        }
...

oneadmin@frontend01:/tmp/opennebula-2.2.1$ scons mysql=yes parsers=yes

oneadmin@frontend01:/tmp/opennebula-2.2.1$ ./install.sh -d /srv/cloud/one

Before compiling OpenNebula, it is necessary to fix a severe mistake into a file from the source code (LibVirtDriverVMware.cc). This bug does not allow to deploy virtual machines on VMware ESXi, since when OpenNebula makes the deployment templates includes the raw format as the default DRIVER.

oneadmin@frontend01:~/templates$ virsh -c esx://esxi01/?no_verify=1
Enter username for esxi01 [root]: oneadmin
Enter oneadmin's password for esxi01:
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # define /srv/cloud/one/var/0/deployment.0
error: Failed to define domain from /srv/cloud/one/var/0/deployment.0
error: internal error Unknown driver name 'raw'

The last step before starting OpenNebula, is to set up (within the oned.conf file) the MySQL connection parameters (remember that in this case, you have to comment the line related to SQLite).

oneadmin@frontend01:/tmp/opennebula-2.2.1$ cat /srv/cloud/one/etc/oned.conf
...
# DB = [ backend = "sqlite" ]

DB = [ backend = "mysql",
   server  = "localhost",
   port    = 0,
   user    = "oneadmin",
   passwd  = "xxxxxx",
   db_name = "opennebula" ]
...

By means of the one script (situated in $ONE_LOCATION/bin), we can run and stop the OpenNebula daemon (oned) and the scheduler (mm_sched). Also say that the log files are located in $ONE_LOCATION/var.

oneadmin@frontend01:~$ one start

oneadmin@frontend01:~$ one stop

And finally, mention too that if we want that the operating system to start automatically OpenNebula during the boot, we must create a LSB init script for this purpose.

root@frontend01:~# cat /etc/init.d/opennebula
#!/bin/bash

### BEGIN INIT INFO
# Provides:          OpenNebula
# Required-Start:    $remote_fs $syslog $network
# Required-Stop:     $remote_fs $syslog $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start daemon at boot time
# Description:       Enable service provided by daemon.
### END INIT INFO

export ONE_LOCATION=/srv/cloud/one
export ONE_AUTH=$ONE_LOCATION/.one/one_auth
export ONE_XMLRPC=http://localhost:2633/RPC2
export PATH=$ONE_LOCATION/bin:$PATH

RETVAL=0

start()
{
     su oneadmin -s /bin/bash -c '$ONE_LOCATION/bin/one start' ; RETVAL=$?
     return $RETVAL
}

stop()
{
     su oneadmin -s /bin/bash -c '$ONE_LOCATION/bin/one stop' ; RETVAL=$?
}

case "$1" in
     start)
             sleep 5
             start
             ;;
     stop)
             stop
             ;;
     restart)
             stop
             start
             ;;
     *)
             echo $"Usage: service opennebula {start stop restart}"
esac

exit $RETVAL


root@frontend01:~# chmod +x /etc/init.d/opennebula

root@frontend01:~# update-rc.d opennebula start 90 2 3 4 5 . stop 10 0 1 6 .

OpenNebula installation on Ubuntu (I)

2011-06-14T13:21:00.036+02:00

After presenting the OpenNebula cloud computing architecture, let's start indicating the versions which will be used.

All Linux machines (frontend01, storage01 and kvm01) have an Ubuntu Server 11.04 (64 bits) installed on them, and esxi01, a VMware ESXi 4.1 hypervisor. The OpenNebula version employed for the tests is 2.2.1 and it will be compiled and installed directly from its source code.

First of all, we are going to set up the shared storage on storage01, by means of NFS protocol. We need too an OpenNebula administrator user (oneadmin) which must be added on all machines (for this purpose, its UID and GID have to be the same for all them - 1001 in my case). The group of this user will be cloud.

root@storage01:~# mkdir -p /srv/cloud/one

root@storage01:~# groupadd --gid 1001 cloud
root@storage01:~# useradd --uid 1001 -g cloud -s /bin/bash -d /srv/cloud/one oneadmin
root@storage01:~# chown -R oneadmin:cloud /srv/cloud

root@storage01:~# aptitude install nfs-kernel-server

root@storage01:~# cat /etc/exports
/srv/cloud      192.168.1.0/255.255.255.0(rw,anonuid=1001,anongid=1001)

root@storage01:~# /etc/init.d/nfs-kernel-server restart

Now I may export the cloud directory to any machine belonging to the subnet (192.168.1.0/24).

Afterwards we must mount that shared on frontend01.

root@frontend01:~# aptitude install nfs-common ; modprobe nfs

root@frontend01:~# mkdir -p /srv/cloud

root@frontend01:~# cat /etc/fstab
...
storage01:/srv/cloud /srv/cloud      nfs4    _netdev,auto    0       0

root@frontend01:~# mount -a

Next step is to create an OpenNebula administrator user on the system.

root@frontend01:~# groupadd cloud

root@frontend01:~# useradd -s /bin/bash -d /srv/cloud/one -g cloud oneadmin

root@frontend01:~# id oneadmin
uid=1001(oneadmin) gid=1001(cloud) groups=1001(cloud)

Any OpenNebula account that we add to the system, must have the following environment variables established.

root@frontend01:~# su - oneadmin

oneadmin@frontend01:~$ cat .bashrc
# ~/.bashrc

if [ -f /etc/bash.bashrc ]; then
  . /etc/bash.bashrc
fi

export ONE_AUTH=$HOME/.one/one_auth
export ONE_LOCATION=/srv/cloud/one
export ONE_XMLRPC=http://localhost:2633/RPC2
export PATH=$ONE_LOCATION/bin:$PATH

oneadmin@frontend01:~$ cat .profile
# ~/.profile

if [ -n "$BASH_VERSION" ]; then
  if [ -f "$HOME/.bashrc" ]; then
          . "$HOME/.bashrc"
  fi
fi

OpenNebula is started by using the ONE_AUTH information.

oneadmin@frontend01:~$ mkdir .one

oneadmin@frontend01:~$ cat .one/one_auth
oneadmin:xxxxxx

We need to generate ssh keys for oneadmin user, in order to be able to connect with the rest of servers without typing a password. By means of the hushlogin file, we avoid the SSH welcome banner, and through the StrictHostKeyChecking directive, the SSH client not to ask about adding hosts to known_hosts file.

oneadmin@frontend01:~$ ssh-keygen

oneadmin@frontend01:~$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys

oneadmin@frontend01:~$ cat .ssh/config
Host *
    StrictHostKeyChecking no

oneadmin@frontend01:~$ touch .hushlogin

And finally, we are going to install MySQL (and its necessary dependences for the OpenNebula compilation) and set up a database called opennebula. It will be manage by OpenNebula so as to store its data.

root@frontend01:~# aptitude install mysql-server libmysql++-dev libxml2-dev

root@frontend01:~# mysql_secure_installation

root@frontend01:~# mysql -u root -p
...
mysql> GRANT ALL PRIVILEGES ON opennebula.* TO 'oneadmin' IDENTIFIED BY 'xxxxxx';
Query OK, 0 rows affected (0.00 sec)

Also say that as in any cluster, all nodes have to be synchronized.

root@frontend01:~# crontab -e
...
0 * * * * ntpdate pool.ntp.org

root@storage01:~# crontab -e
...
0 * * * * ntpdate pool.ntp.org

Cloud computing with OpenNebula

2011-06-07T23:20:00.003+02:00

OpenNebula is an IaaS (Infraestructure as a Service) open source solution which allows to build private, public and hybrid clouds. It has been designed to be able to integrate with any kind of network or storage system and supports the main types of hypervisors: KVM, VMware vSphere (ESXi) and Xen.

Next figure shows a typical schema of an OpenNebula infrastructure, which I am going to develop in future articles.

The server named fronted01 runs OpenNebula and the cluster services. The principal components of OpenNebula are the daemon (manage the life cycle of virtual machines, network, storage and hypervisors), the scheduler (manage the deployment of virtual machines) and the drivers (manage the hypervisor interfaces - VMM -, monitoring - IM - and virtual machines transfers - TM -).

OpenNebula needs too a database to save the information. We have two options: SQLite and MySQL. In my architecture, I will use MySQL and it will be installed on frontend01.

With respect to the storage, OpenNebula works with three possibilities: shared - NFS (there is a shared data area accessible by OpenNebula server and computational nodes), non-shared - SSH (there is no shared area - live migrations cannot be used) and LVM (there must be a block device available in all nodes).

In the articles that I will write about it, I will configure a NFS shared into the storage01 server. It is normal as well to find some architecture where the storage is established inside the front-end. Also point out that the storage is used for keeping the virtual images and machines.

As in any classical IaaS solution, we require the computing nodes (also known as worker nodes), which supply the raw computing power and where the virtual machines are run. In this example, I will employ two hypervisors: KVM (kvm01) and VMware vSphere (esxi01). OpenNebula must be able to start, control and monitor the virtual machines. The communication between OpenNebula and nodes will be carry out through the drivers previously configured.

We can appreciate that OpenNebula is a fully scalable system, since we can add more computational nodes or storage servers based on our needs.

Other features are portability and interoperability, due to we may utilize most of the existing hardware to set up the clusters.

And finally, we face an open and standard architecture model, and besides, it can operate with other public clouds such as Amazon.

Zabbix server installation on Ubuntu (II)

2011-06-01T13:14:00.014+02:00

We are going to conclude the last part of the Zabbix server installation on Ubuntu by setting up a new Apache web site for Zabbix.

root@ubuntu-server:~# cat /etc/apache2/sites-available/zabbix
<VirtualHost *:80>
Alias /zabbix /usr/share/zabbix
ErrorLog /var/log/apache2/zabbix-error.log
CustomLog /var/log/apache2/zabbix-access.log common
</VirtualHost>


root@ubuntu-server:~# a2dissite default

root@ubuntu-server:~# a2ensite zabbix

Besides it is also necessary to modify the PHP configuration file for adjusting it with the Zabbix requirements.

root@ubuntu-server:~# cat /etc/php5/apache2/php.ini
...
memory_limit = 256M

post_max_size = 32M

upload_max_filesize = 16M

max_execution_time = 600

max_input_time = 600

date.timezone = Europe/Madrid

Finally, we have to open a web browser, point to the Zabbix URL (http://ubuntu-server/zabbix in my case) and fulfill the wizard. In the first screen, Zabbix checks the pre-requisites and warns us if something is wrong.

In the fourth step (Configure DB connection), we have to enter the configuration parameters for the database connection.

At the end of the wizard, we must download the Zabbix PHP configuration file (zabbix.conf.php) by clicking on the Save configuration file button.

Then we have to copy that file into the /usr/share/zabbix/conf directory and fix it the suitable permissions.

root@ubuntu-server:~# chmod 600 /usr/share/zabbix/conf/zabbix.conf.php

root@ubuntu-server:~# chown www-data:www-data /usr/share/zabbix/conf/zabbix.conf.php

Taking snapshots on KVM with LVM

2011-05-22T17:50:00.021+02:00

In one article, we already saw how to take snapshots on KVM with libvirt. In this post, I am presenting another possible alternative to libvirt and it is by means of LVM (Logical Volume Management).

The idea is to create the virtual machine on a LV (Logical Volume) and afterwards, by using a feature called snapshot LV, to get an exact copy of that volume.

For my tests, I will utilize Kubuntu 11.04 (64 bits). First of all, I am going to set a LV so as to make up a virtual machine on it. Remember that if you want to set up a LV on a partition, this one must be marked as Linux LVM.

javi@kubuntu:~$ sudo fdisk -l
...
/dev/sdb1            7945        8992     8417280   8e  Linux LVM

In order to create a LV on a partition, we can follow the next steps. At the end, we will format the volume as ext4 (or another kind of format that you prefer).

javi@kubuntu:~$ pvcreate /dev/sdb1

javi@kubuntu:~$ vgcreate VolGroup00 /dev/sdb1

javi@kubuntu:~$ sudo lvcreate -n LogVol00 -L 2G VolGroup00

javi@kubuntu:~$ sudo mkfs.ext4 /dev/VolGroup00/LogVol00

The following figure shows a stage of the Virtual Machine Manager wizard, specifically the part where you have to pick the storage out. I have choosen the previous LV created. On this LV, I will make a new virtual machine (UbuntuServer_10.10, 2 GB virtual hard disk) which will be used before for the snapshotting.

At this point, we are ready to take a snapshot through LVM. Really easy.

javi@kubuntu:~$ sudo lvcreate -n UbuntuServer_10.10-22052011 -L 512M -s /dev/VolGroup00/LogVol00

By taking the snapshot with the preceding command, a new LV is created on the same VG (Volume Group) that LogVol01, that is to say, VolGroup00 in my case. For this reason, we must make sure that there is enough free space on the VG.

You have to take into account how a snapshot works. When you take a snapshot, the original virtual disk (LogVol01) is frozen and all changes are stored into the snapshot (UbuntuServer_10.10-22052011). So as to demonstrate it, we are going to display the LVs state once the snapshot has been taken.

javi@kubuntu:~$ sudo lvdisplay
--- Logical volume ---
LV Name                /dev/VolGroup00/LogVol00
VG Name                VolGroup00
LV UUID                Ag34Yq-990o-eyei-ClnF-OjCA-QltD-BrI39w
LV Write Access        read/write
LV snapshot status     source of
      /dev/VolGroup00/UbuntuServer_10.10-22052011 [active]
LV Status              available
# open                 0
LV Size                2,00 GiB
Current LE             512
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           252:0

--- Logical volume ---
LV Name                /dev/VolGroup00/UbuntuServer_10.10-22052011
VG Name                VolGroup00
LV UUID                PSpJH0-ANEu-HW4W-YnLj-00Ta-g2Gz-WedRqi
LV Write Access        read/write
LV snapshot status     active destination for /dev/VolGroup00/LogVol00
LV Status              available
# open                 0
LV Size                2,00 GiB
Current LE             512
COW-table size         512,00 MiB
COW-table LE           128
Allocated to snapshot  0,00%
Snapshot chunk size    4,00 KiB
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           252:1

The previous output indicates us that we can save up to 512 MB of data for our snapshot (COW-table size), and 0% of that space is being used (Allocated to snapshot). Then we are going to make a little change inside the virtual machine, for example to create a new file of 256 MB.

javi@ubuntu-server:~$ dd if=/dev/zero of=file bs=1M count=256

Now if we take a look at the LV state again, we can see that the 50% of the snapshot has already been allocated.

javi@kubuntu:~$ sudo lvdisplay /dev/VolGroup00/UbuntuServer_10.10-22052011
--- Logical volume ---
LV Name                /dev/VolGroup00/UbuntuServer_10.10-22052011
VG Name                VolGroup00
LV UUID                PSpJH0-ANEu-HW4W-YnLj-00Ta-g2Gz-WedRqi
LV Write Access        read/write
LV snapshot status     active destination for /dev/VolGroup00/LogVol00
LV Status              available
# open                 0
LV Size                2,00 GiB
Current LE             512
COW-table size         512,00 MiB
COW-table LE           128
Allocated to snapshot  50,45%
Snapshot chunk size    4,00 KiB
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           252:1

And finally, if we want to bring back a specific snapshot, we must execute the following sentence.

javi@kubuntu:~$ sudo lvconvert --merge VolGroup00/UbuntuServer_10.10-22052011

Zabbix server installation on Ubuntu (I)

2011-05-14T20:28:00.010+02:00

Some time ago I wrote an article about the installation of Zabbix server from its source code on CentOS.

Now I wanted to explain how to install it but this time, on Ubuntu. For my tests, I am going to use an Ubuntu Server 11.04 (64 bits) and Zabbix 1.8.5. For this infraestructure, we need MySQL and Apache. Let's start installing the necessary packages.

root@ubuntu-server:~# aptitude install build-essential apache2 mysql-server libmysqld-dev snmpd libsnmp-dev php5 php5-mysql php5-gd libcurl4-openssl-dev libiksemel-dev libopenipmi-dev libssh2-1-dev fping

root@ubuntu-server:~# mysql_secure_installation

As well it is important to run the mysql_secure_installation script in order to remove the anonymous user and the test database.

First of all we have to set up the database which will be used by Zabbix.

root@ubuntu-server:~# mysql -u root -p
...
mysql> CREATE DATABASE zabbix;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER 'zabbix'@'localhost' IDENTIFIED BY 'xxxxxx';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON zabbix.* TO 'zabbix'@'localhost';
Query OK, 0 rows affected (0.00 sec)

Once we have downloaded the Zabbix source code and decompressed it, we have just to compile and install it. If we want to have the Zabbix client, we must mark the --enable-agent parameter.

root@ubuntu-server:~/zabbix-1.8.5# ./configure --enable-agent  --enable-ipv6  --enable-server --with-mysql --with-libcurl --with-net-snmp --with-jabber --with-ssh2 --with-openipmi

root@ubuntu-server:~/zabbix-1.8.5# make ; make install

Next step is to create the needed directories and copy the configuration files into them. We must also add a new user (zabbix) to the system and dump the data and schemas within the Zabbix database.

root@ubuntu-server:~/zabbix-1.8.5# mkdir -p /etc/zabbix/alert.d /etc/zabbix/externalscripts /var/log/zabbix /var/run/zabbix /usr/share/zabbix

root@ubuntu-server:~/zabbix-1.8.5# useradd -r -d /var/run/zabbix -s /sbin/nologin zabbix

root@ubuntu-server:~/zabbix-1.8.5# cp -a misc/conf/zabbix_server.conf misc/conf/zabbix_agentd.conf /etc/zabbix/

root@ubuntu-server:~/zabbix-1.8.5# cp -r frontends/php/* /usr/share/zabbix

root@ubuntu-server:~/zabbix-1.8.5# chown zabbix:zabbix /var/run/zabbix /var/log/zabbix

root@ubuntu-server:~/zabbix-1.8.5# (echo "USE zabbix;" ; cat create/schema/mysql.sql ; cat create/data/data.sql ; cat create/data/images_mysql.sql) | mysql -h 127.0.0.1 -u zabbix --password=xxxxxx

Below we can see the minimum setting for both the server and client.

root@ubuntu-server:~# cat /etc/zabbix/zabbix_server.conf
...
# Zabbix server log file
LogFile=/var/log/zabbix/zabbix_server.log

# Zabbix server PID file
PidFile=/var/run/zabbix/zabbix_server.pid

# Zabbix database user and password
DBUser=zabbix
DBPassword=xxxxxx

# Location of alert scripts
AlertScriptsPath=/etc/zabbix/alert.d/

# Location of external scripts
ExternalScripts=/etc/zabbix/externalscripts


root@ubuntu-server:~# cat /etc/zabbix/zabbix_agentd.conf
...
# Zabbix client PID file
PidFile=/var/run/zabbix/zabbix_agentd.pid

# Zabbix client log file
LogFile=/var/log/zabbix/zabbix_agentd.log

# Allow remote commands from zabbix server
EnableRemoteCommands=1

# Maximum time for processing
Timeout=10

# System hostname
Hostname=ubuntu-server

# Zabbix server IP
Server=::ffff:127.0.0.1


root@ubuntu-server:~# chmod 600 /etc/zabbix/zabbix_server.conf

So as to be able to automatically start and stop the Zabbix agent and server, we have to create an Upstart file for this task. The Zabbix source code already provides the suitable script for Upstart, but I prefer to employ my own files (then you can see them - I have set some dependences which I consider important).

root@ubuntu-server:~# cat /etc/init/zabbix-server.conf
# Start zabbix server

pre-start script
if [ ! -d /var/run/zabbix ]; then
     mkdir -p /var/run/zabbix
     chown zabbix:zabbix /var/run/zabbix
fi
end script

start on started mysql
stop on stopping mysql
respawn
expect daemon
exec /usr/local/sbin/zabbix_server


root@ubuntu-server:~# cat /etc/init/zabbix-agent.conf
# Start zabbix agent

pre-start script
if [ ! -d /var/run/zabbix ]; then
     mkdir -p /var/run/zabbix
     chown zabbix:zabbix /var/run/zabbix
fi
end script

start on filesystem
stop on starting shutdown
respawn
expect daemon
exec /usr/local/sbin/zabbix_agentd

Now we can end the part of the Zabbix binary installation by registering the services and booting the processes up.

root@ubuntu-server:~# echo "zabbix-agent    10050/tcp  Zabbix Agent"   >> /etc/services
root@ubuntu-server:~# echo "zabbix-agent    10050/udp  Zabbix Agent"   >> /etc/services
root@ubuntu-server:~# echo "zabbix-trapper  10051/tcp  Zabbix Trapper" >> /etc/services
root@ubuntu-server:~# echo "zabbix-trapper  10051/udp  Zabbix Trapper" >> /etc/services


root@ubuntu-server:~# start zabbix-server

root@ubuntu-server:~# start zabbix-agent

Looking for web security breaches with Skipfish (II)

2011-05-08T13:28:00.010+02:00

I am going to finish my article about looking for web security breaches with Skipfish. Once we have got a global sight of skipfish, I will run a test against a default MediaWiki installation.

First, I must create a dictionary although it will not be used in this test. One interesting option that I have chosen is -I, in order to only follow those URLs which match the string associated with the parameter.

javi@ubuntu-server:~/skipfish-1.86b$ cp -a dictionaries/complete.wl dictionary.wl

javi@ubuntu-server:~/skipfish-1.86b$ ./skipfish -W /dev/null -I http://192.168.122.104/mediawiki -o mediawiki_dir http://192.168.122.104/mediawiki

If you do not set this option and skipfish figures out more sites, it will scan them as well. In case you want to shut out a specific URL, you must establish it by means of the -X parameter.

During the crawling, skipfish shows information in real time about its analysis.

skipfish version 1.86b by <lcamtuf@google.com>                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                 
- 192.168.122.104 -                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                 
Scan statistics:                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                 
Scan time : 0:59:38.542                                                                                                                                                                                                                                           
HTTP requests : 34669 (10.4/s), 100769 kB in, 12487 kB out (31.6 kB/s)                                                                                                                                                                                                
Compression : 77967 kB in, 255451 kB out (53.2% gain)                                                                                                                                                                                                               
HTTP faults : 0 net errors, 0 proto errors, 0 retried, 0 drops                                                                                                                                                                                                      
TCP handshakes : 351 total (152.0 req/conn)                                                                                                                                                                                                                            
TCP faults : 0 failures, 0 timeouts, 3 purged                                                                                                                                                                                                                      
External links : 202 skipped                                                                                                                                                                                                                                           
Reqs pending : 18692                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                 
Database statistics:                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                 
  Pivots : 880 total, 462 done (52.50%)                                                                                                                                                                                                                          
In progress : 34 pending, 145 init, 221 attacks, 18 dict                                                                                                                                                                                                            
Missing nodes : 4 spotted                                                                                                                                                                                                                                             
Node types : 1 serv, 186 dir, 544 file, 16 pinfo, 76 unkn, 57 par, 0 val                                                                                                                                                                                           
Issues found : 11 info, 75 warn, 57 low, 0 medium, 128 high impact                                                                                                                                                                                                   
Dict size : 263 words (263 new), 4 extensions, 256 candidates

At the end of the process, skipfish will dump all the data collected within the mediawiki_dir directory (defined by the -o option), that in turn contains an HTML file (index.html) which allows to view the report generated.

In the previous outcome, skipfish has only found out severe problems related to HTTP PUTs accepted.

So as to be able to understand the results offered by skipfish and if you do not have deep knowledge about web security (like me), you might take a look at the Browser Security Handbook, written and maintained by the same author who is developing skipfish.

Other interesting parameter is for example -A, used for passing HTTP authentication credentials.

And finally, also point out that you can tune skipfish in networking or crawling scopes, through different options which allow to set up for instance parameters related to TCP connections or the depth of the analysis. For getting more information you can check the project documentation.

Kubuntu 11.04 Natty Narwhal

2011-05-03T23:00:00.004+02:00

Here is the last version of Kubuntu, 11.04, also known as Natty Narwhal.

This release comes with important features such as a new kernel (2.6.38), the last KDE stable version (4.6.2), OpenOffice.org has been replaced by LibreOffice 3.3.2 (it is about time!), Upstart (0.9.7), LVM2 2.02.66 (very important for taking snapshots with LVM), Firefox 4.0.1 and so on.

It is significant to mention the new kernel, since fixes the Big Kernel Lock problem. Also point out the new Samba filesharing system, which allows easyly to share a directory in Dolphin, by configuring it through the Properties option.

My first impressions are very good, we have got a hardy and useful system, well worked out and really light.

Looking for web security breaches with Skipfish (I)

2011-04-24T21:40:00.008+02:00

When we have to secure a system, it is very important to employ all the security tools we know, in order to protect it all the best we can. Skipfish is one of those applications which always has to be near.

It is a web security scanner, developed in C, which allows to discover lots of security holes on a web site by performing several kinds of tests:

High risk: SQL / PHP / XML / shell script injections, etc.

Medium risk: XSS (Cross-Site Scripting), CSS attacks, MIME type problems, etc.

Low risk: indexed directories, certificate problems, HTTP credentials, etc.

Due to it is written in pure C, it can reach an high performance: around 500 requests per second against targets located on Internet, about 2000 on local networks and more than 7000 on local hosts. Skipfish generates a sitemap with all the discovered paths, a summary of the document types and a set of security breaches.

I am going to try skipfish (1.86b version) out on an Ubuntu Server 10.10, by crawling a default MediaWiki installation. So as to install skipfish, we must fulfill some dependences on our system and then, compile it.

javi@ubuntu-server:~$ sudo aptitude install build-essential libssl-dev libidn11-dev

javi@ubuntu-server:~$ wget http://skipfish.googlecode.com/files/skipfish-1.86b.tgz

javi@ubuntu-server:~/skipfish-1.86b$ tar xvzf skipfish-1.86b.tgz

javi@ubuntu-server:~/skipfish-1.86b$ make

By typing the -h argument, we can take a look at all the available options by skipfish.

javi@ubuntu-server:~/skipfish-1.86b$ ./skipfish -h
skipfish version 1.86b by <lcamtuf@google.com>
Usage: ./skipfish [ options ... ] -o output_dir start_url [ start_url2 ... ]

Authentication and access options:

-A user:pass   - use specified HTTP authentication credentials
-F host=IP     - pretend that 'host' resolves to 'IP'
-C name=val    - append a custom cookie to all requests
-H name=val    - append a custom HTTP header to all requests
-b (i|f|p)     - use headers consistent with MSIE / Firefox / iPhone
-N             - do not accept any new cookies

Crawl scope options:

-d max_depth   - maximum crawl tree depth (16)
-c max_child   - maximum children to index per node (512)
-x max_desc    - maximum descendants to index per branch (8192)
-r r_limit     - max total number of requests to send (100000000)
-p crawl%      - node and link crawl probability (100%)
-q hex         - repeat probabilistic scan with given seed
-I string      - only follow URLs matching 'string'
-X string      - exclude URLs matching 'string'
-K string      - do not fuzz parameters named 'string'
-D domain      - crawl cross-site links to another domain
-B domain      - trust, but do not crawl, another domain
-Z             - do not descend into 5xx locations
-O             - do not submit any forms
-P             - do not parse HTML, etc, to find new links

Reporting options:

-o dir         - write output to specified directory (required)
-M             - log warnings about mixed content / non-SSL passwords
-E             - log all HTTP/1.0 / HTTP/1.1 caching intent mismatches
-U             - log all external URLs and e-mails seen
-Q             - completely suppress duplicate nodes in reports
-u             - be quiet, disable realtime progress stats

Dictionary management options:

-W wordlist    - load an alternative wordlist (skipfish.wl)
-L             - do not auto-learn new keywords for the site
-V             - do not update wordlist based on scan results
-Y             - do not fuzz extensions in directory brute-force
-R age         - purge words hit more than 'age' scans ago
-T name=val    - add new form auto-fill rule
-G max_guess   - maximum number of keyword guesses to keep (256)

Performance settings:

-g max_conn    - max simultaneous TCP connections, global (40)
-m host_conn   - max simultaneous connections, per target IP (10)
-f max_fail    - max number of consecutive HTTP errors (100)
-t req_tmout   - total request response timeout (20 s)
-w rw_tmout    - individual network I/O timeout (10 s)
-i idle_tmout  - timeout on idle HTTP connections (10 s)
-s s_limit     - response size limit (200000 B)
-e             - do not keep binary responses for reporting

Basically we can see that the skipfish command is made up by a set of options, an output directory for saving the results, and a series of URLs to be analyzed.

We have to take into account when skipfish is working out, it just tests those found links, but we also have the possibility to probe other URLs by means of a brute-force attack, by mixing names (index, doc, etc.) and extensions (pdf, bat, etc.). For this purpose, skipfish affords four dictionaries.

javi@ubuntu-server:~/skipfish-1.86b$ ls dictionaries/*.wl                                                                                                                                                                                                                     
dictionaries/complete.wl  dictionaries/extensions-only.wl  dictionaries/medium.wl  dictionaries/minimal.wl

By default, skipfish will treat to use a dictionary named skipfish.wl and situated on the work directory. Therefore either we can copy one of these dictionaries into the work directory under this name, or use the -W option to define the wordlist path or on the contrary, not to use a dictionary.

As the application is crawling, adds new words within the dictionary.

Benchmarking with Phoronix Test Suite (II)

2011-04-17T12:48:00.009+02:00

This is the second and final part of the article Benchmarking with Phoronix Test Suite.

If we want to carry out the test or suite with the default options, we must aggregate the default-run parameter. Also mention that we can type multiple tests or suites with the run or default-run order.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite run iozone

Phoronix Test Suite v3.0.1
IOzone Test Configuration


Record Size:

1: 4Kb
2: 64Kb
3: 1MB
4: Test All Options

Enter Your Choice: 2

File Size:

1: 512MB
2: 2GB
3: 4GB
4: 8GB
5: Test All Options

Enter Your Choice: 1

Disk Test:

1: Write Performance
2: Read Performance
3: Test All Options

Enter Your Choice: 3
Would you like to save these test results (Y/n): Y
Enter a name to save these results: ubuntu-server

Current Test Identifiers:
- apache

Enter a unique name for this test run: iozone
...

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite default-run iozone

Where are the results and the test environments stored on the filesystem?

root@ubuntu-server:~# tree -d .phoronix-test-suite/test-results/
.phoronix-test-suite/test-results/
├── pts-results-viewer
└── ubuntu-server
├── result-graphs
└── system-logs
├── apache
└── iozone

root@ubuntu-server:~# tree -d .phoronix-test-suite/installed-tests/
.phoronix-test-suite/installed-tests/
└── pts
├── apache-1.3.0
└── iozone-1.7.0

So as to get information about the saved test results and the installed tests and their usage, we have the next options.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-saved-results
                                                                                                                                                                                                                                                             
Phoronix Test Suite v3.0.1                                                                                                                                                                                                                                            
1 Saved Results                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                             
Saved Name: ubuntu-server      Title: ubuntu-server                                                                                                                                                                                                                   
- apache                                                                                                                                                                                                                                                      
- iozone

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-test-usage

Phoronix Test Suite v3.0.1
2 Tests Installed

TEST                 VERSION  INSTALL DATE  LAST RUN    AVG RUN-TIME  TIMES RUN
pts/apache-1.3.0   - 1.3.0    2011-04-01    2011-04-01  5m15s         1
pts/iozone-1.7.0   - 1.7.0    2011-04-02    2011-04-02  2m10s         2

And further, if we want to acquire more details about an executed result, the info parameter will be our choice.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite info ubuntu-server

Title: ubuntu-server
Identifier: ubuntu-server

Test Result Identifiers:
- apache
- iozone

Contained Tests:
- Apache Benchmark
- IOzone

Finally, we also have the option to export the results in other formats (csv, text and pdf).

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite result-file-to-text ubuntu-server > ubuntu-server.txt

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite result-file-to-csv ubuntu-server > ubuntu-server.csv

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite result-file-to-pdf ubuntu-server

Saved To: /root/ubuntu-server.pdf

As well, we will always have the possibility to treat directly the results by means of the html structure.

root@ubuntu-server:~# ls -l .phoronix-test-suite/test-results/ubuntu-server/
total 32
-rw-r--r-- 1 root root 3291 2011-04-02 18:00 composite.xml
-rw-r--r-- 1 root root  192 2011-04-02 18:00 index.html
-rw-r--r-- 1 root root 5163 2011-04-02 21:32 pts-results-viewer.xsl
drwxr-xr-x 2 root root 4096 2011-04-02 21:32 result-graphs
drwxr-xr-x 4 root root 4096 2011-04-02 18:00 system-logs
-rw-r--r-- 1 root root 1491 2011-04-02 17:51 test-1.xml
-rw-r--r-- 1 root root 2210 2011-04-02 18:00 test-2.xml

With html or pdf formats, apart from the numeric data, we will be able to appreciate that Phoronix generates several graphics through the results.

Also point out another useful utility: we can merge several results and obtain a new combination from all of them. For this purpose, we first have to copy the other outcomes into the test-results directory.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite merge-results ubuntu-server centos
Merged Results Saved To: /root/.phoronix-test-suite/test-results/merge-8136/composite.xml

The key of this benchmarking application is to apply correctly in each moment the suitable tests. Due to the large variety of available tests, we can get much information. On the contrary, the main problem under my personal opinion is that the options which you can choose to run a test are very limited. If you wanted to gather more information from a specific tool (for instance iozone), you should execute it manually.

But in short, I think that Phoronix Test Suite is a great tool for measuring the perfomance of a system.

Automatic updates on Ubuntu with unattended-upgrades

2011-04-11T13:27:00.019+02:00

Some time ago I talked about the importance of having correctly our Linux systems up to date (at least automatically), specifically those issues related to security, focusing on CentOS/RHEL distributions. For this purpose I wrote an article named yum-security plugin.

For systems based on Debian/Ubuntu, you have got a package denominated unattended-upgrades, which allows to apply automatic updates (stable, security, updates and proposed-updates).

When we install an Ubuntu release, on the one hand we can mark the option for the system to automatically install the security updates. In this case, Ubuntu will install the unattended-upgrades package on the server and manage this subject.

And on the other, we can directly install it later and fit it based on our needs.

root@ubuntu-server:~# aptitude install unattended-upgrades

Through its configuration file (50unattended-upgrades), we can fit the types of updates (stable and security), the list of packages which must not be updated (mysql-server and apache2), an optional email address for warning about any problem, band with and so on.

root@ubuntu-server:~# cat /etc/apt/apt.conf.d/50unattended-upgrades
// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {
  "${distro_id} stable";
  "${distro_id} ${distro_codename}-security";
//      "${distro_id} ${distro_codename}-updates";
//      "${distro_id} ${distro_codename}-proposed-updates";
};

// List of packages to not update
Unattended-Upgrade::Package-Blacklist {
  "mysql-server";
  "apache2";
};

// Send email to this address for problems or packages upgrades
Unattended-Upgrade::Mail "admin@ubuntu-server.local";

// Do automatic removal of new unused dependencies after the upgrade
//Unattended-Upgrade::Remove-Unused-Dependencies "false";

// Automatically reboot *WITHOUT CONFIRMATION* if a
// the file /var/run/reboot-required is found after the upgrade
Unattended-Upgrade::Automatic-Reboot "false";

// Use apt bandwidth limit feature, this example limits the download
// speed to 70kb/sec
//Acquire::http::Dl-Limit "70";

In order to set the update period (in days), we have to edit the 20auto-upgrades file. In the following example, the packages which can be updated will be downloaded everyday, but the automatic updates will just be applied once a week. The downloaded packages will be removed every 15 days.

root@ubuntu-server:~# cp -a /usr/share/unattended-upgrades/20auto-upgrades /etc/apt/apt.conf.d/

root@ubuntu-server:~# cat /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::Unattended-Upgrade "7";
APT::Periodic::AutocleanInterval "15";

We can manually execute the unattended-upgrade daemon as well, by means of the next order.

root@ubuntu-server:~# unattended-upgrade -d
Initial blacklisted packages: mysql-server apache2
Starting unattended upgrades script
Allowed origins are: ["('Ubuntu', 'stable')", "('Ubuntu', 'maverick-security')"]
pkgs that look like they should be upgraded:
Fetched 0B in 0s (0B/s)                                                                                                                                                           
blacklist: ['mysql-server', 'apache2']
InstCount=0 DelCount=0 BrokenCout=0
No packages found that can be upgraded unattended

And finally, also say this application will be run via cron (/etc/cron.daily/apt). All output will be logged into the /var/log/unattended-ugprades.log file.

Benchmarking with Phoronix Test Suite (I)

2011-04-05T11:04:00.012+02:00

The Phoronix Test Suite is an interesting benchmarking platform aimed at testing and measuring the performance of multiple stuff, such as computers, graphics cards, processors, operating systems and so on.

I have used (in my professional work) and talked about this tool several times, but I had never dedicated any article to evaluate it. Let's get going!

For my tests, I am going to utilize an Ubuntu Server 10.10 (64 bits) and Phoronix Test Suite 3.0.1. When you download the application and descompress it, you have two options: on the one hand you can install it over the operating system (install.sh), and on the other, you may directly run the appropiate script (phoronix-test-suite) and carry out all the options that Phoronix provides. Regardless the choice, first of all you must install a serie of necessary dependences.

root@ubuntu-server:~# aptitude install php5-cli php5-gd php5-curl php-fpdf

root@ubuntu-server:~# tar xvzf phoronix-test-suite-3.0.1.tar.gz

root@ubuntu-server:~/phoronix-test-suite# ./install-sh

In my case, I will choose the second. Also say that if we are behind a proxy, we have to execute the script with the network-setup option, in order to set the proxy configuration up. In addition, we can check out the information about our system hardware and software.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite network-setup

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite system-info

Phoronix Test Suite v3.0.1
System Information

Hardware:
Processor: QEMU Virtual 0.12.5 @ 3.00GHz (1 Core), Motherboard: Bochs, Chipset: Red Hat Virtio, Memory: 1 x 2048 MB RAM, Disk: 15GB, Graphics: Cirrus Logic GD 5446

Software:
OS: Ubuntu 10.10, Kernel: 2.6.35-28-server (x86_64), Display Driver: cirrus, File-System: ext4

Phoronix supplies a set of suites (cpu, database, kernel, memory, etc.) that in turn, make up a group of tests (compress-gzip, dbench, sqlite, etc.), also known as profiles. In total, we have around more than 60 suites and 130 tests.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-available-suites

Phoronix Test Suite v3.0.1
Available Suites

pts/audio-encoding               - Audio Encoding                   System
pts/chess                        - Chess Test Suite                 Processor
pts/compilation                  - Timed Code Compilation           Processor
...

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-available-tests

Phoronix Test Suite v3.0.1
Available Tests

pts/aio-stress               - AIO-Stress                          Disk
pts/apache                   - Apache Benchmark                    System
pts/battery-power-usage      - Battery Power Usage                 System
...

If we want to know more details about a concrete suite or test, we can run the following order (in this example, the memory suite consists of three tests).

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite info memory

Phoronix Test Suite v3.0.1
Memory Test Suite

Run Identifier: pts/memory-1.0.2
Suite Version: 1.0.2
Maintainer: Michael Larabel
Suite Type: Memory
Unique Tests: 3
Suite Description: The system memory test suite consists of tests designed to test the computer's system memory (RAM) performance. Among these tests are RAMspeed and Bandwidth.

pts/memory-1.0.2
* pts/ramspeed
* pts/stream
* pts/cachebench

And if we want to install a suite or test, we must add the install parameter to the script. This sentence will firstly resolve and install the necessary dependences and afterwards, download, compile and install the test. If you select a suite, this order will install all tests belonging to the suite.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite install apache

The following dependencies are needed and will be installed:

- build-essential
...
Phoronix Test Suite v3.0.1

To Install: pts/apache-1.3.0
...

Then we are going to see the execution of the apache test (we can also use this command to launch a suite).

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite run apache
Would you like to save these test results (Y/n): Y
Enter a name to save these results: ubuntu-server
Enter a unique name for this test run: apache

Phoronix Test Suite v3.0.1
If you wish, enter a new description below.
Press ENTER to proceed without changes.

Current Description: Running pts/apache-1.3.0.

New Description:


Apache Benchmark:
pts/apache-1.3.0
Test 1 of 1
Expected Trial Run Count: 3
Running Pre-Test Script @ 15:45:48
Started Run 1 @ 15:45:53
Started Run 2 @ 15:47:34
Started Run 3 @ 15:49:17  [Std. Dev: 2.00%]
Running Post-Test Script @ 15:51:00

Test Results:
7156.08
7026.62
6876.06

Average: 7019.59 Requests Per Second

In the previous output, we can make out that the script has asked for a name to save the results (under this name, we will be able to save other tests) and a specific name for the test. In other tests as follows, we can see that the script requests other types of data, such as the record size, file size and disk test.

Booting the system with Upstart (II)

2011-03-27T19:08:00.007+02:00

Continuing with the article Booting the system with Upstart (I) and in particular, the part of the events, we can also set many other types of triggers.

root@ubuntu-server:~# cat /etc/init/job1.conf
# Start when the loading modules have concluded
start on stopped load-modules

# Start when the filesystem is mounted
start on filesystem

# Start when the filesystem is mounted and the network is started
start on filesystem and started network

# Start under the runlevels 2, 3 and 5
start on runlevel [235]
...

The initctl utility allows to pass down a concrete event.

root@ubuntu-server:~# initctl emit myevent

If we take a look at the files dropped off within the /etc/init directory, we will be able to find out many other events. For instance, if we open the mountall.conf file, we can see that this job mounts the filesystems during the boot and emits events as local-filesystems, all-swaps, etc.

All the theory described for "start on", also can be applied when we want to stop a job.

root@ubuntu-server:~# cat /etc/init/job1.conf
stop on starting shutdown
...

This command as well can be used to list the status of all jobs handled by Upstart.

root@ubuntu-server:~# initctl list
mountall-net stop/waiting
rc stop/waiting
rsyslog start/running, process 687
...

Another interesting configuration option is the respawn tag, which is utilized to start automatically a job if it stops abnormally. We can also set that if a job is respawned more that X times in Y seconds, it will be stopped definitely.

root@ubuntu-server:~# cat /etc/init/job1.conf
respawn

respawn limit 10 120
...

What happens if we want to perform some previous task to start a job? For this case, Upstart provides the pre-start directive, which is run before the job reaches the starting state.

root@ubuntu-server:~# cat /etc/init/job1.conf
pre-start script
   rm /tmp/job*
end script
...

Instead of a script, we can also execute a binary.

root@ubuntu-server:~# cat /etc/init/job1.conf
pre-start exec /usr/local/sbin/nessusd
...

In addition, we can specify other kinds of pre/post actions.

root@ubuntu-server:~# cat /etc/init/job1.conf
# Before the job is started
post-start exec|script

# Before the job is stopping
pre-stop exec|script

# Before the job is stopped
post-stop exec|script
...

You can get more information as always... by accessing the man.

root@ubuntu-server:~# man 5 init

Taking snapshots on KVM with libvirt

2011-03-20T14:23:00.006+01:00

There is an interesting feature related to the KVM virtualization, the snapshots, which take the disk, memory and device state of a concrete domain at a specific moment. This can have many use cases, such as saving a copy of a virtual machine, preserving a domain's state before a potentially dangerous operation, etc. Snapshots are identified with an unique name and can be taken with the virtual machine turned on.

This feature can only be used with the virsh command and not with the graphical tool (the version of Virtual Machine Manager included on Kubuntu 10.10 is the 0.8.4).

So as to create a snapshot for a particular domain, we must set a XML file with at least a name and description for the snapshot.

javi@javi-kubuntu:~$ cat UbuntuServer_10.10-ss.xml
<domainsnapshot>
   <name>UbuntuServer_10.10-16032011</name>
   <description>Snapshot of OS install and updates</description>
</domainsnapshot>

javi@javi-kubuntu:~$ virsh snapshot-create UbuntuServer_10.10 UbuntuServer_10.10-ss.xml
error: Requested operation is not valid: Disk '/var/lib/libvirt/images/UbuntuServer_10.10.img' does not support snapshotting

It is possible that as in the previous output, we get an error message about the virtual disk does not support snapshotting. What is the problem in this case?

The sort of the disk image is raw (default format when you create a virtual machine), which is very simple and does not allow snapshotting. In contradistinction to raw, qcow2 (QEMU image format) supports to have smaller images, encryption, compression, snapshots and so on.

In order to make a qcow2 virtual disk, we can run the following command. In this case, we have reserved all the space for the virtual disk. If we do not want to preallocate the image with metadata, we must take away the "preallocation" option.

javi@javi-kubuntu:~$ sudo qemu-img create -f qcow2 -o preallocation=metadata /var/lib/libvirt/images/UbuntuServer_10.10.qcow2 8G

javi@javi-kubuntu:~$ sudo chown libvirt-qemu:kvm /var/lib/libvirt/images/UbuntuServer_10.10.qcow2

If we want to convert an existing image (for example raw to qcow2), we can use the next order.

javi@javi-kubuntu:/var/lib/libvirt/images$ sudo qemu-img convert -f raw -O qcow2 -o preallocation=metadata UbuntuServer_10.10.raw UbuntuServer_10.10.qcow2

javi@javi-kubuntu:/var/lib/libvirt/images$ sudo chown libvirt-qemu:kvm UbuntuServer_10.10.qcow2

In this moment, we will be able to set a snapshot up with the virsh snapshot-create command. This operation will generate a new XML file.

javi@javi-kubuntu:~$ ls -l /var/lib/libvirt/qemu/snapshot/UbuntuServer_10.10/UbuntuServer_10.10-16032011.xml
-rw------- 1 root root 307 2011-03-16 11:40 /var/lib/libvirt/qemu/snapshot/UbuntuServer_10.10/UbuntuServer_10.10-16032011.xml

javi@javi-kubuntu:~$ virsh snapshot-dumpxml UbuntuServer_10.10 UbuntuServer_10.10-16032011
<domainsnapshot>
   <name>UbuntuServer_10.10-16032011</name>
   <description>Snapshot of OS install and updates</description>
   <state>shutoff</state>
   <creationTime>1300272022</creationTime>
   <domain>
      <uuid>d19af827-30ca-b3a0-8304-231f8cf5dd8b</uuid>
   </domain>
</domainsnapshot>

So as to list all the available snapshots for a certain domain, we can type the following order.

javi@javi-kubuntu:~$ virsh snapshot-list UbuntuServer_10.10
Name                        Creation time             Status
---------------------------------------------------------------
UbuntuServer_10.10-16032011 2011-03-16 11:40:22 +0100 shutoff

If at any point we want to bring back a snapshot, we must run the next command.

javi@javi-kubuntu:~$ virsh snapshot-revert UbuntuServer_10.10 UbuntuServer_10.10-16032011

And finally, we also have the option to remove a snapshot.

javi@javi-kubuntu:~$ virsh snapshot-delete UbuntuServer_10.10 UbuntuServer_10.10-16032011

System monitoring with nmon

2011-03-12T12:42:00.003+01:00

Nmon is another interesting monitoring tool for Linux systems which can present many information related to the CPU, memory, network, etc. through an organized screen.

I have tested the 13g version on Ubuntu Server 10.10. When you start the application, this shows you a little menu with different options, in order to configure your own monitoring panel.

The next figure is a dump of my setting. I have used CPU utilization by processor, memory and swap stats, kernel stats and load average, network and disk input/output and top processes.

And finally, also say that with nmon you can take the data and dump them into a CSV file. For instance, in the following case I have run nmon in background to capture the data each 5 sg and a total of 200 times. Besides I have specified the file name with the '-F' option.

root@ubuntu-server:~# nmon -t -F `hostname`.csv -s 5 -c 200

Booting the system with Upstart (I)

2011-03-05T12:44:00.010+01:00

Upstart is a management daemon which leads the starting and stopping of the system services, and besides, handles them whereas they are running. It is an application actually used on system such as Ubuntu, Fedora, RHEL, etc. and it has replaced the traditional System V init daemon (SysV).

It turns out that SysV has not been really taken away, since while there are tasks based on SysV, both systems will have to live together. Indeed, there is a job defined by Upstart and named rc-sysinit.conf, which controls the execution of the existing traditional SysV scripts. An Upstart system runs by default in runlevel 2.

root@ubuntu-server:~# runlevel
N 2

What is the main advantage of Upstart against SysV under my personal opinion? In contradistinction to SysV, Upstart allows to launch the services or tasks (here named jobs) in parallel, that is to say, they have just to wait for certain events happen. For example, when the Upstart init daemon has finished its initialization (upstart), when the filesystem is mounted (filesystem), MySQL daemon is started (started mysqld), and so on.

Let's see how Upstart works. All Upstart jobs are files made up of a well known structure and stored into the /etc/init directory. For example, we are going to develop an easy job to write the date within a tmp file. Its name will be job1.

root@ubuntu-server:~# cat /etc/init/job1.conf
script
   date > /tmp/job1.tmp ; sleep 10
end script

One of the ways to start this job is manually.

root@ubuntu-server:~# start job1
job1 start/running, process 1705

If we take a look at its state before 10 sg, we will see which is start/running. Afterwards, it will be stop/waiting, since in this case, it is not a daemon such as httpd, but a process which finishes when completes theirs defined tasks.

root@ubuntu-server:~# status job1
job1 start/running, process 1710

root@ubuntu-server:~# status job1
job1 stop/waiting

Upstart allows to define a job or by a script section (as the previous example - remember that the scripting code will be launch under "sh -e") either using the exec directive.

root@ubuntu-server:~# cat /etc/init/job1.conf
script
   exec date > /tmp/job1.tmp
end script

This sort of stanza can be used for instance to run a script or start a daemon like mysqld (zero or more arguments can be passed to it). Other manner to start a job is through an event. For example, we can run our job when job2 or job3 are started.

root@ubuntu-server:~# cat /etc/init/job1.conf
start on started job2 or started job3
...

By default, a job is a service that during its life cycle will be able to pass for different stages: starting, started, stopping and stopped, and in addition, will be able to be respawned if it exits with a zero status.

If we want to specify that the job is a task (it is considered to have finished when it has been run and stopped again - it will terminate with zero status but it will not be able to be respawned), we will have to add the task parameter into the configuration file. Otherwise, Upstart will treat the job as a service.

Following up network connections with conntrack (II)

2011-02-27T13:26:00.005+01:00

Let's finish the previous article about Following up network connections with conntrack (I). Other important parameters which can be changed to optimize the system are related to the time of the different types of connections.

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
432000

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait
120

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait
60

The first parameter sets up the maximum lifetime for an already established connection (432000 sg can be long; 28800 could be enough). The second and third are the maximum lifetime for a waiting connection and for the remote endpoint closes the socket.

So as to list all variables based on the conntrack module, type the next order.

root@ubuntu-server:~# sysctl -a | grep conntrack | grep ipv4
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2 = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 300
net.ipv4.netfilter.ip_conntrack_tcp_loose = 1
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 0
net.ipv4.netfilter.ip_conntrack_tcp_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30
net.ipv4.netfilter.ip_conntrack_max = 15768
net.ipv4.netfilter.ip_conntrack_count = 2
net.ipv4.netfilter.ip_conntrack_buckets = 4096
net.ipv4.netfilter.ip_conntrack_checksum = 1
net.ipv4.netfilter.ip_conntrack_log_invalid = 0

And if you want to change the value of any variable, you must add it within the sysctl.conf file and reload the settings.

root@ubuntu-server:~# cat /etc/sysctl.conf
...
net.ipv4.netfilter.ip_conntrack_max = 131072

root@ubuntu-server:~# sysctl -p

One interesting option for the conntrack command is the possibility to get the statistics about the connection tracking.

root@ubuntu-server:~# conntrack -S
entries                 2  
searched                0  
found                   1107
new                     4  
invalid                 0  
ignore                  0  
delete                  2  
delete_list             2  
insert                  4  
insert_failed           0  
drop                    0  
early_drop              0  
icmp_error              0  
expect_new              0  
expect_create           0  
expect_delete           0  
search_restart          0

Another useful feature for conntrack is to output the connection state on real-time, similar to when you run a "tail -f" on a file.

root@ubuntu-server:~# conntrack -E

We can conclude with this couple of articles that the conntrack module is other helpful way to improve the Linux performance.

Cached memory in Linux

2011-02-19T12:48:00.009+01:00

I am sure that there are many people that if they saw the following output of 'top', they would say that I have a problem with my free memory... they would be wrong.

[root@zabbix ~]# top
...
Mem:   4044540k total,  4007540k used,    37000k free,   156336k buffers
Swap:  2097144k total,        0k used,  2097144k free,  2086808k cached
...

The previous data correspond to a Zabbix installation where the free memory is around 37 MB, but the cached memory is more than 2 GB. What is happening here?

The answer is straightforward: Linux always tries to use all available memory, and thereby, it caches all read data. If at any moment an application needs memory, Linux will free it from the cached memory. This way of acting is pretty good because you will have better performance having the more frecuent data into the memory.

Really you will have a serious problem when your free memory is low, further your cached memory too and on top of all that, your system begins to swap.

Also say that other excellent data from the above 'top' is the swap memory value: 0. A well configured Linux system never should utilize swap. The key parameter for this purpose is swappiness. I usually set it with a value of 20.

[root@zabbix ~]# cat /proc/sys/vm/swappiness
20

There is a quick manner to force to the operating system so as to free the cached memory: changing the value of the drop_caches variable.

[root@zabbix ~]# cat  /proc/sys/vm/drop_caches
0

[root@zabbix ~]# sync ; echo 1 > /proc/sys/vm/drop_caches

[root@zabbix ~]# top
...
Mem:   4044540k total,  1756996k used,  2287544k free,     1572k buffers
Swap:  2097144k total,        0k used,  2097144k free,    62564k cached
...

We can demonstrate the cached memory operation through an easy bash script. The program will search twice the word "test" inside all files of the /var/log/httpd directory.

[root@zabbix ~]# cat script.sh
#!/bin/bash

for (( i=0; i<2; i++))
do
   free -o -m
   /usr/bin/time -f "\nSeek time: %e sg\n" grep -r test /var/log/httpd/ > /dev/null
done

[root@zabbix ~]# ./script.sh
             total       used       free     shared    buffers     cached
Mem:          3949       1712       2237          0          2        101
Swap:         2047          0       2047

Seek time: 3.76 sg

             total       used       free     shared    buffers     cached
Mem:          3949       2003       1946          0          3        412
Swap:         2047          0       2047

Seek time: 0.50 s

In the first loop you can see that the cached memory was 101 MB and it spent 3.76 sg looping through all files. But in the second loop, the 'grep' command spent 0.5 sg because the files were already cached on memory (its size grew from 101 MB to 412 MB).

If we take a look at the size of the /var/log/httpd directory, we can appreciate that the cached memory increment practically matchs with the size of that directory.

[root@zabbix ~]# du -shx /var/log/httpd/
318M     /var/log/httpd/

Following up network connections with conntrack (I)

2011-02-13T13:15:00.005+01:00

Linux has got the ability to perform a monitoring of existing connections by means of the conntrack module, which is compiled but not installed in distributions such as RHEL or CentOS. In order to load it, you can run the next order.

[root@centos ~]# modprobe ip_conntrack

In other operating systems like Debian or Ubuntu Server, first of all you must install the conntrack package and load the nf_conntrack_ipv4 module (if you want to work with IPv6, you will have to load the nf_conntrack_ipv6 module).

root@ubuntu-server:~# aptitude install conntrack

root@ubuntu-server:~# modprobe nf_conntrack_ipv4

The conntrack module allows the kernel to register in a table all network connections of the system (established, time_wait, close, etc.). It used by several applications such as iptstate (it shows information about the state of the system connections) or Shorewall (firewall).

Another example of use for this module it is for instance, when the server has to realize NAT tasks with iptables and it is necessary to keep a table of connections implicated.

The file where conntrack logs all connections is /proc/net/ip_conntrack.

root@ubuntu-server:~# cat /proc/net/ip_conntrack
tcp      6 89 TIME_WAIT src=192.168.1.11 dst=192.168.1.12 sport=59302 dport=10050 packets=5 bytes=291 src=192.168.1.12 dst=192.168.1.11 sport=10050 dport=59302 packets=5 bytes=289 [ASSURED] mark=0 secmark=0 use=1
...

root@ubuntu-server:~# conntrack -L
tcp      6 89 TIME_WAIT src=192.168.1.11 dst=192.168.1.12 sport=59302 dport=10050 packets=5 bytes=291 src=192.168.1.12 dst=192.168.1.11 sport=10050 dport=59302 packets=5 bytes=289 [ASSURED] mark=0 secmark=0 use=1
...

The two first fields are the connection protocol (TCP, 6) and then is the connection state (TIME_WAIT). The rest of the fields represent the IP addresses and ports involved, as well as the number of packets and bytes exchanged between the two points of the connection.

You have also to take into account that Linux saves the connection state in memory, and each of them uses around 350 bytes.

If you want to know how many open connections has got the system, you can utilize the following sentences.

root@ubuntu-server:~# cat /proc/net/ip_conntrack | wc -l
856

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
856

root@ubuntu-server:~# conntrack -C
856

This value is quite important because if at any moment we appreciate that any of our services works slowly (for instance Apache) or many connections are rejected, it can be due to which the number of open connections exceeds the maximum number of connections allowed.

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
65536

The size of the hash table is also limited.

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
16384

If you want to modify it, you must do it when the module is loaded.

root@ubuntu-server:~# modprobe nf_conntrack_ipv4 hashsize=32768

Fitting virtual disks after reducing

2011-02-06T13:21:00.005+01:00

This is the continuation of the article Shrinking virtual disks with LVM. It turns out that once you have mirrored your virtual disk to another smaller, it is normal to have wasted some space in the process, that is to say, an area of the new virtual disk which has been unused.

For this reason, we are going to learn in this article how to recover it. At this moment, the available size of our new disk is 16 GB.

[root@centos ~]# lvs
LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert
LogVol00 VolGroup00 -wi-ao 16,00G                            
LogVol01 VolGroup00 -wi-ao  1,00G

The idea is to create a new partition on the disk in order to take that unutilized space. The partition type must be Linux LVM (8e hexadecimal code).

[root@centos ~]# fdisk /dev/sda
...
Orden (m para obtener ayuda): n
Acción de la orden
e   Partición extendida
p   Partición primaria (1-4)
p
Número de partición (1-4): 3
Primer cilindro (2241-2480, valor predeterminado 2241):
Se está utilizando el valor predeterminado 2241
Último cilindro o +tamaño o +tamañoM o +tamañoK (2241-2480, valor predeterminado 2480):
Se está utilizando el valor predeterminado 2480

Orden (m para obtener ayuda): t
Número de partición (1-4): 3
Código hexadecimal (escriba L para ver los códigos): 8e
Se ha cambiado el tipo de sistema de la partición 3 por 8e (Linux LVM)

Orden (m para obtener ayuda): w

[root@centos ~]# partprobe

[root@centos ~]# fdisk -l

Disco /dev/sda: 20.4 GB, 20401094656 bytes
255 heads, 63 sectors/track, 2480 cylinders
Unidades = cilindros de 16065 * 512 = 8225280 bytes

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        2241    17891328+  8e  Linux LVM
/dev/sda3            2241        2480     1924849   8e  Linux LVM

Next step is to set up a PV (physical volume) on that new partition.

[root@centos ~]# pvcreate /dev/sda3

Now we have to add the new PV created to the existing VG (volume group) and then, extend the LV (logical volume).

[root@centos ~]# vgextend VolGroup00 /dev/sda3

[root@centos ~]# lvextend -l +100%FREE /dev/VolGroup00/LogVol00

And finally, we must only to expand the ext3 filesystem.

[root@centos ~]# resize2fs /dev/VolGroup00/LogVol00

After the operation, we have retrieved 1,81 GB.

[root@centos ~]# lvs
LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert
LogVol00 VolGroup00 -wi-ao 17,81G                             
LogVol01 VolGroup00 -wi-ao  1,00G

So as to realize the task, it is not necessary to put the machine into rescue mode and it can be performed on hot.

Remote management on KVM with SSH

2011-01-30T13:21:00.006+01:00

When we set up virtual machines with libvirt/KVM on production environments, it is typical that the operating system where we want to create the virtual machines, it does not have a graphical user interface as Gnome or KDE, where we can run the Virtual Machine Manager in order to make or handle them.

For this reason, it is necessary to connect it remotely through Virtual Machine Manager so as to set up new virtual machines where a graphical wizard is required.

In this article, we are going to establish a connection over SSH from a Kubuntu 10.10 to a KVM hypervisor located on an Ubuntu Server 10.10. In this way, the libvirt management connection will be securely tunneled over an SSH connection.

In the following figure, you can see the needed packages which must be installed on each computer. Remember that I already presented a series of articles related to KVM virtualization (I, II, III and IV).

First of all, we must start the libvirtd daemon on the Ubuntu Server. We will also ensure that in successive starts, the daemon will run automatically.

javi@ubuntu-server:~$ sudo service libvirt-bin start

javi@ubuntu-server:~$ sudo update-rc.d libvirt-bin defaults

Then, we must generate a public key pair on the computer (kubuntu) where the Virtual Machine Manager (virt-manager package) will be used. In the next step, we have to copy the keys to the machine (ubuntu-server) where libvirtd will be running.

javi@kubuntu:~$ ssh-keygen -t rsa

javi@kubuntu:~$ ssh-copy-id -i ~/.ssh/id_rsa.pub javi@ubuntu-server

In the previous output, we have copied the keys to the javi user home, where 'javi' is a simple user (no root) of ubuntu-server with rights to manage libvirt (by default, any user belonging to libvirtd group can handle libvirt).

And finally, we only have to to make a remote connection over SSH from the Virtual Machine Manager to the hypervisor located on the Ubuntu Server.

Shrinking virtual disks with LVM

2011-01-23T17:25:00.005+01:00

It is possible that what I am going to tell next can be done in other ways, but I am really sure that it will be very useful for many people. Have you ever thought how to reduce a virtual disk on VMware, KVM, Xen, etc. with a Linux filesystem created inside?

If the virtual disk just contains a filesystem such as ext3, ext4, btrfs, etc., the solution is easy: use any partition tool like GParted, shrink the partition or partitions and copy them to another virtual disk smaller.

But what happens if that virtual disk has a filesystem over a Logical Volume (LV)? The solution is not trivial, since partion tools do not support Logical Volume Management (LVM).

Then I am going to explain my solution. For my tests, I will use a CentOS 5.5 virtual machine under VMware vSphere, with a virtual disk of 64 GB (sda). That virtual disk will have two partitions: sda1 (107 MB) and sda2 (63,88 GB).

[root@centos ~]# fdisk -l

Disco /dev/sda: 68.7 GB, 68719476736 bytes
255 heads, 63 sectors/track, 8354 cylinders
Unidades = cilindros de 16065 * 512 = 8225280 bytes

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        8354    66999082+  8e  Linux LVM

The second partition (sda2) will have two LVs, LogVol00 (data area) and LogVol01 (swap).

[root@centos ~]# lvs
LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert
LogVol00 VolGroup00 -wi-ao 62,88G              
LogVol01 VolGroup00 -wi-ao  1,00G

My goal will be to decrease the size of the virtual disk from 64 GB to 19 GB.

In order to be able to resize the ext3 filesystem, LV, VG (Volume Group), PV (Physical Volume) and sda2 partition, you must boot the computer in rescue mode (using for example a Live CD).

boot: linux rescue

During the boot process, we will not mount the existing Linux installation and skip directly to the command shell. Then, we have to activate all known volume groups in the system and check the filesystem to rule out possible errors on it.

sh-3.2# lvm vgchange -a y

sh-3.2# e2fsck -f /dev/VolGroup00/LogVol00

Afterwards, first we must resize the filesystem from 62,88 GB to 16 GB and then, the LV.

sh-3.2# resize2fs /dev/VolGroup00/LogVol00 16G

sh-3.2# lvm lvresize --size 16G /dev/VolGroup00/LogVol00

Because we have reduced the LogVol00 size, now there is a gap between both volumes and it is better that we remove LogVol01 and recreate it again.

sh-3.2# lvm lvremove /dev/VolGroup00/LogVol01

sh-3.2# lvm lvcreate --size 1G --name LogVol01 VolGroup00

sh-3.2# mkswap /dev/VolGroup00/LogVol01

Next step is to decrease the size of the PV. We need 16 GB for the data area and 1 GB for the swap.

sh-3.2# lvm pvresize /dev/sda2 --setphysicalvolumesize 17G
/dev/sda2: cannot resize to 511 extens as 544 are allocated.
0 physical volume(s) resized / 1 physical volume(s) not resized

We can see that we must fit correctly that space... it is easy, a simple rule of three (17*544/511).

sh-3.2# lvm pvresize /dev/sda2 --setphysicalvolumesize 17.03G
Physical volume "/dev/sda2" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

And finally, we have to resize that sda2 partition. To calculate the end sector, first we must take a look at the partition map in sectors (one sector is 512 bytes), get the starting point of the sda2 partition (208845s), add it the size of the PV (35651584s) and also add a security margin of around 64 MB (131072s).

sh-3.2# lvm pvs --units s
PV         VG         Fmt  Attr PSize     PFree
/dev/sda2  VolGroup00 lvm2 a-   35651584S    0S

sh-3.2# parted /dev/sda unit s print
Disk /dev/sda: 134217727s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start    End         Size        Type     File system  Flags
1      63s      208844s     208782s     primary  ext3         boot
2      208845s  134207009s  133998165s  primary               lvm

Now we can resize the partition: 208845 + 35651584 + 131072 = 35991501.

sh-3.2# parted /dev/sda rm 2

sh-3.2# parted /dev/sda mkpart primary 208845s 35991501s

sh-3.2# parted /dev/sda set 2 lvm on

sh-3.2# parted /dev/sda print
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
1      32.3kB  107MB   107MB   primary  ext3         boot
2      107MB   18.4GB  18.3GB  primary               lvm

It is necessary check again the filesystem to see that all is right.

sh-3.2# e2fsck -f /dev/VolGroup00/LogVol00

Now we have to add a second virtual disk (19 GB) to the system and copy the data from sda to sdb.

sh-3.2# dd if=/dev/sda of=/dev/sdb bs=1M &

When the task is complete, we must turn off the virtual machine, delete the first virtual disk (64 GB) and put the second (19 GB) as primary for the next boot.

Then we will finish starting the virtual machine.

Proxmox VE cluster

2011-01-17T22:31:00.010+01:00

Other interesting feature related to Proxmox VE is the possibility to join all Proxmox VE installations in an unique cluster which centralizes and manages all virtual machines. Besides, we will also be able to move the virtual instances between Proxmox VE nodes.

In the following tests we are going to set up a cluster using two Proxmox VE 1.7 servers (proxmox1.local, with IP address 192.168.122.211/24 and proxmox2.local, with IP address 192.168.122.212/24). The two Proxmox VE nodes will be two virtual machines running under KVM.

The cluster setting does not support web management, therefore the only way is to utilize the bash console (pveca command).

First of all, we have to define the master node (for instance proxmox1.local).

proxmox1:~# pveca -c
cluster master successfully created

proxmox1:~# pveca -l
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---DISK
1 : 192.168.122.211 M     A           00:13   0.00    16%    20%

Now we can add new slaves to the master. In our case, we will assign one only node to the cluster, proxmox2.local.

proxmox2:~# pveca -a -h 192.168.122.211
cluster node successfully created

proxmox2:~# pveca -l           
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---DISK
1 : 192.168.122.211 M     A           00:23   0.00    18%    20%
2 : 192.168.122.212 N     A           00:23   0.00    17%    20%

If we open a web browser for the master node, we will be able to see in the main screen the cluster state.

If we want to make a virtual machine, we will be able to choose the server where the virtual machine will be created.

Other useful orders provided by the pveca command are the next:

proxmox2:~# pveca --help
...
pveca -s [-h IP]     # sync cluster configuration from master (or IP)
pveca -d ID          # delete a node
pveca -m             # force local node to become master
pveca -i             # print node info (CID NAME IP ROLE)

And finally, we also have the option of live migration in order to move virtual machines between physical servers. This is an interesting characteristic because in this way, we can locate a virtual machine in other Proxmox node with better hardware, or simply for maintenance tasks.

For example, we are going to imagine that we have a CentOS 5 virtual machine installed on proxmox1.local, and we want to migrate it to proxmox2.local.

For this purpose, we have to open VM Manager, Virtual Machines section and select the Migrate tab. Then, we must select the source and target nodes and the VMID to migrate and press the migrate button (the virtual machine to be moved can be turned on).

ufw (uncomplicated firewall)

2011-01-11T17:26:00.006+01:00

Uncomplicated firewall (ufw) is the default tool included on Ubuntu distributions used to secure all incoming, outgoing and internal network traffic, providing appropiate IPv4 and IPv6 rules based on iptables.

Its files and directories structure is showed then (I have used an Ubuntu Server 10.10 for the tests).

root@ubuntu-server:~# tree /etc/ufw/
/etc/ufw/
├── after6.rules
├── after.rules
├── applications.d
│   └── openssh-server
├── before6.rules
├── before.rules
├── sysctl.conf
└── ufw.conf

By default, the firewall is disabled. So as to enable it, you must run the following order. If you want to turn off the firewall, you must add the disable parameter.

root@ubuntu-server:~# ufw enable
root@ubuntu-server:~# ufw disable

root@ubuntu-server:~# ufw status verbose
Estado: activo
Acceso: on (low)
Por defecto: deny (Entrada), allow (Salida)

With the 'status verbose', we can see that the default policy is to deny for incoming traffic and to allow for outgoing traffic. We can also change these default policies:

root@ubuntu-server:~# ufw default allow|deny|reject incoming|outgoing

Now we are going to view several examples. For instance, to set up a rule in order to allow the incoming mail traffic (any of the three possibilities is valid - the service names are declared into the /etc/servicies file).

root@ubuntu-server:~# ufw allow 25

root@ubuntu-server:~# ufw allow 25/tcp

root@ubuntu-server:~# ufw allow smtp

To remove the rule:

root@ubuntu-server:~# ufw delete allow 25

To add a rule in a specific position (fourth in the following example):

root@ubuntu-server:~# ufw insert 4 allow 22

To define the protocol, the source and destination addresses in order to deny certain traffic:

root@ubuntu-server:~# ufw deny proto esp from 192.168.1.0/24 to any

With ufw you can also specify the log level (the traces will be dumped to the syslog file with low level by default).

root@ubuntu-server:~# ufw logging on|off|LEVEL

LEVEL can be off, low, medium, high and full.

Another interesting feature of ufw is the possibility to define applications. For example, I am going to create an application named 'myapps' with a series of services:

root@ubuntu-server:~# vim /etc/ufw/applications.d/myapps
[myapps-1]
title=My applications
description=my applications: Artifactory, Hudson, Sonar, Redmine, actiTIME, Daisy
ports=8081,8080,9000,3000,7000,8888/tcp

Then I have to update the firewall with that profile information.

root@ubuntu-server:~# ufw app update myapps-1

And finally, I can already set new rules using this application.

root@ubuntu-server:~# ufw allow from 10.0.0.0/8 to any app myapps-1

In order to list all applications or to show information about a certain profile, we can run the following commands:

root@ubuntu-server:~# ufw app list

root@ubuntu-server:~# ufw info myapps-1

Making virtual machines with Proxmox VE

2011-01-04T00:08:00.004+01:00

In the previous article, I presented Proxmox Virtualization Environment, a professional virtualization platform used to make virtual machines based on KVM and OpenVZ technologies.

Today we are going to learn how you can rapidly create a virtual machine from an OpenVZ template (I cannot utilize KVM because I have installed Proxmox VE 1.7 on a KVM/libvirt virtual machine and therefore, that virtual CPU does not have Intel VT / AMD-V support. Don't worry because the KVM process is similar to OpenVZ).

First, we have to open the VM Manager, Appliance Templates section and select the Download tab. Here you can see a list of templates ordered by means of different categories: certified appliances, admin, system and web. Also we can get more OpenVZ templates from lots of web sites and manually upload them to the Proxmox VE data area.

In my test, I am going to choose a CentOS 5 (standard) distribution and download it since Proxmox VE.

Afterwards we have to move to the VM Manager, Virtual Machines, Create section and fill the several fields that you can look in the following figure.

Most of the options are also typical in other virtualization products: hostname, memory, disk space, etc., but I am going to stress two: on the one hand we have the type of virtual machine; I picked out Container (OpenVZ) because it is the only allowed option due to the virtual CPU issue that I commented before. And on the other we must select the sort of network for our virtual machine.

With Proxmox VE you have got two option for the network devices: Virtual Network (venet) or Bridged Ethernet (veth). Basically, venet (virtual network device) provides a point-to-point connection between the guest and the host with better performance and more security, but on the contrary, there is no MAC address and full support of IPv6 stack regarding veth (Virtual eTHernet). In the next article, you can read more information about this topic: Differences between venet and veth.

And finally, we must press the create button and in a few seconds, we will have our virtual machine totally created and perfectly working.

In order to launch it, we must go to the VM Manager, Virtual Machines, List section, make click on the virtual machine that we want to turn on, and press the Start button. In this area we will be able to change different components of the virtual machine.

The Open VNC console option is used to pop up a new window with graphical access to the virtual machine.

Proxmox Virtualization Environment

2010-12-28T15:52:00.013+01:00

Proxmox VE is an open source virtualization platform based on Linux and used to run virtual appliances and virtual machines. It is a product aimed at companies and production environments, since it intends to be an easy deployment and management solution.

Proxmox VE supports three types of virtualization technologies:

Container virtualization (OpenVZ): it allows that a physical server can run multiple insolated operating system instances as well known as containers. Its main problem is these containers or guests can just be Linux instances. However, the OpenVZ kernel provides better performance (penalty between 1% and 3% compared to a standalone server) in contradistinction to other alternatives.

Full virtualization (KVM): it can run Linux and Windows guests but unlike OpenVZ, Intel VT or AMD-V CPU is needed.

Paravirtualization (KVM): it presents a software interface similar to the underlying hardware in order to try to reduce the execution time of certain operations.

In my personal opinion, I think that OpenVZ is a plus because it has got better performance than KVM and the containers idea is fantastic: the size of these templates is really small and you can get a ready virtual machine in few minutes.

You can install Proxmox VE from an ISO image or directly on an existing Debian distribution (32 or 64 bits). I have tested Proxmox VE 1.7 (bare-metal ISO installer based on Debian Lenny) on a KVM virtual machine under my Kubuntu 10.10.

The process is very simple. The Proxmox installer is based on a graphical wizard with several stages: location and time zone selection, password and email address definition, and network configuration.

What are the main features of this product?

Web based administration: easy deployment and management (web based management and virtual console, backup and restore with LVM2 snapshots, etc.).

Virtual appliances: they are fully pre-installed and pre-configured applications including the operating system environment. You can create your own container, get from the community, use Linux OS instances or buy certified appliances.

Proxmox VE cluster: it allows to gather multiple physical server in one VE cluster (central web management and login, cluster synchronization, easy cluster setup, live migration of virtual machines between physical servers, etc.).

When you finish the installation process, you must reboot the machine and update the system.

proxmox:~# aptitude update ; aptitude dist-upgrade

In order to manage Proxmox VE, you must open a web browser and type the IP address configured during the wizard (the default user is 'root').

The Proxmox web interface is very useful. It is formed by three principal sections, VM Manager, Configuration and Administration.

In the VM Manager area, you can upload ISO images and OpenVZ templates, download certified appliances and create and handle virtual machines.

In the Configuration area, you can set up the different parameters of the system (network, DNS, time, administrator options, language, proxy, etc.), add and manage data storages (ISCSI targets, NFS shares, LVM groups and directories) and create new backup jobs.

And finally, in the Administration area you can control the Proxmox VE certificates and services (ClusterSync, ClusterTunnel, NTP, SMTP, SSH and WWW), take a look at the logs and monitor the cluster nodes.

Frag needed and DF set

2010-12-22T11:53:00.013+01:00

I was remembering a curious problem that we had at work the last year.

There was an HTTPS service that when you tried to access it by means of a web browser, the screen did not show anything. The display was blank waiting...

The main inconvenience was that we had not access to the web server to check it out. At that moment, I ran a tcpdump on the client in order to capture all network traffic and try to find out what was happening.

When I analyzed the packets I could see that the TCP connection was established correctly but then, the web server was sending frames with "TCP Previous segment lost", "TCP Dup ACK", "TCP Retransmission" messages. It is a pity because I do not keep these network captures to output them here...

Well, then it was clear that there was some device in the middle of the route between client and server, which was causing a leak of network packets in that communication.

Our network architecture was similar to the schema of the following image.

Taking a look at the firewall logs, we could see that there were ICMP packets related to the problematic HTTPS connection which were being dropped.

The router was sending ICMP packets (type 3, destination unreachable - code 4, fragmentation needed) to the source, in order to warn it that its MTU was smaller than the size of the packets, and on top of all that, the DF (prohibit fragmentation) was set to 1.

When that ICMP packet reached to the firewall, it was dropped and the source never knew that it had to reduce the data field for the TCP/IP packets. The solution was to allow the ICMP traffic (ICMP protocol with type 3 and code 4).

We can realize a little test in our computer. For example, attempting to send a packet whose size exceeds our network MTU (1500) and besides, DF=1 (192.168.1.100 is my PC and 192.168.1.1 the destination).

javi@kubuntu:~$ ping -c 1 -s 2000 -M do 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 2000(2028) bytes of data.
From 192.168.1.100 icmp_seq=1 Frag needed and DF set (mtu = 1500)

If I run a tcpdump, I can see that I receive an ICMP datagram noting me that I have to lower the size of the packets.

javi@kubuntu:~$ sudo tcpdump -ni lo icmp -s0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
18:59:33.014773 IP 192.168.1.100 > 192.168.1.100: ICMP 192.168.1.1 unreachable
- need to frag (mtu 1500), length 556

Linux load average

2010-12-13T13:30:00.005+01:00

I wanted to write an article about a topic that over time, it is still generating much confusion: the load average.

The system load average is a set of three numerical values which are provided by tools such as uptime or top. These values represent the average number of system processes that during the last 1, 5 and 15 minutes have been waiting for any resource of the system (CPU, disk access, network, etc.)

[root@centos ~]# uptime
15:43:45 up 9 days,  5:19,  1 user,  load average: 1.62, 1.49, 1.39

[root@centos ~]# top
top - 15:44:32 up 9 days,  5:20,  1 user,  load average: 1.38, 1.43, 1.37
...

In the previous output of the uptime command, 1.49 means that during the last 5 minutes, an average of 1.49 processes have been blocked waiting for some resource allocation.

For instance, I usually set a trigger in Zabbix for when the average load during the last 5 minutes has been higher than the number of cores available on the monitored machine.

When an alarm of this type is raised, does not necessarily have to mean that the CPU is overloaded. At that moment we have to use other Linux tools, such as top, vmstat, iostat, vnstat, etc, in order to find out what process or processes are involved and what resources are affected.

Finally, to emphasize the word of the title (Linux...), since these values for UNIX systems represent the number of processes which have utilized (running) the CPU or have been expecting for it (runnable).

KVM virtualization on Ubuntu (IV)

2010-12-07T11:51:00.015+01:00

This is the last article about KVM virtualization on Ubuntu (I, II, III).

In the first post, I choose as hypervisor localhost (QEMU). If you pick out the other hypervisor available (localhost (QEMU Usermode)) and try to set up a virtual machine, in the last step you will get an error with the following text: "Imposible completar la instalación: «internal error Process exited while reading console log output: char device redirected to /dev/pts/1 qemu: could not open disk image /var/lib/libvirt/images/Ubuntu.img: No such file or directory".

The problem is when you select this hypervisor, the virtual machine is created with your user account, and if you take a look at the permissions on the /var/lib/lubvirt/images directory, you will see that only the root user can write in it.

javi@javi-kubuntu:~$ ls -l /var/lib/libvirt/
total 12
drwxr-xr-x 2 root         root 4096 2010-09-25 00:52 boot
drwxr-xr-x 2 root         root 4096 2010-09-25 00:52 images
drwxr-xr-x 4 libvirt-qemu kvm  4096 2010-12-07 12:28 qemu

The solution consists in changing the directory from root to libvirtd group and adding write permissions.

javi@javi-kubuntu:~$ sudo chown :libvirtd /var/lib/libvirt/images

javi@javi-kubuntu:~$ sudo chmod g+w /var/lib/libvirt/images

Remember that with this sort of connection (localhost (QEMU Usermode)), the storage will be into the /var/lib/libvirt/images directory, but the configuration and log files will reside in the user home.

javi@javi-kubuntu:~$ tree .libvirt/
.libvirt/
├── qemu
│   ├── cache
│   ├── lib
│   ├── log
│   │   └── Ubuntu.log
│   ├── run
│   ├── save
│   ├── snapshot
│   └── Ubuntu.xml
└── storage
├── autostart
│   └── default.xml -> /home/javi/.libvirt/storage/default.xml
└── default.xml

During four articles, we have learnt how to make virtual machines with KVM, libvirt and Virtual Machine Manager. I still have to present several articles related to snapshots, access to hypervisors running on remote machines, types of virtual networks, etc.

KVM is a great option to handle virtual machines. It would be a good idea to carry out some tests in order to meter the performance of different types of hypervisors, such as Xen or VMware.

Invalid or incomplete multibyte or wide character

2010-11-29T10:52:00.002+01:00

The other day I downloaded a video with JDownloader and when I tried to move it into my usb memory, the system did not recognize the file and it could not move it.

The error message which Linux returned was "Caracter multibyte amplio inválido ó incompleto" (I have configured my system in Spanish), "Invalid or incomplete multibyte or wide character" in English.

javi@javi-kubuntu:~$ mv .jdownloader/downloads/TICs_espa�olas.avi /media/disk-2/
mv: no se puede efectuar «stat» sobre «/media/disk-2/TICs_espa\244olas.avi»: Caracter multibyte amplio inválido ó incompleto

This is a serious problem, because in addition to not be able to copy or move the file, you can neither remove it.

The issue is that file is encoded with ISO-8859-15, and my system is configured as UTF-8. The solution consists in converting the file name encoding to UTF-8. In order to carry out this operation, we can use the convmv utility.

javi@javi-kubuntu:~$ sudo aptitude install convmv

javi@javi-kubuntu:~/.jdownloader/downloads$ convmv -f iso-8859-15 -t utf-8 TICs_espa�olas.avi
Your Perl version has fleas #37757 #49830                                                                                                                                                                                                                                
Starting a dry run without changes...                                                                                                                                                                                                                                    
mv "./TICs_espa�olas.avi"    "./TICs_espa€olas.avi"                                                                
No changes to your files done. Use --notest to finally rename the files.

The previous output shows us the final file name after the conversion. So as to apply the changes permanently, you have to add the --notest argument.

javi@javi-kubuntu:~/.jdownloader/downloads$ convmv --notest -f iso-8859-15 -t utf-8 TICs_espa�olas.avi
Your Perl version has fleas #37757 #49830                                                                                                                                                                                                                                  
mv "./TICs_�olas.avi"    "./TICs_€olas.avi"                                                                
Ready!

KVM virtualization on Ubuntu (III)

2010-11-22T11:56:00.004+01:00

Let's continue exploring more stuff about KVM virtualization on Ubuntu (remember that there have been posted two previous articles, I and II). We will start with an interesting characteristic: cloning existing virtual machine images with identical virtual hardware configurations.

This is a typical operation because it is normal that we have to install several times the same operating system for different functions. For example, we can need three servers running under a Linux distribution: mail, web and data storage. For this purpose, first we would make a virtual machine with that Linux OS, then we would clone that virtual machine twice and finally, we would install the rest of services on the base systems.

The following order generates a new virtual machine named cloned_BT, from BackTrack4 domain. It is necessary to specify the hypervisor by means of the --connect parameter and to have turned the virtual machine off.

javi@javi-kubuntu:~$ virt-clone --connect qemu:///system --original BackTrack4 --auto-clone --name cloned_BT

We can also perform this task through the Virtual Machine Manager.

Other interesting feature is related to the possibility of saving a running domain into a state file, with the aim of being able to be restored later. When we execute this operation, the domain will be automatically switched off.

javi@javi-kubuntu:~$ virsh save BackTrack4 BT4.save

javi@javi-kubuntu:~$ virsh restore BT4.save

We can use for example this characteristic to back up our virtual machine before a critical task, such as a system update. Therefore if we want to restore a domain from a state file, we will add the restore option (the virtual machine must be turned off).

And finally, I will present another useful tool in order to show the state of the virtualized domains. This utility is similar to top.

javi@javi-kubuntu:~$ sudo aptitude install virt-top

javi@javi-kubuntu:~$ virt-top --connect qemu:///system
virt-top 18:01:14 - x86_64 2/2CPU 2000MHz 3961MB
2 domains, 2 active, 2 running, 0 sleeping, 0 paused, 0 inactive D:0 O:0 X:0
CPU: 1,0%  Mem: 1024 MB (1024 MB by guests)

 ID S RDRQ WRRQ RXBY TXBY %CPU %MEM    TIME   NAME                                                                                                                                                                                                                         
  1 R    0    0   52    0  0,5 12,0   0:41.37 BackTrack4
  2 R    0    0   52    0  0,5 12,0   0:30.69 UbuntuServer_10.10

We can also get certain information about a concrete virtual machine selecting the Performance console, in the Virtual Machine Manager.

KVM virtualization on Ubuntu (II)

2010-11-13T12:28:00.004+01:00

In the previous article about KVM virtualization on Ubuntu (I), I exposed how to set up virtual machines with KVM. Now we will learn how to handle them, either through the command line or by means of the Virtual Machine Manager graphical tool.

First of all, we must know where are located the most important files used by KVM (configuration, virtual images and log files). We can see that the characteristics of the virtual machines are stored into XML files.

javi@javi-kubuntu:~$ ls -lh /etc/libvirt/qemu/
total 12K
-rw------- 1 root root 2,0K 2010-10-12 23:00 BackTrack4.xml
drwxr-xr-x 3 root root 4,0K 2010-10-12 18:22 networks
-rw------- 1 root root 2,0K 2010-11-06 16:08 UbuntuServer_10.10.xml

javi@javi-kubuntu:~$ ls -lh /var/lib/libvirt/images/
total 21G
-rw------- 1 root root  12G 2010-10-12 23:02 BackTrack4.img
-rw------- 1 root root 8,0G 2010-11-06 17:04 UbuntuServer_10.10.img

javi@javi-kubuntu:~$ ls -lh /var/log/libvirt/qemu/
total 12K
-rw------- 1 root root    0 2010-10-13 21:38 BackTrack4.log
-rw------- 1 root root 2,3K 2010-11-06 17:04 UbuntuServer_10.10.log

It is also important to know the basic information about the hardware (number and type of CPU, size of the physical memory, etc.) where we will create the virtual machines.

javi@javi-kubuntu:~$ virsh nodeinfo
Modelo del CPU:      x86_64
CPU(s):              2
Frecuencia de CPU:   2000 MHz
Zócalo(s) de CPU:   1
Núcleo(s) por ranura: 2
Hilo(s) por núcleo: 1
Celda(s) NUMA:       1
Tamaño de memoria:  4056208 kB

Remember that in KVM, virtual machines are also known as domains. In order to list the state (running, idle, paused, shutdown, shut off, crashed or dying) of all existing domains, we must run the following order:

javi@javi-kubuntu:~$ virsh list --all
Id Nombre               Estado
----------------------------------
- BackTrack4           apagado
- UbuntuServer_10.10   apagado

If we want to edit the features (processor, memory, boot options, disk, NICs, monitor, etc.) of a concrete domain or add new virtual hardware components (storage, network, graphics, serial, parallel, watchdog, etc.), we can use Virtual Machine Manager.

Other way to modify the details of a virtual machine can be by means of the virsh command. This option is equivalent to output the domain information as an XML dump, edit that dump file, validate the XML file and define the domain from the XML file.

javi@javi-kubuntu:~$ virsh edit BackTrack4

javi@javi-kubuntu:~$ virsh dumpxml BackTrack4 > domain.xml
javi@javi-kubuntu:~$ vi domain.xml
javi@javi-kubuntu:~$ virt-xml-validate domain.xml
javi@javi-kubuntu:~$ virsh define domain.xml

Then we are going to present the necessary commands to start an inactive domain, suspend a running virtual machine in memory, move it out of the suspended state (resume), reboot it and shut it down.

javi@javi-kubuntu:~$ virsh start UbuntuServer_10.10

javi@javi-kubuntu:~$ virsh suspend UbuntuServer_10.10

javi@javi-kubuntu:~$ virsh resume UbuntuServer_10.10

javi@javi-kubuntu:~$ virsh reboot UbuntuServer_10.10

javi@javi-kubuntu:~$ virsh shutdown UbuntuServer_10.10

Other interesting option for a specific domain is dominfo, which returns basic information about the virtual machine.

javi@javi-kubuntu:~$ virsh dominfo BackTrack4
Id:             -
Nombre:         BackTrack4
UUID:           cc951529-f630-b8ea-1da8-9a75e382190d
Tipo de sistema operatuvo: hvm
Estado:         apagado
CPU(s):         1
Memoria máxima: 524288 kB
Memoria utilizada: 524288 kB
Persistente:    sí
Autoinicio:     desactivar
Modelo de seguridad: apparmor
DOI de seguridad: 0

And finally, if we want the domain to be automatically started at boot, we must run the next order. If we want to disable this option, we must add the --disable parameter.

javi@javi-kubuntu:~$ virsh autostart BackTrack4

javi@javi-kubuntu:~$ virsh autostart --disable BackTrack4

KVM virtualization on Ubuntu (I)

2010-11-06T10:34:00.011+01:00

I do not like too much Oracle, I have to recognize it, particulary its business model. For that reason, I think that I have to try (whenever I can) to use other alternative products. The future of certain applications as OpenOffice, MySQL, VirtualBox is not clear...

In this article we are going to relate how to set up virtual machines on Ubuntu with KVM. I think that KVM is the future of the virtualization in the open source world, especially because it is a technology developed by Red Hat and this company is betting very hard for it.

First of all, we have to make sure whether our hardware supports virtualization (I am going to carry out the testing on Kubuntu 10.10 64 bits). For this purpose, we must check out if the vmx or svm flags appear into the cupinfo file.

javi@javi-kubuntu:~$ cat /proc/cpuinfo | egrep "(vmx | svm)" | wc -l
2

In my case, the result is two (my processor has got two cores). Then, we have to install the necessary packages:

javi@javi-kubuntu:~$ sudo aptitude install kvm libvirt-bin virt-manager

Now, we can see a new virtual network interface (virbr0) and a new group (libvirtd) in the system. My user name has also been added to the libvirtd group.

javi@javi-kubuntu:~$ ip a
...
19: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 0e:af:bc:3a:c8:6a brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
inet6 fe80::caf:bcff:fe3a:c86a/64 scope link
valid_lft forever preferred_lft forever

javi@javi-kubuntu:~$ cat /etc/group
...
libvirtd:x:124:javi

We have to restart the sesson for the user changes take effect. In order to verify that everything is right, we can type the following command:

javi@javi-kubuntu:~$ virsh -c qemu:///system list
Id Nombre               Estado
-------------------------------

Now we can create our first virtual machine opening the Virtual Machine Manager (Applications, System, Virtual Machine Manager), picking out localhost (QEMU) - Not Connected option and making double click on it (this is the hypervisor to which we connect - qemu:///system). Then we must press on Create a new virtual machine button and complete the different steps of the wizard.

Through the wizard, we will have to choose the features or resources for the virtual machine, such as the name, installation source (ISO, CD-ROM, network, PXE, importing the image, etc.), type and version of the operating system, amount of memory and CPUs, size of the virtual hard disk, etc.

In the last step, we can see a little summary about the selected characteristics and besides, pick out the type of virtualization (kvm, qemu - we will choose kvm), the sort of architecture (x86_64 or i686 - depends on the operating system to be installed) and the kind of network (NAT by default). In future articles, I will explain how to configure different types of virtual networks.

During the virtual store step, we can select a tab about allocating all the disk size now... In my personal opinion, we must mark this option because the virtual machine performance will be better if all the disk space is reserved from the beginning.

At the end of this process, we will have a screen with the new virtual machine embedded within it and we will be able to performe several operations on it: run, pause, shut down, clone, migrate, etc.