MySQL benchmark with SysBench (II)

Let's carry on the preceding article about MySQL benchmark with SysBench by putting into action the above scenario.

For the first case, I am going to use the values recommended through the first four articles that I wrote about MySQL optimization. So according to them, the values used will be as follows. Let's call this first settings A.

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
key_buffer              = 32M
thread_cache_size       = 512
table_cache             = 512
table_definition_cache  = 512
open_files_limit        = 1024
tmp_table_size          = 64M
max_heap_table_size     = 32M
query_cache_limit       = 4M
query_cache_size        = 256M
query_cache_type        = 1
innodb_buffer_pool_size = 1280M

For the second configuration, called B, I am going to set by means of the parameter innodb-flush-log-at-trx-commit, when the log buffer will be written out to the log file and the flush to disk operation will be effected.

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
innodb-flush-log-at-trx-commit = 0

For the third combination (C), a new parameter will be added: innodb_buffer_pool_instances. With this option, it is possible to define the number of regions that the InnoDB area will be divided into.

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
innodb_buffer_pool_instances = 2

And finally, the size for the log file will be increased by means of innodb_log_file_size, and in this way, the graph D will be turned out (in order to be able to apply this change, first up you have to stop MySQL and remove any existing log file).

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
innodb_log_file_size = 256M

root@ubuntu-server:~# rm -rf /var/lib/mysql/ib_logfile*

Below you can observe the graphs generated for the different cases.

As you can appreciate in the figure, the best improvements are achieved when the size of the log file is increased from its default value (5M) to 256M. The option C, that is, to separate the InnoDB buffer into two regions gets worse in respect of the previous alternative (B).

MySQL benchmark with SysBench (I)

By taking advantage of the previous article about MySQL optimization, I am going to introduce a handy tool called SysBench and aimed at measuring the performance of a MySQL database among other things. In addition, it is also able to evaluate the I/O, scheduler and threads implementation performance, and memory allocation and transfer speed.

So I am going to use this tool in order to verify the improvements commented in the preceding article and related to some parameters of MySQL. The test will be run on Ubuntu Server 12.10 virtualized through VMware. The virtual machine will made up by a 6 GB hard drive, 2 GB of RAM and a couple of virtual cores.

First of all, let's install MySQL and SysBench and increase the default number of maximum connections to 512.

root@ubuntu-server:~# aptitude install mysql-server sysbench

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
max_connections = 512

root@ubuntu-server:~# service mysql restart

Now let's create a table of 1.000.000 of rows in a database called test by using SysBench.

root@ubuntu-server:~# sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=xxxxxx prepare

Then a straightforward script taken care of running the tests will be developed. This bash script executes the OLTP (OnLine Transaction Processing) test on a table of 1.000.000 of rows. The time limit for the whole execution will be 300 seconds and read, update, delete and insert queries will be performed. The total number of maximum requests will be unlimited.

root@ubuntu-server:~# cat script.sh 
#!/bin/bash

for i in 8 16 32 64 128 256 512
do
    service mysql restart ; sleep 5
    sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=xxxxxx --max-time=300 --oltp-read-only=off --max-requests=0 --num-threads=$i run
done

As you can see in the above script, the number of worked threads to be created will be different in each loop iteration, from 8 to 512. So the idea is to run the script with various MySQL combinations and calculate the number of transactions per second.

Zabbix poller processes more than 75% busy and queue delay (III)

Let's complete the last article about Zabbix poller processes more than 75% busy and queue delay. In this section, I am going to tackle the part of the client, that is, those things which can be modified on the agent so as to remove or attenuate the issues mentioned in the first article.

Remember that this is the continuation of the two previous articles:

• Zabbix poller processes more than 75% busy and queue delay (I)
• Zabbix poller processes more than 75% busy and queue delay (II)

First up, I changed the number of pre-forked instances of the Zabbix client which process passive checks (StartAgents) to 64. This parameter is really meaningful, because its default value is 5, that is to say, only five processes will be started in order to obtain the data requested by the server. So if you have a lot of items and a small monitoring period (as my case), you will need more processes to be able to attend all requests.

root@zabbix-client:~# cat /etc/zabbix/zabbix_agentd.conf
...
StartAgents=64

So let's see now in the graphs, how this change impacts on the results. Let's first with the Zabbix server performance.

And then, the Zabbix data gathering process.

As you can see on the first picture, the server has gone from a Zabbix queue of 30 to 0 (although you can observe 5 on the figure, think that the graph has been cut out). And on the second one, the Zabbix busy poller processes went from 24% to 0%.

Other parameters that you can play with are the number of seconds that the data can be stored in the buffer and its maximum number of values.

root@zabbix-client:~# cat /etc/zabbix/zabbix_agentd.conf
...
BufferSend=3600

BufferSize=65535

Also keep in mind that you should have a small value for the timeout (I am using five seconds on my installation).

Lastly, in order to solve the problem that I mentioned in the first article about from time to time, the processes break down and the zabbix agent is stopped, I developed a simple bash script to work around this issue.

root@zabbix-client:~# tail -f /var/log/zabbix/zabbix_agentd.log
...
zabbix_agentd [17271]: [file:'cpustat.c',line:155] lock failed: [22] Invalid argument
 17270:20121015:092010.216 One child process died (PID:17271,exitcode/signal:255). Exiting ...
...
 17270:20121015:092012.216 Zabbix Agent stopped. Zabbix 2.0.3 (revision 30485).


root@zabbix-client:~# cat /etc/zabbix/monitor_zabbix.sh
#!/bin/bash

while [ 1 ];
do
        if ! pgrep -f "/usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf" &> /dev/null ; then
                /etc/zabbix/zabbix.sh start
        fi
        sleep 15
done

This script is run in batch mode and takes care of monitoring the status of the agent processes and starting over when they drop . It uses another bash script to start and stop the agents.

root@zabbix-client:~# cat /etc/zabbix/zabbix.sh
#!/bin/bash

case $1 in
        "start")
                taskset -c $(($(cat /proc/cpuinfo | grep processor | wc -l) - 1)) /usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf;;
        "stop")
                pkill -f "/usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf";;
        *)
                printf "./zabbix.sh start|stop\n\n"
esac

Zabbix poller processes more than 75% busy and queue delay (II)

After putting forward the issues turned up on my current Zabbix installation and related to its performance (Zabbix poller processes more than 75% busy and queue delay), I am going to explain to you how I solved it.

First of all, I tried out to increase the number of pre-forked instances of pollers for the Zabbix server, that is, I changed its default value from 5 to 256 (remember that for that case, you have to set the the number of maximum connections in MySQL - max_connections - higher than 256, since every single poller opens a dedicated connection to the database).

root@zabbix-server:~# cat /etc/zabbix/zabbix_server.conf
...
# StartPollers=5
StartPollers=256

root@zabbix-server:~# cat /etc/mysql/my.cnf
...
max_connextions = 512

Below you can see the outcome after applying it (Zabbix server performance).

And the Zabbix data gathering process.

In the first figure, you can observe that the Zabbix queue has gone from 48 to 30 (approximately), and for the second one, the Zabbix busy poller processes went from 100% to 24%. So it is clear that if you have a server with enough resources, there is no problem to start many pollers. These kind of processes are responsible for requesting the data defined in the items, so the more pollers have available, the less overloaded the system is.

Other Zabbix server parameter that you ought to take into account is for example the Timeout (specifies how log pollers wait for agent responses). Try not to assign a very high value. Otherwise, the system might get overloaded.

Next week, I will end up this series of articles by accomplishing the part of the client.

sysstat vs top vs ps (III)

Let's finish the series of articles related to the differences in the measure of sysstat, top and ps.

What about the system CPU time (%sy) of the first top from the previous article? It is 39.7%. It is right as well. You have to take into account that, because this server has a couple of cores, that data represents the average from both cores. You can see this point by running top into interactive mode and pressing the number 1. Then you will be able to obtain the consumption for both cores.

root@ubuntu-server:~# top
top - 20:04:37 up 47 min,  1 user,  load average: 0.40, 0.54, 0.60
Tasks:  87 total,   1 running,  86 sleeping,   0 stopped,   0 zombie
Cpu0  :  2.9%us, 37.7%sy,  0.0%ni, 59.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  1.6%us, 40.8%sy,  0.0%ni, 57.1%id,  0.0%wa,  0.0%hi,  0.5%si,  0.0%st
Mem:   1024800k total,   207680k used,   817120k free,    23532k buffers
Swap:  1048572k total,        0k used,  1048572k free,   100320k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1561 root      20   0 29280 4180 2416 S  128  0.4  56:57.23 script.py

At any rate, the sum of both percentages does not match the %CPU used by the script. This might be due to the sampling frequencies commented in the preceding article.

Finally, I would like to remark that in order to monitor threads, I like to use pidstat with the -t parameter.

root@ubuntu-server:~# pidstat -u -t -p 1561 1 1
Linux 3.2.0-30-generic-pae (ubuntu-server)     09/30/2012     _i686_    (2 CPU)

08:14:47 PM      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
08:14:48 PM      1561         -    9.00  119.00    0.00  128.00     1  script.py
08:14:48 PM         -      1561    0.00    0.00    0.00    0.00     1  |__script.py
08:14:48 PM         -      1562    4.00   60.00    0.00   64.00     0  |__script.py
08:14:48 PM         -      1563    4.00   59.00    0.00   63.00     1  |__script.py

This tool is great because you can see the core where the thread (or process) is being executed.

Another interesting choice is to use directly the ps command with the suitable parameters (-L shows threads and sgi_p the processor where the process is currently executing on).

root@ubuntu-server:~# ps -Leo pid,pcpu,sgi_p,command | grep '^ 1561'
 1561  0.0 * /usr/bin/python ./script.py
 1561 65.6 0 /usr/bin/python ./script.py
 1561 65.2 0 /usr/bin/python ./script.py

Also point out for the above output that, ps does not offer the number of TID (Thread ID), in contradistinction to top and pidstat (remember that a thread does not have PID).

Zabbix poller processes more than 75% busy and queue delay (I)

In my previous job, I had to set up a Zabbix infrastructure in order to monitor more than 400 devices between switches and servers. The main feature of this architecture was that there were a lot of machines, but the update interval was large (around 30 seconds) and the number of items small.

For this purpose, I wrote down a couple of articles related to this issue:

• Tuning Zabbix to improve its performance (I)
• Tuning Zabbix to improve its performance (II)

But in my current position, I am starting to introduce Zabbix (2.0.3 on Ubuntu Server 12.04) with the aim of controlling few devices where a large number of items and a small monitoring period are required. This situation leads to an overload of the Zabbix server, on the one hand by increasing the number of monitored elements delayed in the queue, and on the other, turning out that the poller processes are busy long.

In addition, I have been able to observe that, from time to time, the agent goes down in an unexpected way. If you take a look at the log file from the client (debug mode), the following error lines are dumped.

root@zabbix-client:~# tail -f /var/log/zabbix/zabbix_agentd.log
...
zabbix_agentd [17271]: [file:'cpustat.c',line:155] lock failed: [22] Invalid argument
 17270:20121015:092010.216 One child process died (PID:17271,exitcode/signal:255). Exiting ...
 17270:20121015:092010.216 zbx_on_exit() called
 17272:20121015:092010.216 Got signal [signal:15(SIGTERM),sender_pid:17270,sender_uid:0,reason:0]. Exiting ...
 17273:20121015:092010.216 Got signal [signal:15(SIGTERM),sender_pid:17270,sender_uid:0,reason:0]. Exiting ...
 17274:20121015:092010.216 Got signal [signal:15(SIGTERM),sender_pid:17270,sender_uid:0,reason:0]. Exiting ...
 17270:20121015:092012.216 Zabbix Agent stopped. Zabbix 2.0.3 (revision 30485).

Below you can observe a figure which shows the Zabbix server performance (queue) for the aforementioned case.

And the other one, reflects the Zabbix data gathering process (pay attention to the data Zabbix busy poller processes, in %).

For the first case, the Zabbix queue has averaged more than 50 monitored items delayed, and for the second one, the poller processes are busy about 100% of the time. This situation can produce that, sometimes, Zabbix draws sporadic dots rather than lines in the graphs. Another effect that you can get from this condition is that if you set a short update interval for an item, you could run into lack of data when you check the values gathered later.

Also say that I followed the tuning guide that I mentioned before, but as you can see, Zabbix server was acting up.

sysstat vs top vs ps (II)

Following up on the previous article, sysstat vs top vs ps (I), a curious case that I would like to talk about is when you use more than one core. Let's create a simple script in Python which runs a couple of threads a little bit overloaded.

root@ubuntu-server:~# nproc 
2

root@ubuntu-server:~# cat script.py 
#!/usr/bin/python

import threading, time

def sleep():
    while True:
        time.sleep(0.000001)

t1 = threading.Thread(target=sleep)
t2 = threading.Thread(target=sleep)

t1.start()
t2.start()

t1.join()
t2.join()

root@ubuntu-server:~# ./script.py &
[1] 1561

If we take a look now at the status of this process by means of top, we can see as follows.

root@ubuntu-server:~# top -b -n 1 -p 1561
top - 19:41:19 up 24 min,  1 user,  load average: 0.74, 0.66, 0.48
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.3%us, 39.7%sy,  0.0%ni, 55.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1024800k total,   205596k used,   819204k free,    23192k buffers
Swap:  1048572k total,        0k used,  1048572k free,    98712k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1561 root      20   0 29280 4180 2416 S  129  0.4  26:23.26 script.py

So what would be the first weird thing that you can observe from the previous screen? The script is consuming 129% of the CPU. This is right because you have to remember that this virtual machine has two cores and the script, rather, its two threads, are using the two ones and that figure is the combination of the CPU utilization from both cores. You can appreciate this situation much better if you execute top with the -H option.

root@ubuntu-server:~# top -b -H -n 1 -p 1561
top - 19:44:22 up 27 min,  1 user,  load average: 0.69, 0.72, 0.54
Tasks:   3 total,   2 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.3%us, 38.9%sy,  0.0%ni, 56.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1024800k total,   205624k used,   819176k free,    23192k buffers
Swap:  1048572k total,        0k used,  1048572k free,    98712k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1563 root      20   0 29280 4180 2416 R   65  0.4  15:10.19 script.py
 1562 root      20   0 29280 4180 2416 R   64  0.4  15:12.39 script.py
 1561 root      20   0 29280 4180 2416 S    0  0.4   0:00.05 script.py

sysstat vs top vs ps (I)

I have always been using several tools to get the CPU utilization of Linux processes through different tools such as top, ps, sar, etc., but so far, I did not realise that the results obtained from them can vary considerably.

For example, I am going to run a job and measure its CPU usage afterwards. Also say that a virtual machine (Ubuntu Server 12.04) with only one core will be used.

root@ubuntu-server:~# nproc 
1

root@ubuntu-server:~# while [ 1 ] ; do sleep 0.0001 ; done &
[1] 5605

Now let's show its performance by means of top, ps and pidstat (this command belongs to the sysstat package, which also provides the sar utility, used to collect, report, or save system activity information).

root@ubuntu-server:~# top -b -n 1 -p 5605 | grep bash | awk '{print $9}'
37.9

root@ubuntu-server:~# ps -eo pid,pcpu | grep '^ 5605' | awk '{print $2}'
37.8

root@ubuntu-server:~# pidstat -u -p 5605 1 1 | grep bash | head -n 1 | awk '{print $7}'
45.12

If you go over the definitions of these measures, you can read as follows:

• top (%CPU): the task's share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time.

• ps (%CPU): cpu utilization of the process in "##.#" format. Currently, it is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage.

• pidstat (%CPU): total percentage of CPU time used by the task. In an SMP environment, the task's CPU usage will be divided by the total number of CPU's if option -I has been entered on the command line.

What is my opinion? All data turned out try to display the CPU utilization of a process during a period of time, but the key is that period of time taken to work out the result, and I think that for pidstat is different that for top and ps.

So my conclusion is that all aforementioned tools are correctly valid, and they will give back you a correct idea about the behaviour of a process in terms of CPU.

Apache performance tuning: benchmarking

Let's continue with the series of articles about Apache performance tuning. Remember that in the previous ones, I approached the topics about dynamic modules (I and II) and directives (I and II).

So after reviewing the directives which allow to fit the global configuration of Apache, I am going to focus on a couple of tools used to adjust the values of those directives, based on the available hardware and the kind of service which wants to be offered.

The first application to be presented is ab (Apache benchmark), normally utilized in order to get hold of the performance values which the web server will be able to achieve. It is designed to give an impression of how your current Apache server works, specially by showing how many requests per second which is able to serve.

This tool puts forward multiple options, but the normal operation format so as to carry out performance tests will be as follows (there are other many useful options that you can check by going to the man):

[root@localhost ~]# ab -c concurrency -n request [-k] [http[s]://]hostname[:port]/path

Through the '-c' option is specified the number of multiple requests to perform at once, that is to say, it stands for the concurrence whereby the web server will be requested. For example, '-c 100' means that one hundred users will be launching requests simultaneously.

By means of the '-n' option, you may establish the number of requests to perform for the benchmarking session, that is, the total number of requests which will be run. For instance, '-c 20 -n 1000' means that 50 groups of 20 requests per group will be executed. And with the '-k' option, the HTTP keepalive feature is enabled. In this way, it is possible to handle multiple requests within one HTTP session.

[root@localhost ~]# ab -c 200 -n 1000 http://localhost/index.html
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Apache/2.2.15
Server Hostname:        localhost
Server Port:            80

Document Path:          /index.html
Document Length:        1656 bytes

Concurrency Level:      200
Time taken for tests:   1.696 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      1934625 bytes
HTML transferred:       1664280 bytes
Requests per second:    589.67 [#/sec] (mean)
Time per request:       339.175 [ms] (mean)
Time per request:       1.696 [ms] (mean, across all concurrent requests)
Transfer rate:          1114.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    8  10.0      4      38
Processing:    14  163 353.1     85    1645
Waiting:       11  160 353.2     81    1638
Total:         51  171 354.7     89    1660

Percentage of the requests served within a certain time (ms)
  50%     89
  66%     92
  75%     94
  80%     95
  90%    102
  95%   1637
  98%   1645
  99%   1651
 100%   1660 (longest request)

The most important values output by ab are Failed requests, Requests per second and Time per request. As you can distinguish in the above example, all requests have been completed successfully, with an average of 589 requests per second and an average time of 1.696 ms for each request.

Another interesting tool used to assess the performance of a web server is curl, which takes care of finding out the response time for an unique request of a single element. Curl transfers data from or to a server, by using one of the supported protocols (HTTP, HTTPS, FTP, etc.).

[root@localhost ~]# curl -o /dev/null -s -w %{time_connect}:%{time_starttransfer}:%{time_total} http://localhost/index.html
0.002:0.003:0.004

In the preceding example, the time taken to process the request and start sending data has been 0.002 - 0.001 = 0.001 sg, and the time for sending 0.004 - 0.003 = 0.001 sg.

Apache performance tuning: directives (II)

Let's carry on with the second part of the article Apache performance tuning: directives (I).

If you take a look at the previous output, you may identify that the first process is the parent and the other ones, its children. The most important column is RSS (Resident Set Size), whereby you can see that the size of memory taken up by a child process is around 2 MB. Thus, you might work out the maximum number of Apache processes which can be simultaneously running on your system:

MaxClients = 338 MB / 2 MB = 169

This value is relative and never should be taken into account exactly. You have to consider that at any given time, other applications running on the system (mail, FTP, database, etc.) can ask for free memory. So in the preceding case, it would be reasonable to grab 200 MB instead of 338 MB. Furthermore, you must think that is not the same task to serve dynamic than static content.

If all processes are busy, Apache launches a series of spare processes (MinSpareServers). If a given time many processes are idle, these are removed with the aim of not exceeding a maximum number of spare processes (MaxSpareServers). In general, it must be satisfied that MinSpareServers ≤ StartServer ≤ MaxSpareServers, and MaxSpareServers should be 50% higher than MinSpareServers.

When a process has handled too many requests (MaxRequestsPerChild), this is deleted and another one is created again. An optimum value for MaxRequestsPerChild is 4000, although it is appropriate to fit it based on the data served.

What happens if a connection does not respond or keeps inactive indefinitely? A connection is another resource for the system and as such, must be freed up in case of not being used. The default timeout is usually very high (300 seconds). In order to enhance the server performance is useful to turn down this value to 20 (nowadays, the speed of the networks is huge, and it does not make sense to set such a high Timeout).

Lastly, we are going to treat an important issue for a web server: the persistence. When a web page is loaded, it is normal that the web browser establishes an only TCP connection and downloads all web elements, by carrying out multiple requests over the same connection.

So as to manage this feature, Apache provides three different parameters in its configuration file.

• KeepAlive: enables (On) or disables (Off) persistent connections.


• MaxKeepAliveRequests: maximum number of requests allowed on a persistent connection.


• KeepAliveTimeout: amount of time the server will keep an inactive persistent connection before closing it.

It is recommended to use a very high value for the KeepAliveTimeout parameter (2 or 3 seconds). In respect of MaxKeepAliveRequests, will depend on the amount of content that web pages have on average, but it should not usually exceed 1000.

Apache performance tuning: directives (I)

In this article about Apache performance tuning, I am going to deal with some Apache directives. Remember that in the previous articles (I and II), I focused on the Apache modules, and we learnt on the one hand, that we can reduce the memory used by each process by removing those modules which are not necessary, and in this way, they will be lighter. And on the other, if we take those unused modules out, we will have less software exposed to possible attacks.

One of the most important tasks when we are setting Apache up is to fit its directives, which allow to control the overall running of the web server. These directives always depend on the available hardware, as well as the type of content that you want to serve. They are located inside the httpd.conf file; bellow, I will show you the most meaningful directives.

• StartServers: number of child server processes created on startup. This value is related to the initial HTTP load of the machine, whether there are another configured services, and so on. For example, if the computer only works as a web server, there is no problem to start with 50 processes, but if there are other running services such as FTP, mail, database, etc., and in addition, it runs low on computational resources, we should start with a small number of active processes, and afterwards, on real time, Apache itself would have to launch the needed ones.


• MaxClients: maximum number of simultaneous processes that can be running at the same time.


• ServerLimit: maximum number of processes for the MaxClients parameter.


• MinSpareServers: minimum number of idle child server processes.


• MaxSpareServers: maximum number of idle child server processes.


• MaxRequestsPerChild: maximum number of requests that an individual child server will handle during its life.


• Timeout: amount of time the server will keep an inactive connection before closing it.

When Apache is started, several processes are created (StartServers) in order to listen for potential connections. When a request reaches the server, this is attended by a child process, and then, the process switches to idle state, waiting for new connections. If at a given time the number of requests is greater than the number of available processes (MaxClients), these requests are queued. Must be satisfied that MaxClients ≤ ServerLimit.

We can work out the maximum number of httpd processes that our system can run based on the available memory. You have to divide the free memory of the server by the amount of memory taken by an unique Apache process.

So as to calculate the free memory of the system, you can use the following command.

[root@centos ~]# cat /proc/meminfo | grep MemFree
MemFree:          346492 kB

And in order to find out the amount of memory grabbed by an Apache process, you may run the next order.

[root@centos ~]# ps -ylC httpd
S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
S     0  1870     1  0  80   0  3404  2792 -      ?        00:00:00 httpd
S    48  1873  1870  0  80   0  2072  2792 -      ?        00:00:00 httpd
S    48  1874  1870  0  80   0  2072  2792 -      ?        00:00:00 httpd
S    48  1875  1870  0  80   0  2072  2792 -      ?        00:00:00 httpd

Apache performance tuning: dynamic modules (II)

Let's continue with the second part of the article titled Apache Performance tuning: dynamic modules (I). Remember that this paper is aimed at reviewing the different modules belonging to Apache, so as to determine whether they are useful for our requirements. To that end, we will be able to fit the amount of memory used by Apache processes.

The most important point is to be aware of that one only process consumes little memory, but if our Apache installation requires lots of processes, the total memory grabbed by Apache will be huge. So if we get hold of turning down the initial memory with which a process is created, afterwards it will run lighter and besides, we will have that free memory available in order to be allocated for other things.

mod_ext_filter

Forwards the response body to an external program before sending it out to the client.

# LoadModule ext_filter_module modules/mod_ext_filter.so

mod_include

Filters files before delivering them to the client.

# LoadModule include_module modules/mod_include.so

mod_info

Provides a comprehensive overview of the web server configuration.

# LoadModule info_module modules/mod_info.so

mod_ldap

Improves the performance of websites by pooling LDAP connections and caching responses.

# LoadModule ldap_module modules/mod_ldap.so

mod_logio

Logs the input and output number of bytes received/sent per request.

# LoadModule logio_module modules/mod_logio.so

mod_proxy

Puts into action a a proxy/gateway.

# LoadModule proxy_module modules/mod_proxy.so
# LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
# LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
# LoadModule proxy_http_module modules/mod_proxy_http.so
# LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
# LoadModule proxy_connect_module modules/mod_proxy_connect.so

mod speling

Tries to correct erroneous URLs that users could have typed by overlooking capitalization and allowing up to one misspelling.

# LoadModule speling_module modules/mod_speling.so

mod_status

Provides statistics about the activity and performance of the web server.

# LoadModule status_module modules/mod_status.so

mod_suexec

Allows CGI scripts to run as a concrete user and group.

# LoadModule suexec_module modules/mod_suexec.so

mod_userdir

Allows user directories can be accessed through the web server.

# LoadModule userdir_module modules/mod_userdir.so

mod_usertrack

Logs user activity.

# LoadModule usertrack_module modules/mod_usertrack.so

After disabling these modules, the memory used by one Apache process (and owned by apache user) went from 2.02 to 1.46 MB, that is to say, we have gained around 0.6 MB. If you take into account that a large number of processes can be running on the system at any given time, the saved memory might be appreciable. In addition, you have to consider that from now on, a process is much lighter, with what its startup and performance will be much better.

Apache performance tuning: dynamic modules (I)

Apache is a cross-platform, modular and open source web server, widely used around the world for its quality, robustness and stability. But like most of the applications, it is installed with a default configuration which is not the most adequate. And I am going to say more: I have never seen an Apache installation where the administrator has set it up correctly later.

During several articles, you are going to learn how to properly optimize Apache, in order to achieve the best performance. The tests will be carried out on CentOS 6.2 (32 bits) with Apache 2.2.15. I am going to break up this first article relative to dynamic modules in two separate parts.

Apache has got two main operating modes, also known as multi-processing modules (MPMs):

• Prefork: an unique Apache process (httpd) launchs child processes which take care of listening for potential connections and serving them. Apache keeps several idle processes ready to attend incoming requests. Thereby, a client does not need to wait for new children are forked. Another advantage of this operation mode is that if there is a problem in any process, this will not affect other processes (each child is independent of the rest). 

• Worker: as in the previous case, an only control process creates several child processes, and in turn, each child process handles a listener thread which passes the inbound connections to other server threads managed as well by the same child process. This mode is faster and more scalable, but in contrast, it is more fault tolerant (several threads share the same memory area, and if there is any problem in the parent, it will involve the rest).

You can install Apache either by compiling it from its source code or by getting directly the binary file from a repository. I for one prefer this second option, because in this way, any kind of update (security or bugfix) will be able to be applied without compiling it again.

A typical installation of Apache via yum comes with the following pre-compiled modules. As you may appreciate, prefork will be the default operating mode (you can change this by modifying the /etc/sysconfig/httpd file).

[root@centos ~]# httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

It is basic to know the funcionality of each module so as to figure out if it can be left out. Then we are going to put forward what modules can be ruled out in the most of the cases. Also point out that all directives showed below, are included into the Apache configuration file (httpd.conf). In many cases, the related modules will be also disabled, aside from the principal one.

mod_actions

Allows the execution of CGI scripts based on the MIME content type and the request method.

# LoadModule actions_module modules/mod_actions.so

mod_auth_basic

Limits access to certain users by using HTTP Basic Authentication. I usually disable its dependencies.

LoadModule auth_basic_module modules/mod_auth_basic.so
# LoadModule authn_file_module modules/mod_authn_file.so
# LoadModule authn_alias_module modules/mod_authn_alias.so
# LoadModule authn_anon_module modules/mod_authn_anon.so
# LoadModule authn_dbm_module modules/mod_authn_dbm.so
# LoadModule authn_default_module modules/mod_authn_default.so
# LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
# LoadModule authn_dbd_module modules/mod_authn_dbd.so

mod_auth_digest

Limits access to certain users by using MD5 Digest Authentication.

# LoadModule auth_digest_module modules/mod_auth_digest.so

mod_authz_*

Limits access to certain groups based on different origins (DBM or plaintext files, hostname or IP address, etc.). I get used to remove all less mod_authz_host.

LoadModule authz_host_module modules/mod_authz_host.so
# LoadModule authz_user_module modules/mod_authz_user.so
# LoadModule authz_owner_module modules/mod_authz_owner.so
# LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
# LoadModule authz_dbm_module modules/mod_authz_dbm.so
# LoadModule authz_default_module modules/mod_authz_default.so

mod_cache

Manages the content cache.

# LoadModule cache_module modules/mod_cache.so
# LoadModule disk_cache_module modules/mod_disk_cache.so

mod_cgi

Allows the execution of CGI scripts.

# LoadModule cgi_module modules/mod_cgi.so

mod_dav

Implements the WebDAV (Web-based Distributed Authoring and Versioning) funcionality.

# LoadModule dav_module modules/mod_dav.so
# LoadModule dav_fs_module modules/mod_dav_fs.so

mod_env

Controls the internal environment variables which are sent out to CGI scripts and SSI pages.

# LoadModule env_module modules/mod_env.so

Tuning Zabbix to improve its performance (II)

Let's continue with the last article about tuning Zabbix to improve its performance. First of all, I am going to set the suitable kernel parameters into the sysctl.conf file.

root@zbx01:~# cat /etc/sysctl.conf
...
# Maximum percentage of physical memory usage before going to swap
vm.swappiness = 10

# Number of open files for all processes
fs.file-max = 407020

# Minimum, default and maximum size of the send/receive buffer used by each TCP socket
net.ipv4.tcp_wmem = 8192        87380   16777216
net.ipv4.tcp_rmem = 8192        87380   16777216

# Maximum number of queued connection requests which have still not received an ACK (three-way handshake)
net.ipv4.tcp_max_syn_backlog = 2048

# Number of seconds to wait for a final FIN packet before the socket is forcibly closed
net.ipv4.tcp_fin_timeout = 25

# Number of seconds a connection needs to be idle before TCP begins sending out keep-alive probes
net.ipv4.tcp_keepalive_time = 1200

# Maximum TCP send window
net.core.wmem_max = 16777216

# Maximum TCP receive window
net.core.rmem_max = 16777216

# Maximum size in bytes of a message queue
kernel.msgmnb = 65536

# Maximum size for a message text
kernel.msgmax = 65536

# Maximum size in bytes for a shared memory segment
kernel.shmmax = 68719476736

# System wide maximum of shared memory pages
kernel.shmall = 4294967296

Then I am going to fit the values of MySQL by means of its configuration file. This part is really important if you want to achieve a good performance.

In order to adjust them, I have been following the status of the database throughout several weeks, by using tuning tools such as MySQL Performance Tuning Primer Script or MySQLTuner.

root@zbx01:~# cat /etc/mysql/my.cnf
...
# Size of the buffer used for index blocks
key_buffer = 16M

# Maximum size of one packet or any generated/intermediate string
max_allowed_packet = 16M

# Number of threads the server should cache for reuse
thread_cache_size = 64

# Maximum allowed number of simultaneous client connections
max_connections = 256

# Number of open tables for all threads
table_cache = 1024

# Number of table definitions that can be stored in the definition cache
table_definition_cache = 1024

# Do not cache results that are larger than this number of bytes
query_cache_limit = 16M

# Amount of memory allocated for caching query results
query_cache_size = 1024M

# Minimum size (in bytes) for blocks allocated by the query cache
query_cache_min_res_unit = 512

# 0: do not cache
# 1: cache all cacheable query results except for those that begin with SELECT SQL_NO_CACHE
# 2: cache results only for cacheable queries that begin with SELECT SQL_CACHE
query_cache_type = 1

# Slow queries are logged
log_slow_queries = /var/log/mysql/mysql-slow.log

# If a query takes longer than this value (seconds), the server logs the query
long_query_time = 5

# Queries that are expected to retrieve all rows are logged
log-queries-not-using-indexes

# Size in bytes of the memory buffer that InnoDB uses to cache data and indexes of its tables
innodb_buffer_pool_size = 4096M

With respect to MySQL, stand out that it is also important to defragment the query cache to enhance its utilization, by carrying out a "flush query cache" on the database. In my installation, I have seen that the optimum period is every hour.

root@zbx01:~# crontab -e
...
0 */1 * * * mysql -u root -pxxxxxx -e "flush query cache"

And finally, I have changed certain parameters from the Zabbix configuration file. The most important variable is related to the pre-forked pollers.

If this number is not enough, your Zabbix server will not be able to save all monitored data and you will find lack of many values. This is due to if the server runs out of sufficient processes to attend the requests, they will be ruled out and not registered.

root@zbx01:~# cat /etc/zabbix/zabbix_server.conf
...
# Number of pre-forked instances of pollers
StartPollers=96

# Shared memory size for storing hosts and items data
CacheSize=64M

# Shared memory size for storing history data
HistoryCacheSize=8M

# Shared memory size for storing trends data.
TrendCacheSize=8M

# Shared memory size for storing character, text or log history data
HistoryTextCacheSize=8M

Regarding Housekeeping, I have not modified any default parameter. In this way, the housekeeping procedure runs every hour and deletes all unnecessary values into the database.

If you note that your server does not work properly because it is using up lots of resources (CPU, memory, I/O) in this task, you will have to fit these options.

root@zbx01:~# cat /etc/zabbix/zabbix_server.conf
...
# Housekeeping is removing unnecessary information from history, alert, and alarm tables
# HousekeepingFrequency=1

# No more than MaxHousekeeperDelete rows will be deleted per one task in one housekeeping cycle
# MaxHousekeeperDelete=500

# Enable/disable housekeeping
# DisableHousekeeping=0

Tuning Zabbix to improve its performance (I)

I am really looking forward to this article. I think it is going to be really useful for Zabbix administrators.

When you have to control a small group of machines, it is enough to install Zabbix (either from the repositories or the source code) and not modify any parameter. But when the number of monitored machines or items is very large, it is necessary to fit some values related to the operating system, the database and the Zabbix itself. Otherwise it is possible that your system acts up or the performance is not expected.

Bellow you can see the status of my Zabbix server at work (Zabbix 1.8.5 with MySQL 5.1, on Ubuntu 11.04 - 64 bits). I am monitoring around 430 devices, between servers and switches, and you can distinguish that the requeried server performance (new values per second) is really huge: 1687.




This configuration would not be possible with a Zabbix base installation. Also point out the hardware features of the server: 4 vCPUs (2.66 GHz), 8 GB RAM and 254 GB of storage.

First of all, we are going to take a look at several graphics of the server. Let's get started with the memory consumption during a typical day. The figure shows that the average available memory is around 1.73 GB and the system is not swapping.




Regarding the CPU, I have chosen a period of 6 hours so as to explain the concept of Housekeeping in Zabbix. As you can make out in the next chart, the normal use of CPU is about 20-25%, but each hour, there is a strong increment. This situation coincides with a rise of the Input/Output operations.




The Housekeeping is a task run by Zabbix which takes care of removing the unnecessary data of the history, alerts and alarms tables. Taking a look at the zabbix log, you can find out how many records are deleted from the database.

root@zbx01:~# egrep 'housekeeper|Deleted' /var/log/zabbix/zabbix_server.log
1599:20110719:230307.692 Executing housekeeper
1599:20110719:231127.392 Deleted 1522478 records from history and trends
1599:20110720:001127.393 Executing housekeeper
1599:20110720:001927.742 Deleted 1480673 records from history and trends
...

This procedure is configured by means of different parameters into the zabbix_server.conf file.

Through the load average graph, we can also appreciate this issue, where the load average (1 min) reaches maximum increases of 1.30.




And finally, the following graphic represents the status of the Zabbix cache during a week. Its values are rightly suited too.




In the next article, I will teach how to set up correctly the parameters related to the Linux kernel, MySQL and Zabbix.

Benchmarking with Phoronix Test Suite (II)

This is the second and final part of the article Benchmarking with Phoronix Test Suite.

If we want to carry out the test or suite with the default options, we must aggregate the default-run parameter. Also mention that we can type multiple tests or suites with the run or default-run order.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite run iozone

Phoronix Test Suite v3.0.1
IOzone Test Configuration


Record Size:

1: 4Kb
2: 64Kb
3: 1MB
4: Test All Options

Enter Your Choice: 2

File Size:

1: 512MB
2: 2GB
3: 4GB
4: 8GB
5: Test All Options

Enter Your Choice: 1

Disk Test:

1: Write Performance
2: Read Performance
3: Test All Options

Enter Your Choice: 3
Would you like to save these test results (Y/n): Y
Enter a name to save these results: ubuntu-server

Current Test Identifiers:
- apache

Enter a unique name for this test run: iozone
...

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite default-run iozone

Where are the results and the test environments stored on the filesystem?

root@ubuntu-server:~# tree -d .phoronix-test-suite/test-results/
.phoronix-test-suite/test-results/
├── pts-results-viewer
└── ubuntu-server
├── result-graphs
└── system-logs
├── apache
└── iozone

root@ubuntu-server:~# tree -d .phoronix-test-suite/installed-tests/
.phoronix-test-suite/installed-tests/
└── pts
├── apache-1.3.0
└── iozone-1.7.0

So as to get information about the saved test results and the installed tests and their usage, we have the next options.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-saved-results
                                                                                                                                                                                                                                                             
Phoronix Test Suite v3.0.1                                                                                                                                                                                                                                            
1 Saved Results                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                             
Saved Name: ubuntu-server      Title: ubuntu-server                                                                                                                                                                                                                   
- apache                                                                                                                                                                                                                                                      
- iozone

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-test-usage

Phoronix Test Suite v3.0.1
2 Tests Installed

TEST                 VERSION  INSTALL DATE  LAST RUN    AVG RUN-TIME  TIMES RUN
pts/apache-1.3.0   - 1.3.0    2011-04-01    2011-04-01  5m15s         1
pts/iozone-1.7.0   - 1.7.0    2011-04-02    2011-04-02  2m10s         2

And further, if we want to acquire more details about an executed result, the info parameter will be our choice.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite info ubuntu-server

Title: ubuntu-server
Identifier: ubuntu-server

Test Result Identifiers:
- apache
- iozone

Contained Tests:
- Apache Benchmark
- IOzone

Finally, we also have the option to export the results in other formats (csv, text and pdf).

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite result-file-to-text ubuntu-server > ubuntu-server.txt

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite result-file-to-csv ubuntu-server > ubuntu-server.csv

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite result-file-to-pdf ubuntu-server

Saved To: /root/ubuntu-server.pdf

As well, we will always have the possibility to treat directly the results by means of the html structure.

root@ubuntu-server:~# ls -l .phoronix-test-suite/test-results/ubuntu-server/
total 32
-rw-r--r-- 1 root root 3291 2011-04-02 18:00 composite.xml
-rw-r--r-- 1 root root  192 2011-04-02 18:00 index.html
-rw-r--r-- 1 root root 5163 2011-04-02 21:32 pts-results-viewer.xsl
drwxr-xr-x 2 root root 4096 2011-04-02 21:32 result-graphs
drwxr-xr-x 4 root root 4096 2011-04-02 18:00 system-logs
-rw-r--r-- 1 root root 1491 2011-04-02 17:51 test-1.xml
-rw-r--r-- 1 root root 2210 2011-04-02 18:00 test-2.xml

With html or pdf formats, apart from the numeric data, we will be able to appreciate that Phoronix generates several graphics through the results.

Also point out another useful utility: we can merge several results and obtain a new combination from all of them. For this purpose, we first have to copy the other outcomes into the test-results directory.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite merge-results ubuntu-server centos
Merged Results Saved To: /root/.phoronix-test-suite/test-results/merge-8136/composite.xml

The key of this benchmarking application is to apply correctly in each moment the suitable tests. Due to the large variety of available tests, we can get much information. On the contrary, the main problem under my personal opinion is that the options which you can choose to run a test are very limited. If you wanted to gather more information from a specific tool (for instance iozone), you should execute it manually.

But in short, I think that Phoronix Test Suite is a great tool for measuring the perfomance of a system.

Benchmarking with Phoronix Test Suite (I)

The Phoronix Test Suite is an interesting benchmarking platform aimed at testing and measuring the performance of multiple stuff, such as computers, graphics cards, processors, operating systems and so on.

I have used (in my professional work) and talked about this tool several times, but I had never dedicated any article to evaluate it. Let's get going!

For my tests, I am going to utilize an Ubuntu Server 10.10 (64 bits) and Phoronix Test Suite 3.0.1. When you download the application and descompress it, you have two options: on the one hand you can install it over the operating system (install.sh), and on the other, you may directly run the appropiate script (phoronix-test-suite) and carry out all the options that Phoronix provides. Regardless the choice, first of all you must install a serie of necessary dependences.

root@ubuntu-server:~# aptitude install php5-cli php5-gd php5-curl php-fpdf

root@ubuntu-server:~# tar xvzf phoronix-test-suite-3.0.1.tar.gz

root@ubuntu-server:~/phoronix-test-suite# ./install-sh

In my case, I will choose the second. Also say that if we are behind a proxy, we have to execute the script with the network-setup option, in order to set the proxy configuration up. In addition, we can check out the information about our system hardware and software.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite network-setup

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite system-info

Phoronix Test Suite v3.0.1
System Information

Hardware:
Processor: QEMU Virtual 0.12.5 @ 3.00GHz (1 Core), Motherboard: Bochs, Chipset: Red Hat Virtio, Memory: 1 x 2048 MB RAM, Disk: 15GB, Graphics: Cirrus Logic GD 5446

Software:
OS: Ubuntu 10.10, Kernel: 2.6.35-28-server (x86_64), Display Driver: cirrus, File-System: ext4

Phoronix supplies a set of suites (cpu, database, kernel, memory, etc.) that in turn, make up a group of tests (compress-gzip, dbench, sqlite, etc.), also known as profiles. In total, we have around more than 60 suites and 130 tests.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-available-suites

Phoronix Test Suite v3.0.1
Available Suites

pts/audio-encoding               - Audio Encoding                   System
pts/chess                        - Chess Test Suite                 Processor
pts/compilation                  - Timed Code Compilation           Processor
...

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-available-tests

Phoronix Test Suite v3.0.1
Available Tests

pts/aio-stress               - AIO-Stress                          Disk
pts/apache                   - Apache Benchmark                    System
pts/battery-power-usage      - Battery Power Usage                 System
...

If we want to know more details about a concrete suite or test, we can run the following order (in this example, the memory suite consists of three tests).

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite info memory

Phoronix Test Suite v3.0.1
Memory Test Suite

Run Identifier: pts/memory-1.0.2
Suite Version: 1.0.2
Maintainer: Michael Larabel
Suite Type: Memory
Unique Tests: 3
Suite Description: The system memory test suite consists of tests designed to test the computer's system memory (RAM) performance. Among these tests are RAMspeed and Bandwidth.

pts/memory-1.0.2
* pts/ramspeed
* pts/stream
* pts/cachebench

And if we want to install a suite or test, we must add the install parameter to the script. This sentence will firstly resolve and install the necessary dependences and afterwards, download, compile and install the test. If you select a suite, this order will install all tests belonging to the suite.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite install apache

The following dependencies are needed and will be installed:

- build-essential
...
Phoronix Test Suite v3.0.1

To Install: pts/apache-1.3.0
...

Then we are going to see the execution of the apache test (we can also use this command to launch a suite).

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite run apache
Would you like to save these test results (Y/n): Y
Enter a name to save these results: ubuntu-server
Enter a unique name for this test run: apache

Phoronix Test Suite v3.0.1
If you wish, enter a new description below.
Press ENTER to proceed without changes.

Current Description: Running pts/apache-1.3.0.

New Description:


Apache Benchmark:
pts/apache-1.3.0
Test 1 of 1
Expected Trial Run Count: 3
Running Pre-Test Script @ 15:45:48
Started Run 1 @ 15:45:53
Started Run 2 @ 15:47:34
Started Run 3 @ 15:49:17  [Std. Dev: 2.00%]
Running Post-Test Script @ 15:51:00

Test Results:
7156.08
7026.62
6876.06

Average: 7019.59 Requests Per Second

In the previous output, we can make out that the script has asked for a name to save the results (under this name, we will be able to save other tests) and a specific name for the test. In other tests as follows, we can see that the script requests other types of data, such as the record size, file size and disk test.

Following up network connections with conntrack (II)

Let's finish the previous article about Following up network connections with conntrack (I). Other important parameters which can be changed to optimize the system are related to the time of the different types of connections.

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
432000

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait
120

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait
60

The first parameter sets up the maximum lifetime for an already established connection (432000 sg can be long; 28800 could be enough). The second and third are the maximum lifetime for a waiting connection and for the remote endpoint closes the socket.

So as to list all variables based on the conntrack module, type the next order.

root@ubuntu-server:~# sysctl -a | grep conntrack | grep ipv4
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2 = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 300
net.ipv4.netfilter.ip_conntrack_tcp_loose = 1
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 0
net.ipv4.netfilter.ip_conntrack_tcp_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30
net.ipv4.netfilter.ip_conntrack_max = 15768
net.ipv4.netfilter.ip_conntrack_count = 2
net.ipv4.netfilter.ip_conntrack_buckets = 4096
net.ipv4.netfilter.ip_conntrack_checksum = 1
net.ipv4.netfilter.ip_conntrack_log_invalid = 0

And if you want to change the value of any variable, you must add it within the sysctl.conf file and reload the settings.

root@ubuntu-server:~# cat /etc/sysctl.conf
...
net.ipv4.netfilter.ip_conntrack_max = 131072

root@ubuntu-server:~# sysctl -p

One interesting option for the conntrack command is the possibility to get the statistics about the connection tracking.

root@ubuntu-server:~# conntrack -S
entries                 2  
searched                0  
found                   1107
new                     4  
invalid                 0  
ignore                  0  
delete                  2  
delete_list             2  
insert                  4  
insert_failed           0  
drop                    0  
early_drop              0  
icmp_error              0  
expect_new              0  
expect_create           0  
expect_delete           0  
search_restart          0

Another useful feature for conntrack is to output the connection state on real-time, similar to when you run a "tail -f" on a file.

root@ubuntu-server:~# conntrack -E

We can conclude with this couple of articles that the conntrack module is other helpful way to improve the Linux performance.

Following up network connections with conntrack (I)

Linux has got the ability to perform a monitoring of existing connections by means of the conntrack module, which is compiled but not installed in distributions such as RHEL or CentOS. In order to load it, you can run the next order.

[root@centos ~]# modprobe ip_conntrack

In other operating systems like Debian or Ubuntu Server, first of all you must install the conntrack package and load the nf_conntrack_ipv4 module (if you want to work with IPv6, you will have to load the nf_conntrack_ipv6 module).

root@ubuntu-server:~# aptitude install conntrack

root@ubuntu-server:~# modprobe nf_conntrack_ipv4

The conntrack module allows the kernel to register in a table all network connections of the system (established, time_wait, close, etc.). It used by several applications such as iptstate (it shows information about the state of the system connections) or Shorewall (firewall).

Another example of use for this module it is for instance, when the server has to realize NAT tasks with iptables and it is necessary to keep a table of connections implicated.

The file where conntrack logs all connections is /proc/net/ip_conntrack.

root@ubuntu-server:~# cat /proc/net/ip_conntrack
tcp      6 89 TIME_WAIT src=192.168.1.11 dst=192.168.1.12 sport=59302 dport=10050 packets=5 bytes=291 src=192.168.1.12 dst=192.168.1.11 sport=10050 dport=59302 packets=5 bytes=289 [ASSURED] mark=0 secmark=0 use=1
...

root@ubuntu-server:~# conntrack -L
tcp      6 89 TIME_WAIT src=192.168.1.11 dst=192.168.1.12 sport=59302 dport=10050 packets=5 bytes=291 src=192.168.1.12 dst=192.168.1.11 sport=10050 dport=59302 packets=5 bytes=289 [ASSURED] mark=0 secmark=0 use=1
...

The two first fields are the connection protocol (TCP, 6) and then is the connection state (TIME_WAIT). The rest of the fields represent the IP addresses and ports involved, as well as the number of packets and bytes exchanged between the two points of the connection.

You have also to take into account that Linux saves the connection state in memory, and each of them uses around 350 bytes.

If you want to know how many open connections has got the system, you can utilize the following sentences.

root@ubuntu-server:~# cat /proc/net/ip_conntrack | wc -l
856

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
856

root@ubuntu-server:~# conntrack -C
856

This value is quite important because if at any moment we appreciate that any of our services works slowly (for instance Apache) or many connections are rejected, it can be due to which the number of open connections exceeds the maximum number of connections allowed.

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
65536

The size of the hash table is also limited.

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
16384

If you want to modify it, you must do it when the module is loaded.

root@ubuntu-server:~# modprobe nf_conntrack_ipv4 hashsize=32768

Benchmark IOzone: Sistemas de almacenamiento compartido en red (III)

Y ya para terminar los resultados de la herramienta IOzone para las pruebas de benchmarking sobre sistemas de almacenamiento compartido, vamos a emplear esta aplicación para estudiar el comportamiento del sistema ante la lectura/escritura secuencial de varios ficheros de 512 MB por múltiples procesos.

Mediante los siguientes comandos, cada uno de los procesos lanzados (desde uno hasta doce) realizará la escritura y lectura secuencial de un fichero de 512 MB a través de registros de 1024 KB.

[root@centos02 shared02]# iozone –Rc –r 1024 –s 512M –l 1 –u 50 –i 0 –i 1 –b shared02.xls

[root@centos03 shared03]# iozone –Rc –r 1024 –s 512M –l 1 –u 12 –i 0 –i 1 –b shared03.xls

Escritura secuencial de múltiples procesos






Re-Escritura secuencial de múltiples procesos






Lectura secuencial de múltiples procesos






Re-Lectura secuencial de múltiples procesos