Sizing up sockets in Python

Last month I have to face the following issue at work related to sockets. We needed some control servers in order to use them in our test environment, but with the peculiarity that they had to be really fast, to be able to serve thousands of requests per second.

The first thought is that perhaps, this kind of server will have to be developed in C, but on the other hand, to perform a quick test in order to measure its behaviour in Python does not take long. So in this way, I made up a couple of Python scripts (client and server) so as to check it out. Let's see first of all, the part of the server.

from socket import socket, AF_INET, SOCK_STREAM
import datetime
import time

PORT = 22222
PACKET_SIZE = 1024
DELAY = 0
MAX_RETRIES = 10


def server():
    sock = socket(AF_INET, SOCK_STREAM)
    sock.bind(('', PORT))
    sock.listen(1)
    conn, _ = sock.accept()
    
    retries = 0
    data_list = []    
    
    n1=datetime.datetime.now()
    while True:
        data = conn.recv(PACKET_SIZE)
        time.sleep(DELAY)
        if data:
            data_list.append(data)
        else:
            retries += 1
            if retries == MAX_RETRIES:
                n2=datetime.datetime.now()
                total_time = str(n2-n1).split(":")[2]
                
                print "TOTAL TIME (sg): " + str(total_time)
                print "NUMBER OF PACKETS RECEIVED: " + str(len(data_list))
                print "RATE (pkts/sg): " + str(len(data_list) / float(total_time))
                
                break


if __name__ == "__main__":
    
    server()

As you can observe above, the server receives packets of a fix size, in this case 1 KB, and each packet is processed (inserted into a list) without any type of delay. When the input buffer is empty (the program has reached the maximum number of retries), the script will show a little summary related to the test (total time, number of received packets and rate). Also point out that the execution time is metered by means of datetime objects.

Let's take a look at now the side of the client.

from socket import socket, AF_INET, SOCK_STREAM

PORT = 22222
PACKET_SIZE = 1024
PACKETS_NUMBER = 10000


if __name__ == "__main__":

    padding = ""
    for i in range(PACKET_SIZE):
        padding += str(1)

    sock = socket(AF_INET, SOCK_STREAM)
    sock.connect(("172.16.75.132", PORT))

    for i in range(PACKETS_NUMBER):
        sock.send(padding)

The client script just sends a fix number of packets (10.000 in my case) with a concrete size (1 KB).

For the test, I am going to run the server on a virtual machine with Ubuntu Server 12.10 installed on it, and the client will be executed on my personal PC against the aforementioned server. This is the output from the server.

$ python server.py 
TOTAL TIME (sg): 00.138241
NUMBER OF PACKETS RECEIVED: 10005
RATE (pkts/sg): 72373.6084085

As you can see, this is an excellent result. More than 70.000 packets have been received and inserted into a list by the server, in spite of being working with sockets in Python. This result came out more than sufficient for our purposes and allowed us to be able to put into practice our required servers in Python.

Updating the key name in a Python dictionary

This is going to be a short text just to remind me how to modify quickly the name of a key in a Python dictionary. It is straightforward operation that I would like to comment. The idea consists in popping the key and its value, and then, putting it into the dictionary again with a new name.

Let's see an example. The following dictionary has a couple of keys, protocol and size, and I want to change the "protocol" key to "inet_protocol", and in addition, preserve its content.

>>> my_dict = {"protocol": "HTTP", "size": 1024}
>>> my_dict["inet_protocol"] = my_dict.pop("protocol")
>>>
>>> print my_dict
{'inet_protocol': 'HTTP', 'size': 1024}

Getting hold of a remote PID through Paramiko

By means of this article, I would like to share with you a little trick so as to obtain the PID of a remote process run through Paramiko. That is, figure for a moment that you launch an application on a remote system by using Paramiko, and you would like to kill that process after a certain timeout.

To be able to do that, you will have to grab the PID of that process after running it, and if you do not want to use "grep" or other kind of filters like this, the shell provides a curious mechanism which consists in executing that process in a new shell and afterwards, reading the PID of that process which has just been run.

Let's see an example.

import paramiko

if __name__ == "__main__":
    
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect("ubuntu-server.local", username="javi", password="xxxxxx")
    
    _, stdout, _ = ssh.exec_command("echo $$ ; exec python test.py")
    pid = stdout.readline()
    print "PID of the remote process: " + pid
    
    _, stdout, _ = ssh.exec_command("ps -eo pid,command | grep %s" % pid)
    print "Looking for the process:\n" + stdout.read()
    
    ssh.exec_command("kill -s SIGINT %s" % pid)
    ssh.close()

As you can see above, I am running on a remote server a straightforward script called test.py with the following lines.

while True:
    pass

Before executing the script through exec (replaces the current process image with a new one), I have put the order "echo $$", and in this way, I will be able to dump the PID of that process. Finally, I am using that PID to search for that process by means of ps and kill it. Let's run the module.

$ python remote_pid.py 
PID of the remote process: 6020

Looking for the process:
 6020 python test.py
 6021 bash -c ps -eo pid,command | grep 6020 
 6023 grep 6020

Setting up a CVS server on Ubuntu

Installing a local CVS server can be really useful although you do not have huge information to control or many people working on the same project. You can just want to have controlled all your documents. For this case and many others, you can quickly set up a version controller such as CVS.

So let's get started by installing the CVS client and server (Ubuntu Server 12.10). During the installation, you have to choose a name for your repository (or repositories). In my case I went with "myrepos".

$ sudo aptitude install cvs cvsd

Then you have to initialize the repository and change the directory where CVS lock files will be dropped off (instead of directly in the repository by default). Remember to create that new directory.

$ sudo cvs -d /var/lib/cvsd/myrepos init

$ sudo vim /var/lib/cvsd/myrepos/CVSROOT/config
...
LockDir=/tmp/myrepos

$ sudo mkdir /var/lib/cvsd/tmp/myrepos

Now you only have to create an user for the repository and modify its owner.

$ sudo cvsd-passwd /var/lib/cvsd/myrepos javi
/usr/sbin/cvsd-passwd: adding user 'javi' to '/var/lib/cvsd/myrepos/CVSROOT/passwd'

$ sudo chown -R cvsd:cvsd /var/lib/cvsd

Let's restart the server and check it out from the localhost.

$ sudo /etc/init.d/cvsd restart

$ sudo cvs -d :pserver:javi@localhost:/myrepos login
Logging in to :pserver:javi@localhost:2401/myrepos
CVS password: 

Speeding up Python with Cython (III)

Let's carry on with the last article about speeding up Python with Cython. After running the test.py script again by using now the new version of the module, we obtain as follows.

cython$ python test.py 
0.0394711971283

Now the improvement achieves the 98% apropos of the first round. Let's take a look at the new HTML file in order to discern which parts of the code have been directly turned into pure C code.

From the previous picture, it can be asserted that apart from the two static types, a couple of critical structures have been also translated into C: the loops. Cython also provides another C-style syntax so as to write loops. The following file uses this kind of structure, but remember that will not produce any improvement because those loops have already been translated into C.

def calculate_primes(int limit):
    primes = []
    cdef int number = 0
    cdef int divisor = 0

    for number from 2 <= number <= limit:
        for divisor from 2 <= divisor <= number:
            if number % divisor == 0 and number == divisor:
                primes.append(number)
                break
            elif number % divisor == 0 and number != divisor:
                break

    return primes

You can check the documentation located on the Cython website and learn lots of code structures provided by Cython. For example, if this method was embedded in the main script (not as an imported module), you might develop the next optimized method, where the function returns a pointer to an array, and that array has been defined as static and not as a Python list.

cdef int *calculate_primes(int limit):
    cdef int primes[10000]
    cdef int number = 0
    cdef int divisor = 0
    cdef int i = 0

    for number from 2 <= number <= limit:
        for divisor from 2 <= divisor <= number:
            if number % divisor == 0 and number == divisor:
                primes[i] = number
                i += 1
                break
            elif number % divisor == 0 and number != divisor:
                break

    return primes

Let's test out again the HTML file generated for this method.

Speeding up Python performance with Cython (II)

Following up on the last article and as you can see in the previous script, first of all the full path of the Cython installation has been added to the PYTHONPATH. By running this script, both a C file (calculate_primes1.c) and a loadable module by Python (calculate_primes1.so) are been generated.

cython$ python setup.py build_ext --inplace
running build_ext
cythoning calculate_primes1.pyx to calculate_primes1.c
building 'calculate_primes1' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -fwrapv -fPIC -I/usr/include/python2.6 -c calculate_primes1.c -o build/temp.linux-x86_64-2.6/calculate_primes1.o
gcc -pthread -shared build/temp.linux-x86_64-2.6/calculate_primes1.o -L/usr/lib64 -lpython2.6 -o calculate_primes1.so

cython$ ls -l
drwxr-xr-x  3 javi javi  4096 Feb 16 17:46 build
-rw-r--r--  1 javi javi 68295 Feb 17 11:27 calculate_primes1.c
-rw-r--r--  1 javi javi   393 Feb 17 11:06 calculate_primes1.pyx
-rwxr-xr-x  1 javi javi 53388 Feb 17 11:27 calculate_primes1.so
-rwxr-xr-x  1 javi javi   393 Feb 16 21:43 calculate_primes.py
-rwxr-xr-x  1 javi javi   530 Feb 16 21:49 calculate_primes.pyc
drwxr-xr-x 10 javi javi  4096 Feb 16 16:34 Cython-0.18
-rwxr-xr-x  1 javi javi   314 Feb 17 11:06 setup.py
-rwxr-xr-x  1 javi javi   426 Feb 16 21:53 test.py

If you take a look at the C file, you will appreciate that is a huge file (compared with the Python version) where some parts of the code have been translated to C and others keep as Python API calls.

A handy tool aimed at turning out an HTML report which shows the Cython code interlined with the C code, is the own compiler executed with the "-a" option.

cython$ Cython-0.18/cython.py -a calculate_primes1.pyx

If you open the come out HTML file, you will see that lines are colored according to the level of "typedness" (white lines translates to pure C without any Python API calls).

Now let's run again this new version of the module compiled with Cython (remember to change the name of the module inside the test.py file). As you can derive from the result, the performance has been enhanced about 30%.

cython$ python test.py 
1.20011019707

Let's go a step further and add some static types to a second version of our module (called now calculate_primes2).

def calculate_primes(int limit):
    primes = []
    cdef int number = 0
    cdef int divisor = 0

    for number in range(limit):
        for divisor in range(2, number+1):
            if number % divisor == 0 and number == divisor:
                primes.append(number)
                break
            elif number % divisor == 0 and number != divisor:
                break

    return primes

Speeding up Python performance with Cython (I)

Cython is a programming language which creates C/C++ extensions for Python, that is, is able to translate some parts of a Python program into C code, and in this way, increasing considerably the execution time of a Python program. In addition, provides some predefined constructions that you can use directly when you develop in Cython.

Let's see a Python method which takes care of calculating all prime numbers within a range (calculate_primes.py).

def calculate_primes(limit):
    primes = []
    number = 0
    divisor = 0

    for number in range(limit):
        for divisor in range(2, number+1):
            if number % divisor == 0 and number == divisor:
                primes.append(number)
                break
            elif number % divisor == 0 and number != divisor:
                break

    return primes

Now let's import this module in another program and run it in order to work out the time spent in searching for all prime numbers between 2 and 10000 (the application will be executed five times and averaged it).

$ cat test.py 
from timeit import Timer  

timer = Timer("calculate_primes(10000)",
              "from calculate_primes import calculate_primes")
print sum(timer.repeat(repeat=5, number=1)) / 5


$ python test.py 
1.7978372097

The next step will be to compile this program with Cython in order to generate C extensions for our program. So as to install Cython, you have a couple of options: either to grab directly its source code from the web page of the project, or install it from some repository. In my case, I am going to download its source code and thereby, to be able to have the most recent version (0.18 at present).

After downloading and uncompressing it, you will be able to install it on your system by running the setup.py script, or as in my case, just decompressing it on the home directory and using that path later. Also point out that you must have installed on your system all tools needed to compile C/C++ programs (for example, the package build-essential in case of Ubuntu).

$ mkdir cython ; cd cython

cython$ wget http://cython.org/release/Cython-0.18.zip ; unzip Cython-0.18.zip

The first test will be to compile the module called calculate_primes with Cython, so as to generate on the one hand a C file, and on the other, a module loadable by Python. First up, you need to change the extension of the module from py to pyx. And to compile it, you might do it by hand, but it is much better to create a tiny script responsible of accomplishing it.

cython$ cp -a calculate_primes.py calculate_primes1.pyx

cython$ cat setup.py 
import sys
sys.path.insert(0, "/home/javi/cython/Cython-0.18")
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("calculate_primes1", ["calculate_primes1.pyx"])]
)

Paramiko object with Process (RNG must be re-initialized after fork)

When you create an object in Python which will be used later by an object of type Process, that works correctly. Let's see an example where an object of type Process prints an attribute provided by another object belonging to the class A.

from multiprocessing import Process

class A():
    def __init__(self):
        self.my_dict = {"day": "Tuesday"}

class B():
    def __init__(self):
        self.a = A()
        self.__process = None

    def start(self):
        self.__process = Process(target=self.__run_process)
        self.__process.start()
        
    def __run_process(self):
        print self.a.my_dict
        
if __name__ == "__main__":
    b = B()
    b.start()


$ python test.py 
{'day': 'Tuesday'}

But when that process has to handle a Paramiko object where the connection has been initialized before, will not work.

import paramiko
from multiprocessing import Process

class B():
    def __init__(self):
        self.__process = None
        self.__ssh = paramiko.SSHClient()
        self.__ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        self.__ssh.connect('127.0.0.1', username='javi', password='xxxxxx')
        
    def start(self):
        self.__process = Process(target=self.__run_process)
        self.__process.start()
        
    def __run_process(self):
        try:
            _, stdout, _ = self.__ssh.exec_command("hostname")
        except:
            print "Failed the execution"
        else:
            print stdout.read()
        
if __name__ == "__main__":
    b = B()
    b.start()


$ python test.py 
Failed the execution

If you take a look at the output generated by the execution without catching it through an exception, you could observe an error as follows.

$ python test.py 
Process Process-1:
Traceback (most recent call last):
...
    raise AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
AssertionError: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()

This is a well-known issue in Paramiko that has been apparently fixed in recent versions (for my tests, I am using the version provided by Ubuntu 12.10 by default: 1.7.7.1-3). A possible workaround will be for example to open a new SSH connection for that process. Let's grab an example.

import paramiko
from multiprocessing import Process

class B():
    def __init__(self):
        self.__process = None
        self.__ssh = paramiko.SSHClient()
        self.__ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        
    def start(self):
        self.__process = Process(target=self.__run_process)
        self.__process.start()
        
    def __run_process(self):
        self.__ssh.connect('127.0.0.1', username='javi', password='xxxxxx')
        try:
            _, stdout, _ = self.__ssh.exec_command("hostname")
        except:
            print "Failed the execution"
        else:
            print stdout.read()
        
if __name__ == "__main__":
    b = B()
    b.start()


$ python test.py 
javi-pc

As you can see above, the definition of the Paramiko object takes place within the constructor of the class B, but the connection is eventually initialized by the forked process.

Merging dictionaries in Python with lists within

Because I have just set out to program in Python at work (in depth), I am going to write down on my blog those interesting things that I have to work out related to this activity.

And for example, this week I had to cope a curious problem about merging dictionaries with any kind of element within, such as lists or other dictionaries. Browsing the Internet, I was able to find some functions to combine dictionaries, but the problem was that those methods did not take into account the option of having lists inside.

So in order to get over this issue, I had to develop the following two recursive methods.

def merge_dict(dict1, dict2):   
    if isinstance(dict1, dict) and isinstance(dict2, dict):
        for k, v in dict2.iteritems():
            if k not in dict1:
                dict1[k] = v
            else:
                dict1[k] = merge_dict(dict1[k], v)
    elif isinstance(dict1, list) and isinstance(dict2, list):
        dict1 = merge_list(dict1, dict2)
    elif dict2 == None:
        return dict1
    else:
        return dict2
        
    return dict1

def merge_list(list1, list2):
    if isinstance(list1, list) and isinstance(list2, list):
        for i, item in enumerate(list2):
            if len(list1) > i:
                if isinstance(list1[i], dict) and isinstance(item, dict):
                    list1[i] = merge_dict(list1[i], item)
                elif isinstance(list1[i], list) and isinstance(item, list):
                    list1[i] = merge_list(list1[i], item)
                else:
                    list1[i] = item
            else:
                list1.append(list2[i])
    elif isinstance(list1, dict) and isinstance(list2, dict):
        list1 = merge_dict(list1, list2)
    else:
        return list2

    return list1

As you can see in the above code, the method merge_dict loops through the dictionary and if it comes across a list, it will have a function able to merge the list. Also say that dict2 and list2 will have more priority that dict1 and list1 respectively.

Let's see an example where there are a couple of dictionaries with several elements within.

dict1 = {"country": [{"name": "Spain", "capital": "Madrid"}, {"name": "France"}]}
dict2 = {"country": [{"name": "Germany"}], "continent": "Europe"}

print merge_dict(dict1, dict2)

This is the output for the preceding script.

$ python test.py 

{'country': [{'name': 'Germany', 'capital': 'Madrid'}, {'name': 'France'}], 'continent': 'Europe'}

MySQL benchmark with SysBench (II)

Let's carry on the preceding article about MySQL benchmark with SysBench by putting into action the above scenario.

For the first case, I am going to use the values recommended through the first four articles that I wrote about MySQL optimization. So according to them, the values used will be as follows. Let's call this first settings A.

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
key_buffer              = 32M
thread_cache_size       = 512
table_cache             = 512
table_definition_cache  = 512
open_files_limit        = 1024
tmp_table_size          = 64M
max_heap_table_size     = 32M
query_cache_limit       = 4M
query_cache_size        = 256M
query_cache_type        = 1
innodb_buffer_pool_size = 1280M

For the second configuration, called B, I am going to set by means of the parameter innodb-flush-log-at-trx-commit, when the log buffer will be written out to the log file and the flush to disk operation will be effected.

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
innodb-flush-log-at-trx-commit = 0

For the third combination (C), a new parameter will be added: innodb_buffer_pool_instances. With this option, it is possible to define the number of regions that the InnoDB area will be divided into.

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
innodb_buffer_pool_instances = 2

And finally, the size for the log file will be increased by means of innodb_log_file_size, and in this way, the graph D will be turned out (in order to be able to apply this change, first up you have to stop MySQL and remove any existing log file).

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
innodb_log_file_size = 256M

root@ubuntu-server:~# rm -rf /var/lib/mysql/ib_logfile*

Below you can observe the graphs generated for the different cases.

As you can appreciate in the figure, the best improvements are achieved when the size of the log file is increased from its default value (5M) to 256M. The option C, that is, to separate the InnoDB buffer into two regions gets worse in respect of the previous alternative (B).

MySQL benchmark with SysBench (I)

By taking advantage of the previous article about MySQL optimization, I am going to introduce a handy tool called SysBench and aimed at measuring the performance of a MySQL database among other things. In addition, it is also able to evaluate the I/O, scheduler and threads implementation performance, and memory allocation and transfer speed.

So I am going to use this tool in order to verify the improvements commented in the preceding article and related to some parameters of MySQL. The test will be run on Ubuntu Server 12.10 virtualized through VMware. The virtual machine will made up by a 6 GB hard drive, 2 GB of RAM and a couple of virtual cores.

First of all, let's install MySQL and SysBench and increase the default number of maximum connections to 512.

root@ubuntu-server:~# aptitude install mysql-server sysbench

root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
max_connections = 512

root@ubuntu-server:~# service mysql restart

Now let's create a table of 1.000.000 of rows in a database called test by using SysBench.

root@ubuntu-server:~# sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=xxxxxx prepare

Then a straightforward script taken care of running the tests will be developed. This bash script executes the OLTP (OnLine Transaction Processing) test on a table of 1.000.000 of rows. The time limit for the whole execution will be 300 seconds and read, update, delete and insert queries will be performed. The total number of maximum requests will be unlimited.

root@ubuntu-server:~# cat script.sh 
#!/bin/bash

for i in 8 16 32 64 128 256 512
do
    service mysql restart ; sleep 5
    sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=xxxxxx --max-time=300 --oltp-read-only=off --max-requests=0 --num-threads=$i run
done

As you can see in the above script, the number of worked threads to be created will be different in each loop iteration, from 8 to 512. So the idea is to run the script with various MySQL combinations and calculate the number of transactions per second.

MySQL optimization (V)

I am going to reopen the last article that I wrote about MySQL optimization because I came across three new parameters (for me) which enhance the performance of a MySQL database, and I would like to note down them on my blog.

The first parameter is innodb-flush-log-at-trx-commit, which manages both when the log buffer is written out to the log file and the flush to disk operation is performed. Its default value is 1, which means that the log buffer is dumped to the log file at each transaction commit and the flush to disk operation is carried out directly on the log file.

When its value is 0, the log buffer is sent to the log file once per second, so in this way, you are turning down the disk accesses. In respect of the flush to disk, the operation is also effected on the log file but not coinciding with the commit, but taking advantage of free periods. And when it takes the value of 2 (less aggressive than 0), the log buffer is written out to the log file at each commit but as in the previous case, the flush is done at any free moment for the server.

The other parameter that I would like to talk about is innodb_buffer_pool_instances (in case of you are using the InnoDB engine), which represents the number of regions that the InnoDB buffer is broken up.  This parameter is really useful when you are using a server with several cores, and thereby, each core (thread) can work on a separate instance. A good recommendation is to set it to the same value as the number of cores, but another popular option is to follow the next rule: (innodb_buffer_pool_size [in GB] + number of cores) / 2.

And finally, the last parameter is innodb_log_file_size, related to the InnoDB log file. Its default value is 5 MB and I consider that is not enough for production environments. The larger the value is, the less control flush activity is needed in the buffer pool, saving disk I/O operations. I think that a right value would be between 64 and 256 MB.

Zabbix poller processes more than 75% busy and queue delay (III)

Let's complete the last article about Zabbix poller processes more than 75% busy and queue delay. In this section, I am going to tackle the part of the client, that is, those things which can be modified on the agent so as to remove or attenuate the issues mentioned in the first article.

Remember that this is the continuation of the two previous articles:

• Zabbix poller processes more than 75% busy and queue delay (I)
• Zabbix poller processes more than 75% busy and queue delay (II)

First up, I changed the number of pre-forked instances of the Zabbix client which process passive checks (StartAgents) to 64. This parameter is really meaningful, because its default value is 5, that is to say, only five processes will be started in order to obtain the data requested by the server. So if you have a lot of items and a small monitoring period (as my case), you will need more processes to be able to attend all requests.

root@zabbix-client:~# cat /etc/zabbix/zabbix_agentd.conf
...
StartAgents=64

So let's see now in the graphs, how this change impacts on the results. Let's first with the Zabbix server performance.

And then, the Zabbix data gathering process.

As you can see on the first picture, the server has gone from a Zabbix queue of 30 to 0 (although you can observe 5 on the figure, think that the graph has been cut out). And on the second one, the Zabbix busy poller processes went from 24% to 0%.

Other parameters that you can play with are the number of seconds that the data can be stored in the buffer and its maximum number of values.

root@zabbix-client:~# cat /etc/zabbix/zabbix_agentd.conf
...
BufferSend=3600

BufferSize=65535

Also keep in mind that you should have a small value for the timeout (I am using five seconds on my installation).

Lastly, in order to solve the problem that I mentioned in the first article about from time to time, the processes break down and the zabbix agent is stopped, I developed a simple bash script to work around this issue.

root@zabbix-client:~# tail -f /var/log/zabbix/zabbix_agentd.log
...
zabbix_agentd [17271]: [file:'cpustat.c',line:155] lock failed: [22] Invalid argument
 17270:20121015:092010.216 One child process died (PID:17271,exitcode/signal:255). Exiting ...
...
 17270:20121015:092012.216 Zabbix Agent stopped. Zabbix 2.0.3 (revision 30485).


root@zabbix-client:~# cat /etc/zabbix/monitor_zabbix.sh
#!/bin/bash

while [ 1 ];
do
        if ! pgrep -f "/usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf" &> /dev/null ; then
                /etc/zabbix/zabbix.sh start
        fi
        sleep 15
done

This script is run in batch mode and takes care of monitoring the status of the agent processes and starting over when they drop . It uses another bash script to start and stop the agents.

root@zabbix-client:~# cat /etc/zabbix/zabbix.sh
#!/bin/bash

case $1 in
        "start")
                taskset -c $(($(cat /proc/cpuinfo | grep processor | wc -l) - 1)) /usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf;;
        "stop")
                pkill -f "/usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf";;
        *)
                printf "./zabbix.sh start|stop\n\n"
esac

Zabbix poller processes more than 75% busy and queue delay (II)

After putting forward the issues turned up on my current Zabbix installation and related to its performance (Zabbix poller processes more than 75% busy and queue delay), I am going to explain to you how I solved it.

First of all, I tried out to increase the number of pre-forked instances of pollers for the Zabbix server, that is, I changed its default value from 5 to 256 (remember that for that case, you have to set the the number of maximum connections in MySQL - max_connections - higher than 256, since every single poller opens a dedicated connection to the database).

root@zabbix-server:~# cat /etc/zabbix/zabbix_server.conf
...
# StartPollers=5
StartPollers=256

root@zabbix-server:~# cat /etc/mysql/my.cnf
...
max_connextions = 512

Below you can see the outcome after applying it (Zabbix server performance).

And the Zabbix data gathering process.

In the first figure, you can observe that the Zabbix queue has gone from 48 to 30 (approximately), and for the second one, the Zabbix busy poller processes went from 100% to 24%. So it is clear that if you have a server with enough resources, there is no problem to start many pollers. These kind of processes are responsible for requesting the data defined in the items, so the more pollers have available, the less overloaded the system is.

Other Zabbix server parameter that you ought to take into account is for example the Timeout (specifies how log pollers wait for agent responses). Try not to assign a very high value. Otherwise, the system might get overloaded.

Next week, I will end up this series of articles by accomplishing the part of the client.

sysstat vs top vs ps (III)

Let's finish the series of articles related to the differences in the measure of sysstat, top and ps.

What about the system CPU time (%sy) of the first top from the previous article? It is 39.7%. It is right as well. You have to take into account that, because this server has a couple of cores, that data represents the average from both cores. You can see this point by running top into interactive mode and pressing the number 1. Then you will be able to obtain the consumption for both cores.

root@ubuntu-server:~# top
top - 20:04:37 up 47 min,  1 user,  load average: 0.40, 0.54, 0.60
Tasks:  87 total,   1 running,  86 sleeping,   0 stopped,   0 zombie
Cpu0  :  2.9%us, 37.7%sy,  0.0%ni, 59.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  1.6%us, 40.8%sy,  0.0%ni, 57.1%id,  0.0%wa,  0.0%hi,  0.5%si,  0.0%st
Mem:   1024800k total,   207680k used,   817120k free,    23532k buffers
Swap:  1048572k total,        0k used,  1048572k free,   100320k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1561 root      20   0 29280 4180 2416 S  128  0.4  56:57.23 script.py

At any rate, the sum of both percentages does not match the %CPU used by the script. This might be due to the sampling frequencies commented in the preceding article.

Finally, I would like to remark that in order to monitor threads, I like to use pidstat with the -t parameter.

root@ubuntu-server:~# pidstat -u -t -p 1561 1 1
Linux 3.2.0-30-generic-pae (ubuntu-server)     09/30/2012     _i686_    (2 CPU)

08:14:47 PM      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
08:14:48 PM      1561         -    9.00  119.00    0.00  128.00     1  script.py
08:14:48 PM         -      1561    0.00    0.00    0.00    0.00     1  |__script.py
08:14:48 PM         -      1562    4.00   60.00    0.00   64.00     0  |__script.py
08:14:48 PM         -      1563    4.00   59.00    0.00   63.00     1  |__script.py

This tool is great because you can see the core where the thread (or process) is being executed.

Another interesting choice is to use directly the ps command with the suitable parameters (-L shows threads and sgi_p the processor where the process is currently executing on).

root@ubuntu-server:~# ps -Leo pid,pcpu,sgi_p,command | grep '^ 1561'
 1561  0.0 * /usr/bin/python ./script.py
 1561 65.6 0 /usr/bin/python ./script.py
 1561 65.2 0 /usr/bin/python ./script.py

Also point out for the above output that, ps does not offer the number of TID (Thread ID), in contradistinction to top and pidstat (remember that a thread does not have PID).

Zabbix poller processes more than 75% busy and queue delay (I)

In my previous job, I had to set up a Zabbix infrastructure in order to monitor more than 400 devices between switches and servers. The main feature of this architecture was that there were a lot of machines, but the update interval was large (around 30 seconds) and the number of items small.

For this purpose, I wrote down a couple of articles related to this issue:

• Tuning Zabbix to improve its performance (I)
• Tuning Zabbix to improve its performance (II)

But in my current position, I am starting to introduce Zabbix (2.0.3 on Ubuntu Server 12.04) with the aim of controlling few devices where a large number of items and a small monitoring period are required. This situation leads to an overload of the Zabbix server, on the one hand by increasing the number of monitored elements delayed in the queue, and on the other, turning out that the poller processes are busy long.

In addition, I have been able to observe that, from time to time, the agent goes down in an unexpected way. If you take a look at the log file from the client (debug mode), the following error lines are dumped.

root@zabbix-client:~# tail -f /var/log/zabbix/zabbix_agentd.log
...
zabbix_agentd [17271]: [file:'cpustat.c',line:155] lock failed: [22] Invalid argument
 17270:20121015:092010.216 One child process died (PID:17271,exitcode/signal:255). Exiting ...
 17270:20121015:092010.216 zbx_on_exit() called
 17272:20121015:092010.216 Got signal [signal:15(SIGTERM),sender_pid:17270,sender_uid:0,reason:0]. Exiting ...
 17273:20121015:092010.216 Got signal [signal:15(SIGTERM),sender_pid:17270,sender_uid:0,reason:0]. Exiting ...
 17274:20121015:092010.216 Got signal [signal:15(SIGTERM),sender_pid:17270,sender_uid:0,reason:0]. Exiting ...
 17270:20121015:092012.216 Zabbix Agent stopped. Zabbix 2.0.3 (revision 30485).

Below you can observe a figure which shows the Zabbix server performance (queue) for the aforementioned case.

And the other one, reflects the Zabbix data gathering process (pay attention to the data Zabbix busy poller processes, in %).

For the first case, the Zabbix queue has averaged more than 50 monitored items delayed, and for the second one, the poller processes are busy about 100% of the time. This situation can produce that, sometimes, Zabbix draws sporadic dots rather than lines in the graphs. Another effect that you can get from this condition is that if you set a short update interval for an item, you could run into lack of data when you check the values gathered later.

Also say that I followed the tuning guide that I mentioned before, but as you can see, Zabbix server was acting up.

sysstat vs top vs ps (II)

Following up on the previous article, sysstat vs top vs ps (I), a curious case that I would like to talk about is when you use more than one core. Let's create a simple script in Python which runs a couple of threads a little bit overloaded.

root@ubuntu-server:~# nproc 
2

root@ubuntu-server:~# cat script.py 
#!/usr/bin/python

import threading, time

def sleep():
    while True:
        time.sleep(0.000001)

t1 = threading.Thread(target=sleep)
t2 = threading.Thread(target=sleep)

t1.start()
t2.start()

t1.join()
t2.join()

root@ubuntu-server:~# ./script.py &
[1] 1561

If we take a look now at the status of this process by means of top, we can see as follows.

root@ubuntu-server:~# top -b -n 1 -p 1561
top - 19:41:19 up 24 min,  1 user,  load average: 0.74, 0.66, 0.48
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.3%us, 39.7%sy,  0.0%ni, 55.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1024800k total,   205596k used,   819204k free,    23192k buffers
Swap:  1048572k total,        0k used,  1048572k free,    98712k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1561 root      20   0 29280 4180 2416 S  129  0.4  26:23.26 script.py

So what would be the first weird thing that you can observe from the previous screen? The script is consuming 129% of the CPU. This is right because you have to remember that this virtual machine has two cores and the script, rather, its two threads, are using the two ones and that figure is the combination of the CPU utilization from both cores. You can appreciate this situation much better if you execute top with the -H option.

root@ubuntu-server:~# top -b -H -n 1 -p 1561
top - 19:44:22 up 27 min,  1 user,  load average: 0.69, 0.72, 0.54
Tasks:   3 total,   2 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.3%us, 38.9%sy,  0.0%ni, 56.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1024800k total,   205624k used,   819176k free,    23192k buffers
Swap:  1048572k total,        0k used,  1048572k free,    98712k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1563 root      20   0 29280 4180 2416 R   65  0.4  15:10.19 script.py
 1562 root      20   0 29280 4180 2416 R   64  0.4  15:12.39 script.py
 1561 root      20   0 29280 4180 2416 S    0  0.4   0:00.05 script.py