Sizing up sockets in Python

Last month I have to face the following issue at work related to sockets. We needed some control servers in order to use them in our test environment, but with the peculiarity that they had to be really fast, to be able to serve thousands of requests per second.

The first thought is that perhaps, this kind of server will have to be developed in C, but on the other hand, to perform a quick test in order to measure its behaviour in Python does not take long. So in this way, I made up a couple of Python scripts (client and server) so as to check it out. Let's see first of all, the part of the server.

from socket import socket, AF_INET, SOCK_STREAM
import datetime
import time

PORT = 22222
PACKET_SIZE = 1024
DELAY = 0
MAX_RETRIES = 10


def server():
    sock = socket(AF_INET, SOCK_STREAM)
    sock.bind(('', PORT))
    sock.listen(1)
    conn, _ = sock.accept()
    
    retries = 0
    data_list = []    
    
    n1=datetime.datetime.now()
    while True:
        data = conn.recv(PACKET_SIZE)
        time.sleep(DELAY)
        if data:
            data_list.append(data)
        else:
            retries += 1
            if retries == MAX_RETRIES:
                n2=datetime.datetime.now()
                total_time = str(n2-n1).split(":")[2]
                
                print "TOTAL TIME (sg): " + str(total_time)
                print "NUMBER OF PACKETS RECEIVED: " + str(len(data_list))
                print "RATE (pkts/sg): " + str(len(data_list) / float(total_time))
                
                break


if __name__ == "__main__":
    
    server()

As you can observe above, the server receives packets of a fix size, in this case 1 KB, and each packet is processed (inserted into a list) without any type of delay. When the input buffer is empty (the program has reached the maximum number of retries), the script will show a little summary related to the test (total time, number of received packets and rate). Also point out that the execution time is metered by means of datetime objects.

Let's take a look at now the side of the client.

from socket import socket, AF_INET, SOCK_STREAM

PORT = 22222
PACKET_SIZE = 1024
PACKETS_NUMBER = 10000


if __name__ == "__main__":

    padding = ""
    for i in range(PACKET_SIZE):
        padding += str(1)

    sock = socket(AF_INET, SOCK_STREAM)
    sock.connect(("172.16.75.132", PORT))

    for i in range(PACKETS_NUMBER):
        sock.send(padding)

The client script just sends a fix number of packets (10.000 in my case) with a concrete size (1 KB).

For the test, I am going to run the server on a virtual machine with Ubuntu Server 12.10 installed on it, and the client will be executed on my personal PC against the aforementioned server. This is the output from the server.

$ python server.py 
TOTAL TIME (sg): 00.138241
NUMBER OF PACKETS RECEIVED: 10005
RATE (pkts/sg): 72373.6084085

As you can see, this is an excellent result. More than 70.000 packets have been received and inserted into a list by the server, in spite of being working with sockets in Python. This result came out more than sufficient for our purposes and allowed us to be able to put into practice our required servers in Python.

Updating the key name in a Python dictionary

This is going to be a short text just to remind me how to modify quickly the name of a key in a Python dictionary. It is straightforward operation that I would like to comment. The idea consists in popping the key and its value, and then, putting it into the dictionary again with a new name.

Let's see an example. The following dictionary has a couple of keys, protocol and size, and I want to change the "protocol" key to "inet_protocol", and in addition, preserve its content.

>>> my_dict = {"protocol": "HTTP", "size": 1024}
>>> my_dict["inet_protocol"] = my_dict.pop("protocol")
>>>
>>> print my_dict
{'inet_protocol': 'HTTP', 'size': 1024}

Getting hold of a remote PID through Paramiko

By means of this article, I would like to share with you a little trick so as to obtain the PID of a remote process run through Paramiko. That is, figure for a moment that you launch an application on a remote system by using Paramiko, and you would like to kill that process after a certain timeout.

To be able to do that, you will have to grab the PID of that process after running it, and if you do not want to use "grep" or other kind of filters like this, the shell provides a curious mechanism which consists in executing that process in a new shell and afterwards, reading the PID of that process which has just been run.

Let's see an example.

import paramiko

if __name__ == "__main__":
    
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect("ubuntu-server.local", username="javi", password="xxxxxx")
    
    _, stdout, _ = ssh.exec_command("echo $$ ; exec python test.py")
    pid = stdout.readline()
    print "PID of the remote process: " + pid
    
    _, stdout, _ = ssh.exec_command("ps -eo pid,command | grep %s" % pid)
    print "Looking for the process:\n" + stdout.read()
    
    ssh.exec_command("kill -s SIGINT %s" % pid)
    ssh.close()

As you can see above, I am running on a remote server a straightforward script called test.py with the following lines.

while True:
    pass

Before executing the script through exec (replaces the current process image with a new one), I have put the order "echo $$", and in this way, I will be able to dump the PID of that process. Finally, I am using that PID to search for that process by means of ps and kill it. Let's run the module.

$ python remote_pid.py 
PID of the remote process: 6020

Looking for the process:
 6020 python test.py
 6021 bash -c ps -eo pid,command | grep 6020 
 6023 grep 6020

Setting up a CVS server on Ubuntu

Installing a local CVS server can be really useful although you do not have huge information to control or many people working on the same project. You can just want to have controlled all your documents. For this case and many others, you can quickly set up a version controller such as CVS.

So let's get started by installing the CVS client and server (Ubuntu Server 12.10). During the installation, you have to choose a name for your repository (or repositories). In my case I went with "myrepos".

$ sudo aptitude install cvs cvsd

Then you have to initialize the repository and change the directory where CVS lock files will be dropped off (instead of directly in the repository by default). Remember to create that new directory.

$ sudo cvs -d /var/lib/cvsd/myrepos init

$ sudo vim /var/lib/cvsd/myrepos/CVSROOT/config
...
LockDir=/tmp/myrepos

$ sudo mkdir /var/lib/cvsd/tmp/myrepos

Now you only have to create an user for the repository and modify its owner.

$ sudo cvsd-passwd /var/lib/cvsd/myrepos javi
/usr/sbin/cvsd-passwd: adding user 'javi' to '/var/lib/cvsd/myrepos/CVSROOT/passwd'

$ sudo chown -R cvsd:cvsd /var/lib/cvsd

Let's restart the server and check it out from the localhost.

$ sudo /etc/init.d/cvsd restart

$ sudo cvs -d :pserver:javi@localhost:/myrepos login
Logging in to :pserver:javi@localhost:2401/myrepos
CVS password: 

Speeding up Python with Cython (III)

Let's carry on with the last article about speeding up Python with Cython. After running the test.py script again by using now the new version of the module, we obtain as follows.

cython$ python test.py 
0.0394711971283

Now the improvement achieves the 98% apropos of the first round. Let's take a look at the new HTML file in order to discern which parts of the code have been directly turned into pure C code.

From the previous picture, it can be asserted that apart from the two static types, a couple of critical structures have been also translated into C: the loops. Cython also provides another C-style syntax so as to write loops. The following file uses this kind of structure, but remember that will not produce any improvement because those loops have already been translated into C.

def calculate_primes(int limit):
    primes = []
    cdef int number = 0
    cdef int divisor = 0

    for number from 2 <= number <= limit:
        for divisor from 2 <= divisor <= number:
            if number % divisor == 0 and number == divisor:
                primes.append(number)
                break
            elif number % divisor == 0 and number != divisor:
                break

    return primes

You can check the documentation located on the Cython website and learn lots of code structures provided by Cython. For example, if this method was embedded in the main script (not as an imported module), you might develop the next optimized method, where the function returns a pointer to an array, and that array has been defined as static and not as a Python list.

cdef int *calculate_primes(int limit):
    cdef int primes[10000]
    cdef int number = 0
    cdef int divisor = 0
    cdef int i = 0

    for number from 2 <= number <= limit:
        for divisor from 2 <= divisor <= number:
            if number % divisor == 0 and number == divisor:
                primes[i] = number
                i += 1
                break
            elif number % divisor == 0 and number != divisor:
                break

    return primes

Let's test out again the HTML file generated for this method.

Speeding up Python performance with Cython (II)

Following up on the last article and as you can see in the previous script, first of all the full path of the Cython installation has been added to the PYTHONPATH. By running this script, both a C file (calculate_primes1.c) and a loadable module by Python (calculate_primes1.so) are been generated.

cython$ python setup.py build_ext --inplace
running build_ext
cythoning calculate_primes1.pyx to calculate_primes1.c
building 'calculate_primes1' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -fwrapv -fPIC -I/usr/include/python2.6 -c calculate_primes1.c -o build/temp.linux-x86_64-2.6/calculate_primes1.o
gcc -pthread -shared build/temp.linux-x86_64-2.6/calculate_primes1.o -L/usr/lib64 -lpython2.6 -o calculate_primes1.so

cython$ ls -l
drwxr-xr-x  3 javi javi  4096 Feb 16 17:46 build
-rw-r--r--  1 javi javi 68295 Feb 17 11:27 calculate_primes1.c
-rw-r--r--  1 javi javi   393 Feb 17 11:06 calculate_primes1.pyx
-rwxr-xr-x  1 javi javi 53388 Feb 17 11:27 calculate_primes1.so
-rwxr-xr-x  1 javi javi   393 Feb 16 21:43 calculate_primes.py
-rwxr-xr-x  1 javi javi   530 Feb 16 21:49 calculate_primes.pyc
drwxr-xr-x 10 javi javi  4096 Feb 16 16:34 Cython-0.18
-rwxr-xr-x  1 javi javi   314 Feb 17 11:06 setup.py
-rwxr-xr-x  1 javi javi   426 Feb 16 21:53 test.py

If you take a look at the C file, you will appreciate that is a huge file (compared with the Python version) where some parts of the code have been translated to C and others keep as Python API calls.

A handy tool aimed at turning out an HTML report which shows the Cython code interlined with the C code, is the own compiler executed with the "-a" option.

cython$ Cython-0.18/cython.py -a calculate_primes1.pyx

If you open the come out HTML file, you will see that lines are colored according to the level of "typedness" (white lines translates to pure C without any Python API calls).

Now let's run again this new version of the module compiled with Cython (remember to change the name of the module inside the test.py file). As you can derive from the result, the performance has been enhanced about 30%.

cython$ python test.py 
1.20011019707

Let's go a step further and add some static types to a second version of our module (called now calculate_primes2).

def calculate_primes(int limit):
    primes = []
    cdef int number = 0
    cdef int divisor = 0

    for number in range(limit):
        for divisor in range(2, number+1):
            if number % divisor == 0 and number == divisor:
                primes.append(number)
                break
            elif number % divisor == 0 and number != divisor:
                break

    return primes

Speeding up Python performance with Cython (I)

Cython is a programming language which creates C/C++ extensions for Python, that is, is able to translate some parts of a Python program into C code, and in this way, increasing considerably the execution time of a Python program. In addition, provides some predefined constructions that you can use directly when you develop in Cython.

Let's see a Python method which takes care of calculating all prime numbers within a range (calculate_primes.py).

def calculate_primes(limit):
    primes = []
    number = 0
    divisor = 0

    for number in range(limit):
        for divisor in range(2, number+1):
            if number % divisor == 0 and number == divisor:
                primes.append(number)
                break
            elif number % divisor == 0 and number != divisor:
                break

    return primes

Now let's import this module in another program and run it in order to work out the time spent in searching for all prime numbers between 2 and 10000 (the application will be executed five times and averaged it).

$ cat test.py 
from timeit import Timer  

timer = Timer("calculate_primes(10000)",
              "from calculate_primes import calculate_primes")
print sum(timer.repeat(repeat=5, number=1)) / 5


$ python test.py 
1.7978372097

The next step will be to compile this program with Cython in order to generate C extensions for our program. So as to install Cython, you have a couple of options: either to grab directly its source code from the web page of the project, or install it from some repository. In my case, I am going to download its source code and thereby, to be able to have the most recent version (0.18 at present).

After downloading and uncompressing it, you will be able to install it on your system by running the setup.py script, or as in my case, just decompressing it on the home directory and using that path later. Also point out that you must have installed on your system all tools needed to compile C/C++ programs (for example, the package build-essential in case of Ubuntu).

$ mkdir cython ; cd cython

cython$ wget http://cython.org/release/Cython-0.18.zip ; unzip Cython-0.18.zip

The first test will be to compile the module called calculate_primes with Cython, so as to generate on the one hand a C file, and on the other, a module loadable by Python. First up, you need to change the extension of the module from py to pyx. And to compile it, you might do it by hand, but it is much better to create a tiny script responsible of accomplishing it.

cython$ cp -a calculate_primes.py calculate_primes1.pyx

cython$ cat setup.py 
import sys
sys.path.insert(0, "/home/javi/cython/Cython-0.18")
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("calculate_primes1", ["calculate_primes1.pyx"])]
)