tag:blogger.com,1999:blog-14944773468859324312024-03-14T07:16:06.702+01:00Redes Privadas VirtualesIn computing, a single second can be a lifetimeJavier Andrés Alonsohttp://www.blogger.com/profile/15322743276117860784noreply@blogger.comBlogger228125tag:blogger.com,1999:blog-1494477346885932431.post-6917764905397290042013-04-14T17:10:00.000+02:002013-04-14T17:12:45.377+02:00Sizing up sockets in PythonLast month I have to face the following issue at work related to sockets. We needed some control servers in order to use them in our test environment, but with the peculiarity that they had to be really fast, to be able to serve thousands of requests per second.<br />
<br />
The first thought is that perhaps, this kind of server will have to be developed in C, but on the other hand, to perform a quick test in order to measure its behaviour in Python does not take long. So in this way, I made up a couple of Python scripts (client and server) so as to check it out. Let's see first of all, the part of the server.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">from</span> <span style="color: #0e84b5; font-weight: normal;">socket</span> <span style="color: green; font-weight: normal;">import</span> socket, AF_INET, SOCK_STREAM
<span style="color: green; font-weight: normal;">import</span> <span style="color: #0e84b5; font-weight: normal;">datetime</span>
<span style="color: green; font-weight: normal;">import</span> <span style="color: #0e84b5; font-weight: normal;">time</span>
PORT <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">22222</span>
PACKET_SIZE <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">1024</span>
DELAY <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
MAX_RETRIES <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">10</span>
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">server</span>():
sock <span style="color: #303030;">=</span> socket(AF_INET, SOCK_STREAM)
sock<span style="color: #303030;">.</span>bind((<span style="background-color: #fff0f0;">''</span>, PORT))
sock<span style="color: #303030;">.</span>listen(<span style="color: #0000d0; font-weight: normal;">1</span>)
conn, _ <span style="color: #303030;">=</span> sock<span style="color: #303030;">.</span>accept()
retries <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
data_list <span style="color: #303030;">=</span> []
n1<span style="color: #303030;">=</span>datetime<span style="color: #303030;">.</span>datetime<span style="color: #303030;">.</span>now()
<span style="color: green; font-weight: normal;">while</span> <span style="color: #007020;">True</span>:
data <span style="color: #303030;">=</span> conn<span style="color: #303030;">.</span>recv(PACKET_SIZE)
time<span style="color: #303030;">.</span>sleep(DELAY)
<span style="color: green; font-weight: normal;">if</span> data:
data_list<span style="color: #303030;">.</span>append(data)
<span style="color: green; font-weight: normal;">else</span>:
retries <span style="color: #303030;">+=</span> <span style="color: #0000d0; font-weight: normal;">1</span>
<span style="color: green; font-weight: normal;">if</span> retries <span style="color: #303030;">==</span> MAX_RETRIES:
n2<span style="color: #303030;">=</span>datetime<span style="color: #303030;">.</span>datetime<span style="color: #303030;">.</span>now()
total_time <span style="color: #303030;">=</span> <span style="color: #007020;">str</span>(n2<span style="color: #303030;">-</span>n1)<span style="color: #303030;">.</span>split(<span style="background-color: #fff0f0;">":"</span>)[<span style="color: #0000d0; font-weight: normal;">2</span>]
<span style="color: green; font-weight: normal;">print</span> <span style="background-color: #fff0f0;">"TOTAL TIME (sg): "</span> <span style="color: #303030;">+</span> <span style="color: #007020;">str</span>(total_time)
<span style="color: green; font-weight: normal;">print</span> <span style="background-color: #fff0f0;">"NUMBER OF PACKETS RECEIVED: "</span> <span style="color: #303030;">+</span> <span style="color: #007020;">str</span>(<span style="color: #007020;">len</span>(data_list))
<span style="color: green; font-weight: normal;">print</span> <span style="background-color: #fff0f0;">"RATE (pkts/sg): "</span> <span style="color: #303030;">+</span> <span style="color: #007020;">str</span>(<span style="color: #007020;">len</span>(data_list) <span style="color: #303030;">/</span> <span style="color: #007020;">float</span>(total_time))
<span style="color: green; font-weight: normal;">break</span>
<span style="color: green; font-weight: normal;">if</span> __name__ <span style="color: #303030;">==</span> <span style="background-color: #fff0f0;">"__main__"</span>:
server()
</pre>
</div>
<br />
<br />
As you can observe above, the server receives packets of a fix size, in this case 1 KB, and each packet is processed (inserted into a list) without any type of delay. When the input buffer is empty (the program has reached the maximum number of retries), the script will show a little summary related to the test (total time, number of received packets and rate). Also point out that the execution time is metered by means of datetime objects.<br />
<br />
Let's take a look at now the side of the client.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">from</span> <span style="color: #0e84b5; font-weight: normal;">socket</span> <span style="color: green; font-weight: normal;">import</span> socket, AF_INET, SOCK_STREAM
PORT <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">22222</span>
PACKET_SIZE <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">1024</span>
PACKETS_NUMBER <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">10000</span>
<span style="color: green; font-weight: normal;">if</span> __name__ <span style="color: #303030;">==</span> <span style="background-color: #fff0f0;">"__main__"</span>:
padding <span style="color: #303030;">=</span> <span style="background-color: #fff0f0;">""</span>
<span style="color: green; font-weight: normal;">for</span> i <span style="color: black; font-weight: normal;">in</span> <span style="color: #007020;">range</span>(PACKET_SIZE):
padding <span style="color: #303030;">+=</span> <span style="color: #007020;">str</span>(<span style="color: #0000d0; font-weight: normal;">1</span>)
sock <span style="color: #303030;">=</span> socket(AF_INET, SOCK_STREAM)
sock<span style="color: #303030;">.</span>connect((<span style="background-color: #fff0f0;">"172.16.75.132"</span>, PORT))
<span style="color: green; font-weight: normal;">for</span> i <span style="color: black; font-weight: normal;">in</span> <span style="color: #007020;">range</span>(PACKETS_NUMBER):
sock<span style="color: #303030;">.</span>send(padding)
</pre>
</div>
<br />
<br />
The client script just sends a fix number of packets (10.000 in my case) with a concrete size (1 KB).<br />
<br />
For the test, I am going to run the server on a virtual machine with Ubuntu Server 12.10 installed on it, and the client will be executed on my personal PC against the aforementioned server. This is the output from the server.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #f0a0a0; color: #f00000;">$</span> python server<span style="color: #303030;">.</span>py
TOTAL TIME (sg): <span style="color: #6000e0; font-weight: normal;">00.138241</span>
NUMBER OF PACKETS RECEIVED: <span style="color: #0000d0; font-weight: normal;">10005</span>
RATE (pkts<span style="color: #303030;">/</span>sg): <span style="color: #6000e0; font-weight: normal;">72373.6084085</span>
</pre>
</div>
<br />
<br />
As you can see, this is an excellent result. More than 70.000 packets have been received and inserted into a list by the server, in spite of being working with sockets in Python. This result came out more than sufficient for our purposes and allowed us to be able to put into practice our required servers in Python.<br />
<br />
<br />Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-1494477346885932431.post-5021863318547782602013-04-06T11:46:00.000+02:002013-04-06T11:46:02.343+02:00Updating the key name in a Python dictionaryThis is going to be a short text just to remind me how to modify quickly the name of a key in a Python dictionary. It is straightforward operation that I would like to comment. The idea consists in popping the key and its value, and then, putting it into the dictionary again with a new name.<br />
<br />
Let's see an example. The following dictionary has a couple of keys, protocol and size, and I want to change the "protocol" key to "inet_protocol", and in addition, preserve its content.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #303030;">>>></span> my_dict <span style="color: #303030;">=</span> {<span style="background-color: #fff0f0;">"protocol"</span>: <span style="background-color: #fff0f0;">"HTTP"</span>, <span style="background-color: #fff0f0;">"size"</span>: <span style="color: #0000d0; font-weight: normal;">1024</span>}
<span style="color: #303030;">>>></span> my_dict[<span style="background-color: #fff0f0;">"inet_protocol"</span>] <span style="color: #303030;">=</span> my_dict<span style="color: #303030;">.</span>pop(<span style="background-color: #fff0f0;">"protocol"</span>)
<span style="color: #303030;">>>></span>
<span style="color: #303030;">>>></span> <span style="color: green; font-weight: normal;">print</span> my_dict
{<span style="background-color: #fff0f0;">'inet_protocol'</span>: <span style="background-color: #fff0f0;">'HTTP'</span>, <span style="background-color: #fff0f0;">'size'</span>: <span style="color: #0000d0; font-weight: normal;">1024</span>}
</pre>
</div>
<br />
<br />Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-1494477346885932431.post-33408626198414180212013-03-28T11:34:00.000+01:002013-03-28T11:34:55.982+01:00Getting hold of a remote PID through ParamikoBy means of this article, I would like to share with you a little trick so as to obtain the PID of a remote process run through Paramiko. That is, figure for a moment that you launch an application on a remote system by using Paramiko, and you would like to kill that process after a certain timeout.<br />
<br />
To be able to do that, you will have to grab the PID of that process after running it, and if you do not want to use "grep" or other kind of filters like this, the shell provides a curious mechanism which consists in executing that process in a new shell and afterwards, reading the PID of that process which has just been run.<br />
<br />
Let's see an example.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">import</span> <span style="color: #0e84b5; font-weight: normal;">paramiko</span>
<span style="color: green; font-weight: normal;">if</span> __name__ <span style="color: #303030;">==</span> <span style="background-color: #fff0f0;">"__main__"</span>:
ssh <span style="color: #303030;">=</span> paramiko<span style="color: #303030;">.</span>SSHClient()
ssh<span style="color: #303030;">.</span>set_missing_host_key_policy(paramiko<span style="color: #303030;">.</span>AutoAddPolicy())
ssh<span style="color: #303030;">.</span>connect(<span style="background-color: #fff0f0;">"ubuntu-server.local"</span>, username<span style="color: #303030;">=</span><span style="background-color: #fff0f0;">"javi"</span>, password<span style="color: #303030;">=</span><span style="background-color: #fff0f0;">"xxxxxx"</span>)
_, stdout, _ <span style="color: #303030;">=</span> ssh<span style="color: #303030;">.</span>exec_command(<span style="background-color: #fff0f0;">"echo $$ ; exec python test.py"</span>)
pid <span style="color: #303030;">=</span> stdout<span style="color: #303030;">.</span>readline()
<span style="color: green; font-weight: normal;">print</span> <span style="background-color: #fff0f0;">"PID of the remote process: "</span> <span style="color: #303030;">+</span> pid
_, stdout, _ <span style="color: #303030;">=</span> ssh<span style="color: #303030;">.</span>exec_command(<span style="background-color: #fff0f0;">"ps -eo pid,command | grep </span><span style="background-color: #e0e0e0;">%s</span><span style="background-color: #fff0f0;">"</span> <span style="color: #303030;">%</span> pid)
<span style="color: green; font-weight: normal;">print</span> <span style="background-color: #fff0f0;">"Looking for the process:</span><span style="background-color: #fff0f0; color: #606060; font-weight: normal;">\n</span><span style="background-color: #fff0f0;">"</span> <span style="color: #303030;">+</span> stdout<span style="color: #303030;">.</span>read()
ssh<span style="color: #303030;">.</span>exec_command(<span style="background-color: #fff0f0;">"kill -s SIGINT </span><span style="background-color: #e0e0e0;">%s</span><span style="background-color: #fff0f0;">"</span> <span style="color: #303030;">%</span> pid)
ssh<span style="color: #303030;">.</span>close()
</pre>
</div>
<br />
<br />
As you can see above, I am running on a remote server a straightforward script called test.py with the following lines.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">while</span> <span style="color: #007020;">True</span>:
<span style="color: green; font-weight: normal;">pass</span>
</pre>
</div>
<br />
<br />
Before executing the script through exec (replaces the current process image with a new one), I have put the order "echo $$", and in this way, I will be able to dump the PID of that process. Finally, I am using that PID to search for that process by means of ps and kill it. Let's run the module.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #f0a0a0; color: #f00000;">$</span> python remote_pid<span style="color: #303030;">.</span>py
PID of the remote process: <span style="color: #0000d0; font-weight: normal;">6020</span>
Looking <span style="color: green; font-weight: normal;">for</span> the process:
<span style="color: #0000d0; font-weight: normal;">6020</span> python test<span style="color: #303030;">.</span>py
<span style="color: #0000d0; font-weight: normal;">6021</span> bash <span style="color: #303030;">-</span>c ps <span style="color: #303030;">-</span>eo pid,command <span style="color: #303030;">|</span> grep <span style="color: #0000d0; font-weight: normal;">6020</span>
<span style="color: #0000d0; font-weight: normal;">6023</span> grep <span style="color: #0000d0; font-weight: normal;">6020</span>
</pre>
</div>
<br />
<br />Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-1494477346885932431.post-3143633583879458062013-03-17T12:14:00.000+01:002013-03-17T12:14:49.259+01:00Setting up a CVS server on UbuntuInstalling a local CVS server can be really useful although you do not have huge information to control or many people working on the same project. You can just want to have controlled all your documents. For this case and many others, you can quickly set up a version controller such as CVS.<br />
<br />
So let's get started by installing the CVS client and server (Ubuntu Server 12.10). During the installation, you have to choose a name for your repository (or repositories). In my case I went with "myrepos".<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #f0a0a0; color: #f00000;">$</span> sudo aptitude install cvs cvsd
</pre>
</div>
<br />
<br />
Then you have to initialize the repository and change the directory where CVS lock files will be dropped off (instead of directly in the repository by default). Remember to create that new directory.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #f0a0a0; color: #f00000;">$</span> sudo cvs <span style="color: #303030;">-</span>d <span style="color: #303030;">/</span>var<span style="color: #303030;">/</span>lib<span style="color: #303030;">/</span>cvsd<span style="color: #303030;">/</span>myrepos init
<span style="background-color: #f0a0a0; color: #f00000;">$</span> sudo vim <span style="color: #303030;">/</span>var<span style="color: #303030;">/</span>lib<span style="color: #303030;">/</span>cvsd<span style="color: #303030;">/</span>myrepos<span style="color: #303030;">/</span>CVSROOT<span style="color: #303030;">/</span>config
<span style="color: #303030;">...</span>
LockDir<span style="color: #303030;">=/</span>tmp<span style="color: #303030;">/</span>myrepos
<span style="background-color: #f0a0a0; color: #f00000;">$</span> sudo mkdir <span style="color: #303030;">/</span>var<span style="color: #303030;">/</span>lib<span style="color: #303030;">/</span>cvsd<span style="color: #303030;">/</span>tmp<span style="color: #303030;">/</span>myrepos
</pre>
</div>
<br />
<br />
Now you only have to create an user for the repository and modify its owner.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #f0a0a0; color: #f00000;">$</span> sudo cvsd<span style="color: #303030;">-</span>passwd <span style="color: #303030;">/</span>var<span style="color: #303030;">/</span>lib<span style="color: #303030;">/</span>cvsd<span style="color: #303030;">/</span>myrepos javi
<span style="color: #303030;">/</span>usr<span style="color: #303030;">/</span>sbin<span style="color: #303030;">/</span>cvsd<span style="color: #303030;">-</span>passwd: adding user <span style="background-color: #fff0f0;">'javi'</span> to <span style="background-color: #fff0f0;">'/var/lib/cvsd/myrepos/CVSROOT/passwd'</span>
<span style="background-color: #f0a0a0; color: #f00000;">$</span> sudo chown <span style="color: #303030;">-</span>R cvsd:cvsd <span style="color: #303030;">/</span>var<span style="color: #303030;">/</span>lib<span style="color: #303030;">/</span>cvsd
</pre>
</div>
<br />
<br />
Let's restart the server and check it out from the localhost.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #f0a0a0; color: #f00000;">$</span> sudo <span style="color: #303030;">/</span>etc<span style="color: #303030;">/</span>init<span style="color: #303030;">.</span>d<span style="color: #303030;">/</span>cvsd restart
<span style="background-color: #f0a0a0; color: #f00000;">$</span> sudo cvs <span style="color: #303030;">-</span>d :pserver:javi<span style="color: #505050; font-weight: bold;">@localhost</span>:<span style="color: #303030;">/</span>myrepos login
Logging <span style="color: black; font-weight: bold;">in</span> to :pserver:javi<span style="color: #505050; font-weight: bold;">@localhost</span>:<span style="color: #0000d0; font-weight: bold;">2401</span><span style="color: #303030;">/</span>myrepos
CVS password:
</pre>
</div>
<br />
<br />Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-1494477346885932431.post-58626621225077114242013-03-02T17:14:00.001+01:002013-03-02T17:14:39.438+01:00Speeding up Python with Cython (III)Let's carry on with the last article about speeding up Python with Cython. After running the test.py script again by using now the new version of the module, we obtain as follows.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">cython<span style="background-color: #f0a0a0; color: #f00000;">$</span> python test<span style="color: #303030;">.</span>py
<span style="color: #6000e0; font-weight: normal;">0.0394711971283</span>
</pre>
</div>
<br />
<br />
Now the improvement achieves the 98% apropos of the first round. Let's take a look at the new HTML file in order to discern which parts of the code have been directly turned into pure C code.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-DoYao2OHNC0/USDzWMni5KI/AAAAAAAAAFE/htLAZJeX9NU/s1600/cython2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="348" src="http://4.bp.blogspot.com/-DoYao2OHNC0/USDzWMni5KI/AAAAAAAAAFE/htLAZJeX9NU/s640/cython2.png" width="640" /></a></div>
<br />
<br />
From the previous picture, it can be asserted that apart from the two static types, a couple of critical structures have been also translated into C: the loops. Cython also provides another C-style syntax so as to write loops. The following file uses this kind of structure, but remember that will not produce any improvement because those loops have already been translated into C.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">calculate_primes</span>(<span style="color: #007020;">int</span> limit):
primes <span style="color: #303030;">=</span> []
cdef <span style="color: #007020;">int</span> number <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
cdef <span style="color: #007020;">int</span> divisor <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
<span style="color: green; font-weight: normal;">for</span> number <span style="color: green; font-weight: normal;">from</span> <span style="background-color: #f0a0a0; color: #f00000;">2</span> <span style="background-color: #f0a0a0; color: #f00000;"><=</span> <span style="color: #0e84b5; font-weight: normal;">number</span> <span style="background-color: #f0a0a0; color: #f00000;"><=</span> <span style="color: #0e84b5; font-weight: normal;">limit</span><span style="background-color: #f0a0a0; color: #f00000;">:</span>
<span style="color: green; font-weight: normal;">for</span> divisor <span style="color: green; font-weight: normal;">from</span> <span style="background-color: #f0a0a0; color: #f00000;">2</span> <span style="background-color: #f0a0a0; color: #f00000;"><=</span> <span style="color: #0e84b5; font-weight: normal;">divisor</span> <span style="background-color: #f0a0a0; color: #f00000;"><=</span> <span style="color: #0e84b5; font-weight: normal;">number</span><span style="background-color: #f0a0a0; color: #f00000;">:</span>
<span style="color: green; font-weight: normal;">if</span> number <span style="color: #303030;">%</span> divisor <span style="color: #303030;">==</span> <span style="color: #0000d0; font-weight: normal;">0</span> <span style="color: black; font-weight: normal;">and</span> number <span style="color: #303030;">==</span> divisor:
primes<span style="color: #303030;">.</span>append(number)
<span style="color: green; font-weight: normal;">break</span>
<span style="color: green; font-weight: normal;">elif</span> number <span style="color: #303030;">%</span> divisor <span style="color: #303030;">==</span> <span style="color: #0000d0; font-weight: normal;">0</span> <span style="color: black; font-weight: normal;">and</span> number <span style="color: #303030;">!=</span> divisor:
<span style="color: green; font-weight: normal;">break</span>
<span style="color: green; font-weight: normal;">return</span> primes
</pre>
</div>
<br />
<br />
You can check the documentation located on the Cython website and learn lots of code structures provided by Cython. For example, if this method was embedded in the main script (not as an imported module), you might develop the next optimized method, where the function returns a pointer to an array, and that array has been defined as static and not as a Python list.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">cdef <span style="color: #007020;">int</span> <span style="color: #303030;">*</span>calculate_primes(<span style="color: #007020;">int</span> limit):
cdef <span style="color: #007020;">int</span> primes[<span style="color: #0000d0; font-weight: normal;">10000</span>]
cdef <span style="color: #007020;">int</span> number <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
cdef <span style="color: #007020;">int</span> divisor <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
cdef <span style="color: #007020;">int</span> i <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
<span style="color: green; font-weight: normal;">for</span> number <span style="color: green; font-weight: normal;">from</span> <span style="background-color: #f0a0a0; color: #f00000;">2</span> <span style="background-color: #f0a0a0; color: #f00000;"><=</span> <span style="color: #0e84b5; font-weight: normal;">number</span> <span style="background-color: #f0a0a0; color: #f00000;"><=</span> <span style="color: #0e84b5; font-weight: normal;">limit</span><span style="background-color: #f0a0a0; color: #f00000;">:</span>
<span style="color: green; font-weight: normal;">for</span> divisor <span style="color: green; font-weight: normal;">from</span> <span style="background-color: #f0a0a0; color: #f00000;">2</span> <span style="background-color: #f0a0a0; color: #f00000;"><=</span> <span style="color: #0e84b5; font-weight: normal;">divisor</span> <span style="background-color: #f0a0a0; color: #f00000;"><=</span> <span style="color: #0e84b5; font-weight: normal;">number</span><span style="background-color: #f0a0a0; color: #f00000;">:</span>
<span style="color: green; font-weight: normal;">if</span> number <span style="color: #303030;">%</span> divisor <span style="color: #303030;">==</span> <span style="color: #0000d0; font-weight: normal;">0</span> <span style="color: black; font-weight: normal;">and</span> number <span style="color: #303030;">==</span> divisor:
primes[i] <span style="color: #303030;">=</span> number
i <span style="color: #303030;">+=</span> <span style="color: #0000d0; font-weight: normal;">1</span>
<span style="color: green; font-weight: normal;">break</span>
<span style="color: green; font-weight: normal;">elif</span> number <span style="color: #303030;">%</span> divisor <span style="color: #303030;">==</span> <span style="color: #0000d0; font-weight: normal;">0</span> <span style="color: black; font-weight: normal;">and</span> number <span style="color: #303030;">!=</span> divisor:
<span style="color: green; font-weight: normal;">break</span>
<span style="color: green; font-weight: normal;">return</span> primes
</pre>
</div>
<br />
<br />
Let's test out again the HTML file generated for this method.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-oN78SwkO3cM/USEAxqMyRcI/AAAAAAAAAFU/uFHy6k1Tm14/s1600/cython3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="374" src="http://4.bp.blogspot.com/-oN78SwkO3cM/USEAxqMyRcI/AAAAAAAAAFU/uFHy6k1Tm14/s640/cython3.png" width="640" /></a></div>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-90352056714795298512013-02-23T18:17:00.000+01:002013-02-23T18:17:20.488+01:00Speeding up Python performance with Cython (II)Following up on the last article and as you can see in the previous script, first of all the full path of the Cython installation has been added to the PYTHONPATH. By running this script, both a C file (calculate_primes1.c) and a loadable module by Python (calculate_primes1.so) are been generated.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">cython<span style="background-color: #f0a0a0; color: #f00000;">$</span> python setup<span style="color: #303030;">.</span>py build_ext <span style="color: #303030;">--</span>inplace
running build_ext
cythoning calculate_primes1<span style="color: #303030;">.</span>pyx to calculate_primes1<span style="color: #303030;">.</span>c
building <span style="background-color: #fff0f0;">'calculate_primes1'</span> extension
gcc <span style="color: #303030;">-</span>pthread <span style="color: #303030;">-</span>fno<span style="color: #303030;">-</span>strict<span style="color: #303030;">-</span>aliasing <span style="color: #303030;">-</span>DNDEBUG <span style="color: #303030;">-</span>fmessage<span style="color: #303030;">-</span>length<span style="color: #303030;">=</span><span style="color: #0000d0; font-weight: normal;">0</span> <span style="color: #303030;">-</span>O2 <span style="color: #303030;">-</span>Wall <span style="color: #303030;">-</span>D_FORTIFY_SOURCE<span style="color: #303030;">=</span><span style="color: #0000d0; font-weight: normal;">2</span> <span style="color: #303030;">-</span>fstack<span style="color: #303030;">-</span>protector <span style="color: #303030;">-</span>funwind<span style="color: #303030;">-</span>tables <span style="color: #303030;">-</span>fasynchronous<span style="color: #303030;">-</span>unwind<span style="color: #303030;">-</span>tables <span style="color: #303030;">-</span>g <span style="color: #303030;">-</span>fwrapv <span style="color: #303030;">-</span>fPIC <span style="color: #303030;">-</span>I<span style="color: #303030;">/</span>usr<span style="color: #303030;">/</span>include<span style="color: #303030;">/</span>python2<span style="color: #303030;">.</span><span style="color: #0000d0; font-weight: normal;">6</span> <span style="color: #303030;">-</span>c calculate_primes1<span style="color: #303030;">.</span>c <span style="color: #303030;">-</span>o build<span style="color: #303030;">/</span>temp<span style="color: #303030;">.</span>linux<span style="color: #303030;">-</span>x86_64<span style="color: #303030;">-</span><span style="color: #6000e0; font-weight: normal;">2.6</span><span style="color: #303030;">/</span>calculate_primes1<span style="color: #303030;">.</span>o
gcc <span style="color: #303030;">-</span>pthread <span style="color: #303030;">-</span>shared build<span style="color: #303030;">/</span>temp<span style="color: #303030;">.</span>linux<span style="color: #303030;">-</span>x86_64<span style="color: #303030;">-</span><span style="color: #6000e0; font-weight: normal;">2.6</span><span style="color: #303030;">/</span>calculate_primes1<span style="color: #303030;">.</span>o <span style="color: #303030;">-</span>L<span style="color: #303030;">/</span>usr<span style="color: #303030;">/</span>lib64 <span style="color: #303030;">-</span>lpython2<span style="color: #303030;">.</span><span style="color: #0000d0; font-weight: normal;">6</span> <span style="color: #303030;">-</span>o calculate_primes1<span style="color: #303030;">.</span>so
cython<span style="background-color: #f0a0a0; color: #f00000;">$</span> ls <span style="color: #303030;">-</span>l
drwxr<span style="color: #303030;">-</span>xr<span style="color: #303030;">-</span>x <span style="color: #0000d0; font-weight: normal;">3</span> javi javi <span style="color: #0000d0; font-weight: normal;">4096</span> Feb <span style="color: #0000d0; font-weight: normal;">16</span> <span style="color: #0000d0; font-weight: normal;">17</span>:<span style="color: #0000d0; font-weight: normal;">46</span> build
<span style="color: #303030;">-</span>rw<span style="color: #303030;">-</span>r<span style="color: #303030;">--</span>r<span style="color: #303030;">--</span> <span style="color: #0000d0; font-weight: normal;">1</span> javi javi <span style="color: #0000d0; font-weight: normal;">68295</span> Feb <span style="color: #0000d0; font-weight: normal;">17</span> <span style="color: #0000d0; font-weight: normal;">11</span>:<span style="color: #0000d0; font-weight: normal;">27</span> calculate_primes1<span style="color: #303030;">.</span>c
<span style="color: #303030;">-</span>rw<span style="color: #303030;">-</span>r<span style="color: #303030;">--</span>r<span style="color: #303030;">--</span> <span style="color: #0000d0; font-weight: normal;">1</span> javi javi <span style="color: #0000d0; font-weight: normal;">393</span> Feb <span style="color: #0000d0; font-weight: normal;">17</span> <span style="color: #0000d0; font-weight: normal;">11</span>:<span style="color: #4000e0; font-weight: normal;">06</span> calculate_primes1<span style="color: #303030;">.</span>pyx
<span style="color: #303030;">-</span>rwxr<span style="color: #303030;">-</span>xr<span style="color: #303030;">-</span>x <span style="color: #0000d0; font-weight: normal;">1</span> javi javi <span style="color: #0000d0; font-weight: normal;">53388</span> Feb <span style="color: #0000d0; font-weight: normal;">17</span> <span style="color: #0000d0; font-weight: normal;">11</span>:<span style="color: #0000d0; font-weight: normal;">27</span> calculate_primes1<span style="color: #303030;">.</span>so
<span style="color: #303030;">-</span>rwxr<span style="color: #303030;">-</span>xr<span style="color: #303030;">-</span>x <span style="color: #0000d0; font-weight: normal;">1</span> javi javi <span style="color: #0000d0; font-weight: normal;">393</span> Feb <span style="color: #0000d0; font-weight: normal;">16</span> <span style="color: #0000d0; font-weight: normal;">21</span>:<span style="color: #0000d0; font-weight: normal;">43</span> calculate_primes<span style="color: #303030;">.</span>py
<span style="color: #303030;">-</span>rwxr<span style="color: #303030;">-</span>xr<span style="color: #303030;">-</span>x <span style="color: #0000d0; font-weight: normal;">1</span> javi javi <span style="color: #0000d0; font-weight: normal;">530</span> Feb <span style="color: #0000d0; font-weight: normal;">16</span> <span style="color: #0000d0; font-weight: normal;">21</span>:<span style="color: #0000d0; font-weight: normal;">49</span> calculate_primes<span style="color: #303030;">.</span>pyc
drwxr<span style="color: #303030;">-</span>xr<span style="color: #303030;">-</span>x <span style="color: #0000d0; font-weight: normal;">10</span> javi javi <span style="color: #0000d0; font-weight: normal;">4096</span> Feb <span style="color: #0000d0; font-weight: normal;">16</span> <span style="color: #0000d0; font-weight: normal;">16</span>:<span style="color: #0000d0; font-weight: normal;">34</span> Cython<span style="color: #303030;">-</span><span style="color: #6000e0; font-weight: normal;">0.18</span>
<span style="color: #303030;">-</span>rwxr<span style="color: #303030;">-</span>xr<span style="color: #303030;">-</span>x <span style="color: #0000d0; font-weight: normal;">1</span> javi javi <span style="color: #0000d0; font-weight: normal;">314</span> Feb <span style="color: #0000d0; font-weight: normal;">17</span> <span style="color: #0000d0; font-weight: normal;">11</span>:<span style="color: #4000e0; font-weight: normal;">06</span> setup<span style="color: #303030;">.</span>py
<span style="color: #303030;">-</span>rwxr<span style="color: #303030;">-</span>xr<span style="color: #303030;">-</span>x <span style="color: #0000d0; font-weight: normal;">1</span> javi javi <span style="color: #0000d0; font-weight: normal;">426</span> Feb <span style="color: #0000d0; font-weight: normal;">16</span> <span style="color: #0000d0; font-weight: normal;">21</span>:<span style="color: #0000d0; font-weight: normal;">53</span> test<span style="color: #303030;">.</span>py
</pre>
</div>
<br />
<br />
If you take a look at the C file, you will appreciate that is a huge file (compared with the Python version) where some parts of the code have been translated to C and others keep as Python API calls.<br />
<br />
A handy tool aimed at turning out an HTML report which shows the Cython code interlined with the C code, is the own compiler executed with the "-a" option.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">cython<span style="background-color: #f0a0a0; color: #f00000;">$</span> Cython<span style="color: #303030;">-</span><span style="color: #6000e0; font-weight: normal;">0.18</span><span style="color: #303030;">/</span>cython<span style="color: #303030;">.</span>py <span style="color: #303030;">-</span>a calculate_primes1<span style="color: #303030;">.</span>pyx
</pre>
</div>
<br />
<br />
If you open the come out HTML file, you will see that lines are colored according to the level of "typedness" (white lines translates to pure C without any Python API calls).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-QZn1ZDPBhL4/USC75mG1DmI/AAAAAAAAAE0/ATVkF_Y_3i4/s1600/cython1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="345" src="http://3.bp.blogspot.com/-QZn1ZDPBhL4/USC75mG1DmI/AAAAAAAAAE0/ATVkF_Y_3i4/s640/cython1.png" width="640" /></a></div>
<br />
<br />
Now let's run again this new version of the module compiled with Cython (remember to change the name of the module inside the test.py file). As you can derive from the result, the performance has been enhanced about 30%.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">cython<span style="background-color: #f0a0a0; color: #f00000;">$</span> python test<span style="color: #303030;">.</span>py
<span style="color: #6000e0; font-weight: normal;">1.20011019707</span>
</pre>
</div>
<br />
<br />
Let's go a step further and add some static types to a second version of our module (called now calculate_primes2).<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">calculate_primes</span>(<span style="color: #007020;">int</span> limit):
primes <span style="color: #303030;">=</span> []
cdef <span style="color: #007020;">int</span> number <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
cdef <span style="color: #007020;">int</span> divisor <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
<span style="color: green; font-weight: normal;">for</span> number <span style="color: black; font-weight: normal;">in</span> <span style="color: #007020;">range</span>(limit):
<span style="color: green; font-weight: normal;">for</span> divisor <span style="color: black; font-weight: normal;">in</span> <span style="color: #007020;">range</span>(<span style="color: #0000d0; font-weight: normal;">2</span>, number<span style="color: #303030;">+</span><span style="color: #0000d0; font-weight: normal;">1</span>):
<span style="color: green; font-weight: normal;">if</span> number <span style="color: #303030;">%</span> divisor <span style="color: #303030;">==</span> <span style="color: #0000d0; font-weight: normal;">0</span> <span style="color: black; font-weight: normal;">and</span> number <span style="color: #303030;">==</span> divisor:
primes<span style="color: #303030;">.</span>append(number)
<span style="color: green; font-weight: normal;">break</span>
<span style="color: green; font-weight: normal;">elif</span> number <span style="color: #303030;">%</span> divisor <span style="color: #303030;">==</span> <span style="color: #0000d0; font-weight: normal;">0</span> <span style="color: black; font-weight: normal;">and</span> number <span style="color: #303030;">!=</span> divisor:
<span style="color: green; font-weight: normal;">break</span>
<span style="color: green; font-weight: normal;">return</span> primes
</pre>
</div>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-20912615461369677262013-02-17T17:29:00.000+01:002013-02-17T17:39:38.798+01:00Speeding up Python performance with Cython (I)Cython is a programming language which creates C/C++ extensions for Python, that is, is able to translate some parts of a Python program into C code, and in this way, increasing considerably the execution time of a Python program. In addition, provides some predefined constructions that you can use directly when you develop in Cython.<br />
<br />
Let's see a Python method which takes care of calculating all prime numbers within a range (calculate_primes.py).<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">calculate_primes</span>(limit):
primes <span style="color: #303030;">=</span> []
number <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
divisor <span style="color: #303030;">=</span> <span style="color: #0000d0; font-weight: normal;">0</span>
<span style="color: green; font-weight: normal;">for</span> number <span style="color: black; font-weight: normal;">in</span> <span style="color: #007020;">range</span>(limit):
<span style="color: green; font-weight: normal;">for</span> divisor <span style="color: black; font-weight: normal;">in</span> <span style="color: #007020;">range</span>(<span style="color: #0000d0; font-weight: normal;">2</span>, number<span style="color: #303030;">+</span><span style="color: #0000d0; font-weight: normal;">1</span>):
<span style="color: green; font-weight: normal;">if</span> number <span style="color: #303030;">%</span> divisor <span style="color: #303030;">==</span> <span style="color: #0000d0; font-weight: normal;">0</span> <span style="color: black; font-weight: normal;">and</span> number <span style="color: #303030;">==</span> divisor:
primes<span style="color: #303030;">.</span>append(number)
<span style="color: green; font-weight: normal;">break</span>
<span style="color: green; font-weight: normal;">elif</span> number <span style="color: #303030;">%</span> divisor <span style="color: #303030;">==</span> <span style="color: #0000d0; font-weight: normal;">0</span> <span style="color: black; font-weight: normal;">and</span> number <span style="color: #303030;">!=</span> divisor:
<span style="color: green; font-weight: normal;">break</span>
<span style="color: green; font-weight: normal;">return</span> primes
</pre>
</div>
<br />
<br />
Now let's import this module in another program and run it in order to work out the time spent in searching for all prime numbers between 2 and 10000 (the application will be executed five times and averaged it).<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #f0a0a0; color: #f00000;">$</span> cat test<span style="color: #303030;">.</span>py
<span style="color: green; font-weight: normal;">from</span> <span style="color: #0e84b5; font-weight: normal;">timeit</span> <span style="color: green; font-weight: normal;">import</span> Timer
timer <span style="color: #303030;">=</span> Timer(<span style="background-color: #fff0f0;">"calculate_primes(10000)"</span>,
<span style="background-color: #fff0f0;">"from calculate_primes import calculate_primes"</span>)
<span style="color: green; font-weight: normal;">print</span> <span style="color: #007020;">sum</span>(timer<span style="color: #303030;">.</span>repeat(repeat<span style="color: #303030;">=</span><span style="color: #0000d0; font-weight: normal;">5</span>, number<span style="color: #303030;">=</span><span style="color: #0000d0; font-weight: normal;">1</span>)) <span style="color: #303030;">/</span> <span style="color: #0000d0; font-weight: normal;">5</span>
<span style="background-color: #f0a0a0; color: #f00000;">$</span> python test<span style="color: #303030;">.</span>py
<span style="color: #6000e0; font-weight: normal;">1.7978372097</span>
</pre>
</div>
<br />
<br />
The next step will be to compile this program with Cython in order to generate C extensions for our program. So as to install Cython, you have a couple of options: either to grab directly its source code from the web page of the project, or install it from some repository. In my case, I am going to download its source code and thereby, to be able to have the most recent version (0.18 at present).<br />
<br />
After downloading and uncompressing it, you will be able to install it on your system by running the setup.py script, or as in my case, just decompressing it on the home directory and using that path later. Also point out that you must have installed on your system all tools needed to compile C/C++ programs (for example, the package build-essential in case of Ubuntu).<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #f0a0a0; color: #f00000;">$</span> mkdir cython ; cd cython
cython<span style="background-color: #f0a0a0; color: #f00000;">$</span> wget http:<span style="color: #303030;">//</span>cython<span style="color: #303030;">.</span>org<span style="color: #303030;">/</span>release<span style="color: #303030;">/</span>Cython<span style="color: #303030;">-</span><span style="color: #6000e0; font-weight: bold;">0.18</span><span style="color: #303030;">.</span>zip ; unzip Cython<span style="color: #303030;">-</span><span style="color: #6000e0; font-weight: bold;">0.18</span><span style="color: #303030;">.</span>zip
</pre>
</div>
<br />
<br />
The first test will be to compile the module called calculate_primes with Cython, so as to generate on the one hand a C file, and on the other, a module loadable by Python. First up, you need to change the extension of the module from py to pyx. And to compile it, you might do it by hand, but it is much better to create a tiny script responsible of accomplishing it.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">cython<span style="background-color: #f0a0a0; color: #f00000;">$</span> cp <span style="color: #303030;">-</span>a calculate_primes<span style="color: #303030;">.</span>py calculate_primes1<span style="color: #303030;">.</span>pyx
cython<span style="background-color: #f0a0a0; color: #f00000;">$</span> cat setup<span style="color: #303030;">.</span>py
<span style="color: green; font-weight: normal;">import</span> <span style="color: #0e84b5; font-weight: normal;">sys</span>
sys<span style="color: #303030;">.</span>path<span style="color: #303030;">.</span>insert(<span style="color: #0000d0; font-weight: normal;">0</span>, <span style="background-color: #fff0f0;">"/home/javi/cython/Cython-0.18"</span>)
<span style="color: green; font-weight: normal;">from</span> <span style="color: #0e84b5; font-weight: normal;">distutils.core</span> <span style="color: green; font-weight: normal;">import</span> setup
<span style="color: green; font-weight: normal;">from</span> <span style="color: #0e84b5; font-weight: normal;">distutils.extension</span> <span style="color: green; font-weight: normal;">import</span> Extension
<span style="color: green; font-weight: normal;">from</span> <span style="color: #0e84b5; font-weight: normal;">Cython.Distutils</span> <span style="color: green; font-weight: normal;">import</span> build_ext
setup(
cmdclass <span style="color: #303030;">=</span> {<span style="background-color: #fff0f0;">'build_ext'</span>: build_ext},
ext_modules <span style="color: #303030;">=</span> [Extension(<span style="background-color: #fff0f0;">"calculate_primes1"</span>, [<span style="background-color: #fff0f0;">"calculate_primes1.pyx"</span>])]
)
</pre>
</div>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-57392495752798543862013-02-10T17:44:00.000+01:002013-02-16T21:34:45.904+01:00Paramiko object with Process (RNG must be re-initialized after fork)When you create an object in Python which will be used later by an object of type Process, that works correctly. Let's see an example where an object of type Process prints an attribute provided by another object belonging to the class A.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;"><pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">from</span> <span style="color: #0e84b5; font-weight: normal;">multiprocessing</span> <span style="color: green; font-weight: normal;">import</span> Process
<span style="color: green; font-weight: normal;">class</span> <span style="color: #b00060; font-weight: normal;">A</span>():
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">__init__</span>(<span style="color: #007020;">self</span>):
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>my_dict <span style="color: #303030;">=</span> {<span style="background-color: #fff0f0;">"day"</span>: <span style="background-color: #fff0f0;">"Tuesday"</span>}
<span style="color: green; font-weight: normal;">class</span> <span style="color: #b00060; font-weight: normal;">B</span>():
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">__init__</span>(<span style="color: #007020;">self</span>):
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>a <span style="color: #303030;">=</span> A()
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__process <span style="color: #303030;">=</span> <span style="color: #007020;">None</span>
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">start</span>(<span style="color: #007020;">self</span>):
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__process <span style="color: #303030;">=</span> Process(target<span style="color: #303030;">=</span><span style="color: #007020;">self</span><span style="color: #303030;">.</span>__run_process)
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__process<span style="color: #303030;">.</span>start()
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">__run_process</span>(<span style="color: #007020;">self</span>):
<span style="color: green; font-weight: normal;">print</span> <span style="color: #007020;">self</span><span style="color: #303030;">.</span>a<span style="color: #303030;">.</span>my_dict
<span style="color: green; font-weight: normal;">if</span> __name__ <span style="color: #303030;">==</span> <span style="background-color: #fff0f0;">"__main__"</span>:
b <span style="color: #303030;">=</span> B()
b<span style="color: #303030;">.</span>start()
<span style="background-color: #f0a0a0; color: #f00000;">$</span> python test<span style="color: #303030;">.</span>py
{<span style="background-color: #fff0f0;">'day'</span>: <span style="background-color: #fff0f0;">'Tuesday'</span>}
</pre></div><br />
<br />
But when that process has to handle a Paramiko object where the connection has been initialized before, will not work.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;"><pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">import</span> <span style="color: #0e84b5; font-weight: normal;">paramiko</span>
<span style="color: green; font-weight: normal;">from</span> <span style="color: #0e84b5; font-weight: normal;">multiprocessing</span> <span style="color: green; font-weight: normal;">import</span> Process
<span style="color: green; font-weight: normal;">class</span> <span style="color: #b00060; font-weight: normal;">B</span>():
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">__init__</span>(<span style="color: #007020;">self</span>):
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__process <span style="color: #303030;">=</span> <span style="color: #007020;">None</span>
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__ssh <span style="color: #303030;">=</span> paramiko<span style="color: #303030;">.</span>SSHClient()
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__ssh<span style="color: #303030;">.</span>set_missing_host_key_policy(paramiko<span style="color: #303030;">.</span>AutoAddPolicy())
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__ssh<span style="color: #303030;">.</span>connect(<span style="background-color: #fff0f0;">'127.0.0.1'</span>, username<span style="color: #303030;">=</span><span style="background-color: #fff0f0;">'javi'</span>, password<span style="color: #303030;">=</span><span style="background-color: #fff0f0;">'xxxxxx'</span>)
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">start</span>(<span style="color: #007020;">self</span>):
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__process <span style="color: #303030;">=</span> Process(target<span style="color: #303030;">=</span><span style="color: #007020;">self</span><span style="color: #303030;">.</span>__run_process)
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__process<span style="color: #303030;">.</span>start()
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">__run_process</span>(<span style="color: #007020;">self</span>):
<span style="color: green; font-weight: normal;">try</span>:
_, stdout, _ <span style="color: #303030;">=</span> <span style="color: #007020;">self</span><span style="color: #303030;">.</span>__ssh<span style="color: #303030;">.</span>exec_command(<span style="background-color: #fff0f0;">"hostname"</span>)
<span style="color: green; font-weight: normal;">except</span>:
<span style="color: green; font-weight: normal;">print</span> <span style="background-color: #fff0f0;">"Failed the execution"</span>
<span style="color: green; font-weight: normal;">else</span>:
<span style="color: green; font-weight: normal;">print</span> stdout<span style="color: #303030;">.</span>read()
<span style="color: green; font-weight: normal;">if</span> __name__ <span style="color: #303030;">==</span> <span style="background-color: #fff0f0;">"__main__"</span>:
b <span style="color: #303030;">=</span> B()
b<span style="color: #303030;">.</span>start()
<span style="background-color: #f0a0a0; color: #f00000;">$</span> python test<span style="color: #303030;">.</span>py
Failed the execution
</pre></div><br />
<br />
If you take a look at the output generated by the execution without catching it through an exception, you could observe an error as follows.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;"><pre style="line-height: 125%; margin: 0;"><span style="background-color: #f0a0a0; color: #f00000;">$</span> python test<span style="color: #303030;">.</span>py
Process Process<span style="color: #303030;">-</span><span style="color: #0000d0; font-weight: normal;">1</span>:
Traceback (most recent call last):
<span style="color: #303030;">...</span>
<span style="color: green; font-weight: normal;">raise</span> <span style="color: #f00000; font-weight: normal;">AssertionError</span>(<span style="background-color: #fff0f0;">"PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()"</span>)
<span style="color: #f00000; font-weight: normal;">AssertionError</span>: PID check failed<span style="color: #303030;">.</span> RNG must be re<span style="color: #303030;">-</span>initialized after fork()<span style="color: #303030;">.</span> Hint: Try Random<span style="color: #303030;">.</span>atfork()
</pre></div><br />
<br />
This is a well-known issue in Paramiko that has been apparently fixed in recent versions (for my tests, I am using the version provided by Ubuntu 12.10 by default: 1.7.7.1-3). A possible workaround will be for example to open a new SSH connection for that process. Let's grab an example.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;"><pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">import</span> <span style="color: #0e84b5; font-weight: normal;">paramiko</span>
<span style="color: green; font-weight: normal;">from</span> <span style="color: #0e84b5; font-weight: normal;">multiprocessing</span> <span style="color: green; font-weight: normal;">import</span> Process
<span style="color: green; font-weight: normal;">class</span> <span style="color: #b00060; font-weight: normal;">B</span>():
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">__init__</span>(<span style="color: #007020;">self</span>):
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__process <span style="color: #303030;">=</span> <span style="color: #007020;">None</span>
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__ssh <span style="color: #303030;">=</span> paramiko<span style="color: #303030;">.</span>SSHClient()
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__ssh<span style="color: #303030;">.</span>set_missing_host_key_policy(paramiko<span style="color: #303030;">.</span>AutoAddPolicy())
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">start</span>(<span style="color: #007020;">self</span>):
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__process <span style="color: #303030;">=</span> Process(target<span style="color: #303030;">=</span><span style="color: #007020;">self</span><span style="color: #303030;">.</span>__run_process)
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__process<span style="color: #303030;">.</span>start()
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">__run_process</span>(<span style="color: #007020;">self</span>):
<span style="color: #007020;">self</span><span style="color: #303030;">.</span>__ssh<span style="color: #303030;">.</span>connect(<span style="background-color: #fff0f0;">'127.0.0.1'</span>, username<span style="color: #303030;">=</span><span style="background-color: #fff0f0;">'javi'</span>, password<span style="color: #303030;">=</span><span style="background-color: #fff0f0;">'xxxxxx'</span>)
<span style="color: green; font-weight: normal;">try</span>:
_, stdout, _ <span style="color: #303030;">=</span> <span style="color: #007020;">self</span><span style="color: #303030;">.</span>__ssh<span style="color: #303030;">.</span>exec_command(<span style="background-color: #fff0f0;">"hostname"</span>)
<span style="color: green; font-weight: normal;">except</span>:
<span style="color: green; font-weight: normal;">print</span> <span style="background-color: #fff0f0;">"Failed the execution"</span>
<span style="color: green; font-weight: normal;">else</span>:
<span style="color: green; font-weight: normal;">print</span> stdout<span style="color: #303030;">.</span>read()
<span style="color: green; font-weight: normal;">if</span> __name__ <span style="color: #303030;">==</span> <span style="background-color: #fff0f0;">"__main__"</span>:
b <span style="color: #303030;">=</span> B()
b<span style="color: #303030;">.</span>start()
<span style="background-color: #f0a0a0; color: #f00000;">$</span> python test<span style="color: #303030;">.</span>py
javi<span style="color: #303030;">-</span>pc
</pre></div><br />
<br />
As you can see above, the definition of the Paramiko object takes place within the constructor of the class B, but the connection is eventually initialized by the forked process.<br />
<br />
<br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-35964090621720710462013-02-03T16:41:00.002+01:002013-02-16T21:35:27.427+01:00Merging dictionaries in Python with lists withinBecause I have just set out to program in Python at work (in depth), I am going to write down on my blog those interesting things that I have to work out related to this activity.<br />
<br />
And for example, this week I had to cope a curious problem about merging dictionaries with any kind of element within, such as lists or other dictionaries. Browsing the Internet, I was able to find some functions to combine dictionaries, but the problem was that those methods did not take into account the option of having lists inside.<br />
<br />
So in order to get over this issue, I had to develop the following two recursive methods.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; css: background:#eeeeee; overflow: auto; padding: .2em .6em; width: auto;"><pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">merge_dict</span>(dict1, dict2):
<span style="color: green; font-weight: normal;">if</span> <span style="color: #007020;">isinstance</span>(dict1, <span style="color: #007020;">dict</span>) <span style="color: black; font-weight: normal;">and</span> <span style="color: #007020;">isinstance</span>(dict2, <span style="color: #007020;">dict</span>):
<span style="color: green; font-weight: normal;">for</span> k, v <span style="color: black; font-weight: normal;">in</span> dict2<span style="color: #303030;">.</span>iteritems():
<span style="color: green; font-weight: normal;">if</span> k <span style="color: black; font-weight: normal;">not</span> <span style="color: black; font-weight: normal;">in</span> dict1:
dict1[k] <span style="color: #303030;">=</span> v
<span style="color: green; font-weight: normal;">else</span>:
dict1[k] <span style="color: #303030;">=</span> merge_dict(dict1[k], v)
<span style="color: green; font-weight: normal;">elif</span> <span style="color: #007020;">isinstance</span>(dict1, <span style="color: #007020;">list</span>) <span style="color: black; font-weight: normal;">and</span> <span style="color: #007020;">isinstance</span>(dict2, <span style="color: #007020;">list</span>):
dict1 <span style="color: #303030;">=</span> merge_list(dict1, dict2)
<span style="color: green; font-weight: normal;">elif</span> dict2 <span style="color: #303030;">==</span> <span style="color: #007020;">None</span>:
<span style="color: green; font-weight: normal;">return</span> dict1
<span style="color: green; font-weight: normal;">else</span>:
<span style="color: green; font-weight: normal;">return</span> dict2
<span style="color: green; font-weight: normal;">return</span> dict1
<span style="color: green; font-weight: normal;">def</span> <span style="color: #0060b0; font-weight: normal;">merge_list</span>(list1, list2):
<span style="color: green; font-weight: normal;">if</span> <span style="color: #007020;">isinstance</span>(list1, <span style="color: #007020;">list</span>) <span style="color: black; font-weight: normal;">and</span> <span style="color: #007020;">isinstance</span>(list2, <span style="color: #007020;">list</span>):
<span style="color: green; font-weight: normal;">for</span> i, item <span style="color: black; font-weight: normal;">in</span> <span style="color: #007020;">enumerate</span>(list2):
<span style="color: green; font-weight: normal;">if</span> <span style="color: #007020;">len</span>(list1) <span style="color: #303030;">></span> i:
<span style="color: green; font-weight: normal;">if</span> <span style="color: #007020;">isinstance</span>(list1[i], <span style="color: #007020;">dict</span>) <span style="color: black; font-weight: normal;">and</span> <span style="color: #007020;">isinstance</span>(item, <span style="color: #007020;">dict</span>):
list1[i] <span style="color: #303030;">=</span> merge_dict(list1[i], item)
<span style="color: green; font-weight: normal;">elif</span> <span style="color: #007020;">isinstance</span>(list1[i], <span style="color: #007020;">list</span>) <span style="color: black; font-weight: normal;">and</span> <span style="color: #007020;">isinstance</span>(item, <span style="color: #007020;">list</span>):
list1[i] <span style="color: #303030;">=</span> merge_list(list1[i], item)
<span style="color: green; font-weight: normal;">else</span>:
list1[i] <span style="color: #303030;">=</span> item
<span style="color: green; font-weight: normal;">else</span>:
list1<span style="color: #303030;">.</span>append(list2[i])
<span style="color: green; font-weight: normal;">elif</span> <span style="color: #007020;">isinstance</span>(list1, <span style="color: #007020;">dict</span>) <span style="color: black; font-weight: normal;">and</span> <span style="color: #007020;">isinstance</span>(list2, <span style="color: #007020;">dict</span>):
list1 <span style="color: #303030;">=</span> merge_dict(list1, list2)
<span style="color: green; font-weight: normal;">else</span>:
<span style="color: green; font-weight: normal;">return</span> list2
<span style="color: green; font-weight: normal;">return</span> list1
</pre></div><br />
<br />
As you can see in the above code, the method merge_dict loops through the dictionary and if it comes across a list, it will have a function able to merge the list. Also say that dict2 and list2 will have more priority that dict1 and list1 respectively.<br />
<br />
Let's see an example where there are a couple of dictionaries with several elements within.<br />
<br />
<div style="background: #fafafa; border: 1px dashed #999999; color: black; overflow: auto; padding: .2em .6em; width: auto;"><pre style="line-height: 125%; margin: 0;">dict1 <span style="color: #303030;">=</span> {<span style="background-color: #fff0f0;">"country"</span>: [{<span style="background-color: #fff0f0;">"name"</span>: <span style="background-color: #fff0f0;">"Spain"</span>, <span style="background-color: #fff0f0;">"capital"</span>: <span style="background-color: #fff0f0;">"Madrid"</span>}, {<span style="background-color: #fff0f0;">"name"</span>: <span style="background-color: #fff0f0;">"France"</span>}]}
dict2 <span style="color: #303030;">=</span> {<span style="background-color: #fff0f0;">"country"</span>: [{<span style="background-color: #fff0f0;">"name"</span>: <span style="background-color: #fff0f0;">"Germany"</span>}], <span style="background-color: #fff0f0;">"continent"</span>: <span style="background-color: #fff0f0;">"Europe"</span>}
<span style="color: green; font-weight: normal;">print</span> merge_dict(dict1, dict2)
</pre></div><br />
<br />
This is the output for the preceding script.<br />
<br />
<div style="background: #fafafa; background: #eeeeee; border: 1px dashed #999999; color: black; overflow: auto; padding: .2em .6em; width: auto;"><pre style="line-height: 125%; margin: 0;"><span style="color: #906030;">$ </span>python test.py
<span style="color: #303030;">{</span><span style="background-color: #fff0f0;">'country'</span>: <span style="color: #303030;">[{</span><span style="background-color: #fff0f0;">'name'</span>: <span style="background-color: #fff0f0;">'Germany'</span>, <span style="background-color: #fff0f0;">'capital'</span>: <span style="background-color: #fff0f0;">'Madrid'</span><span style="color: #303030;">}</span>, <span style="color: #303030;">{</span><span style="background-color: #fff0f0;">'name'</span>: <span style="background-color: #fff0f0;">'France'</span><span style="color: #303030;">}]</span>, <span style="background-color: #fff0f0;">'continent'</span>: <span style="background-color: #fff0f0;">'Europe'</span><span style="color: #303030;">}</span>
</pre></div><br />
<br />
Unknownnoreply@blogger.com7tag:blogger.com,1999:blog-1494477346885932431.post-77899878291943542262012-12-02T14:43:00.001+01:002012-12-02T14:43:15.668+01:00MySQL benchmark with SysBench (II)Let's carry on the preceding article about <a href="http://redes-privadas-virtuales.blogspot.com.es/2012/11/mysql-benchmark-with-sysbench-i.html">MySQL benchmark with SysBench</a> by putting into action the above scenario.<br />
<br />
For the first case, I am going to use the values recommended through the first four articles that I wrote about MySQL optimization. So according to them, the values used will be as follows. Let's call this first settings A.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
key_buffer = 32M
thread_cache_size = 512
table_cache = 512
table_definition_cache = 512
open_files_limit = 1024
tmp_table_size = 64M
max_heap_table_size = 32M
query_cache_limit = 4M
query_cache_size = 256M
query_cache_type = 1
innodb_buffer_pool_size = 1280M
</code></pre>
<br />
For the second configuration, called B, I am going to set by means of the parameter <i>innodb-flush-log-at-trx-commit</i>, when the log buffer will be written out to the log file and the flush to disk operation will be effected.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
innodb-flush-log-at-trx-commit = 0
</code></pre>
<br />
For the third combination (C), a new parameter will be added: <i>innodb_buffer_pool_instances</i>. With this option, it is possible to define the number of regions that the InnoDB area will be divided into.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
innodb_buffer_pool_instances = 2
</code></pre>
<br />
And finally, the size for the log file will be increased by means of <i>innodb_log_file_size</i>, and in this way, the graph D will be turned out (in order to be able to apply this change, first up you have to stop MySQL and remove any existing log file).<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
innodb_log_file_size = 256M
root@ubuntu-server:~# rm -rf /var/lib/mysql/ib_logfile*
</code></pre>
<br />
Below you can observe the graphs generated for the different cases.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-zrTe2gkvvLg/ULJACcrKDbI/AAAAAAAAAEA/BPYvJ5FzbNA/s1600/MySQL+benchmark+with+SysBench.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="412" src="http://3.bp.blogspot.com/-zrTe2gkvvLg/ULJACcrKDbI/AAAAAAAAAEA/BPYvJ5FzbNA/s640/MySQL+benchmark+with+SysBench.png" width="640" /></a></div>
<br />
As you can appreciate in the figure, the best improvements are achieved when the size of the log file is increased from its default value (5M) to 256M. The option C, that is, to separate the InnoDB buffer into two regions gets worse in respect of the previous alternative (B).<br />
<br />
<br />Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-1494477346885932431.post-38399957994010979092012-11-25T16:39:00.001+01:002012-11-25T16:39:21.617+01:00MySQL benchmark with SysBench (I)By taking advantage of the previous article about <a href="http://redes-privadas-virtuales.blogspot.com.es/2012/11/mysql-optimization-v.html">MySQL optimization</a>, I am going to introduce a handy tool called SysBench and aimed at measuring the performance of a MySQL database among other things. In addition, it is also able to evaluate the I/O, scheduler and threads implementation performance, and memory allocation and transfer speed.<br />
<br />
So I am going to use this tool in order to verify the improvements commented in the preceding article and related to some parameters of MySQL. The test will be run on Ubuntu Server 12.10 virtualized through VMware. The virtual machine will made up by a 6 GB hard drive, 2 GB of RAM and a couple of virtual cores.<br />
<br />
First of all, let's install MySQL and SysBench and increase the default number of maximum connections to 512.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# aptitude install mysql-server sysbench
root@ubuntu-server:~# vim /etc/mysql/my.cnf
...
max_connections = 512
root@ubuntu-server:~# service mysql restart
</code></pre>
<br />
Now let's create a table of 1.000.000 of rows in a database called test by using SysBench.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=xxxxxx prepare
</code></pre>
<br />
Then a straightforward script taken care of running the tests will be developed. This bash script executes the OLTP (OnLine Transaction Processing) test on a table of 1.000.000 of rows. The time limit for the whole execution will be 300 seconds and read, update, delete and insert queries will be performed. The total number of maximum requests will be unlimited.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# cat script.sh
#!/bin/bash
for i in 8 16 32 64 128 256 512
do
service mysql restart ; sleep 5
sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=xxxxxx --max-time=300 --oltp-read-only=off --max-requests=0 --num-threads=$i run
done
</code></pre>
<br />
As you can see in the above script, the number of worked threads to be created will be different in each loop iteration, from 8 to 512. So the idea is to run the script with various MySQL combinations and calculate the number of transactions per second.<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-37314293171689078852012-11-18T18:56:00.001+01:002012-11-18T18:57:29.018+01:00MySQL optimization (V)I am going to reopen the last article that I wrote about <a href="http://redes-privadas-virtuales.blogspot.com.es/2010/10/mysql-optimization-iv.html">MySQL optimization</a> because I came across three new parameters (for me) which enhance the performance of a MySQL database, and I would like to note down them on my blog.<br />
<br />
The first parameter is <i>innodb-flush-log-at-trx-commit</i>, which manages both when the log buffer is written out to the log file and the flush to disk operation is performed. Its default value is 1, which means that the log buffer is dumped to the log file at each transaction commit and the flush to disk operation is carried out directly on the log file.<br />
<br />
When its value is 0, the log buffer is sent to the log file once per second, so in this way, you are turning down the disk accesses. In respect of the flush to disk, the operation is also effected on the log file but not coinciding with the commit, but taking advantage of free periods. And when it takes the value of 2 (less aggressive than 0), the log buffer is written out to the log file at each commit but as in the previous case, the flush is done at any free moment for the server.<br />
<br />
The other parameter that I would like to talk about is <i>innodb_buffer_pool_instances</i> (in case of you are using the InnoDB engine), which represents the number of regions that the InnoDB buffer is broken up. This parameter is really useful when you are using a server with several cores, and thereby, each core (thread) can work on a separate instance. A good recommendation is to set it to the same value as the number of cores, but another popular option is to follow the next rule: (innodb_buffer_pool_size [in GB] + number of cores) / 2.<br />
<br />
And finally, the last parameter is <i>innodb_log_file_size</i>, related to the InnoDB log file. Its default value is 5 MB and I consider that is not enough for production environments. The larger the value is, the less control flush activity is needed in the buffer pool, saving disk I/O operations. I think that a right value would be between 64 and 256 MB.<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-89952418095689442322012-11-04T12:25:00.001+01:002012-11-04T12:41:15.615+01:00Zabbix poller processes more than 75% busy and queue delay (III)Let's complete the last article about Zabbix poller processes more than 75% busy and queue delay. In this section, I am going to tackle the part of the client, that is, those things which can be modified on the agent so as to remove or attenuate the issues mentioned in the first article.<br />
<br />
Remember that this is the continuation of the two previous articles:<br />
<br />
<ul>
<li><a href="http://redes-privadas-virtuales.blogspot.com.es/2012/10/zabbix-poller-processes-more-than-75_15.html">Zabbix poller processes more than 75% busy and queue delay (I)</a></li>
<li><a href="http://redes-privadas-virtuales.blogspot.com.es/2012/10/zabbix-poller-processes-more-than-75_28.html">Zabbix poller processes more than 75% busy and queue delay (II)</a></li>
</ul>
<br />
First up, I changed the number of pre-forked instances of the Zabbix client which process passive checks (<i>StartAgents</i>) to 64. This parameter is really meaningful, because its default value is 5, that is to say, only five processes will be started in order to obtain the data requested by the server. So if you have a lot of items and a small monitoring period (as my case), you will need more processes to be able to attend all requests.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@zabbix-client:~# cat /etc/zabbix/zabbix_agentd.conf
...
StartAgents=64
</code></pre>
<br />
So let's see now in the graphs, how this change impacts on the results. Let's first with the Zabbix server performance.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-J3RacPkDmJc/UI1jbcWERbI/AAAAAAAAADo/2cxA58370Eo/s1600/zabbix6.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="208" src="http://3.bp.blogspot.com/-J3RacPkDmJc/UI1jbcWERbI/AAAAAAAAADo/2cxA58370Eo/s640/zabbix6.jpg" width="640" /></a></div>
<br />
<br />
And then, the Zabbix data gathering process.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-dRiL3lITDic/UI1jt39k59I/AAAAAAAAADw/xmAv2pO7wXo/s1600/zabbix7.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="286" src="http://2.bp.blogspot.com/-dRiL3lITDic/UI1jt39k59I/AAAAAAAAADw/xmAv2pO7wXo/s640/zabbix7.jpg" width="640" /></a></div>
<br />
<br />
As you can see on the first picture, the server has gone from a Zabbix queue of 30 to 0 (although you can observe 5 on the figure, think that the graph has been cut out). And on the second one, the Zabbix busy poller processes went from 24% to 0%.<br />
<br />
Other parameters that you can play with are the number of seconds that the data can be stored in the buffer and its maximum number of values.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@zabbix-client:~# cat /etc/zabbix/zabbix_agentd.conf
...
BufferSend=3600
BufferSize=65535
</code></pre>
<br />
Also keep in mind that you should have a small value for the timeout (I am using five seconds on my installation).<br />
<br />
Lastly, in order to solve the problem that I mentioned in the first article about from time to time, the processes break down and the zabbix agent is stopped, I developed a simple bash script to work around this issue.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@zabbix-client:~# tail -f /var/log/zabbix/zabbix_agentd.log
...
zabbix_agentd [17271]: [file:'cpustat.c',line:155] lock failed: [22] Invalid argument
17270:20121015:092010.216 One child process died (PID:17271,exitcode/signal:255). Exiting ...
...
17270:20121015:092012.216 Zabbix Agent stopped. Zabbix 2.0.3 (revision 30485).
root@zabbix-client:~# cat /etc/zabbix/monitor_zabbix.sh
#!/bin/bash
while [ 1 ];
do
if ! pgrep -f "/usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf" &> /dev/null ; then
/etc/zabbix/zabbix.sh start
fi
sleep 15
done
</code></pre>
<br />
This script is run in batch mode and takes care of monitoring the status of the agent processes and starting over when they drop . It uses another bash script to start and stop the agents.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@zabbix-client:~# cat /etc/zabbix/zabbix.sh
#!/bin/bash
case $1 in
"start")
taskset -c $(($(cat /proc/cpuinfo | grep processor | wc -l) - 1)) /usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf;;
"stop")
pkill -f "/usr/local/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf";;
*)
printf "./zabbix.sh start|stop\n\n"
esac
</code></pre>
<br />
<br />Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-1494477346885932431.post-36149990690589088102012-10-28T12:21:00.002+01:002012-10-28T12:21:53.593+01:00Zabbix poller processes more than 75% busy and queue delay (II)After putting forward the issues turned up on my current Zabbix installation and related to its performance (<a href="http://redes-privadas-virtuales.blogspot.com.es/2012/10/zabbix-poller-processes-more-than-75_15.html">Zabbix poller processes more than 75% busy and queue delay</a>), I am going to explain to you how I solved it.<br />
<br />
First of all, I tried out to increase the number of pre-forked instances of pollers for the Zabbix server, that is, I changed its default value from 5 to 256 (remember that for that case, you have to set the the number of maximum connections in MySQL - <i>max_connections</i> - higher than 256, since every single poller opens a dedicated connection to the database).<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@zabbix-server:~# cat /etc/zabbix/zabbix_server.conf
...
# StartPollers=5
StartPollers=256
root@zabbix-server:~# cat /etc/mysql/my.cnf
...
max_connextions = 512
</code></pre>
<br />
Below you can see the outcome after applying it (Zabbix server performance).<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-pqYntLtZ-p0/UIz6lzvV-aI/AAAAAAAAADY/KGUTZHJ5IRU/s1600/zabbix5.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="210" src="http://4.bp.blogspot.com/-pqYntLtZ-p0/UIz6lzvV-aI/AAAAAAAAADY/KGUTZHJ5IRU/s640/zabbix5.jpg" width="640" /></a></div>
<br />
<br />
And the Zabbix data gathering process.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-7waXJT9ZSmk/UIz5MXcjxQI/AAAAAAAAADI/iEqoW6BQDiw/s1600/zabbix4.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="288" src="http://3.bp.blogspot.com/-7waXJT9ZSmk/UIz5MXcjxQI/AAAAAAAAADI/iEqoW6BQDiw/s640/zabbix4.jpg" width="640" /></a></div>
<br />
<br />
In the first figure, you can observe that the Zabbix queue has gone from 48 to 30 (approximately), and for the second one, the Zabbix busy poller processes went from 100% to 24%. So it is clear that if you have a server with enough resources, there is no problem to start many pollers. These kind of processes are responsible for requesting the data defined in the items, so the more pollers have available, the less overloaded the system is.<br />
<br />
Other Zabbix server parameter that you ought to take into account is for example the <i>Timeout</i> (specifies how log pollers wait for agent responses). Try not to assign a very high value. Otherwise, the system might get overloaded.<br />
<br />
Next week, I will end up this series of articles by accomplishing the part of the client.<br />
<br />
<br />Unknownnoreply@blogger.com15tag:blogger.com,1999:blog-1494477346885932431.post-36443179668978176472012-10-21T20:21:00.001+02:002012-10-21T20:25:36.234+02:00sysstat vs top vs ps (III)Let's finish the series of articles related to the differences in the measure of sysstat, top and ps.<br />
<br />
What about the system CPU time (%sy) of the first top from the previous article? It is 39.7%. It is right as well. You have to take into account that, because this server has a couple of cores, that data represents the average from both cores. You can see this point by running top into interactive mode and pressing the number 1. Then you will be able to obtain the consumption for both cores.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# top
top - 20:04:37 up 47 min, 1 user, load average: 0.40, 0.54, 0.60
Tasks: 87 total, 1 running, 86 sleeping, 0 stopped, 0 zombie
<b>Cpu0 : 2.9%us, 37.7%sy, 0.0%ni, 59.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 1.6%us, 40.8%sy, 0.0%ni, 57.1%id, 0.0%wa, 0.0%hi, 0.5%si, 0.0%st</b>
Mem: 1024800k total, 207680k used, 817120k free, 23532k buffers
Swap: 1048572k total, 0k used, 1048572k free, 100320k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1561 root 20 0 29280 4180 2416 S 128 0.4 56:57.23 script.py
</code></pre>
<br />
At any rate, the sum of both percentages does not match the %CPU used by the script. This might be due to the sampling frequencies commented in the preceding article.<br />
<br />
Finally, I would like to remark that in order to monitor threads, I like to use pidstat with the -t parameter.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# pidstat -u -t -p 1561 1 1
Linux 3.2.0-30-generic-pae (ubuntu-server) 09/30/2012 _i686_ (2 CPU)
08:14:47 PM TGID TID %usr %system %guest %CPU <b>CPU</b> Command
08:14:48 PM 1561 - 9.00 119.00 0.00 128.00 <b>1</b> script.py
08:14:48 PM - 1561 0.00 0.00 0.00 0.00 <b>1</b> |__script.py
08:14:48 PM - 1562 4.00 60.00 0.00 64.00 <i><b>0</b></i> |__script.py
08:14:48 PM - 1563 4.00 59.00 0.00 63.00 <b>1</b> |__script.py
</code></pre>
<br />
This tool is great because you can see the core where the thread (or process) is being executed.<br />
<br />
Another interesting choice is to use directly the ps command with the suitable parameters (-L shows threads and sgi_p the processor where the process is currently executing on).<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# ps -Leo pid,pcpu,sgi_p,command | grep '^ 1561'
1561 0.0 * /usr/bin/python ./script.py
1561 65.6 0 /usr/bin/python ./script.py
1561 65.2 0 /usr/bin/python ./script.py
</code></pre>
<br />
Also point out for the above output that, ps does not offer the number of TID (Thread ID), in contradistinction to top and pidstat (remember that a thread does not have PID).<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-54668092147617728642012-10-15T22:34:00.001+02:002012-10-28T12:24:22.786+01:00Zabbix poller processes more than 75% busy and queue delay (I)In my previous job, I had to set up a Zabbix infrastructure in order to monitor more than 400 devices between switches and servers. The main feature of this architecture was that there were a lot of machines, but the update interval was large (around 30 seconds) and the number of items small.<br />
<br />
For this purpose, I wrote down a couple of articles related to this issue:<br />
<br />
<ul>
<li><a href="http://redes-privadas-virtuales.blogspot.com.es/2011/07/tuning-zabbix-to-improve-its_25.html">Tuning Zabbix to improve its performance (I)</a></li>
<li><a href="http://redes-privadas-virtuales.blogspot.com.es/2011/08/tuning-zabbix-to-improve-its.html">Tuning Zabbix to improve its performance (II)</a></li>
</ul>
<br />
But in my current position, I am starting to introduce Zabbix (2.0.3 on Ubuntu Server 12.04) with the aim of controlling few devices where a large number of items and a small monitoring period are required. This situation leads to an overload of the Zabbix server, on the one hand by increasing the number of monitored elements delayed in the queue, and on the other, turning out that the poller processes are busy long.<br />
<br />
In addition, I have been able to observe that, from time to time, the agent goes down in an unexpected way. If you take a look at the log file from the client (debug mode), the following error lines are dumped.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@zabbix-client:~# tail -f /var/log/zabbix/zabbix_agentd.log
...
zabbix_agentd [17271]: [file:'cpustat.c',line:155] lock failed: [22] Invalid argument
17270:20121015:092010.216 One child process died (PID:17271,exitcode/signal:255). Exiting ...
17270:20121015:092010.216 zbx_on_exit() called
17272:20121015:092010.216 Got signal [signal:15(SIGTERM),sender_pid:17270,sender_uid:0,reason:0]. Exiting ...
17273:20121015:092010.216 Got signal [signal:15(SIGTERM),sender_pid:17270,sender_uid:0,reason:0]. Exiting ...
17274:20121015:092010.216 Got signal [signal:15(SIGTERM),sender_pid:17270,sender_uid:0,reason:0]. Exiting ...
17270:20121015:092012.216 Zabbix Agent stopped. Zabbix 2.0.3 (revision 30485).
</code></pre>
<br />
Below you can observe a figure which shows the Zabbix server performance (queue) for the aforementioned case.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-K63MosFrv00/UHr--bmNtEI/AAAAAAAAACg/nyXXYaucFPs/s1600/Zabbix+server+performance+(queue).jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="206" src="http://3.bp.blogspot.com/-K63MosFrv00/UHr--bmNtEI/AAAAAAAAACg/nyXXYaucFPs/s640/Zabbix+server+performance+(queue).jpg" width="640" /></a></div>
<br />
<br />
And the other one, reflects the Zabbix data gathering process (pay attention to the data Zabbix busy poller processes, in %).<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-yzs_lSsnERM/UHr_E_DkNRI/AAAAAAAAACo/8P863ypboBo/s1600/Zabbix+data+gathering+process.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="283" src="http://1.bp.blogspot.com/-yzs_lSsnERM/UHr_E_DkNRI/AAAAAAAAACo/8P863ypboBo/s640/Zabbix+data+gathering+process.jpg" width="640" /></a></div>
<br />
<br />
For the first case, the Zabbix queue has averaged more than 50 monitored items delayed, and for the second one, the poller processes are busy about 100% of the time. This situation can produce that, sometimes, Zabbix draws sporadic dots rather than lines in the graphs. Another effect that you can get from this condition is that if you set a short update interval for an item, you could run into lack of data when you check the values gathered later.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-TWNMf41Tl0k/UHsGpUCVHRI/AAAAAAAAAC4/WwX7eVdN_qs/s1600/Zabbix+sporadic+dots.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="282" src="http://4.bp.blogspot.com/-TWNMf41Tl0k/UHsGpUCVHRI/AAAAAAAAAC4/WwX7eVdN_qs/s400/Zabbix+sporadic+dots.jpg" width="400" /></a></div>
<br />
<br />
Also say that I followed the tuning guide that I mentioned before, but as you can see, Zabbix server was acting up.<br />
<br />
<br />Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-1494477346885932431.post-4332864547778117792012-10-06T20:08:00.001+02:002012-10-06T20:08:30.987+02:00sysstat vs top vs ps (II)Following up on the previous article, <a href="http://redes-privadas-virtuales.blogspot.com.es/2012/09/sysstat-vs-top-vs-ps-i.html">sysstat vs top vs ps (I)</a>, a curious case that I would like to talk about is when you use more than one core. Let's create a simple script in Python which runs a couple of threads a little bit overloaded.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# nproc
2
root@ubuntu-server:~# cat script.py
#!/usr/bin/python
import threading, time
def sleep():
while True:
time.sleep(0.000001)
t1 = threading.Thread(target=sleep)
t2 = threading.Thread(target=sleep)
t1.start()
t2.start()
t1.join()
t2.join()
root@ubuntu-server:~# ./script.py &
[1] 1561
</code></pre>
<br />
If we take a look now at the status of this process by means of top, we can see as follows.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# top -b -n 1 -p 1561
top - 19:41:19 up 24 min, 1 user, load average: 0.74, 0.66, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.3%us, <b>39.7%sy</b>, 0.0%ni, 55.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1024800k total, 205596k used, 819204k free, 23192k buffers
Swap: 1048572k total, 0k used, 1048572k free, 98712k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1561 root 20 0 29280 4180 2416 S <b>129</b> 0.4 26:23.26 script.py
</code></pre>
<br />
So what would be the first weird thing that you can observe from the previous screen? The script is consuming 129% of the CPU. This is right because you have to remember that this virtual machine has two cores and the script, rather, its two threads, are using the two ones and that figure is the combination of the CPU utilization from both cores. You can appreciate this situation much better if you execute top with the -H option.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# top -b -H -n 1 -p 1561
top - 19:44:22 up 27 min, 1 user, load average: 0.69, 0.72, 0.54
Tasks: 3 total, 2 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.3%us, 38.9%sy, 0.0%ni, 56.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1024800k total, 205624k used, 819176k free, 23192k buffers
Swap: 1048572k total, 0k used, 1048572k free, 98712k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1563 root 20 0 29280 4180 2416 R 65 0.4 15:10.19 script.py
1562 root 20 0 29280 4180 2416 R 64 0.4 15:12.39 script.py
1561 root 20 0 29280 4180 2416 S 0 0.4 0:00.05 script.py
</code></pre>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-34233664766933092772012-09-30T16:46:00.000+02:002012-09-30T16:46:19.882+02:00sysstat vs top vs ps (I)I have always been using several tools to get the CPU utilization of Linux processes through different tools such as top, ps, sar, etc., but so far, I did not realise that the results obtained from them can vary considerably.<br />
<br />
For example, I am going to run a job and measure its CPU usage afterwards. Also say that a virtual machine (Ubuntu Server 12.04) with only one core will be used.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# nproc
1
root@ubuntu-server:~# while [ 1 ] ; do sleep 0.0001 ; done &
[1] 5605
</code></pre>
<br />
Now let's show its performance by means of top, ps and pidstat (this command belongs to the sysstat package, which also provides the sar utility, used to collect, report, or save system activity information).<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# top -b -n 1 -p 5605 | grep bash | awk '{print $9}'
37.9
root@ubuntu-server:~# ps -eo pid,pcpu | grep '^ 5605' | awk '{print $2}'
37.8
root@ubuntu-server:~# pidstat -u -p 5605 1 1 | grep bash | head -n 1 | awk '{print $7}'
45.12
</code></pre>
<br />
If you go over the definitions of these measures, you can read as follows:<br />
<br />
<ul>
<li>top (%CPU): <i>the task's share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time</i>.</li>
</ul>
<ul>
<li>ps (%CPU): <i>cpu utilization of the process in "##.#" format. Currently, it is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage</i>.</li>
</ul>
<ul>
<li>pidstat (%CPU): <i>total percentage of CPU time used by the task. In an SMP environment, the task's CPU usage will be divided by the total number of CPU's if option -I has been entered on the command line</i>.</li>
</ul>
<br />
What is my opinion? All data turned out try to display the CPU utilization of a process during a period of time, but the key is that period of time taken to work out the result, and I think that for pidstat is different that for top and ps.<br />
<br />
So my conclusion is that all aforementioned tools are correctly valid, and they will give back you a correct idea about the behaviour of a process in terms of CPU.<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-9375639706180874072012-09-16T22:09:00.001+02:002012-09-16T22:09:31.895+02:00Managing passwords with MyPasswordsFrom a long time, I was looking for a tool in order to handle all my passwords, and by trying out different options, I came across <a href="http://sourceforge.net/projects/mypasswords7/">MyPasswords</a>, an easy and handy application which allows you to store your credentials within a Derby database.<br />
<br />
What can I highlight from this tool? First of all, it is really fast and does not require any installation, that is, we are talking about a java application that can be run on Linux, Unix, Solaris, Mac, Windows, etc. Secondly, you can easily export the repository to a XML file, so as to bring it back later. And finally, MyPasswords works with tags, that is to say, a tag can be added to each element stored in the database, and in this way, it is straightforward to locate an item at any given time.<br />
<br />
For this article, I am going to use the latest version available on the website: 2.92. After grabbing and unpacking it, you can execute it by running the shell script called <i>MyPasswords.sh</i> (a simple script which launches the java file). Then, you will be able to see a screen as follows.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-PDqnRdEUUGY/UFYnxQk3oBI/AAAAAAAAACI/epltImGA-Hs/s1600/MyPAsswords+(I).png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="275" src="http://3.bp.blogspot.com/-PDqnRdEUUGY/UFYnxQk3oBI/AAAAAAAAACI/epltImGA-Hs/s400/MyPAsswords+(I).png" width="400" /></a></div>
<br />
<br />
Don't forget to take a look at the <i>readme.txt</i> file, since it is wrote down the default password used to start MyPasswords.<br />
<br />
As you can appreciate in the previous image, the main window allows you to create a new entry, by fulfilling the fields that you want to store for your item, such as the username and password. Pay attention to the <i>Strength</i> field, as MyPasswords is able to warn you about the strength of the password introduced.<br />
<br />
I recommend you to use the password generator utility provided by MyPasswords, and turn out passwords with at least 16 alphanumeric characters (much better if you add symbols as well).<br />
<br />
The <i>Tags</i> field is very practical, since it allows you later to look up your items by browsing a tag tree. In addition, you have the <i>Search</i> option, used to find elements by using titles and tags. Also point out that it is a good idea to export your encrypted repository to a XML file from time to time, as a backup. If so, you will have to supply a password in order to preserve the generated file.<br />
<br />
Lastly, remember to change the default password used by MyPasswords. It is necessary that this password is really strong, as it will be the key to access all your passwords.<br />
<br />
<br />Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-1494477346885932431.post-36675079167523966182012-09-03T19:19:00.000+02:002012-09-03T19:19:37.761+02:00Remote log server via HTTP (IV)Through the following text, let's end up the series of articles related to the installation and configuration of a log server via HTTP (<a href="http://redes-privadas-virtuales.blogspot.com.es/2012/07/remote-log-server-via-http-i.html">I</a>, <a href="http://redes-privadas-virtuales.blogspot.com.es/2012/08/remote-log-server-via-http-ii.html">II</a> and <a href="http://redes-privadas-virtuales.blogspot.com.es/2012/08/remote-log-server-via-http-iii.html">III</a>).<br />
<br />
Next, Apache is going to be installed and tuned based on the kind of service which will be offered (static data), taking out those unnecessary modules, adjusting the parameters of Apache according to the content served and modifying those variables which affect the security of the web server.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# yum install httpd
[root@server ~]# cat /etc/httpd/conf/httpd.conf
...
# Remove the information about the server version
ServerTokens Prod
...
# Do not cache the web pages
ExpiresActive Off
...
# Number of second before receiving and sending a time out
Timeout 20
...
# Not allow persistent connections
KeepAlive Off
...
# prefork MPM
<IfModule prefork.c>
StartServers 50
MinSpareServers 35
MaxSpareServers 70
ServerLimit 512
MaxClients 512
MaxRequestsPerChild 4000
</IfModule>
...
# Name used by the server to identify itself
ServerName localhost
...
# Protect the root directory
<Directory />
Options -FollowSymLinks
Order deny,allow
Deny from all
</Directory>
# Default charset for all content served
AddDefaultCharset ISO-8859-15
...
</code></pre>
<br />
In the configuration file, it can be observed that the ISO-8859-15 standard has been used as charset to offer the data by the web server. That is because with UTF-8, accents are represented with strange characters by Firefox.<br />
<br />
Make sure that the <i>welcome.conf</i> file has got the following lines to allow to index the content and not the welcome page.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# cat /etc/httpd/conf.d/welcome.conf
<LocationMatch "^/+$">
Options Indexes
ErrorDocument 403 /error/noindex.html
</LocationMatch>
</code></pre>
<br />
Finally, a virtual host will be created in order to serve the log files.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# cat /etc/httpd/conf.d/logserver.conf
NameVirtualHost 192.168.1.10:80
<VirtualHost 192.168.1.10:80>
ServerName server.local
DocumentRoot /mnt/shared/logs
ErrorLog /var/log/httpd/logserver-error_log
CustomLog /var/log/httpd/logserver-access_log common
<Directory "/mnt/shared/logs">
Options Indexes
AllowOverride None
EnableSendfile Off
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
</code></pre>
<br />
It is important to highlight the <i>EnableSendfile</i> directive (enabled by default), allowing Apache to use the sendfile support included in the Linux kernel. Through this feature, Apache will not read the static files, but the kernel will offer them directly. But it happens that when Apache serves data from NFS or Samba and network outages take place, the connection can turn into an unstable state. So for this case, it is much better to deactivate it.<br />
<br />
Now you have to run Apache and make it automatically start during the booting of the machine.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# service httpd restart
[root@server ~]# chkconfig httpd on
</code></pre>
<br />
In order to secure the web server, iptables will be configured with the following settings.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# cat /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport ssh -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport http -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -j ACCEPT
-A RH-Firewall-1-INPUT -j LOG
-A RH-Firewall-1-INPUT -j REJECT
COMMIT
[root@server ~]# service iptables restart
[root@server ~]# chkconfig iptables on
</code></pre>
<br />
Lastly, the backup for the logs will be scheduled through cron by running a task with rsync every 15 minutes.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# yum install rsync
[root@server ~]# cat /etc/crontab
...
*/15 * * * * /usr/bin/rsync -altgvb /mnt/shared/logs/nfs /backup/logs/nfs
*/15 * * * * /usr/bin/rsync -altgvb /mnt/shared/logs/samba /backup/logs/samba
</code></pre>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-83901858717019979272012-08-27T23:51:00.001+02:002012-08-27T23:51:19.879+02:00GPT, beyond the MBR (II)Let's follow up on the second part of the article about <a href="http://redes-privadas-virtuales.blogspot.com.es/2012/08/gpt-beyond-mbr-i.html">GPT, beyond the MBR</a>.<br />
<br />
Now, a new GPT partition table is going to be created by means of parted, and afterwards, it will be displayed.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# parted /dev/sdb mklabel gpt
root@ubuntu-server:~# parted /dev/sdb print
Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdb: 1074MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
</code></pre>
<br />
If you try to edit the partition table with fdisk, you will come across a message as follows.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# fdisk /dev/sdb
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.
...
</code></pre>
<br />
Let's create for example ten primary partitions of 10 MB each through a simple bash script.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# j=1 ; for ((i=11; i<=101; i+=10)); do parted /dev/sdb mkpart primary $j $i; j=$i ; done
root@ubuntu-server:~# parted /dev/sdb print
Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdb: 1074MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 10.5MB 9437kB primary
2 10.5MB 21.0MB 10.5MB primary
3 21.0MB 31.5MB 10.5MB primary
4 31.5MB 40.9MB 9437kB primary
5 40.9MB 51.4MB 10.5MB primary
6 51.4MB 60.8MB 9437kB primary
7 60.8MB 71.3MB 10.5MB primary
8 71.3MB 80.7MB 9437kB primary
9 80.7MB 91.2MB 10.5MB primary
10 91.2MB 101MB 9437kB primary
</code></pre>
<br />
If we take a look at the GPT table, we can distinguish the following parts.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# dd if=/dev/sdb bs=512 count=4 | xxd -c 16
0000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
...
00001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U.
0000200: 4546 4920 5041 5254 0000 0100 5c00 0000 EFI PART....\...
...
0000400: a2a0 d0eb e5b9 3344 87c0 68b6 b726 99c7 ......3D..h..&..
0000410: 4e6e ad36 2fec 8046 bc1f 4a42 82d2 8052 Nn.6/..F..JB...R
0000420: 0008 0000 0000 0000 ff4f 0000 0000 0000 .........O......
0000430: 0000 0000 0000 0000 7000 7200 6900 6d00 ........p.r.i.m.
0000440: 6100 7200 7900 0000 0000 0000 0000 0000 a.r.y...........
...
</code></pre>
<br />
First up, it is the legacy MBR, which GPT holds for reasons of compatibility (the code 0x55AA points to the end of the MBR). The second part of the GPT (512 bytes) contains the header information for GUID partitioning. And the first partition entry appears in position 0x400.<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-60666965286305474212012-08-18T20:07:00.000+02:002012-08-18T20:07:57.761+02:00GPT, beyond the MBR (I)Have you ever wondered what are the limits of the MBR (Master Boot Record)? I mean, the maximum size of a partition or a entire hard drive able to be handled. The answer is 2.2 TB.<br />
<br />
Below you can observe the structure of the MBR (a total size of 512 bytes, where each row is 32 bytes).<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# dd if=/dev/sda bs=512 count=1 | xxd -g 4 -c 32
0000000: eb639010 8ed0bc00 b0b80000 8ed88ec0 fbbe007c bf0006b9 0002f3a4 ea210600 .c.................|.........!..
0000020: 00bebe07 3804750b 83c61081 fefe0775 f3eb16b4 02b001bb 007cb280 8a74018b ....8.u........u.........|...t..
0000040: 4c02cd13 ea007c00 00ebfe00 00000000 00000000 00000000 00000080 01000000 L.....|.........................
0000060: 00000000 fffa9090 f6c28074 05f6c270 7402b280 ea797c00 0031c08e d88ed0bc ...........t...pt....y|..1......
0000080: 0020fba0 647c3cff 740288c2 52bb1704 80270374 06be887d e81701be 057cb441 . ..d|<.t...R....'.t...}.....|.A
00000a0: bbaa55cd 135a5272 3d81fb55 aa753783 e1017432 31c08944 04408844 ff894402 ..U..ZRr=..U.u7...t21..D.@.D..D.
00000c0: c7041000 668b1e5c 7c66895c 08668b1e 607c6689 5c0cc744 060070b4 42cd1372 ....f..\|f.\.f..`|f.\..D..p.B..r
00000e0: 05bb0070 eb76b408 cd13730d f6c2800f 84d000be 937de982 00660fb6 c68864ff ...p.v....s..........}...f....d.
0000100: 40668944 040fb6d1 c1e20288 e888f440 8944080f b6c2c0e8 02668904 66a1607c @f.D...........@.D.......f..f.`|
0000120: 6609c075 4e66a15c 7c6631d2 66f73488 d131d266 f774043b 44087d37 fec188c5 f..uNf.\|f1.f.4..1.f.t.;D.}7....
0000140: 30c0c1e8 0208c188 d05a88c6 bb00708e c331dbb8 0102cd13 721e8cc3 601eb900 0........Z....p..1......r...`...
0000160: 018edb31 f6bf0080 8ec6fcf3 a51f61ff 265a7cbe 8e7deb03 be9d7de8 3400bea2 ...1..........a.&Z|..}....}.4...
0000180: 7de82e00 cd18ebfe 47525542 20004765 6f6d0048 61726420 4469736b 00526561 }.......GRUB .Geom.Hard Disk.Rea
00001a0: 64002045 72726f72 0d0a00bb 0100b40e cd10ac3c 0075f4c3 ab9d0600 0000<b>8020</b> d. Error...........<.u.........
00001c0: <b>2100831a 3b1f0008 00000098 0700003b 1b1f051f d00ffea7 07000250 b8000000</b> !...;..........;...........P....
00001e0: <b>00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000</b>55aa ..............................U.
</code></pre>
<br />
The first 440 bytes contains the bootstrap code area, which takes care of starting up the operating system present on the active partition. Then, it is the disk signature, a 4-byte number that is randomly generated when the MBR is first created. It is an identifier which applies to the whole hard drive (not a single partition). After that, there are a couple of bytes set to null, and next, the partition table.<br />
<br />
The partition table is made up of four entries of 16 bytes each which define the position and size of the sectors (LBA or Logical Block Addressing). In order to work around the problem of having just four partitions, these primary partitions can contain an arbitrary number of logical partitions. This schema can lead to problems, because some operating systems can only boot from primary partitions.<br />
<br />
On the other hand, how do I work out the figure of 2.2 TB as top size for a partition? In conjunction with the universal sector size of 512 bytes and the 32-bit LBA pointers used by MBR partitions, you have ( 2^32 ) - 1 sectors * 512 bytes per sector.<br />
<br />
(Also say that the MBR ends with the sequence of two bytes 0x55AA).<br />
<br />
On account of the current size of hard drives and RAID technologies, the problem is really serious and will become more severe over time. In order to overcome it, the GUID partition table (GPT) is the natural successor to the MBR partition table.<br />
<br />
GPT, supported on Linux since the 2.6.25 kernel version, uses 128-byte LBA modern tables, so therefore it is possible to have hard drives up to 8 ZB addressable for a disk with a 512-byte sector size. In addition, GPT can manage up to 128 partitions, so there is no need for extended or logical partitions. If you are interested, you can read up more about this topic on the Internet, since the aim of this article is to put forward how to manage this technology on Linux (also mention that for my tests, I used an Ubuntu Server 12.04 distribution).<br />
<br />
First of all, point out that fdisk does not work with this partitioning scheme, but the good news is that can be got over with other Linux tools, such as parted. Keeping on with the tests, I have added a second hard drive to my system.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>root@ubuntu-server:~# fdisk -l /dev/sdb
Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
</code></pre>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-76652004877214919722012-08-12T18:33:00.000+02:002012-08-12T22:18:04.196+02:00Remote log server via HTTP (III)Once finished the second article about the configuration of Samba, <a href="http://redes-privadas-virtuales.blogspot.com.es/2012/08/remote-log-server-via-http-ii.html">Remote log server via HTTP (II)</a>, now the <i>server</i> is going to be set up in order to be able to import the log directories via NFS and Samba. Furthermore, they will be served via Apache and backed up from time to time.<br />
<br />
First of all, you have to create the directory where the logs will be mounted, as well as the backup directories.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# mkdir -p /mnt/shared/logs
[root@server ~]# mkdir -p /backup/logs/nfs /backup/logs/samba
</code></pre>
<br />
So that SELinux allows Apache to access a directory brought by NFS or Samba, you have to enable the variables <i>httpd_use_nfs</i> and <i>httpd_use_cifs</i>. In addition, you have to change the SELinux security context of each directory imported.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# setsebool -P httpd_use_nfs=on httpd_use_cifs=on
[root@server ~]# chcon -R -u system_u /mnt/shared/logs
[root@server ~]# chcon -R -t httpd_sys_content_t /mnt/shared/logs
</code></pre>
<br />
Because the log server will not share any data through NFS and will offer no service by means of portmap, you will be able to deactivate the nfslock service.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# service nfslock stop
[root@server ~]# chkconfig nfslock off
</code></pre>
<br />
If you want to mount the NFS remote directory from <i>client</i> by hand, run the following order.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# mount -t nfs -o soft,intr client.local:/var/log /mnt/shared/logs
</code></pre>
<br />
And for the case of Samba.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# mount -t cifs -o username=samba_logs,password=xxxxxx,soft //client.local/logs /mnt/shared/logs
</code></pre>
<br />
The problem of mounting a remote directory statically is that the traffic passed down over the network is also increased, since when a file is updated, it is refreshed in the destination in the same way.<br />
<br />
Moreover, you have to take into account another severe problem related to mount file systems via Samba, and is that if the connection is cut off (restarted, some network problem, etc.), Samba does not reconnect and the mount point can remain in an unstable state, thereby any existing synchronization would be lost. Thus, it is really important to always mount file systems by using <a href="http://redes-privadas-virtuales.blogspot.com.es/2012/02/mounting-file-systems-via-samba-cifs-or.html">automount</a>.<br />
<br />
Automount is a useful tool which takes care of mounting a directory when it is really accessed. It has got a timeout (600 sg by default) that when it is completed, the directory is automatically unmounted. This situation leads to reduce the network traffic (there may be long periods where you are not accessing the shared space) and avoid loss of synchronization. Also say that automount is managed by the autofs daemon.<br />
<br />
This is the configuration used by automount to mount the log directory from <i>client</i>.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# yum install autofs
[root@server ~]# vim /etc/auto.master
...
/mnt/shared/logs /etc/auto.logs -g,--timeout=300
[root@server ~]# cat /etc/auto.logs
nfs -fstype=nfs,soft,intr client.local:/var/log
samba -fstype=cifs,username=samba_logs,password=xxxxxx,iocharset=utf8,soft ://client.local/logs
[root@server ~]# chmod 600 /etc/auto.logs
[root@server ~]# service autofs restart
</code></pre>
<br />
The <i>soft</i> option is used for an application which is trying to access the shared area does not keep blocked if the connection is lost, and brings the control back to the system after 0.7 sg. With <i>intr</i>, allows the user to send an interruption signal if the application which uses NFS hangs.<br />
<br />
If instead of hooking up to a Linux machine via Samba you have a Windows machine inside a domain, you would have to specify the domain name through the <i>domain</i> parameter.<br />
<br />
Also mention that when you are mounting a directory from a Windows server, it might happen that strange characters turn up. This is due to the character conversion. So as to fix it, you have to use the <i>iocharset=utf8</i> option for each mount point.<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-41499293920086765982012-08-05T20:48:00.001+02:002012-08-12T22:30:51.788+02:00Remote log server via HTTP (II)Let's keep on with the second article about setting up a <a href="http://redes-privadas-virtuales.blogspot.com.es/2012/07/remote-log-server-via-http-i.html">Remote log server via HTTP</a>. In the preceding part, the NFS daemon was configured in order to be able to export the local log server through NFS, and all this correctly secured by iptables and TCP wrappers. In this article, I am going to continue with the configuration of Samba.<br />
<br />
First up, a new user called <i>samba_logs</i> will be adding to the system. From this user, the <i>server</i> machine will be able to hook up to the log directory via Samba. This user will not have neither a personal directory within <i>home</i> nor a shell.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# useradd -d /dev/null -s /sbin/nologin samba_logs
</code></pre>
<br />
In turn, this user will also be used to create an ACL (Access Control List) on the <i>/var/log</i> directory, granting read permissions to that user.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# setfacl -R -m d:u:samba_logs:r /var/log/
[root@client ~]# getfacl /var/log/
...
default:user:samba_logs:r--
...
</code></pre>
<br />
Then the samba package will be installed and configured.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# yum install samba
[root@client ~]# cat /etc/samba/smb.conf
[global]
...
hosts allow = 192.168.1.
...
[logs]
comment = Log directory
path = /var/log
read only = yes
valid users = samba_logs
</code></pre>
<br />
Finally, the samba service will be restarted and marked as persistent. Furthermore, the user will be added to the local smbpasswd file.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# service smb restart
[root@client ~]# chkconfig smb on
[root@client ~]# smbpasswd -a samba_logs
</code></pre>
<br />
So as to shield the server by iptables, the following rules will be set into the <i>/etc/sysconfig/iptables</i> file (Samba uses the ports 137, 138 and 139 TCP/UDP).<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# cat /etc/sysconfig/iptables
...
-A RH-Firewall-1-INPUT -s server.local -p tcp --dport 137:139 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p udp --dport 137:139 -j ACCEPT
...
[root@client ~]# service iptables restart
</code></pre>
<br />
Remember that is important to keep SELinux and TCP wrappes on. In order SELinux to let read the exported files, it is necessary to activate the variable <i>samba_export_all_ro</i>.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# getenforce
Enforcing
[root@client ~]# setsebool -P samba_export_all_ro on
</code></pre>
<br />
And below you can observe the configuration for iptables.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# cat /etc/sysconfig/iptables
...
-A RH-Firewall-1-INPUT -s server.local -p tcp --dport 137:139 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p udp --dport 137:139 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p tcp --dport 445 -j ACCEPT
...
</code></pre>
<br />
Now we can try out that everything is properly configured by running the next command on <i>server</i>.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# yum install samba-client cifs-utils
[root@server ~]# smbclient -U samba_logs -L client.local
Enter samba_logs's password:
Domain=[MYGROUP] OS=[Unix] Server=[Samba 3.5.10-125.el6]
Sharename Type Comment
--------- ---- -------
logs Disk Log directory
IPC$ IPC IPC Service (Samba Server Version 3.5.10-125.el6)
samba_logs Disk Home Directories
Domain=[MYGROUP] OS=[Unix] Server=[Samba 3.5.10-125.el6]
Server Comment
--------- -------
Workgroup Master
--------- -------
</code></pre>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1494477346885932431.post-36426235355200242192012-07-29T21:26:00.002+02:002012-07-29T21:27:55.968+02:00Remote log server via HTTP (I)When you have a network with multiple servers which can be accessed by several people, for example in order to take a look at their log files, it is desirable to centralize this task on a single machine which is the common point of access for all. In this way, you will improve aspects related to the security of your infrastructure:<br />
<br />
<ul>
<li>Those people will not directly log on the servers.</li>
<li>It will be possible to create a copy of the most important log files in real time.</li>
</ul>
<br />
The schema that I am going to follow so as to develop this series of articles is based on a couple of Linux servers (CentOS 6.3). The first computer, <i>client</i>, will be the machine which will export their log files, either by means of NFS or Samba. And the second one, <i>server</i>, will be the machine will import those log files and will put forward them via HTTP.<br />
<br />
In addition, the log server will carry out a backup each five minutes of the log files through rsync. The aim of backing up is to be able to have a copy of the log files in case of one of the servers was messed up and it was impossible to reach it.<br />
<br />
NFS is a protocol belonging to the application level and used to share volumes between several computers within a network. In return, Samba is an implementation from the SMB (Server Message Block) protocol for Linux machines, renamed to CIFS (Common Internet File System) later, which allows to share resources (directories, printers, etc.) between different computers, authenticate connections to Windows domains, provide Windows Internet Naming Service (WINS), work as PDC (Primary Domain Controller), and so on.<br />
<br />
Using either protocol has its advantages and disadvantages. For this reason, it will be exposed in this articles both protocols. Basically, NFS does not use users and passwords like Samba, but that the only way to perform an access control is through IP addresses or host names. On the other hand, in order to share files across a local area network, NFS can be enough.<br />
<br />
First of all, let's register the names of all implicated nodes inside the file <i>/etc/hosts</i>.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[... ~]# cat /etc/hosts
...
192.168.1.10 server server.local
192.168.1.11 client client.local
</code></pre>
<br />
In this first article, we are going to set out by configuring NFS in the <i>client</i> machine. You will have to make sure to grant read permissions for all users to those elements you want to export.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# chmod -R o+r /var/log/
</code></pre>
<br />
Now you have to set up NFS in order to make public the previous directory. Afterwards, the NFS daemon will have to be started and enabled.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# cat /etc/exports
/var/log/ server.local(ro,sync,root_squash)
[root@client ~]# service nfs restart ; chkconfig nfs on
</code></pre>
<br />
As you can see, by means of the file <i>/etc/exports</i> it has been indicated that the <i>/var/log</i> directory just will be able to be mounted by <i>server.local</i> in read-only mode (ro). Furthermore, requests only will be replied after the changes have been committed to stable storage (sync) and those of them which came from root, will be mapped to anonymous (root_squash).<br />
<br />
In order to secure this server, you can begin with TCP wrappers, thus you have to allow both portmap (converts RPC program numbers into Internet port numbers) and mountd (answers a client request to mount a file system) services only for the IP address where they will be listening to.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# cat /etc/hosts.deny
ALL: ALL
[root@client ~]# cat /etc/hosts.allow
sshd: ALL
portmap: server.local
rpcbind: server.local
mountd: server.local
</code></pre>
<br />
So as to the NFS service can be protected by iptables, you will have to add the following lines to the <i>/etc/sysconfig/nfs</i> file (by default, NFS establishes the link through a random port).<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# cat /etc/sysconfig/nfs
...
MOUNTD_PORT="4002"
STATD_PORT="4003"
LOCKD_TCPPORT="4004"
LOCKD_UDPPORT="4004"
RQUOTAD_PORT="4005"
[root@client ~]# service nfs restart
</code></pre>
<br />
It happens that in NFSv4, the only ports that you need to open are 2049 TCP and 111 UDP. But in order to protect NFSv3 and NFSv2 by a firewall, as well as to be able to use the showmount command, you need to open the previous ports.<br />
<br />
Now you have to add the corresponding rules to the iptables configuration file.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@client ~]# cat /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport ssh -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p tcp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p udp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p tcp --dport 2049 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p udp --dport 2049 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p tcp --dport 4002:4005 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p udp --dport 4002:4005 -j ACCEPT
-A RH-Firewall-1-INPUT -j LOG
-A RH-Firewall-1-INPUT -j REJECT
COMMIT
[root@client ~]# service iptables restart
[root@client ~]# chkconfig iptables on
</code></pre>
<br />
Also remember that it is really important to have SELinux enabled and running in enforcing mode.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@s1 ~]# getenforce
Enforcing
</code></pre>
<br />
And finally, let's try out the directories exported by client from the server machine.<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 97%;"><code>[root@server ~]# showmount -e client.local
Export list for client.local:
/var/log server
</code></pre>
<br />
<br />Unknownnoreply@blogger.com2