sysstat vs top vs ps (I)

I have always been using several tools to get the CPU utilization of Linux processes through different tools such as top, ps, sar, etc., but so far, I did not realise that the results obtained from them can vary considerably.

For example, I am going to run a job and measure its CPU usage afterwards. Also say that a virtual machine (Ubuntu Server 12.04) with only one core will be used.

root@ubuntu-server:~# nproc 
1

root@ubuntu-server:~# while [ 1 ] ; do sleep 0.0001 ; done &
[1] 5605

Now let's show its performance by means of top, ps and pidstat (this command belongs to the sysstat package, which also provides the sar utility, used to collect, report, or save system activity information).

root@ubuntu-server:~# top -b -n 1 -p 5605 | grep bash | awk '{print $9}'
37.9

root@ubuntu-server:~# ps -eo pid,pcpu | grep '^ 5605' | awk '{print $2}'
37.8

root@ubuntu-server:~# pidstat -u -p 5605 1 1 | grep bash | head -n 1 | awk '{print $7}'
45.12

If you go over the definitions of these measures, you can read as follows:

• top (%CPU): the task's share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time.

• ps (%CPU): cpu utilization of the process in "##.#" format. Currently, it is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage.

• pidstat (%CPU): total percentage of CPU time used by the task. In an SMP environment, the task's CPU usage will be divided by the total number of CPU's if option -I has been entered on the command line.

What is my opinion? All data turned out try to display the CPU utilization of a process during a period of time, but the key is that period of time taken to work out the result, and I think that for pidstat is different that for top and ps.

So my conclusion is that all aforementioned tools are correctly valid, and they will give back you a correct idea about the behaviour of a process in terms of CPU.

Managing passwords with MyPasswords

From a long time, I was looking for a tool in order to handle all my passwords, and by trying out different options, I came across MyPasswords, an easy and handy application which allows you to store your credentials within a Derby database.

What can I highlight from this tool? First of all, it is really fast and does not require any installation, that is, we are talking about a java application that can be run on Linux, Unix, Solaris, Mac, Windows, etc. Secondly, you can easily export the repository to a XML file, so as to bring it back later. And finally, MyPasswords works with tags, that is to say, a tag can be added to each element stored in the database, and in this way, it is straightforward to locate an item at any given time.

For this article, I am going to use the latest version available on the website: 2.92. After grabbing and unpacking it, you can execute it by running the shell script called MyPasswords.sh (a simple script which launches the java file). Then, you will be able to see a screen as follows.

Don't forget to take a look at the readme.txt file, since it is wrote down the default password used to start MyPasswords.

As you can appreciate in the previous image, the main window allows you to create a new entry, by fulfilling the fields that you want to store for your item, such as the username and password. Pay attention to the Strength field, as MyPasswords is able to warn you about the strength of the password introduced.

I recommend you to use the password generator utility provided by MyPasswords, and turn out passwords with at least 16 alphanumeric characters (much better if you add symbols as well).

The Tags field is very practical, since it allows you later to look up your items by browsing a tag tree. In addition, you have the Search option, used to find elements by using titles and tags. Also point out that it is a good idea to export your encrypted repository to a XML file from time to time, as a backup. If so, you will have to supply a password in order to preserve the generated file.

Lastly, remember to change the default password used by MyPasswords. It is necessary that this password is really strong, as it will be the key to access all your passwords.

Remote log server via HTTP (IV)

Through the following text, let's end up the series of articles related to the installation and configuration of a log server via HTTP (I, II and III).

Next, Apache is going to be installed and tuned based on the kind of service which will be offered (static data), taking out those unnecessary modules, adjusting the parameters of Apache according to the  content served and modifying those variables which affect the security of the web server.

[root@server ~]# yum install httpd

[root@server ~]# cat /etc/httpd/conf/httpd.conf
...
# Remove the information about the server version
ServerTokens Prod
...
# Do not cache the web pages
ExpiresActive Off
...
# Number of second before receiving and sending a time out
Timeout 20
...
# Not allow persistent connections
KeepAlive Off
...
# prefork MPM
<IfModule prefork.c>
   StartServers          50
   MinSpareServers       35
   MaxSpareServers       70
   ServerLimit           512
   MaxClients            512
   MaxRequestsPerChild   4000
</IfModule>
...
# Name used by the server to identify itself
ServerName localhost
...
# Protect the root directory
<Directory />
   Options -FollowSymLinks
   Order deny,allow
   Deny from all
</Directory>

# Default charset for all content served
AddDefaultCharset ISO-8859-15
...

In the configuration file, it can be observed that the ISO-8859-15 standard has been used as charset to offer the data by the web server. That is because with UTF-8, accents are represented with strange characters by Firefox.

Make sure that the welcome.conf file has got the following lines to allow to index the content and not the welcome page.

[root@server ~]# cat /etc/httpd/conf.d/welcome.conf
<LocationMatch "^/+$">
   Options Indexes
   ErrorDocument 403 /error/noindex.html
</LocationMatch>

Finally, a virtual host will be created in order to serve the log files.

[root@server ~]# cat /etc/httpd/conf.d/logserver.conf
NameVirtualHost 192.168.1.10:80

<VirtualHost 192.168.1.10:80>
   ServerName server.local
   DocumentRoot /mnt/shared/logs
   ErrorLog /var/log/httpd/logserver-error_log
   CustomLog /var/log/httpd/logserver-access_log common
   <Directory "/mnt/shared/logs">
      Options Indexes
      AllowOverride None
      EnableSendfile Off
      Order allow,deny
      Allow from all
   </Directory>
</VirtualHost>

It is important to highlight the EnableSendfile directive (enabled by default), allowing Apache to use the sendfile support included in the Linux kernel. Through this feature, Apache will not read the static files, but the kernel will offer them directly. But it happens that when Apache serves data from NFS or Samba and network outages take place, the connection can turn into an unstable state. So for this case, it is much better to deactivate it.

Now you have to run Apache and make it automatically start during the booting of the machine.

[root@server ~]# service httpd restart

[root@server ~]# chkconfig httpd on

In order to secure the web server, iptables will be configured with the following settings.

[root@server ~]# cat /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport ssh -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport http -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -j ACCEPT
-A RH-Firewall-1-INPUT -j LOG
-A RH-Firewall-1-INPUT -j REJECT
COMMIT

[root@server ~]# service iptables restart

[root@server ~]# chkconfig iptables on

Lastly, the backup for the logs will be scheduled through cron by running a task with rsync every 15 minutes.

[root@server ~]# yum install rsync

[root@server ~]# cat /etc/crontab
...
*/15 * * * * /usr/bin/rsync -altgvb /mnt/shared/logs/nfs /backup/logs/nfs
*/15 * * * * /usr/bin/rsync -altgvb /mnt/shared/logs/samba /backup/logs/samba