Sep 30, 2012

sysstat vs top vs ps (I)

I have always been using several tools to get the CPU utilization of Linux processes through different tools such as top, ps, sar, etc., but so far, I did not realise that the results obtained from them can vary considerably.

For example, I am going to run a job and measure its CPU usage afterwards. Also say that a virtual machine (Ubuntu Server 12.04) with only one core will be used.

root@ubuntu-server:~# nproc 
1

root@ubuntu-server:~# while [ 1 ] ; do sleep 0.0001 ; done &
[1] 5605

Now let's show its performance by means of top, ps and pidstat (this command belongs to the sysstat package, which also provides the sar utility, used to collect, report, or save system activity information).

root@ubuntu-server:~# top -b -n 1 -p 5605 | grep bash | awk '{print $9}'
37.9

root@ubuntu-server:~# ps -eo pid,pcpu | grep '^ 5605' | awk '{print $2}'
37.8

root@ubuntu-server:~# pidstat -u -p 5605 1 1 | grep bash | head -n 1 | awk '{print $7}'
45.12

If you go over the definitions of these measures, you can read as follows:

  • top (%CPU): the task's share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time.
  • ps (%CPU): cpu utilization of the process in "##.#" format. Currently, it is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage.
  • pidstat (%CPU): total percentage of CPU time used by the task. In an SMP environment, the task's CPU usage will be divided by the total number of CPU's if option -I has been entered on the command line.

What is my opinion? All data turned out try to display the CPU utilization of a process during a period of time, but the key is that period of time taken to work out the result, and I think that for pidstat is different that for top and ps.

So my conclusion is that all aforementioned tools are correctly valid, and they will give back you a correct idea about the behaviour of a process in terms of CPU.


Sep 16, 2012

Managing passwords with MyPasswords

From a long time, I was looking for a tool in order to handle all my passwords, and by trying out different options, I came across MyPasswords, an easy and handy application which allows you to store your credentials within a Derby database.

What can I highlight from this tool? First of all, it is really fast and does not require any installation, that is, we are talking about a java application that can be run on Linux, Unix, Solaris, Mac, Windows, etc. Secondly, you can easily export the repository to a XML file, so as to bring it back later. And finally, MyPasswords works with tags, that is to say, a tag can be added to each element stored in the database, and in this way, it is straightforward to locate an item at any given time.

For this article, I am going to use the latest version available on the website: 2.92. After grabbing and unpacking it, you can execute it by running the shell script called MyPasswords.sh (a simple script which launches the java file). Then, you will be able to see a screen as follows.




Don't forget to take a look at the readme.txt file, since it is wrote down the default password used to start MyPasswords.

As you can appreciate in the previous image, the main window allows you to create a new entry, by fulfilling the fields that you want to store for your item, such as the username and password. Pay attention to the Strength field, as MyPasswords is able to warn you about the strength of the password introduced.

I recommend you to use the password generator utility provided by MyPasswords, and turn out passwords with at least 16 alphanumeric characters (much better if you add symbols as well).

The Tags field is very practical, since it allows you later to look up your items by browsing a tag tree. In addition, you have the Search option, used to find elements by using titles and tags. Also point out that it is a good idea to export your encrypted repository to a XML file from time to time, as a backup. If so, you will have to supply a password in order to preserve the generated file.

Lastly, remember to change the default password used by MyPasswords. It is necessary that this password is really strong, as it will be the key to access all your passwords.


Sep 3, 2012

Remote log server via HTTP (IV)

Through the following text, let's end up the series of articles related to the installation and configuration of a log server via HTTP (I, II and III).

Next, Apache is going to be installed and tuned based on the kind of service which will be offered (static data), taking out those unnecessary modules, adjusting the parameters of Apache according to the  content served and modifying those variables which affect the security of the web server.

[root@server ~]# yum install httpd

[root@server ~]# cat /etc/httpd/conf/httpd.conf
...
# Remove the information about the server version
ServerTokens Prod
...
# Do not cache the web pages
ExpiresActive Off
...
# Number of second before receiving and sending a time out
Timeout 20
...
# Not allow persistent connections
KeepAlive Off
...
# prefork MPM
<IfModule prefork.c>
   StartServers          50
   MinSpareServers       35
   MaxSpareServers       70
   ServerLimit           512
   MaxClients            512
   MaxRequestsPerChild   4000
</IfModule>
...
# Name used by the server to identify itself
ServerName localhost
...
# Protect the root directory
<Directory />
   Options -FollowSymLinks
   Order deny,allow
   Deny from all
</Directory>

# Default charset for all content served
AddDefaultCharset ISO-8859-15
...

In the configuration file, it can be observed that the ISO-8859-15 standard has been used as charset to offer the data by the web server. That is because with UTF-8, accents are represented with strange characters by Firefox.

Make sure that the welcome.conf file has got the following lines to allow to index the content and not the welcome page.

[root@server ~]# cat /etc/httpd/conf.d/welcome.conf
<LocationMatch "^/+$">
   Options Indexes
   ErrorDocument 403 /error/noindex.html
</LocationMatch>

Finally, a virtual host will be created in order to serve the log files.

[root@server ~]# cat /etc/httpd/conf.d/logserver.conf
NameVirtualHost 192.168.1.10:80

<VirtualHost 192.168.1.10:80>
   ServerName server.local
   DocumentRoot /mnt/shared/logs
   ErrorLog /var/log/httpd/logserver-error_log
   CustomLog /var/log/httpd/logserver-access_log common
   <Directory "/mnt/shared/logs">
      Options Indexes
      AllowOverride None
      EnableSendfile Off
      Order allow,deny
      Allow from all
   </Directory>
</VirtualHost>

It is important to highlight the EnableSendfile directive (enabled by default), allowing Apache to use the sendfile support included in the Linux kernel. Through this feature, Apache will not read the static files, but the kernel will offer them directly. But it happens that when Apache serves data from NFS or Samba and network outages take place, the connection can turn into an unstable state. So for this case, it is much better to deactivate it.

Now you have to run Apache and make it automatically start during the booting of the machine.

[root@server ~]# service httpd restart

[root@server ~]# chkconfig httpd on

In order to secure the web server, iptables will be configured with the following settings.

[root@server ~]# cat /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport ssh -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport http -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -j ACCEPT
-A RH-Firewall-1-INPUT -j LOG
-A RH-Firewall-1-INPUT -j REJECT
COMMIT

[root@server ~]# service iptables restart

[root@server ~]# chkconfig iptables on

Lastly, the backup for the logs will be scheduled through cron by running a task with rsync every 15 minutes.

[root@server ~]# yum install rsync

[root@server ~]# cat /etc/crontab
...
*/15 * * * * /usr/bin/rsync -altgvb /mnt/shared/logs/nfs /backup/logs/nfs
*/15 * * * * /usr/bin/rsync -altgvb /mnt/shared/logs/samba /backup/logs/samba


Aug 27, 2012

GPT, beyond the MBR (II)

Let's follow up on the second part of the article about GPT, beyond the MBR.

Now, a new GPT partition table is going to be created by means of parted, and afterwards, it will be displayed.

root@ubuntu-server:~# parted /dev/sdb mklabel gpt

root@ubuntu-server:~# parted /dev/sdb print
Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdb: 1074MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

If you try to edit the partition table with fdisk, you will come across a message as follows.

root@ubuntu-server:~# fdisk /dev/sdb 

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.
...

Let's create for example ten primary partitions of 10 MB each through a simple bash script.

root@ubuntu-server:~# j=1 ; for ((i=11; i<=101; i+=10)); do parted /dev/sdb mkpart primary $j $i; j=$i ; done

root@ubuntu-server:~# parted /dev/sdb print
Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdb: 1074MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  10.5MB  9437kB               primary
 2      10.5MB  21.0MB  10.5MB               primary
 3      21.0MB  31.5MB  10.5MB               primary
 4      31.5MB  40.9MB  9437kB               primary
 5      40.9MB  51.4MB  10.5MB               primary
 6      51.4MB  60.8MB  9437kB               primary
 7      60.8MB  71.3MB  10.5MB               primary
 8      71.3MB  80.7MB  9437kB               primary
 9      80.7MB  91.2MB  10.5MB               primary
10      91.2MB  101MB   9437kB               primary

If we take a look at the GPT table, we can distinguish the following parts.

root@ubuntu-server:~# dd if=/dev/sdb bs=512 count=4 | xxd -c 16
0000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
...
00001f0: 0000 0000 0000 0000 0000 0000 0000 55aa  ..............U.
0000200: 4546 4920 5041 5254 0000 0100 5c00 0000  EFI PART....\...
...
0000400: a2a0 d0eb e5b9 3344 87c0 68b6 b726 99c7  ......3D..h..&..
0000410: 4e6e ad36 2fec 8046 bc1f 4a42 82d2 8052  Nn.6/..F..JB...R
0000420: 0008 0000 0000 0000 ff4f 0000 0000 0000  .........O......
0000430: 0000 0000 0000 0000 7000 7200 6900 6d00  ........p.r.i.m.
0000440: 6100 7200 7900 0000 0000 0000 0000 0000  a.r.y...........
...

First up, it is the legacy MBR, which GPT holds for reasons of compatibility (the code 0x55AA points to the end of the MBR). The second part of the GPT (512 bytes) contains the header information for GUID partitioning. And the first partition entry appears in position 0x400.


Aug 18, 2012

GPT, beyond the MBR (I)

Have you ever wondered what are the limits of the MBR (Master Boot Record)? I mean, the maximum size of a partition or a entire hard drive able to be handled. The answer is 2.2 TB.

Below you can observe the structure of the MBR (a total size of 512 bytes, where each row is 32 bytes).

root@ubuntu-server:~# dd if=/dev/sda bs=512 count=1 | xxd -g 4 -c 32
0000000: eb639010 8ed0bc00 b0b80000 8ed88ec0 fbbe007c bf0006b9 0002f3a4 ea210600  .c.................|.........!..
0000020: 00bebe07 3804750b 83c61081 fefe0775 f3eb16b4 02b001bb 007cb280 8a74018b  ....8.u........u.........|...t..
0000040: 4c02cd13 ea007c00 00ebfe00 00000000 00000000 00000000 00000080 01000000  L.....|.........................
0000060: 00000000 fffa9090 f6c28074 05f6c270 7402b280 ea797c00 0031c08e d88ed0bc  ...........t...pt....y|..1......
0000080: 0020fba0 647c3cff 740288c2 52bb1704 80270374 06be887d e81701be 057cb441  . ..d|<.t...R....'.t...}.....|.A
00000a0: bbaa55cd 135a5272 3d81fb55 aa753783 e1017432 31c08944 04408844 ff894402  ..U..ZRr=..U.u7...t21..D.@.D..D.
00000c0: c7041000 668b1e5c 7c66895c 08668b1e 607c6689 5c0cc744 060070b4 42cd1372  ....f..\|f.\.f..`|f.\..D..p.B..r
00000e0: 05bb0070 eb76b408 cd13730d f6c2800f 84d000be 937de982 00660fb6 c68864ff  ...p.v....s..........}...f....d.
0000100: 40668944 040fb6d1 c1e20288 e888f440 8944080f b6c2c0e8 02668904 66a1607c  @f.D...........@.D.......f..f.`|
0000120: 6609c075 4e66a15c 7c6631d2 66f73488 d131d266 f774043b 44087d37 fec188c5  f..uNf.\|f1.f.4..1.f.t.;D.}7....
0000140: 30c0c1e8 0208c188 d05a88c6 bb00708e c331dbb8 0102cd13 721e8cc3 601eb900  0........Z....p..1......r...`...
0000160: 018edb31 f6bf0080 8ec6fcf3 a51f61ff 265a7cbe 8e7deb03 be9d7de8 3400bea2  ...1..........a.&Z|..}....}.4...
0000180: 7de82e00 cd18ebfe 47525542 20004765 6f6d0048 61726420 4469736b 00526561  }.......GRUB .Geom.Hard Disk.Rea
00001a0: 64002045 72726f72 0d0a00bb 0100b40e cd10ac3c 0075f4c3 ab9d0600 00008020  d. Error...........<.u......... 
00001c0: 2100831a 3b1f0008 00000098 0700003b 1b1f051f d00ffea7 07000250 b8000000  !...;..........;...........P....
00001e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 000055aa  ..............................U.

The first 440 bytes contains the bootstrap code area, which takes care of starting up the operating system present on the active partition. Then, it is the disk signature, a 4-byte number that is randomly generated when the MBR is first created. It is an identifier which applies to the whole hard drive (not a single partition). After that, there are a couple of bytes set to null, and next, the partition table.

The partition table is made up of four entries of 16 bytes each which define the position and size of the sectors (LBA or Logical Block Addressing). In order to work around the problem of having just four partitions, these primary partitions can contain an arbitrary number of logical partitions. This schema can lead to problems, because some operating systems can only boot from primary partitions.

On the other hand, how do I work out the figure of 2.2 TB as top size for a partition? In conjunction with the universal sector size of 512 bytes and the 32-bit LBA pointers used by MBR partitions, you have ( 2^32 ) - 1 sectors * 512 bytes per sector.

(Also say that the MBR ends with the sequence of two bytes 0x55AA).

On account of the current size of hard drives and RAID technologies, the problem is really serious and will become more severe over time. In order to overcome it, the GUID partition table (GPT) is the natural successor to the MBR partition table.

GPT, supported on Linux since the 2.6.25 kernel version, uses 128-byte LBA modern tables, so therefore it is possible to have hard drives up to 8 ZB addressable for a disk with a 512-byte sector size. In addition, GPT can manage up to 128 partitions, so there is no need for extended or logical partitions. If you are interested, you can read up more about this topic on the Internet, since the aim of this article is to put forward how to manage this technology on Linux (also mention that for my tests, I used an Ubuntu Server 12.04 distribution).

First of all, point out that fdisk does not work with this partitioning scheme, but the good news is that can be got over with other Linux tools, such as parted. Keeping on with the tests, I have added a second hard drive to my system.

root@ubuntu-server:~# fdisk -l /dev/sdb 

Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table


Aug 12, 2012

Remote log server via HTTP (III)

Once finished the second article about the configuration of Samba, Remote log server via HTTP (II), now the server is going to be set up in order to be able to import the log directories via NFS and Samba. Furthermore, they will be served via Apache and backed up from time to time.

First of all, you have to create the directory where the logs will be mounted, as well as the backup directories.

[root@server ~]# mkdir -p /mnt/shared/logs 

[root@server ~]# mkdir -p /backup/logs/nfs /backup/logs/samba

So that SELinux allows Apache to access a directory brought by NFS or Samba, you have to enable the variables httpd_use_nfs and httpd_use_cifs. In addition, you have to change the SELinux security context of each directory imported.

[root@server ~]# setsebool -P httpd_use_nfs=on httpd_use_cifs=on

[root@server ~]# chcon -R -u system_u /mnt/shared/logs

[root@server ~]# chcon -R -t httpd_sys_content_t /mnt/shared/logs

Because the log server will not share any data through NFS and will offer no service by means of portmap, you will be able to deactivate the nfslock service.

[root@server ~]# service nfslock stop

[root@server ~]# chkconfig nfslock off

If you want to mount the NFS remote directory from client by hand, run the following order.

[root@server ~]# mount -t nfs -o soft,intr client.local:/var/log /mnt/shared/logs

And for the case of Samba.

[root@server ~]# mount -t cifs -o username=samba_logs,password=xxxxxx,soft //client.local/logs /mnt/shared/logs

The problem of mounting a remote directory statically is that the traffic passed down over the network is also increased, since when a file is updated, it is refreshed in the destination in the same way.

Moreover, you have to take into account another severe problem related to mount file systems via Samba, and is that if the connection is cut off (restarted, some network problem, etc.), Samba does not reconnect and the mount point can remain in an unstable state, thereby any existing synchronization would be lost. Thus, it is really important to always mount file systems by using automount.

Automount is a useful tool which takes care of mounting a directory when it is really accessed. It has got a timeout (600 sg by default) that when it is completed, the directory is automatically unmounted. This situation leads to reduce the network traffic (there may be long periods where you are not accessing the shared space) and avoid loss of synchronization. Also say that automount is managed by the autofs daemon.

This is the configuration used by automount to mount the log directory from client.

[root@server ~]# yum install autofs

[root@server ~]# vim /etc/auto.master
...
/mnt/shared/logs    /etc/auto.logs   -g,--timeout=300

[root@server ~]# cat /etc/auto.logs 
nfs    -fstype=nfs,soft,intr    client.local:/var/log
samba  -fstype=cifs,username=samba_logs,password=xxxxxx,iocharset=utf8,soft    ://client.local/logs

[root@server ~]# chmod 600 /etc/auto.logs

[root@server ~]# service autofs restart

The soft option is used for an application which is trying to access the shared area does not keep blocked if the connection is lost, and brings the control back to the system after 0.7 sg. With intr, allows the user to send an interruption signal if the application which uses NFS hangs.

If instead of hooking up to a Linux machine via Samba you have a Windows machine inside a domain, you would have to specify the domain name through the domain parameter.

Also mention that when you are mounting a directory from a Windows server, it might happen that strange characters turn up. This is due to the character conversion. So as to fix it, you have to use the iocharset=utf8 option for each mount point.


Aug 5, 2012

Remote log server via HTTP (II)

Let's keep on with the second article about setting up a Remote log server via HTTP. In the preceding part, the NFS daemon was configured in order to be able to export the local log server through NFS, and all this correctly secured by iptables and TCP wrappers. In this article, I am going to continue with the configuration of Samba.

First up, a new user called samba_logs will be adding to the system. From this user, the server machine will be able to hook up to the log directory via Samba. This user will not have neither a personal directory within home nor a shell.

[root@client ~]# useradd -d /dev/null -s /sbin/nologin samba_logs

In turn, this user will also be used to create an ACL (Access Control List) on the /var/log directory, granting read permissions to that user.

[root@client ~]# setfacl -R -m d:u:samba_logs:r /var/log/

[root@client ~]# getfacl /var/log/
...
default:user:samba_logs:r--
...

Then the samba package will be installed and configured.

[root@client ~]# yum install samba

[root@client ~]# cat /etc/samba/smb.conf
[global]
...
     hosts allow = 192.168.1.
...
[logs]
     comment = Log directory
     path = /var/log
     read only = yes
     valid users = samba_logs

Finally, the samba service will be restarted and marked as persistent. Furthermore, the user will be added to the local smbpasswd file.

[root@client ~]# service smb restart

[root@client ~]# chkconfig smb on

[root@client ~]# smbpasswd -a samba_logs

So as to shield the server by iptables, the following rules will be set into the /etc/sysconfig/iptables file (Samba uses the ports 137, 138 and 139 TCP/UDP).

[root@client ~]# cat /etc/sysconfig/iptables
...
-A RH-Firewall-1-INPUT -s server.local -p tcp --dport 137:139 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p udp --dport 137:139 -j ACCEPT
...

[root@client ~]# service iptables restart

Remember that is important to keep SELinux and TCP wrappes on. In order SELinux to let read the exported files, it is necessary to activate the variable samba_export_all_ro.

[root@client ~]# getenforce
Enforcing

[root@client ~]# setsebool -P samba_export_all_ro on

And below you can observe the configuration for iptables.

[root@client ~]# cat /etc/sysconfig/iptables
...
-A RH-Firewall-1-INPUT -s server.local -p tcp --dport 137:139 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p udp --dport 137:139 -j ACCEPT
-A RH-Firewall-1-INPUT -s server.local -p tcp --dport 445 -j ACCEPT
...

Now we can try out that everything is properly configured by running the next command on server.

[root@server ~]# yum install samba-client cifs-utils

[root@server ~]# smbclient -U samba_logs -L client.local
Enter samba_logs's password: 
Domain=[MYGROUP] OS=[Unix] Server=[Samba 3.5.10-125.el6]

    Sharename       Type      Comment
    ---------       ----      -------
    logs            Disk      Log directory
    IPC$            IPC       IPC Service (Samba Server Version 3.5.10-125.el6)
    samba_logs      Disk      Home Directories
Domain=[MYGROUP] OS=[Unix] Server=[Samba 3.5.10-125.el6]

    Server               Comment
    ---------            -------

    Workgroup            Master
    ---------            -------