Following up network connections with conntrack (II)

Let's finish the previous article about Following up network connections with conntrack (I). Other important parameters which can be changed to optimize the system are related to the time of the different types of connections.

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
432000

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait
120

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait
60

The first parameter sets up the maximum lifetime for an already established connection (432000 sg can be long; 28800 could be enough). The second and third are the maximum lifetime for a waiting connection and for the remote endpoint closes the socket.

So as to list all variables based on the conntrack module, type the next order.

root@ubuntu-server:~# sysctl -a | grep conntrack | grep ipv4
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2 = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 300
net.ipv4.netfilter.ip_conntrack_tcp_loose = 1
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 0
net.ipv4.netfilter.ip_conntrack_tcp_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30
net.ipv4.netfilter.ip_conntrack_max = 15768
net.ipv4.netfilter.ip_conntrack_count = 2
net.ipv4.netfilter.ip_conntrack_buckets = 4096
net.ipv4.netfilter.ip_conntrack_checksum = 1
net.ipv4.netfilter.ip_conntrack_log_invalid = 0

And if you want to change the value of any variable, you must add it within the sysctl.conf file and reload the settings.

root@ubuntu-server:~# cat /etc/sysctl.conf
...
net.ipv4.netfilter.ip_conntrack_max = 131072

root@ubuntu-server:~# sysctl -p

One interesting option for the conntrack command is the possibility to get the statistics about the connection tracking.

root@ubuntu-server:~# conntrack -S
entries                 2  
searched                0  
found                   1107
new                     4  
invalid                 0  
ignore                  0  
delete                  2  
delete_list             2  
insert                  4  
insert_failed           0  
drop                    0  
early_drop              0  
icmp_error              0  
expect_new              0  
expect_create           0  
expect_delete           0  
search_restart          0

Another useful feature for conntrack is to output the connection state on real-time, similar to when you run a "tail -f" on a file.

root@ubuntu-server:~# conntrack -E

We can conclude with this couple of articles that the conntrack module is other helpful way to improve the Linux performance.

Cached memory in Linux

I am sure that there are many people that if they saw the following output of 'top', they would say that I have a problem with my free memory... they would be wrong.

[root@zabbix ~]# top
...
Mem:   4044540k total,  4007540k used,    37000k free,   156336k buffers
Swap:  2097144k total,        0k used,  2097144k free,  2086808k cached
...

The previous data correspond to a Zabbix installation where the free memory is around 37 MB, but the cached memory is more than 2 GB. What is happening here?

The answer is straightforward: Linux always tries to use all available memory, and thereby, it caches all read data. If at any moment an application needs memory, Linux will free it from the cached memory. This way of acting is pretty good because you will have better performance having the more frecuent data into the memory.

Really you will have a serious problem when your free memory is low, further your cached memory too and on top of all that, your system begins to swap.

Also say that other excellent data from the above 'top' is the swap memory value: 0. A well configured Linux system never should utilize swap. The key parameter for this purpose is swappiness. I usually set it with a value of 20.

[root@zabbix ~]# cat /proc/sys/vm/swappiness
20

There is a quick manner to force to the operating system so as to free the cached memory: changing the value of the drop_caches variable.

[root@zabbix ~]# cat  /proc/sys/vm/drop_caches
0

[root@zabbix ~]# sync ; echo 1 > /proc/sys/vm/drop_caches

[root@zabbix ~]# top
...
Mem:   4044540k total,  1756996k used,  2287544k free,     1572k buffers
Swap:  2097144k total,        0k used,  2097144k free,    62564k cached
...

We can demonstrate the cached memory operation through an easy bash script. The program will search twice the word "test" inside all files of the /var/log/httpd directory.

[root@zabbix ~]# cat script.sh
#!/bin/bash

for (( i=0; i<2; i++))
do
   free -o -m
   /usr/bin/time -f "\nSeek time: %e sg\n" grep -r test /var/log/httpd/ > /dev/null
done

[root@zabbix ~]# ./script.sh
             total       used       free     shared    buffers     cached
Mem:          3949       1712       2237          0          2        101
Swap:         2047          0       2047

Seek time: 3.76 sg

             total       used       free     shared    buffers     cached
Mem:          3949       2003       1946          0          3        412
Swap:         2047          0       2047

Seek time: 0.50 s

In the first loop you can see that the cached memory was 101 MB and it spent 3.76 sg looping through all files. But in the second loop, the 'grep' command spent 0.5 sg because the files were already cached on memory (its size grew from 101 MB to 412 MB).

If we take a look at the size of the /var/log/httpd directory, we can appreciate that the cached memory increment practically matchs with the size of that directory.

[root@zabbix ~]# du -shx /var/log/httpd/
318M     /var/log/httpd/

Following up network connections with conntrack (I)

Linux has got the ability to perform a monitoring of existing connections by means of the conntrack module, which is compiled but not installed in distributions such as RHEL or CentOS. In order to load it, you can run the next order.

[root@centos ~]# modprobe ip_conntrack

In other operating systems like Debian or Ubuntu Server, first of all you must install the conntrack package and load the nf_conntrack_ipv4 module (if you want to work with IPv6, you will have to load the nf_conntrack_ipv6 module).

root@ubuntu-server:~# aptitude install conntrack

root@ubuntu-server:~# modprobe nf_conntrack_ipv4

The conntrack module allows the kernel to register in a table all network connections of the system (established, time_wait, close, etc.). It used by several applications such as iptstate (it shows information about the state of the system connections) or Shorewall (firewall).

Another example of use for this module it is for instance, when the server has to realize NAT tasks with iptables and it is necessary to keep a table of connections implicated.

The file where conntrack logs all connections is /proc/net/ip_conntrack.

root@ubuntu-server:~# cat /proc/net/ip_conntrack
tcp      6 89 TIME_WAIT src=192.168.1.11 dst=192.168.1.12 sport=59302 dport=10050 packets=5 bytes=291 src=192.168.1.12 dst=192.168.1.11 sport=10050 dport=59302 packets=5 bytes=289 [ASSURED] mark=0 secmark=0 use=1
...

root@ubuntu-server:~# conntrack -L
tcp      6 89 TIME_WAIT src=192.168.1.11 dst=192.168.1.12 sport=59302 dport=10050 packets=5 bytes=291 src=192.168.1.12 dst=192.168.1.11 sport=10050 dport=59302 packets=5 bytes=289 [ASSURED] mark=0 secmark=0 use=1
...

The two first fields are the connection protocol (TCP, 6) and then is the connection state (TIME_WAIT). The rest of the fields represent the IP addresses and ports involved, as well as the number of packets and bytes exchanged between the two points of the connection.

You have also to take into account that Linux saves the connection state in memory, and each of them uses around 350 bytes.

If you want to know how many open connections has got the system, you can utilize the following sentences.

root@ubuntu-server:~# cat /proc/net/ip_conntrack | wc -l
856

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
856

root@ubuntu-server:~# conntrack -C
856

This value is quite important because if at any moment we appreciate that any of our services works slowly (for instance Apache) or many connections are rejected, it can be due to which the number of open connections exceeds the maximum number of connections allowed.

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
65536

The size of the hash table is also limited.

root@ubuntu-server:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
16384

If you want to modify it, you must do it when the module is loaded.

root@ubuntu-server:~# modprobe nf_conntrack_ipv4 hashsize=32768

Fitting virtual disks after reducing

This is the continuation of the article Shrinking virtual disks with LVM. It turns out that once you have mirrored your virtual disk to another smaller, it is normal to have wasted some space in the process, that is to say, an area of the new virtual disk which has been unused.




For this reason, we are going to learn in this article how to recover it. At this moment, the available size of our new disk is 16 GB.

[root@centos ~]# lvs
LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert
LogVol00 VolGroup00 -wi-ao 16,00G                            
LogVol01 VolGroup00 -wi-ao  1,00G

The idea is to create a new partition on the disk in order to take that unutilized space. The partition type must be Linux LVM (8e hexadecimal code).

[root@centos ~]# fdisk /dev/sda
...
Orden (m para obtener ayuda): n
Acción de la orden
e   Partición extendida
p   Partición primaria (1-4)
p
Número de partición (1-4): 3
Primer cilindro (2241-2480, valor predeterminado 2241):
Se está utilizando el valor predeterminado 2241
Último cilindro o +tamaño o +tamañoM o +tamañoK (2241-2480, valor predeterminado 2480):
Se está utilizando el valor predeterminado 2480

Orden (m para obtener ayuda): t
Número de partición (1-4): 3
Código hexadecimal (escriba L para ver los códigos): 8e
Se ha cambiado el tipo de sistema de la partición 3 por 8e (Linux LVM)

Orden (m para obtener ayuda): w

[root@centos ~]# partprobe

[root@centos ~]# fdisk -l

Disco /dev/sda: 20.4 GB, 20401094656 bytes
255 heads, 63 sectors/track, 2480 cylinders
Unidades = cilindros de 16065 * 512 = 8225280 bytes

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        2241    17891328+  8e  Linux LVM
/dev/sda3            2241        2480     1924849   8e  Linux LVM

Next step is to set up a PV (physical volume) on that new partition.

[root@centos ~]# pvcreate /dev/sda3

Now we have to add the new PV created to the existing VG (volume group) and then, extend the LV (logical volume).

[root@centos ~]# vgextend VolGroup00 /dev/sda3

[root@centos ~]# lvextend -l +100%FREE /dev/VolGroup00/LogVol00

And finally, we must only to expand the ext3 filesystem.

[root@centos ~]# resize2fs /dev/VolGroup00/LogVol00

After the operation, we have retrieved 1,81 GB.

[root@centos ~]# lvs
LV       VG         Attr   LSize  Origin Snap%  Move Log Copy%  Convert
LogVol00 VolGroup00 -wi-ao 17,81G                             
LogVol01 VolGroup00 -wi-ao  1,00G

So as to realize the task, it is not necessary to put the machine into rescue mode and it can be performed on hot.