Apr 24, 2011

Looking for web security breaches with Skipfish (I)

When we have to secure a system, it is very important to employ all the security tools we know, in order to protect it all the best we can. Skipfish is one of those applications which always has to be near.

It is a web security scanner, developed in C, which allows to discover lots of security holes on a web site by performing several kinds of tests:

  • High risk: SQL / PHP / XML / shell script injections, etc.

  • Medium risk: XSS (Cross-Site Scripting), CSS attacks, MIME type problems, etc.

  • Low risk: indexed directories, certificate problems, HTTP credentials, etc.

Due to it is written in pure C, it can reach an high performance: around 500 requests per second against targets located on Internet, about 2000 on local networks and more than 7000 on local hosts. Skipfish generates a sitemap with all the discovered paths, a summary of the document types and a set of security breaches.

I am going to try skipfish (1.86b version) out on an Ubuntu Server 10.10, by crawling a default MediaWiki installation. So as to install skipfish, we must fulfill some dependences on our system and then, compile it.

javi@ubuntu-server:~$ sudo aptitude install build-essential libssl-dev libidn11-dev

javi@ubuntu-server:~$ wget http://skipfish.googlecode.com/files/skipfish-1.86b.tgz

javi@ubuntu-server:~/skipfish-1.86b$ tar xvzf skipfish-1.86b.tgz

javi@ubuntu-server:~/skipfish-1.86b$ make

By typing the -h argument, we can take a look at all the available options by skipfish.

javi@ubuntu-server:~/skipfish-1.86b$ ./skipfish -h
skipfish version 1.86b by <lcamtuf@google.com>
Usage: ./skipfish [ options ... ] -o output_dir start_url [ start_url2 ... ]

Authentication and access options:

-A user:pass   - use specified HTTP authentication credentials
-F host=IP     - pretend that 'host' resolves to 'IP'
-C name=val    - append a custom cookie to all requests
-H name=val    - append a custom HTTP header to all requests
-b (i|f|p)     - use headers consistent with MSIE / Firefox / iPhone
-N             - do not accept any new cookies

Crawl scope options:

-d max_depth   - maximum crawl tree depth (16)
-c max_child   - maximum children to index per node (512)
-x max_desc    - maximum descendants to index per branch (8192)
-r r_limit     - max total number of requests to send (100000000)
-p crawl%      - node and link crawl probability (100%)
-q hex         - repeat probabilistic scan with given seed
-I string      - only follow URLs matching 'string'
-X string      - exclude URLs matching 'string'
-K string      - do not fuzz parameters named 'string'
-D domain      - crawl cross-site links to another domain
-B domain      - trust, but do not crawl, another domain
-Z             - do not descend into 5xx locations
-O             - do not submit any forms
-P             - do not parse HTML, etc, to find new links

Reporting options:

-o dir         - write output to specified directory (required)
-M             - log warnings about mixed content / non-SSL passwords
-E             - log all HTTP/1.0 / HTTP/1.1 caching intent mismatches
-U             - log all external URLs and e-mails seen
-Q             - completely suppress duplicate nodes in reports
-u             - be quiet, disable realtime progress stats

Dictionary management options:

-W wordlist    - load an alternative wordlist (skipfish.wl)
-L             - do not auto-learn new keywords for the site
-V             - do not update wordlist based on scan results
-Y             - do not fuzz extensions in directory brute-force
-R age         - purge words hit more than 'age' scans ago
-T name=val    - add new form auto-fill rule
-G max_guess   - maximum number of keyword guesses to keep (256)

Performance settings:

-g max_conn    - max simultaneous TCP connections, global (40)
-m host_conn   - max simultaneous connections, per target IP (10)
-f max_fail    - max number of consecutive HTTP errors (100)
-t req_tmout   - total request response timeout (20 s)
-w rw_tmout    - individual network I/O timeout (10 s)
-i idle_tmout  - timeout on idle HTTP connections (10 s)
-s s_limit     - response size limit (200000 B)
-e             - do not keep binary responses for reporting

Basically we can see that the skipfish command is made up by a set of options, an output directory for saving the results, and a series of URLs to be analyzed.

We have to take into account when skipfish is working out, it just tests those found links, but we also have the possibility to probe other URLs by means of a brute-force attack, by mixing names (index, doc, etc.) and extensions (pdf, bat, etc.). For this purpose, skipfish affords four dictionaries.

javi@ubuntu-server:~/skipfish-1.86b$ ls dictionaries/*.wl                                                                                                                                                                                                                     
dictionaries/complete.wl  dictionaries/extensions-only.wl  dictionaries/medium.wl  dictionaries/minimal.wl

By default, skipfish will treat to use a dictionary named skipfish.wl and situated on the work directory. Therefore either we can copy one of these dictionaries into the work directory under this name, or use the -W option to define the wordlist path or on the contrary, not to use a dictionary.

As the application is crawling, adds new words within the dictionary.


Apr 17, 2011

Benchmarking with Phoronix Test Suite (II)

This is the second and final part of the article Benchmarking with Phoronix Test Suite.

If we want to carry out the test or suite with the default options, we must aggregate the default-run parameter. Also mention that we can type multiple tests or suites with the run or default-run order.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite run iozone

Phoronix Test Suite v3.0.1
IOzone Test Configuration


Record Size:

1: 4Kb
2: 64Kb
3: 1MB
4: Test All Options

Enter Your Choice: 2

File Size:

1: 512MB
2: 2GB
3: 4GB
4: 8GB
5: Test All Options

Enter Your Choice: 1

Disk Test:

1: Write Performance
2: Read Performance
3: Test All Options

Enter Your Choice: 3
Would you like to save these test results (Y/n): Y
Enter a name to save these results: ubuntu-server

Current Test Identifiers:
- apache

Enter a unique name for this test run: iozone
...

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite default-run iozone

Where are the results and the test environments stored on the filesystem?

root@ubuntu-server:~# tree -d .phoronix-test-suite/test-results/
.phoronix-test-suite/test-results/
├── pts-results-viewer
└── ubuntu-server
├── result-graphs
└── system-logs
├── apache
└── iozone

root@ubuntu-server:~# tree -d .phoronix-test-suite/installed-tests/
.phoronix-test-suite/installed-tests/
└── pts
├── apache-1.3.0
└── iozone-1.7.0

So as to get information about the saved test results and the installed tests and their usage, we have the next options.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-saved-results
                                                                                                                                                                                                                                                             
Phoronix Test Suite v3.0.1                                                                                                                                                                                                                                            
1 Saved Results                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                             
Saved Name: ubuntu-server      Title: ubuntu-server                                                                                                                                                                                                                   
- apache                                                                                                                                                                                                                                                      
- iozone

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-test-usage

Phoronix Test Suite v3.0.1
2 Tests Installed

TEST                 VERSION  INSTALL DATE  LAST RUN    AVG RUN-TIME  TIMES RUN
pts/apache-1.3.0   - 1.3.0    2011-04-01    2011-04-01  5m15s         1
pts/iozone-1.7.0   - 1.7.0    2011-04-02    2011-04-02  2m10s         2

And further, if we want to acquire more details about an executed result, the info parameter will be our choice.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite info ubuntu-server

Title: ubuntu-server
Identifier: ubuntu-server

Test Result Identifiers:
- apache
- iozone

Contained Tests:
- Apache Benchmark
- IOzone

Finally, we also have the option to export the results in other formats (csv, text and pdf).

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite result-file-to-text ubuntu-server > ubuntu-server.txt

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite result-file-to-csv ubuntu-server > ubuntu-server.csv

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite result-file-to-pdf ubuntu-server

Saved To: /root/ubuntu-server.pdf

As well, we will always have the possibility to treat directly the results by means of the html structure.

root@ubuntu-server:~# ls -l .phoronix-test-suite/test-results/ubuntu-server/
total 32
-rw-r--r-- 1 root root 3291 2011-04-02 18:00 composite.xml
-rw-r--r-- 1 root root  192 2011-04-02 18:00 index.html
-rw-r--r-- 1 root root 5163 2011-04-02 21:32 pts-results-viewer.xsl
drwxr-xr-x 2 root root 4096 2011-04-02 21:32 result-graphs
drwxr-xr-x 4 root root 4096 2011-04-02 18:00 system-logs
-rw-r--r-- 1 root root 1491 2011-04-02 17:51 test-1.xml
-rw-r--r-- 1 root root 2210 2011-04-02 18:00 test-2.xml

With html or pdf formats, apart from the numeric data, we will be able to appreciate that Phoronix generates several graphics through the results.

Also point out another useful utility: we can merge several results and obtain a new combination from all of them. For this purpose, we first have to copy the other outcomes into the test-results directory.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite merge-results ubuntu-server centos
Merged Results Saved To: /root/.phoronix-test-suite/test-results/merge-8136/composite.xml

The key of this benchmarking application is to apply correctly in each moment the suitable tests. Due to the large variety of available tests, we can get much information. On the contrary, the main problem under my personal opinion is that the options which you can choose to run a test are very limited. If you wanted to gather more information from a specific tool (for instance iozone), you should execute it manually.

But in short, I think that Phoronix Test Suite is a great tool for measuring the perfomance of a system.


Apr 11, 2011

Automatic updates on Ubuntu with unattended-upgrades

Some time ago I talked about the importance of having correctly our Linux systems up to date (at least automatically), specifically those issues related to security, focusing on CentOS/RHEL distributions. For this purpose I wrote an article named yum-security plugin.

For systems based on Debian/Ubuntu, you have got a package denominated unattended-upgrades, which allows to apply automatic updates (stable, security, updates and proposed-updates).

When we install an Ubuntu release, on the one hand we can mark the option for the system to automatically install the security updates. In this case, Ubuntu will install the unattended-upgrades package on the server and manage this subject.

And on the other, we can directly install it later and fit it based on our needs.

root@ubuntu-server:~# aptitude install unattended-upgrades

Through its configuration file (50unattended-upgrades), we can fit the types of updates (stable and security), the list of packages which must not be updated (mysql-server and apache2), an optional email address for warning about any problem, band with and so on.

root@ubuntu-server:~# cat /etc/apt/apt.conf.d/50unattended-upgrades
// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {
  "${distro_id} stable";
  "${distro_id} ${distro_codename}-security";
//      "${distro_id} ${distro_codename}-updates";
//      "${distro_id} ${distro_codename}-proposed-updates";
};

// List of packages to not update
Unattended-Upgrade::Package-Blacklist {
  "mysql-server";
  "apache2";
};

// Send email to this address for problems or packages upgrades
Unattended-Upgrade::Mail "admin@ubuntu-server.local";

// Do automatic removal of new unused dependencies after the upgrade
//Unattended-Upgrade::Remove-Unused-Dependencies "false";

// Automatically reboot *WITHOUT CONFIRMATION* if a
// the file /var/run/reboot-required is found after the upgrade
Unattended-Upgrade::Automatic-Reboot "false";

// Use apt bandwidth limit feature, this example limits the download
// speed to 70kb/sec
//Acquire::http::Dl-Limit "70";

In order to set the update period (in days), we have to edit the 20auto-upgrades file. In the following example, the packages which can be updated will be downloaded everyday, but the automatic updates will just be applied once a week. The downloaded packages will be removed every 15 days.

root@ubuntu-server:~# cp -a /usr/share/unattended-upgrades/20auto-upgrades /etc/apt/apt.conf.d/

root@ubuntu-server:~# cat /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::Unattended-Upgrade "7";
APT::Periodic::AutocleanInterval "15";

We can manually execute the unattended-upgrade daemon as well, by means of the next order.

root@ubuntu-server:~# unattended-upgrade -d
Initial blacklisted packages: mysql-server apache2
Starting unattended upgrades script
Allowed origins are: ["('Ubuntu', 'stable')", "('Ubuntu', 'maverick-security')"]
pkgs that look like they should be upgraded:
Fetched 0B in 0s (0B/s)                                                                                                                                                           
blacklist: ['mysql-server', 'apache2']
InstCount=0 DelCount=0 BrokenCout=0
No packages found that can be upgraded unattended

And finally, also say this application will be run via cron (/etc/cron.daily/apt). All output will be logged into the /var/log/unattended-ugprades.log file.


Apr 5, 2011

Benchmarking with Phoronix Test Suite (I)

The Phoronix Test Suite is an interesting benchmarking platform aimed at testing and measuring the performance of multiple stuff, such as computers, graphics cards, processors, operating systems and so on.

I have used (in my professional work) and talked about this tool several times, but I had never dedicated any article to evaluate it. Let's get going!

For my tests, I am going to utilize an Ubuntu Server 10.10 (64 bits) and Phoronix Test Suite 3.0.1. When you download the application and descompress it, you have two options: on the one hand you can install it over the operating system (install.sh), and on the other, you may directly run the appropiate script (phoronix-test-suite) and carry out all the options that Phoronix provides. Regardless the choice, first of all you must install a serie of necessary dependences.

root@ubuntu-server:~# aptitude install php5-cli php5-gd php5-curl php-fpdf

root@ubuntu-server:~# tar xvzf phoronix-test-suite-3.0.1.tar.gz

root@ubuntu-server:~/phoronix-test-suite# ./install-sh

In my case, I will choose the second. Also say that if we are behind a proxy, we have to execute the script with the network-setup option, in order to set the proxy configuration up. In addition, we can check out the information about our system hardware and software.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite network-setup

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite system-info

Phoronix Test Suite v3.0.1
System Information

Hardware:
Processor: QEMU Virtual 0.12.5 @ 3.00GHz (1 Core), Motherboard: Bochs, Chipset: Red Hat Virtio, Memory: 1 x 2048 MB RAM, Disk: 15GB, Graphics: Cirrus Logic GD 5446

Software:
OS: Ubuntu 10.10, Kernel: 2.6.35-28-server (x86_64), Display Driver: cirrus, File-System: ext4

Phoronix supplies a set of suites (cpu, database, kernel, memory, etc.) that in turn, make up a group of tests (compress-gzip, dbench, sqlite, etc.), also known as profiles. In total, we have around more than 60 suites and 130 tests.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-available-suites

Phoronix Test Suite v3.0.1
Available Suites

pts/audio-encoding               - Audio Encoding                   System
pts/chess                        - Chess Test Suite                 Processor
pts/compilation                  - Timed Code Compilation           Processor
...

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite list-available-tests

Phoronix Test Suite v3.0.1
Available Tests

pts/aio-stress               - AIO-Stress                          Disk
pts/apache                   - Apache Benchmark                    System
pts/battery-power-usage      - Battery Power Usage                 System
...

If we want to know more details about a concrete suite or test, we can run the following order (in this example, the memory suite consists of three tests).

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite info memory

Phoronix Test Suite v3.0.1
Memory Test Suite

Run Identifier: pts/memory-1.0.2
Suite Version: 1.0.2
Maintainer: Michael Larabel
Suite Type: Memory
Unique Tests: 3
Suite Description: The system memory test suite consists of tests designed to test the computer's system memory (RAM) performance. Among these tests are RAMspeed and Bandwidth.

pts/memory-1.0.2
* pts/ramspeed
* pts/stream
* pts/cachebench

And if we want to install a suite or test, we must add the install parameter to the script. This sentence will firstly resolve and install the necessary dependences and afterwards, download, compile and install the test. If you select a suite, this order will install all tests belonging to the suite.

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite install apache

The following dependencies are needed and will be installed:

- build-essential
...
Phoronix Test Suite v3.0.1

To Install: pts/apache-1.3.0
...

Then we are going to see the execution of the apache test (we can also use this command to launch a suite).

root@ubuntu-server:~/phoronix-test-suite# ./phoronix-test-suite run apache
Would you like to save these test results (Y/n): Y
Enter a name to save these results: ubuntu-server
Enter a unique name for this test run: apache

Phoronix Test Suite v3.0.1
If you wish, enter a new description below.
Press ENTER to proceed without changes.

Current Description: Running pts/apache-1.3.0.

New Description:


Apache Benchmark:
pts/apache-1.3.0
Test 1 of 1
Expected Trial Run Count: 3
Running Pre-Test Script @ 15:45:48
Started Run 1 @ 15:45:53
Started Run 2 @ 15:47:34
Started Run 3 @ 15:49:17  [Std. Dev: 2.00%]
Running Post-Test Script @ 15:51:00

Test Results:
7156.08
7026.62
6876.06

Average: 7019.59 Requests Per Second

In the previous output, we can make out that the script has asked for a name to save the results (under this name, we will be able to save other tests) and a specific name for the test. In other tests as follows, we can see that the script requests other types of data, such as the record size, file size and disk test.