WebTap

Most webservers write access-log files which contain information about source address of the client, date of the access, accessed url and status code of the server response. The data is not specific enough for an effective analysis of the web-communication. Therefore more detailed information is needed. The ModSecurity-Module integrates an audit-logging engine which writes detailed log-data in a format specific to ModSecurity. Unfortunately, writing complex log-data affects the webserver system and might result in performance issues which in return makes the web-applications response seem to be awkward.

To overcome this problem I wrote WebTap. It is basically a tool for reading http-streams via a sniffer-tool and writes the complete audit-log in the format of modsecurity to disk. This enables you to do the audit-logging on a separate machine by the use of a network tap or the monitor-port of your switch. Thus it doesn't affect the webservers performance, but provides you the most detail logging-data.

This way it can act as a sensor for a web-application oriented intrusion detection system. The following graphic shows the intended use and the traffic flow for setting up the WebTap on a dedicated sensor system:

Click on the graphic to get a closer view. The setup can also be done without a mirroring switch by placing the WebTap application directly on the web server system. This will result in more complete audit logs as the mirror port might drop frames (packets) when under heavy load. (Dropped packets are a general problem when monitoring network traffic.)

Also note, that the recording of audit log data is independent of the ModSecurity module. Thus, this setup even allows for auditing of web-applications without ModSecurity.

Features

WebTap basically features

  • Writing audit-events of http-traffic read from tcpick
  • Network access to the audit-stream via secure sockets
  • Writing audit-events of https-traffic read from ssltrace (beta-state)

Please note that support for reading SSL streams is not very well tested (i.e. heavy-loaded networks might loose packets, which in turn might cause the WebTap struggling). As with the web-audit library, the WebTap monitor can also be easily integrated into your own application. If you're interested in integrating WebTap into your own projects, feel free to contact me at chris (at) jwall.org.

Getting Started

For creating audit-events from tcpick, WebTap relies on patched version of tcpick, which is a tcp-sniffer and re-assembling tool. Combined with this tool, WebTap follows http-streams, reads the application-data and can in principle be used to log any part of the communication.

Download

A first release of the WebTap application is already available. Please keep in mind that the software is still under development and not production ready.

There is a signed as well as an unsigned jar available, see the security page for details. The distributed jars already contain the current release of the web-audit library, which is used for parsing and writing the audit-log data.

Running WebTap

Keep in mind that you need a patched tcpick available. For building that tcpick, you can follow the instructions in the Building Tcpick section. After you built tcpick, running WebTap is done by starting tcpick and piping its output to the WebTap:

   tcpick -i eth0 -n -h -bBR "tcp port 80" | java -jar org.jwall.web.tap-0.4.8.jar \
           -o /path/to/log-directory

The command is to be placed in one line, the above line is split due to visibility. In its current state, WebTap will create a serial audit-log file for each observed "Host" header in the streams. Thus, if you have two sites running on www.site1.com and www.site2.com, the traffic recorded for the sites will be written to the files www.site1.com-audit.log and www.site2.com-audit.log within the output-directory given at the command line.

Linux/Unix only: As the tcpick sniffer is currently only available on Unix-platforms, the WebTap is not usable on Windows systems yet. However, there are plans to implement the tcp-reassembly directly within the Java part which will result in an indepedent application that might only depend on java libs (e.g. jpcap).

Auditing HTTPS Streams

Certain SSL streams can be decrypted if the private key of the server is previously known. An open-source library for decrypting SSL streams with a given server-key is provided by SSL-Tech. Additionally to their decryption library they provide a small sample tool, called ssltrace, which I modified to generate output-data similar to tcpick.

With this modified ssltrace it is then possible to audit https accesses to a web application in case the server's private key is available. There are a few instructions on Building ssltrace to be used with the WebTap monitor. After building the ssltrace tool, auditing of a ssl-secured application can be started as follows:

   ssltrace -i eth0 -ip  -port 443 -key /path/to/server.key | \
      java -jar org.jwall.web.tap-0.4.8.jar -o /path/to/log-directory -t ssl

Note the -t ssl option given to the WebTap.

Windows: As there is a Windows version of the DSSL library available at SSL-Tech, auditing https-streams should also be possible on Windows systems. However, in lack of a development environment for Windows I cannot prove/test/provide a Windows binary. If anyone can provide an executable ssltrace for the Windows platform I'd be happy to include this on the web-page.

License

So far the WebTap is put under the GNU Public License (GPLv3). If you're interested in using the library within a closed-source environment then feel free to contact me a chris (at) jwall.org.

Though the software might be related to the ModSecurity module, the WebTap monitor is neither maintained nor otherwise affiliated with ModSecurity, Thinking Stone or Breach Security. It is as such not a product or official implementation of the aforementioned companies.