This document is available on the Internet at:  http://urbanmainframe.com/folders/blog/20041230/folders/blog/20041230/

Web Server Flood Barrier

Date:  30th December, 2004

Tags:

Thames Flood Barrier

It's safe to say that my last attempt to implement "flood" defences on this web server was, well, disastrous. I installed mod_throttle and brought the server to its knees for around 8 hours.

Despite this, ahem, minor setback - I continued to investigate configurations and software that would enable me to maintain server availability, even under DoS and DDoS attacks. I think I've finally managed to get some serviceable defences in place...

Defending Against DoS Attacks

“no resources are allocated to spoofed requests”

DoS and DDoS attacks have knocked several high-profile websites offline, sometimes for several hours. Victims include the likes of Yahoo!, Microsoft and Akamai. The attack on Akamai had a knock-on effect on its clients too, including Apple and even the mighty Google!

Down-time is not a good thing for any website. But if you're running an ecommerce company, even a few minutes offline will likely result in a loss of revenue. While the Urban Mainframe isn't business-critical, its URL is often given to prospective Shapeshifter clients to illustrate the capabilities of our CMS. It doesn't look good for us if the site is unavailable.

How DoS Works

In order to establish a TCP/IP connection between two computers (client and server) both machines participate in a procedure known as the "Three-Way Handshake" (the exchange of three "packets"). The digital conversation looks like this:

  1. Client computer sends a "SYN" (synchronise) packet to initiate the connection;
  2. Server allocates resources to the connection request and responds with a "SYN/ACK" (synchronise request acknowledged) packet;
  3. Client responds with an "ACK" packet to acknowledge the server's response and the connection is made.
diagram illustrating TCP/IP "three-way handshake"

The mechanics of a DoS attack are very simple:

  1. Client computer sends a "SYN" packet with spoofed IP return address;
  2. Server allocates resources to the connection request and responds with a "SYN/ACK" packet;
  3. Server does not receive "ACK" packet back from the client, because its "SYN/ACK" was "lost" to a spoofed IP address;
  4. Server waits until a "timeout" occurs (often as long as 3 minutes) then resends the "SYN/ACK", under the assumption that it was lost or dropped;
  5. Client (or multiple clients) continue to send a high volume of fake "SYN" packets until all the server's resources are consumed;
  6. Server is effectively offline.
diagram illustrating TCP/IP "three-way handshake" during DoS attack

Introducing "SYN cookies"

Fortunately, some very clever people considered the DoS problem and devised a rather elegant and highly effective solution: SYN cookies.

The SYN cookie-enabled TCP/IP stack allocates no resources in response to an incoming "SYN" packet. Instead, it sends a carefully constructed "SYN/ACK" complete with a unique sequence number generated as an MD5 hash of the client's IP address, port number and other information. When the client responds with its normal "ACK", that special sequence number is returned and the server can then verify that the request has originated from a live IP address. Thus, the server first allocates resources on the third packet of the handshake, not the first and no resources are allocated to spoofed requests.

DoS attacks, once considered to be indefensible, are thus handled effectively and gracefully and, even while under attack, the target server should be able to handle requests as normal.

SYN cookies are a standard component of modern Linux and FreeBSD distributions. Unfortunately, they are not enabled by default under Linux. But that's easily remedied. Just add the following command to your "rc.local" file (/etc/rc.d/rc.local), restart the server and SYN cookies will begin defending the computer immediately:

echo 1 > /proc/sys/net/ipv4/tcp_syncookies

Introducing "mod_dosevasive"

So we've now got a server that's resistant to DoS and DDoS attacks from spoofed IP addresses. But how do we deal with floods that SYN cookies ignores becauses their packets aren't spoofed? Such floods can occur when bots index web content or when a user decides to cURL or wget your entire website. High-volume, high frequency requests can be as damaging to a web-server as any DoS attack.

mod_dosevasive, available from the imaginatively named Nuclear Elephant, is an Apache module designed to take evasive action in the event of a HTTP DoS, DDoS or brute force attack.

The module protects the web-server by denying requests, by IP address, from any client if they break these few, simple rules (which are totally configurable by the administrator):

  • Do not request the same page more than twice per second
  • Do not make more than 50 concurrent requests per thread, per second
  • Do not make any requests while blocked

According to Jonathan A. Zdziarski, mod_dosevasive's author, "this method has worked well in both single-server script attacks as well as distributed attacks."

mod_dosevasive is designed to be able to communicate with ipchains, firewalls and routers (etcetera) so, with minimal effort, you can have your firewall's configuration automatically updated as new rogue IP addresses are identified. As an added bonus, mod_dosevasive can issue reports via email and logging facilities.

Installation is a breeze. On my DSO-enabled Apache 1.3 server, I simply extracted the download into my server's /tmp directory before entering the following command:

/usr/local/apache/bin/apxs -iac /tmp/mod_dosevasive.c

Then a quick trip into Apache's "httpd.config", to which I added the following suffix:

# Begin mod_dosevasive Configuration
LoadModule dosevasive_module /usr/local/apache/libexec/mod_dosevasive.so
AddModule mod_dosevasive.c
# End mod_dosevasive Configuration

The I restarted Apache:

cd /usr/local/apache/bin;./apachectl graceful

Close the Windows

If there are two unoccupied houses side-by-side, one with doors and windows locked and one with a window that's open, we all appreciate that a burglar is more likely to practice his skills on the latter. Like a river, he will choose the path of least resistance.

The same is true of electronic predators. It's virtually impossible to keep a skilled and determined hacker at bay - fortunately, the vast majority of Internet criminals aren't members of this elite. The unskilled hackers, the so-called "script kiddies", are only as good as the tools they have at their disposal. Like the aforementioned burglar, they will pick on the easy target first. With SYN cookies and mod_dosevasive we're not only closing our windows, we're bolting them and arming our alarm system too.

As a result of our efforts, we've created a web-server which, while certainly not invulnerable, is hardened against some of the more common DoS-type attacks that could befall it. We can sleep a little easier at night.