flag of the United Kingdom

Experimenting with Distributed Caching

Date:  Sat, 11th-Sep-2004prevnext

Tags: Commentary, Distributed Computing

I am currently experimenting with a couple of distributed content caching (DCC) systems.

DCC systems provide a means to distribute heavily trafficked content to reduce load on a web server, helping to avoid the so-called "Slashdot" effect which can actually knock a server offline.

DCC systems work by storing local copies of content on multiple web-servers and distributing requests for that content amongst this collection of hosts, using highly optimised load-balancing algorithms. The net effect of which is that the principle server (the content producers host) has to service only a percentage of the incoming requests, with the remainder being serviced by the DCC network.

I'm curious as to how effective these systems are. The benefits are obvious: for example, an efficient DCC network could help to alleviate some of the scaling problems of RSS content delivery...

“DCC could help to alleviate some of the scaling problems of RSS”

Therefore I have decided to conduct a little experiment. I am offering two files for download, each of which is served via a different DCC network. I ask you to download either or both of these files and record your downloads in the comments.

I will tally these up and compare them against my access logs in order to quantify the efficiency of the respective DCC networks.

Please invite your friends and colleagues to download these files too (if there are only a few requests for the files it's going to be difficult to determine the effectiveness of the DCC networks).

The files are PDFs (Adobe Reader required to view) of a couple of Apple Computer posters.

The first file is delivered via Coral and weighs in at 760KB:
Download Test File 1 - http://xrl.us/czu7

The second file is delivered via FreeCache and is somewhat larger at 14MB:
Download Test File 2 - http://xrl.us/czu8

Naturally, I will publish my findings here - if I manage to accumulate enough data to support a conclusion.

Thank you for participating.

You can comment on this entry, or read what others have written (6 comments).