WordPress vs. DiggWordPress seems to have a bad reputation when it comes to scalability. Maybe it’s deserved, since a default WordPress installation doesn’t really scale well. But making WordPress scale isn’t hard. I recently hit the Digg home page and got roughly 70,000 pageviews in under 12 hours. Another post hit the home page later the same day, and another 10,000 clickthroughs followed. As a result, I’ve been asked by a few people how I managed to keep my site up under that sort of stress. Honestly, I haven’t done anything that fancy. But for future reference I figured I’d document my configuration, and let people in on one trick that saved my butt.

First, I run WP-Cache. I started using WP-Cache after my first Digg experience and, for me, the performance improvements far outweigh the compatibility issues that may arise. If you’re a real performance junkie you may be interested in a simple hack to enable gzip with WP-Cache.

If you have some content that must be fully dynamic (like the pageview counters that I recently added here under the post title) you can take advantage of the mfunc functionality built into WP-Cache. To include some dynamic code in a cached page, use the following syntax:

<!--mfunc function_name('parameter'); -->
<?php function_name('parameter'); ?>
<!--/mfunc-->

Caching a page, then selectively adding dynamic functionality where it’s necessary will drastically reduce the load on your server.

This site is hosted on a dedicated server, along with another project I’m working on. A second server handles the database back-end for both sites. This setup is capable of handling a tremendous amount of traffic. The bottleneck, surprisingly enough, is the front-end web server (not the database).

In retrospect, it probably would have been wise to upgrade the front-end server to at least 1GB of ram (it currently has only 512MB). With Apache processes running upwards of 20MB each, RAM limits the number of child processes that can be spawned before the system starts thrashing. And since an Apache process can only handle one connection at a time, the number of Apache child processes places an upper bound on the number of concurrent requests the server can handle. Additional requests are backlogged, and when the backlog builds up system performance suffers.

I’ve been experimenting with using mod_backhand in combination with Amazon EC2 to offload some traffic from my primary web server to an elastic compute cloud during periods of heavy traffic. My success with Amazon S3 has fueled my interest in EC2, but I’m still skeptical of EC2’s ability scale rapidly enough to handle traffic spikes (it takes a minute or two to provision an EC2 server — in that time, a front page story on Digg could send upwards of 1,000 page requests your way).

Finally, my little trick. If you’re not running WP-Cache, and one of your posts is about to go viral, try caching the post manually. It’s a simple process:

Create a static version of your post using a tool like wget, or by saving the page in your browser. Duplicate the directory structure that you see in the post URL under the root web directory on your server (e.g., http://immike.net/blog/08/13/post-title would be something like /var/www/blog/08/13/post-title). Name the static copy of the post index.html and store it in your newly created directory.

If you’re using the standard WordPress rewrite rules (the ones WordPress auto-generates), static HTML files will override dynamically generated content. Thus, as soon as the static copy of your post is in place, Apache will start serving it instead of passing requests off to WordPress. Even on modest hardware Apache can handle hundreds of requests for static content per second, so this trick should keep your server up through the storm. Once things calm down, simply remove the files/directories and WordPress will take over once again.