As many of you may have noticed, DailyKos performed admirably yesterday. I saw a diary or comment somewhere asking for a post-mortem of what happened, so here it is.
First, the hardware: There is an image server (single Opteron, 1GB RAM) running thttpd, 10 dual Xeons with 2GB RAM running Apache 1.3 proxies with Apache 2 with mod_perl2 backends, a dual Opteron with 8GB RAM as the master database running MySQL 5.0, a dual Opteron with 4GB RAM as the slave database, and a dual P4 running jotter's search server, and a memcached server with 2GB RAM on a dual Xeon.
Before Election Day, I pursued different ways to aggressively cache content. Some worked well, like moving CSS files to disk to be served by the proxies, caching recent stories and recommended diaries to disk for anonymous users, and using memcached as much as I could. Some things didn't work, like trying to find a way to pool DB connections among apache2 threads in worker mode or caching story data more aggressively.
First thing in the morning on Election Day, I pulled the archives offline. Don't worry, they'll be back soon. This gave an immediate performance boost, because the archive tables are so huge (the archived comments table is over 4GB in size), and I've found that if anything is going to cause a problem in the database, it's the archive. Sure enough, throughout the evening the database had no problems at all.
Eventually, as traffic grew, I had to swap out the index page template with a simpler one that had simpler versions of the recommended diaries and recent diaries boxes. That had an immediate benefit as well. The election results pages were also cached to disk to have the proxies serve up for everyone, thus reducing even more the load on the mod_perl servers. That, I think, proved to be the most important thing of all.
Other than that, I just had to keep an eye on all the servers and massage them occasionally as needed. There were a few brief slowdowns, but other than that the site remained zippy. Unfortunately, I can't rest on this, because the presidential season will be starting up before we know it.