There's actually surprisingly little to say this year. As Markos explained, I had been preparing for Election Day for the past year, so most of the work for the election was done long before. I've discussed the changes previously in The New Technology of Daily Kos, A Way to Scale Your Blog, and The Technology of Daily Kos: Introducing mod_mcpage. Today, we'll go over how the site performed on Election Day itself, and touch on the future briefly.
To be on the safe side, I provisioned three additional servers to add to the pool of webheads. As expected, they came up and configured themselves properly with the exception of improper permissions on /tmp. Beyond that, there were no issues on the webserver side of things. The system load during the busiest part of the night approached 2, but rarely if ever exceeded it. Most of the time, each webserver's load was in between 0.5 and 1.8 or so. There were some slowdowns caused by the ads and a small error in the embedded maps, but those were cleared up quickly. A reader also wrote in with some tips for optimizing the page rendering times, which I have been implementing over time.
There's also not much to say about the database servers. They also ran cool as cucumbers throughout the night, and while I checked up on them periodically, I didn't have to do anything with them.
The big problem of the evening was with the image server and the map. There was a long period of time when the results were simply not coming in because the cron job that was pulling them down from the AP was running really slowly, and it was impossible to even scp or sftp anything to the server without restarting lighttpd. This was very curious, and very bad for performance, so I spent a while investigating different kernel settings, adjusting lighttpd, poking at the CDN, before finally stumbling on the causes of the problem. The first was that the Electoral Scoreboard map still had some references to the origin server rather than the CDN from testing. The second was that it turned out that when we had our new image server set up, it got put on a 100Mbit switch on its public interface. This had never been a problem before, but once traffic on the map started going, it was saturating the network. (In other news, the tubes weren't wide enough.) After some frenzied phone calls with Voxel, we moved the image server to a new switch, made some DNS changes, and adjusted some lighttpd and CDN settings . After that, the problems with the map largely cleared up. There were still some slowness issues, caused by the fact that the map was taking steps to insure it received non-cached versions of the data files, which meant that they had to be refetched constantly from the CDN. Another unforseen issue was the fact that there were nearly 10,000 little files for the live results, which doesn't seem to have helped things at all. While it would have been better to have forseen these issues before that evening, they were largely fixed before things got super crazy. There is room for improvement next time, however. Interestingly, at the height of traffic, the Voxel guys told me the CDN was pushing ~850 Mbit/sec, and the image server was pushing 200MBit/sec. Overall, I would rate my handling of the image server, CDN, and map traffic as satisfactory but not excellent.
Finally, the future: Now that things will be calming down a bit, I'll have a chance to do some of the things I've been wanting to do for a while, but had to hold off on until after the election. Most of these are backend improvements that I have in mind, but some of them will be things visible to the average user. I won't go into detail about what they'll be or when they'll arrive, because no man shall know the day or the hour of their arrival. One thing I will mention is that I have some improvements to mod_mcpage that I want to make to move page caching entirely into the module, so it doesn't require the cooperation of the backend. This will make it more useful for others to use if they'd like.
By and large, Election Night 2008 was a success. Aside from the problems early on with the map and some wonkiness with the ads on the front page, there's not really much to report. I will, however, continue my efforts to keep the site running better than before.
As a bonus, while I write about the tech side of things on Daily Kos occasionally, it's not generally the place for it. However, I come across issues while working on Daily Kos that even Google is not clear on fairly often, and some people have expressed an interest in being able to read about what I've had to do to fix them. Thus, once I manage to make a logo and a halfway acceptable color scheme (because I honestly have no design skills, sadly), I will be quietly starting a blog to post more about tech stuff in greater detail. I don't expect to make a big announcement here about it, but perhaps someday you'll have to Google why on earth you can't get MySQL to use large pages correctly despite following the instructions on the MySQL site, and you'll find your way to a post that will help you.