Skip to main content

There's actually surprisingly little to say this year. As Markos explained, I had been preparing for Election Day for the past year, so most of the work for the election was done long before. I've discussed the changes previously in The New Technology of Daily Kos, A Way to Scale Your Blog, and The Technology of Daily Kos: Introducing mod_mcpage. Today, we'll go over how the site performed on Election Day itself, and touch on the future briefly.

To be on the safe side, I provisioned three additional servers to add to the pool of webheads. As expected, they came up and configured themselves properly with the exception of improper permissions on /tmp. Beyond that, there were no issues on the webserver side of things. The system load during the busiest part of the night approached 2, but rarely if ever exceeded it. Most of the time, each webserver's load was in between 0.5 and 1.8 or so. There were some slowdowns caused by the ads and a small error in the embedded maps, but those were cleared up quickly. A reader also wrote in with some tips for optimizing the page rendering times, which I have been implementing over time.

There's also not much to say about the database servers. They also ran cool as cucumbers throughout the night, and while I checked up on them periodically, I didn't have to do anything with them.

The big problem of the evening was with the image server and the map. There was a long period of time when the results were simply not coming in because the cron job that was pulling them down from the AP was running really slowly, and it was impossible to even scp or sftp anything to the server without restarting lighttpd. This was very curious, and very bad for performance, so I spent a while investigating different kernel settings, adjusting lighttpd, poking at the CDN, before finally stumbling on the causes of the problem. The first was that the Electoral Scoreboard map still had some references to the origin server rather than the CDN from testing. The second was that it turned out that when we had our new image server set up, it got put on a 100Mbit switch on its public interface. This had never been a problem before, but once traffic on the map started going, it was saturating the network. (In other news, the tubes weren't wide enough.) After some frenzied phone calls with Voxel, we moved the image server to a new switch, made some DNS changes, and adjusted some lighttpd and CDN settings . After that, the problems with the map largely cleared up. There were still some slowness issues, caused by the fact that the map was taking steps to insure it received non-cached versions of the data files, which meant that they had to be refetched constantly from the CDN. Another unforseen issue was the fact that there were nearly 10,000 little files for the live results, which doesn't seem to have helped things at all. While it would have been better to have forseen these issues before that evening, they  were largely fixed before things got super crazy. There is room for improvement next time, however. Interestingly, at the height of traffic, the Voxel guys told me the CDN was pushing ~850 Mbit/sec, and the image server was pushing 200MBit/sec. Overall, I would rate my handling of the image server, CDN, and map traffic as satisfactory but not excellent.

Finally, the future: Now that things will be calming down a bit, I'll have a chance to do some of the things I've been wanting to do for a while, but had to hold off on until after the election. Most of these are backend improvements that I have in mind, but some of them will be things visible to the average user. I won't go into detail about what they'll be or when they'll arrive, because no man shall know the day or the hour of their arrival. One thing I will mention is that I have some improvements to mod_mcpage that I want to make to move page caching entirely into the module, so it doesn't require the cooperation of the backend. This will make it more useful for others to use if they'd like.

By and large, Election Night 2008 was a success. Aside from the problems early on with the map and some wonkiness with the ads on the front page, there's not really much to report. I will, however, continue my efforts to keep the site running better than before.

As a bonus, while I write about the tech side of things on Daily Kos occasionally, it's not generally the place for it. However, I come across issues while working on Daily Kos that even Google is not clear on fairly often, and some people have expressed an interest in being able to read about what I've had to do to fix them. Thus, once I manage to make a logo and a halfway acceptable color scheme (because I honestly have no design skills, sadly), I will be quietly starting a blog to post more about tech stuff in greater detail. I don't expect to make a big announcement here about it, but perhaps someday you'll have to Google why on earth you can't get MySQL to use large pages correctly despite following the instructions on the MySQL site, and you'll find your way to a post that will help you.

Originally posted to ct on Sat Nov 08, 2008 at 12:59 PM PST.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

pontificator, ClaudeB, Jake McIntyre, Sylv, vicki, Terri, tmo, northsylvania, CJB, hazey, Bill in Portland Maine, Ray Radlein, Odysseus, taylormattd, jotter, nu, hester, murphy, Rayne, deben, kate mckinnon, Trendar, gogol, Adam B, rhfactor, asimbagirl, thebes, Pandora, TrueBlueMajority, mem from somerville, saraswati, Dems2004, Susan Gardner, RunawayRose, greenbird, dengre, tommurphy, Stoy, rincewind, shayera, eeff, Mnemosyne, TheGryphon, elfling, frisco, lawnorder, sobermom, faedrake, Poika, Caneel, sardonyx, tyler93023, rwsab, bronte17, Naturegal, gladkov, mbayrob, Susan1138, sarahnity, nyceve, jerseyjoew, peace voter, highacidity, Xapulin, stevej, mxwing, ksh01, mkfarkus, roses, chechecule, allysonsta, peraspera, amanuensis, nargel, juslikagrzly, oceanview, Terre, dmsilev, Alna Dem, wader, nio, Tomtech, DemocracyLover in NYC, Redfire, InquisitiveRaven, SneakySnu, psnyder, TexDem, dmsmith, hoolia, duncanidaho, desmoinesdem, GN1927, On The Bus, Catte Nappe, Frankenoid, Liberaljentaps, BlogDog, DH from MD, 313to212, KateCrashes, side pocket, gmhowell, OrangeClouds115, eco, CanYouBeAngryAndStillDream, SanDiegoDem, Marianne Benz, Irish Patti, Timroff, Limelite, BDA in VA, sawgrass727, Gowrie Gal, Green Mountain Flatlander, Big Tex, nailbender, John3, marina, radarlady, Strat, jrooth, yuriwho, SherwoodB, DocGonzo, leeroy, kaye, Brooke In Seattle, dansk47, cfk, Buffalo Girl, ladybug53, sofia, blue jersey mom, Joy Busey, power model, Big Nit Attack, neroden, wiscmass, Shaking the Tree, LithiumCola, Pluto, Lindy, Shiborg, Spathiphyllum, Ekaterin, proudprogressiveCA, JanF, Land of Enchantment, RainyDay, Asinus Asinum Fricat, begone, Mother Mags, trashablanca, Sanuk, Kingsmeg, edwardssl, tobendaro, dharmafarmer, Ellicatt, borkitekt, InsultComicDog, buhdydharma, KenBee, MTmofo, TheBlaz, NBBooks, triv33, imabluemerkin, paul2port, justalittlebitcrazy, Doctor Frog, myrealname, ER Doc, soccergrandmom, dirtfarmer, quantumspin, justiceputnam, DemocraticLuntz, Doggie269, Clive all hat no horse Rodeo, va dare, Dreaming of Better Days, Granny Doc, coolsub, Hedwig, slksfca, markthshark, OHdog, Aaa T Tudeattack, AntKat, AmericanRiverCanyon, One Pissed Off Liberal, asilomar, Cronesense, Loudoun County Dem, tgypsy, Must Have Been The Roses, gloriana, vets74, drbloodaxe, jds1978, la urracca, profmom, flumptytail, Ninepatch, drchelo, Unbozo, Strabo, bnasley, dolphin777, sabishi, RokkiBlueboa, jnhobbs, plembo, millwood, gchaucer2, uciguy30, JML9999, willb48, Empower Ink, mcyrhul, Theghostofkarlafayetucker, Red no more, sand805, alkalinesky, mconvente, condorcet, wayoutinthestix, pepper mint, CDH in Brooklyn, lineatus, Akonitum, Its any one guess, Lujane, bluesheep, asius, caps lock on, Futuristic Dreamer, dmhlt 66, Scubaval, rhutcheson, watercarrier4diogenes, shortgirl, jlms qkw, maggiejean, 1BQ, bhagamu, Neon Vincent, WSComn, Discipline28, CanyonWren, earicicle, mrchumchum, EquationDoc, DefendOurConstitution, goldnata, redtex, Texas Revolutionary, sanglug, Keith Slater, CityLightsLover, audiored, EmmaKY, sophistry makes me tired, Colorado Billy, Larsstephens, cowvindaloo, Lazar, BigVegan, Its the Supreme Court Stupid, BrighidG, KroneckerD, budr, japangypsy, boriquasi, msmo1918, Eddie L, Peyton on the Plateau, President Obama, sullivanst, Lady Libertine, rfall, ThirtyFiveUp, Urtica dioica gracilis, MsGrin, bottles, HylasBrook, Kai99, dietcokehead98, kirbybruno, geffdee, FightingRegistrar, Ruzzian, Word Alchemy, lucia55116

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site