From a Windows CE programmer --

This is just a short diary entry, since for some reason the site is signing me out when I try to comment on the "...from and election judge".  This is an observation from someone who has calibrated touchscreens thousands of times...and might throw a little light on the mechanics (or not).

Calibration errors are ususally small, on most CE hardware (which, based on my understanding, is the family line of many voting machines).  It depends a lot on implementation quality, but even the implementations of the CE codebase five years ago customarily only would go off a few centimeters over time.  More modern devices sometimes never go off calibration, or only a bit over a long period.  Of course, my CE based convergence phone probably had a bigger test budget than a diebold machine, not to mention more competent work on the drivers, so I'd expect some difference favoring the phone...but the current crop of screens and drivers has rendered the CE "calibrate screen" item almost meaningless on the seven current CE devices on my desk at the moment.  By and large, its not something you need to do.  Five years ago, you had to do it a lot.

The main point I want to make is that if I were hired to write a hack for voting machines, I'd (a) introduce a calibration error (hard, low level hardware stuff, but if you could recompile the video driver from source and fiddle, perhaps not so hard -- it could be as simple as frying a register or two at the proper moment), (b) put in some probability based code, way way further up in the system, to flip the vote -- one time in ten, or five, or two, and (c) put in code to turn off the vote flipper for a short time after each recalibration event.  When the video driver got a notification that the screen was recalibrated, it would put the vote flipper to sleep for a short, random interval ("hey look, its fixed!").  The two systems would have nothing to do with each other, except for catching each other's events, and could easily be written by different programmers.  At the end of the day (say, November 8th) the top level vote flipper would erase itself as well as it could on first boot -- and the video problem would stay in place, an easy to find "why".

I almost always prefer stupidity to malice, as an explaination, so I'm not arguing in the least that's what's happening -- but if I were rigging machines based on touchscreens, would not be able to access the machines after shipment, and had access to the video/touchscreen source, its defintely the route I would go.

How could you tell?  Its fairly common sense.  The calibration error is likely to be consistent per area of screen.  In other words, if touching physical area "a" triggers area "b", then it will probably keep doing that for awhile before degrading further.  A vote flipped to republican by one screen arrangement of names will flip democrat in another.  It is very unlikely that the screen calibration errors tie directly to the vote flipping software, so you should be able to create or find a scenario where the calibration favors democrats -- with the same probability it favors republicans.  If you can favor republicans over democrats, with screen geometry being equal before executing "calibrate screen" - then its probably hacked.  If not, its far less likely.

Anyway, hope this contributes something.
Edit note: it was requested I drop the "2600" portion of the lead in. I did so. I want to emphasize this is just a "how would I do it" rather than a full blown conspiracy theory, and a general attempt to encourage anyone with a crack at a malfunctioning system to get their best friend with test experience to try it, in a methodical fashion.

•  My view

  Like you, I look for the stupidity explanation first, and from what I've read it seems sufficient.

  But the error rates on these machines are much, much higher than we should tolerate. Bi-partisan errors are also unacceptable. Votes should count.

  by Elwood Dowd on Tue Oct 31, 2006 at 02:59:27 PM PDT

•  Thank you

  For some reason, there are a lot more developers chiming in this time, which makes me happy.

  A commenter on another thread claims to have seen a circuit board behind a touch-screen voting machine that had no ESD boundary protection. I figured that would knock calibration out of whack pretty quickly, but I have no personal experience developing touch-screen devices, so it was a guess. Was I close? (Even if I wasn't, it could very easily throw a wrench in the works by sending a big signal spike down some random path on the board...)

  by oldjohnbrown on Tue Oct 31, 2006 at 03:07:06 PM PDT

  •  not an issue

    unless you're dealing with voltages on the order of, let's say, a stun gun or taser.  Even then, it probably wouldn't knock the machine out.

    by arbiter on Tue Oct 31, 2006 at 03:12:53 PM PDT

  •  heh

    CE comes with a big pile of source and some reference hardware -- which can be an x86 machine --and the adopting company gets to port it all to their hardware, or buy something from another vendor that it will run on.  To do it the way I'd described, you'd need to set named events from the driver -- albeit no one would know what that event was for, and you could have all sorts of other reasons for it being there. Being a software sort, I can only speak to that, but from my end it would be way easier to predict the size and impact of a software monkey wrench over any sort of hardware hack.  

    by jessical on Tue Oct 31, 2006 at 03:14:19 PM PDT

    •  you're right on target: get this info out there..

      You've outlined a set of tests that can be done to determine if the screen calibration issues are errors or are hacks.  We need to get that information into the hands of every single election-protection group in America.

      Hey Markos and other moderators:  FRONT PAGE THIS DIARY, this is truly vital for election-protection groups and volunteers!  

      Consider election judges and poll workers armed with this information.  They could perform the tests any time a machine "malfunctioned" and report the results to a national organization for tracking purposes.  Ideally there would be local attorneys available to go into court and take whatever steps were needed if machines proved to be hacked.  And ideally we'd have the media on the issue ASAP.  

      PS, Jessical, I'd suggest changing your title because most people don't get the "2600" reference.  Something like:  "Windows CE programmer re. Vote Hacking"   (And hey I gotta question: is it going to be worthwhile upgrading from XP Home to Vista when it comes out?  I'm a cross-platform type, using MacOSX and XP Home on two different machines, if that makes any difference, and will be getting an Apple Intel machine next year....)

      by G2geek on Tue Oct 31, 2006 at 03:38:55 PM PDT

      •  NOOOOOOOOOO!

        XP home won't work on the Intel Mac and it's quite unlikely that Vista will.

        The only dual-boot combo that will work right now is OSX and XP.  Wait for the virtual server package to come out.  Then you can run any OS you like.

        by arbiter on Tue Oct 31, 2006 at 03:47:29 PM PDT

        •  thanks!

          Thanks for the heads-up.  

          Whenever I get a new machine or do an upgrade or reconfiguration, I run it in "guinea-pig mode" for a few days to see how it behaves, before starting to use it for critical tasks.  (Needless to say, all my data files are backed up on an external HDD while all this is going on.)

          Here's me looking forward to one laptop running both systems, that I can pack in my field kit to go to client sites, rather than having to bring two as I do at present.  I guess this will have to wait a while, but I'm patient.

          by G2geek on Tue Oct 31, 2006 at 03:56:40 PM PDT

          •  vmware

            for os x will be out soon:)  Or so they say...

            by jessical on Tue Oct 31, 2006 at 04:00:57 PM PDT

  •  I'd like to know more about the hardware too

    I've calibrated my share of touchscreens too, but they weren't CE based.   Is the hardware shock proof?  Will moving the hardware contribute to cal problems?  We ran into that with some HP equipment at GM, but it was older tech and windows based.

    by dougymi on Tue Oct 31, 2006 at 03:14:33 PM PDT

    •  on CE

      virtually anything can fry calibration.  Heat and dirt are biggies.  But, as I observed, the situation has improved remarkably in a short time.

      I would not expect calibration per se would ever throw an election, its just too damn hard to make something already cheesy behave in a predictably bad way.  That said, there is always serendipity :}

      My emphasis was not so much on calibration per se as the idea that more than one thing may be going on -- in fact, given a compentent CE type geek, I'd seriously expect that -- and the discussion I've seen has all flowed around calibration alone, or around a general judgement of conspiracy theory in general.  Anyway.

      by jessical on Tue Oct 31, 2006 at 03:34:01 PM PDT

•  Good info to have

  Thank you Jessica for this really good information.  I believe the more we understand about the technology of electronic voting machines, the better we are prepared to argue our case against them.

  I'm in Ohio, so we tend to get our hackles up sooner than later some days.  It's going to be a bumpy ride, folks.

  Happy Halloweeeeeeen!

  by Ninepatch on Tue Oct 31, 2006 at 03:07:41 PM PDT

•  I think what you contributed

  was more confusion. This sounds way too complicated. If someone wanted to rig the vote using the DRE machines, the thing to do would be to alter the unseen software so no one would be the wiser that anything untoward was going on. The so-called "vote-flipping" that everyone is going nuts about is calibration error.

  In the more urban counties in Florida, touch screen voting machines were acquired during 2001 (so they could be used in the 2002 election cycle), so what we have is five-year-old technology. It's not like the private sector where equipment is upgraded to maintain efficient operations.

  by Alice in Florida on Tue Oct 31, 2006 at 03:10:15 PM PDT

  •  the software is not "unseen"

    far from it - that's why it is vulnerable.

    He has cleared up a great deal of confusion as to:

    1.  How errors can occur in the absence of "hacking".

    2.  How malicious "hacking" could occur.

    You may not understand how the software works, but it is not hidden or unseeable.  

    by arbiter on Tue Oct 31, 2006 at 03:15:56 PM PDT

  •  I disagree, poliltely

    But you do raise a really important point.

    To my way of thinking, people make the biggest mistakes about complex technical systems when they look for single explainations and "common sense".  You get occam's razor after you understand it, not before.  I spend much of my professional life working with people who want "bottom line" clarity from complex systems, and essentially when you take that approach you bet the whole of your intuition against the whole of the problem.  This is great for politics or people but fails miserably in complex and counterintuitive systems.

    The sort of "solution" I described is how many CE programmers approach stuff -- you have the richness of a computer and the rigidity of an embedded system, so many applications contain hacks that go from the driver to application level in one swoop.

    What I wish I could contribute -- but can't being just some posting geek -- is for someone with actual development experience and/or test experience to take apart the problem.  This is not addressed by anyone in a armchair, myself included, nor by anyone whose take is "common sense".  The only way to address it is with rigorous testing during the voting period.  Now.  With care and lack of preconceptions.

    Flame away...

    by jessical on Tue Oct 31, 2006 at 03:21:01 PM PDT

    •  btw

      my reply was to alice not arbiter!  I woulda stay'ed quiet if I'd seen his post while busy typing :(

      by jessical on Tue Oct 31, 2006 at 03:22:09 PM PDT

      •  ha!

        Love it.  Geeks of the world unite :)

        You understand the essential issue - wanting an easy or intuitive answer for a complex system isn't possible.  A pocket calculator is complex.  An e-voting machine is many orders of magnitude more so.  

        But they can be made to work properly, every time.  It merely requires rigorous testing (that's what I do) which, in all honesty, I don't know if that's been done or not.  There are too many people screaming about agendas to be able to hear the voices of those who have actually done real, unbiased testing (and Bev Harris/BlackBoxVoting, much as I like her, is not a source for competent or unbiased test results).

        If the Repubs start screaming about the machines after they lose the House, then I think we can trust the machines.  If everyone hates them equally, they probably work fine :)

        by arbiter on Tue Oct 31, 2006 at 03:43:53 PM PDT

•  If Republicans were screaming

  about vote flipping I'd be more inclined to give theories like this a favorable hearing.  You don't actually have an explanation that ends up favoring (R) candidates and referendum responses.

  It's also pretty quite on the (R) side of the camp, so apparently no (R) votes have been harmed in the proces.  So Republicans seem ok with the electronic election systems performance, except maybe for the Savannah/Chavez thing.

  by antirove on Tue Oct 31, 2006 at 03:17:35 PM PDT

  •  actually

    you didn't read very carefully.  The vote flipper would be independent and favor Rs, but could easily do so on a percentage basis.   My last sentence in the post sucked and I apologize.  My main point is that we should be testing the null hypothesis here -- that there is a calibration error without ANY other interference -- and in doing so, we can usefully assume that the calibration and vote flipping systems are largely seperate -- they may exchange events, but are unlikely to exhange data.

    I could care less about my specific theory, except as I think there should be someone who knows how to track a bug looking at the thing, at the time of failure.  "how to track a bug" is not the same as "having used buggy software".

    by jessical on Tue Oct 31, 2006 at 03:29:01 PM PDT

    •  So, how do we get good bug trackers inside

      the voting booth on a non-partisan basis?

      Leaving speculation about possible event driven and timer-based interrupt code issues aside, the mere fact that Republicans aren't screaming about their votes shifting to the (D) column needs while Democrats could help justify getting 'bug trackers' into the voting arena...but how do we get them onto specific problems when the problem arises, and still maintain voter privacy and credible non-partisan standing?  We'd probably need many van loads of technically competent people able to get to problems within minutes, to prevent inordinate delays.

      We need to anticipate a possible critical counter-attack.  By voters self-selecting to disclose and express a desire work with a bug tracker, and their predominance of (D) affiliations, we could be inviting concerns our bug-tracker may also be a hacker and that this is an opportunity to hack the vote on that same percentage basis we'd claim is happening against (D) candidates and options.  This needs to be carefully planned and coordinated, except we have just days to figure it out.

      by antirove on Tue Oct 31, 2006 at 04:09:38 PM PDT

      •  Well..

        I figure if fraud is going on at that level, its going to happen this round regardless.  The big solution is probably something a lot simpler, or a lot more complex...

        My hope was that since election folks come from all walks of life, and geeks are geeks, people will be aware and thoughtful if a machine goes wonky, and will try to replicate the error before trying to fix it....but...I'm in an armchair in seattle so what do I know about the possible?

        by jessical on Tue Oct 31, 2006 at 04:15:56 PM PDT

•  Dear Jessica. Good work!

  We need more of this technical expertise, especially when it points to places to look.

  My expertise in high-strung high speed complex machinery, from particle accelerators to IBM Selectrics to racing motorcycles (gas thermodynamics at the speed of sound is fun indeed!) gives me a very strong hunch that your idea about tinkering at the edges of the position recognition algorithm on the touchscreen would work, is simple, and would be hard to prove unless many examples could be found in a large group of subpoenaed machines.

  So perhaps someone should look at getting a bunch of machines suspected of tinkery subpoenaed and examined by technically sophisticated people.

  Like you. Lawyers, anyone?

  by ormondotvos on Tue Oct 31, 2006 at 03:38:18 PM PDT

  •  well...

    Actually, it would most likely be an election volunteer with the engineering experience to track it down on even one or two machines with the behavior -- the lovely thing about computers is that an n of 1 is fully representative of the population :}  Three which had shown flipping behavior at any point would be sufficient for changes introduced in test :}  But by the time you booted on the 8th, if they were any good at all, the story would be over...

    If the guy who saw the florida flip had kept notes and been careful, he could have collected evidence to convince a great number of people, I think...but how would he even know where to begin?

    I suspect we'll end up with a much more transparent and simpler electronic vote system, in time, or else voting tied to personal key. I suspect in the end we will have a choice between true anonymity and fraud.

    I always wished I understood hardware like motorcycles :)

    by jessical on Tue Oct 31, 2006 at 03:48:42 PM PDT

•  FRONTPAGE THIS!

  and  yes, change the 2600 reference to something less /.

  by ormondotvos on Tue Oct 31, 2006 at 03:43:17 PM PDT

•  fyi very old

  ce version for diebold is 3.0? is this old enuf to do lousy calibration.

  i am not a ce prog, but a computer geek for 15+ yrs...

  i have followed the tech parts of the diebold controversy.

  by biscobosco on Tue Oct 31, 2006 at 03:47:50 PM PDT

  •  ah hah

    Yeah, 3.0 would drift.  Ugh.  And badly, I'd expect, on a third party board.  Thanks.  

    If this post moves, would you please comment more??? I doubt it will move up, but if it does, PLEASE do.  I have NOT followed every bump and grind of the diebold controversy, largely because the whole thing has been so theoretical -- I mean, yeah, ok, you can recompile the whole OS and install it from USB, so what?  I was mainly inspired by too many Analog editorials (post campbell lol) about otherwise brilliant people not understanding technology, and the news item from florida about the vote flipping being attributed (without reference to the original observation) to calibration error.  I just didn't get it, sans some additional data.  For all I know, that data exists and this is a big fat red herring.  I just wish that someone with debugging skills would watch it happen and tell us what is really going on.

    by jessical on Tue Oct 31, 2006 at 03:56:04 PM PDT

    •  here's why that will never happen

      don't mean to be wet blanket, but...

      contracturally counties are not allowed typically to submit their diebold equip to 3rd parties for testing.

      in utah where a county clerk did this - he was fired (though elected) and diebold charged the county a bunch of monry (like 40k) to have machines touched by a tech.

      that's why the princeton story was big news. they actually got their hands on a machine to test.

      diebold hates independent 3rd party testing.

      by biscobosco on Tue Oct 31, 2006 at 04:10:14 PM PDT

      •  I sure didn't mean

        for this to become some sort of concrete theory and "test every machine"!  I think truth has power though.  If someone like you followed an error, in detail and with clarity, lots of people like me would get it, right away.  That would not save the election but it would convince a lot of competent people very quickly.

        Really effective fraud would sit between the system and the count, not flip actual votes...

        by jessical on Tue Oct 31, 2006 at 04:36:44 PM PDT

•  moderators: FRONT PAGE THIS

  Markos or anyone: front-page this diary or get it on the Rec list for 24 to 48 hours.  I've been in the geek biz for @ 25 years and this is right on target.  

  This is the best information I've seen about dealing with election hacking issues.  Jessical is proposing tests that could be performed to determine if a very likely type of hack is actually occurring or not.  

  Every election-protection group in the country needs to know about this ASAP.  It will not be complicated to train volunteers to run the tests she is suggesting, any smart 16-year-old should be able to to do it.  

  This could blow the lid off the election hacking issue in a huge way.  Let's get moving on it, not a moment to waste!

  by G2geek on Tue Oct 31, 2006 at 03:47:54 PM PDT

  •  response

    Not sure its worth that lol...I'll drop the 2600 reference and put in an editing note though...

    by jessical on Tue Oct 31, 2006 at 03:50:11 PM PDT

    •  how recently has ce improved

      the calibration issues?  see my above comment. diebold  is using an elderly version of win ce....

      by biscobosco on Tue Oct 31, 2006 at 03:58:27 PM PDT

      •  ...

        already replied.  Above all, I did not want this post to be about one hack, but about methodical testing of problems.  I don't write so well...

        by jessical on Tue Oct 31, 2006 at 04:00:20 PM PDT

  •  nah, not worth it...really...

    Still going to fix it, but feel constrained to observe this is just one explaination -- by way of illustration -- and my own little "huh -- now how would I do that?".  Like many geeks, my view would probably change if I had the diebold reference board in front of me.  It would change a lot if I had their bug database in front of me :))

    by jessical on Tue Oct 31, 2006 at 03:59:23 PM PDT