After all, he lost a major election to one.
No, not THAT guy. He was a saint!
I'm talking about when Carter ran for Governor of Georgia in 1966, as the State Senator from Plains, and came in third in the Primary, right behind this assh*le:

The guy pointing a f*cking gun at a black man (for the crime of trying to eat at his restaurant) went on to beat Carter in the 1966 gubernatorial race in Georgia and win a runoff to become the state's chief executive. That's the man who built an entire political career on being a bigger and more violent racist bastard than anyone else in Georgia. And few people saw him in action closer than Jimmy Carter did.
Lester Maddox is destined to be an enduring stain on our party and my state of origin. Carter lost to him in '66 and was saddled with him as a lieutenant governor from '71-'75 (the Governor and Lt. Governor don't run on the same ballot line in Georgia). When I used to commute from my office to my home in Marietta, I had to cross the Lester and Virginia Maddox Memorial Bridge and take the Larry McDonald Memorial Highway (not so named until 1998, well after it was pretty clear that the odious McCarthyite racist was nobody to celebrate). It's very difficult to honor Georgia's past leaders without giving credit to a fair number of racists--it's a sad but inevitable part of southern political history. (Even Jimmy's own campaign history has its low points in playing to the sizable racist segment of the state's 1970 electorate, though nothing like Maddox's.)
Many people have traditionally used the less vile aspects of Maddox's career to try and excuse his behavior. "Sure, he may have hated the idea of letting black people eat his food so much that he sold his restaurant rather than comply with the law," they say, "but he hired plenty of blacks to his administration when he was governor!" Yeah, he did, and it's to his credit that he stopped pointing guns at people when he started campaigning for Governor. It doesn't make up for being a racist asshole any more than being able to ride a bicycle backwards made him a Tour de France contender.
I've been fed the myth of Lester Maddox my whole life: he wasn't really racist, he just believed in freedom to not associate; he wasn't truly segregationist in his beliefs on government, but only for private business, et cetera. It was a more modern-day version of the "Lost Cause" mythology we were spoon-fed about the Confederacy in our Georgia public schools (you know, that the Civil War wasn't really about slavery, but about states' rights to presumably do some heretofore unknown thing other than allow slavery). And if we put aside things like the fact that Maddox handed ax handles to customers to chase off the black people he was pointing his gun at, the fact that he proudly auctioned off autographed "Pickrick Drumsticks" years later, the fact that he was a prominent charter member of the KKK's political wing, or all of his various racist declarations and positions, then maybe there's something to that revisionist myth.
But let's be clear: Jimmy Carter has seen racism in politics for half a century. He's watched it shift from overt to covert, from proud stands to winks and nods, from front and center in campaign platforms to symbolic gestures in campaign appearances. He watched it transform Lester Maddox from inciting a mob of customers to grab ax handles and follow him as he pointed his revolver at blacks, to advocating a mildly stated support for "voluntary segregation."
And when a man who lost to Lester Maddox says he's watching racism in action, we ought to at least give him the benefit of the doubt.
But is there really any doubt that there's been a strong, racist opposition to Obama, going back to the primary season? Hell, I personally watched this crazy racist nutjob (and her two as-yet anonymous idiot cohorts) rant and heckle for over an hour before being ejected from the DNC Rules & Bylaws Committee meeting in 2008 and telling any camera she could find that, by following the rules of the election, the Democratic Party had just ruined its chances by supporting "an inadequate black male." Since then, Obama's been called a "magic negro." He's been called an Arab. And he's been called...well, pretty much everything under the sun. And nary a peep about this vile racism from the "leaders" of the right (save for the occasional denial that it exists at all).
But the second that a former President, a Nobel laureate, someone who's seen political racism for decades up-close dares to point out the obvious--that the racist opposition to Barack Obama has been a major factor in his national political career all along and is still there for all to see--it's "playing the race card" and "shouting down the opposition."
What it really amounts to is this: Jimmy Carter still has the guts to call it like it is. And nothing infuriates the GOP establishment more than when that happens.