Skip to main content

Russell Tice, former NSA analyst, is talking to Keith Olberman as I type this, providing some stunning revelations about the extent and nature of the Bush administration's domestic spying program.  I'm going to need a transcript to get into all the details, but wanted to write this in order to try to clarify at least one or two technical details he brought up.

Those points aside, though, there are two big takeaways from this:

  1. Domestic spying was pervasive -- far, far larger than anything we knew.
  1. Domestic spying targeted specific known non-terrorist groups: e.g., journalists.

Let's try the technical stuff first.

Mr. Tice talked at some length about the difference between large-scale technical surveillance and more focused directed surveillance.  If I've understood him correctly, then I think I can explain what he was talking about by using email as an example.

If you were interested in screening huge amounts of email, but didn't have the capacity to capture or store it all, you might decide to just content yourself with the metadata.  Metadata is just "data about data".  For instance, in the case of email, some interesting metadata might be: (a) what language it's in (b) the sender's address (c) the recipient's address (d) the length in bytes (e) the length in lines (f) what kinds of attachments, if any (g) what mail program was used to compose it (h) what the Subject line was, and so on.

This sort of metadata is relatively easy to extract and takes up a lot less room than the actual data: the metadata for an email message with 2M of photos attached might fit in 1K.  (And this is the point where it should dawn on you that similar metadata exists for faxes, phone calls, and every other electronic form of communication.)

Metadata can be useful.  Suppose you know that The Bad Guy always uses Eudora 1.1 to compose mail messages and always attaches photos that are 772x448 pixels in JPG format.  If you've extracted the right metadata from billions of messages, you might be able to figure out that the 99.999% of them aren't what you're looking for by using that as a filter.  If you're lucky, the only messages left will be the ones you want -- or the number will be small enough that brute force or maybe a simple search will get you what you want.

But metadata can be abused.  It's possible to use the same collection to reconstruct
the salient details of every message sent by A.  Or from A to B.  Or which has a "Subject:" line containing the string "protest".  And so on.  It enables ad hoc fishing expeditions that are limited only by the scope of the collection, the kind of metadata extracted -- and the restraint of those conducting them, which I think we can safely characterize as "nonexistent".

If I understood Mr. Tice correctly, metadata collection was untargeted and pervasive.
They went for everything they could get.  Which means if you sent a message to Aunt Mary with a photo of the dog on July 17, 2004, they acquired -- or at least tried to acquire -- the metadata for it.

Mr. Tice's further point was that high-level technical analysis like this was used to select specific targets for detailed analysis -- and in that detailed analysis, EVERYTHING was collected.  Not just metadata: everything.  Every phone call, every fax, every email, every instant message, everything.  All captured and stored in a database...somewhere.


This so much worse than what we knew at this time last year that I hardly know where to begin.  Let me just recapitulate part of what Mr. Tice said: he stated that he'd been asked to identify particular groups so that they could be excluded from surveillance...but eventually he realized that this was an internal NSA cover story, and that those were precisely the groups being targeted.  It took him a while to get around to naming one of those groups, but when he did...journalists.  Reporters.  The news media.

So not only did our own government spy on ordinary citizens -- which is bad enough -- it spied on the people most likely to be contacted by whistleblowers passing along the news that the government was spying on ordinary citizens.

All of which leads us to a number of very disturbing questions:

  1. Who authorized this?
  1. Who knew about this?
  1. How was this done?  Who, among telcos and ISPs, collaborated?  Were they served  National Security Letters to silence them?
  1. Is it still going on?
  1. What are all the people and groups who were targeted?
  1. Where's the data?
  1. Who has had/now has access to the data?
  1. What purposes has this data been used for?

I'm sure there are more -- we have, I'm afraid, only begun to hear the smallest portion of this and there is likely much more to come.

Update 1:It's been pointed out to me that I didn't mention one of the most powerful methods of metadata analysis -- one that quite likely was used on this data.  (Is being used?)  It's possible to use the metadata to ask questions like "Who does A talk to?" and "Who do they talk to?" and "Who do they talk to?" and "Does this look like a close-knit group?" and "How often do they talk to each other?" and "What are the characteristics of those communications?"  There are a bunch of terms for this; "network analysis" is the one I know.  It's a way of discerning a connected group in an ocean of data.  Like many tools, it can be used for good, evil and neutral purposes.  But what makes it so powerful is that very few people are aware of its existence, and thus they unknowingly collaborate in creating this metadata and making it useful.

Update 2:Here's a link to the MSNBC site with the interview, and maybe, if I've not botched this (I did -- so pulling the attempt.  Just open the link in another browser tab).

Update 3:KateCrashes points out in the comments that there is a Wikipedia entry on Russell Tice; thanks!   A quick scan of that article reveals a number of links to previous news stories about him as well as considerable backstory about his history as a whistleblower.

Update 4:Thank you for the recommendation; very kind of you.  One more clarification based on a re-listen to the interview.  Mr. Tice was very careful to say, with emphasis, "all computer communications".  That's a massive amount of data/metadata: it could (and probably did) include not just email and IM and blog postings and so on, but every click (HTTP request) sent out, every initiated login session (with HTTPS to a web site, let's say, or SSH to a server), every file retrieved with FTP or HTTP, every file served or retrieved with P2P protocols...and I'm presuming, timestamps on all that to facilitate later sequencing of events.  I've used this kind of tap in the course of my work (network security) and the amount of data it provides on just a single computer (and its users) is amazing; applied on a very large scale, it would put the Panopticon to shame.  If he does come back tomorrow to Countdown, as KO asked, it will be interesting to hear how he augments and clarifies his statement.

Update 5:The morning after, I've had a chance to slowly and carefully read the transcript.  Something else that I think may be important jumped out at me.  Now it's entirely possible I'm way off-base here, but let me quote the relevant text and try to explain.

TICE: Well, as I was going for support for this particular organization, it sort of was dropped to me that, you know, this is 24/7. Because I was saying, you know, I need collection at this time, at this point for, you know, for a window of time. And I would say, will we have the capability at this particular point? And positioning assets, and I was ultimately told we don't have to worry about that, because we've got it covered all the time. And that's when it clicked in my head, this is not something that's being done on a onesy basis, onesy-twosie. This is something that's happening all the time.

If your goal is to tap all email (again, as above, just using that as an example) then when you put the tap(s) depends on the extent of what you're doing.  If, for example, you just wanted everything in and out of Yahoo, then you serve Yahoo with a NSL, and place the taps directly on their mail server infrastructure.  But if the next day, you want everything in and out of AOL, you have to repeat the process there.  This is, I think, what he means when he uses the phrase "positioning assets"; I've seen it used in the past to describe the process of figuring out where taps need to be to monitor a target.

But "positioning assets" is tedious and inefficient if your goal is to tap everything.  It requires the collaboration of an ever-increasing number of people, and as that number grows, so does the probability that one or more will deliberately or accidentally leak what's going on.  So rather than doing this the right way -- the way it's intended, with court approval for limited, targeted taps whose purpose is clear and whose extent is curtailed to avoid collecting any more data than necessary -- it sounds to me, from Mr. Tice's description, that precisely the opposite approach was chosen.

That is: "assets" were placed so that they were minimal in number but maximal in scope.  This probably means that they were placed at network exchange points -- which to a first approximation, you can think of as the backbone of the Internet.  That's why assets didn't need to be repositioned if the target changed: they were already positioned someplace where they could see everything.

I can't imagine a FISA court approving this.  Perhaps that's a failure on my part, and I'm simply not thinking darkly enough, but this seems so completely over-the-top that I can't imagine it.

Update 6:In the comments, Jesselyn Raddack has provided this link to a legal defense of Mr. Tice that she wrote.  I highly recommend reading it.

Originally posted to RiderOnTheStorm on Wed Jan 21, 2009 at 06:14 PM PST.

Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags


More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

skybluewater, Michael D, CA Pol Junkie, Bob Johnson, Kimberley, Louise, RakDaddy, Alumbrados, davej, zzyzx, buffalo soldier, Joe Bob, canso, BrooklynBoy, exsimo2, DeminNewJ, KeithH, nofundy, Cowalker, Mogolori, tankej, nolalily, mbc, grollen, ogre, Rayne, teacherken, slinkerwink, Timaeus, Trendar, Subterranean, glitterscale, lipris, Liberal Thinking, native, tikkun, abarefootboy, limulus, alisonk, gaspare, Unstable Isotope, maynard, saraswati, ScientistMom in NY, theboz, PeterHug, TocqueDeville, greenbird, tnichlsn, JTML, Sprinkles, Emerson, karlpk, tommurphy, Shockwave, Sherri in TX, mindgeek, billlaurelMD, Andrew C White, Lipstick Liberal, Jim W, brendan1963, LEP, Stein, Mnemosyne, RickD, Veritas78, movie buff, TX Unmuzzled, Pompatus, polecat, figdish, Bexley Lane, DFWmom, Voodoo, marjo, lilorphant, SallyCat, Matilda, mataliandy, madwayne, New Sweden, Creosote, dweb8231, silence, Vitarai, strengthANDwisdom, Heart of the Rockies, RubDMC, Scott in NAZ, eyeswideopen, Gustogirl, jackspace, srbaxley, Justina, bronte17, Slacker Gal, conchita, EricS, DickCheneyBeforeHeDicksYou, cyberKosFan, landrew, leveymg, annrose, dionys1, understandinglife, whenwego, maxschell, Morague, macleme, CoolOnion, KMc, Pithy Cherub, Cool Blue Reason, shock, Minerva, mijita, fishwars, Scoopster, Patricia Taylor, MisterOpus1, barath, roses, chechecule, ivote2004, skertso, L0kI, nargel, dqueue, larryrant, karenc, Miss Blue, MJB, jjhalpin, oceanview, Nate Roberts, thingamabob, librarianman, rioduran, enough already, bewert, David Boyle, Cedwyn, Alna Dem, Brit, Eddie C, slackjawedlackey, jdmorg, Braindead, hopesprings, minorityusa, hopjoy, Dittoz, emmasnacker, danthrax, sockpuppet, TexDem, oldjohnbrown, CitizenOfEarth, milton333, november3rd, MA Liberal, brainwave, grannyhelen, kevin22262, Nina, praedor, bloomster, cosette, PaulVA, fuzed, Oy the Billybumbler, Penny Century, noveocanes, Chirons apprentice, Bluehawk, defluxion10, nika7k, joliberal, snakelass, dnn, Calidrissp, papercut, eodell, salinas, lcrp, Green Tea, alizard, Pohjola, Democratic Hawk, inclusiveheart, walkshills, frostieb, Donna in Rome, AllisonInSeattle, KateCrashes, peterj911, ybruti, WV Democrat, side pocket, WisVoter, Lefty Mama, tomjones, JayDean, CanYouBeAngryAndStillDream, Wife of Bath, SanDiegoDem, Deward Hastings, jcrit, TexasLefty, Black Max, Dawgbro, vacantlook, Daddy Bartholomew, malcontent, thereisnospoon, maggie sarah, TekBoss, Armand451, Limelite, iliketodrum, BDA in VA, environmentalist, wolverinethad, cohe, Gowrie Gal, weelzup, luvmovies2000, rapala, tardcat, greenskeeper, nehark, Dirk McQuigley, tergenev, mediaprisoner, Bluesee, Simian, Tarindel, marina, radarlady, 3goldens, Tinfoil Hat, IntotheAbyss, Bensch, greycat, Ckntfld, UFOH1, jfdunphy, freakofsociety, el dorado gal, franziskaner, yuriwho, LarisaW, baccaruda, CTPatriot, subtropolis, Heiuan, furiousxxgeorge, klamothe, arnott, Cook, OpherGopher, PBen, Superpole, jhutson, panicbean, DocGonzo, Northstar, Robin7459, Valtin, tomfool, TN yellow dog, dewtx, Dobber, Laurence Lewis, reflectionsv37, eru, Blissing, owlbear1, cfk, Pam from Calif, Sun Tzu, pasadena beggar, Jaime Frontero, jimstaro, jimreyn, GreyHawk, cassidy3, Ice Blue, QuickSilver, Skid, gerardbellavita, most peculiar mama, brenda, babatunde, antiapollon, Isara, SBandini, FunkyEntropy, Bob B, mikolo, Savvy813, collapse, moodyx, lone star liberal, power model, Shotput8, shiobhan, Ginny in CO, mph2005, FightTheFuture, neroden, wiscmass, Bill Kush, serrano, sodalis, Lisa Lockwood, Shaking the Tree, churchlady, deepsouthdoug, LithiumCola, Cory Bantic, Lindy, ZinZen, kkjohnson, timba, bently, Jim P, begone, Abacab, lcork, RiaD, SoulCatcher, martini, third Party please, occams hatchet, Coherent Viewpoint, kovie, Showman, Shirl In Idaho, redcedar, Sanuk, hlee1169, BachFan, MissInformation, Nightprowlkitty, Do Tell, Compound F, tarheelblue, jsamuel, vigilant meerkat, sherlyle, cybersaur, Clytemnestra, Junkyard Dem, MeMeMeMeMe, tung sol, ferallike, Opakapaka, BlueInARedState, emeraldmaiden, quotemstr, the freak, Kimball Cross, blackthorn, Yellow Canary, liberalconservative, victoria2dc, Samwoman, KenBee, mrobinson, fromer, rhetoricus, Shakludanto, Im with Rosey, Albatross, mcartri, neecie100, greenearth, hideinplainsight, Lefty Coaster, goodasgold, triv33, tecampbell, MJ via Chicago, StrayCat, Tanya, need pr, jasonbl, merryberry, gooderservice, gatorcog, Libby Shaw, imabluemerkin, NC Dem, real world chick, FireCrow, Terminus, Kitten, NearlyNormal, armadillo, bleeding heart, el cid, MarciaJ720, myrealname, ER Doc, Dinclusin, doinaheckuvanutjob, bonesy, TayTay, Pilgrim X, llbear, Timothy J, doingbusinessas, Clive all hat no horse Rodeo, buckeye blue, va dare, Lovo, fiddlingnero, thecoolmacdude, Stripe, means are the ends, Dreaming of Better Days, kurt, chapel hill guy, Jbeaudill, jatkin02, kurious, sasher, lorzie, Nulwee, Opposite Reaction, Aaa T Tudeattack, gtaylor, ammasdarling, Evil Betty, Noor B, YoyogiBear, out of left field, BeninSC, ibonewits, xaxado, asilomar, nobody at all, anotherdemocrat, khereva, Old Gardener, Haningchadus14, Loudoun County Dem, dmh44, Deadicated Marxist, FWIW, uniongal, DocbytheBay, suburi, threegoal, Outrider, DrWolfy, edsbrooklyn, FishOutofWater, Redbear, Duccio, wildweasels, tbirchard, HeartlandLiberal, Calvin Jones and the 13th Apostle, lurks a lot, DWG, londubh, Webster Hodges, newpioneer, Strabo, bnasley, Kentucky Kid, jayden, Heyroot, vbdietz, albaum, ultraslacker, bobswern, jnhobbs, millwood, Moderation, owl06, Phylosoph, MadAsHellMaddie, Rumarhazzit, Zydekos, Captain Nimrod, eyesoars, Bikemom, Hens Teeth, vet, wuod kwatch, Terra Mystica, TomP, Empower Ink, JDWolverton, jwinIL14, MKinTN, FirstValuesThenIssues, rogerdaddy, gundyj, revm3up, mconvente, VelvetElvis, Clio2, condorcet, JeffW, Hippokleides, Captain C, A Simple Man, maizenblue, Phil N DeBlanc, rontun, califdem, Youffraita, Foundmyvoice, Thomas Twinnings, SMWalt, ReEnergizer, filby, OutCarolineStreet, treesrock, Rick Winrod, thetadelta, Akonitum, Its any one guess, beltane, Lujane, tofumagoo, Ministerbruce, boatjones, toddpw, geomoo, daddy4mak, pickandshovel, Missys Brother, Quicksilver2723, Jeff Y, home solar, mofembot, S C B, kyril, xango715, petulans, mspt, Matt Browner Hamlin, kempsternyc, tampamatt, Cobbler, luckylizard, MalachiConstant, Bartimaeus Blue, mary13L, DixieDishrag, Uncle Bob, BYw, nklein, red 83, Tross, A Man Called Gloom, allie123, dont think, In her own Voice, Simply Agrestic, lgcap, CIndyCasella, al75, forgore, lostboyjim, legendmn, Fiddlegirl, who dat, Bule Betawi, Rhysling, Ran3dy, El Yoss, ReallyEvilCanine, number nine dream, Mr Tentacle, denisegreenbay, world dancer, emboyo, litoralis, The Dead Man, An Affirming Flame, WereBear, Shhs, cantelow, bsmechanic, Carol in San Antonio, be the change you seek, bondibox, Stranded Wind, Inky99, Nebraskablue, janmtairy, notrouble, Partisan Progressive, slaney black, ScientistSteve, velvet blasphemy, earicicle, SciVo, Patch Adam, Hopeful Monster, zackamac, WobegonGal, EquationDoc, NCenvtl, Goldie Taylor, Daily Activist, Mercuriousss, John Shade, smartheart, Virginian in Spain, followyourbliss, platypus60, Framlingham, Calfacon, TheOpinionGuy, mksutherland, ZilV, MooseHB, kw7777777, obscuresportsquarterly, kevinpdx, deutschluz, synductive99, histOries Marko, Joeytj, IreGyre, CityLightsLover, stevenwag, notksanymore, diddosMN, ArthurPoet, sherijr, spyguy999, reesespcs, northernlights, jfromga, azadmanish, ravenlore, Livvy5, Adept2u, longtimewatcher, RogerDodger, Cleopatra, Larsstephens, ruscle, canuckster, lompe, gohmak, sulthernao, parse this, Carl Brutanananadilewski, ppl can fly, oohdoiloveyou, unertl, Liberal Pagan, loper2008, Alec82, mikesbuddy, everyoneismad, winkster, flitedocnm, egyinny, Vacationland, robertacker13, dditt, 1000 Points of Fright, Blue Blood, KayEmarr, PurpleMyst, Hamsun, politicalslave, fidellio, eponymouse, stunzeed, Obamacrat, citizen31, chrome327, Crabby Abbey, Ronald Singleterry, ko62, NoVAVoter, ATFILLINOIS, BonnieSchlitz, Dexter, RJP9999, snaglepuss, kinkistyle, ctlrick, Marja E, Chris Carlson, SlackwareGrrl, DrFitz, jawboneblues, cai, JRandomPoster, riptide, abrauer, Publius2008, washunate, jdacal, Betty Pinson, Urtica dioica gracilis, Mebe, MsGrin, kenwards, Floande, Decatur, bottles, nosleep4u, AMfromATL, fiercefilms, ssldenver, slice, Vega, High Tide, samantha in oregon, ribofunk, Pakalolo, loki99, Kai99, BrowniesAreGood, Yo yo yo, Colorado is the Shiznit, CornSyrupAwareness, cylonbabe, RepTracker, heart of a quince, I love OCD, Geoff D, Uju, atheistben, DupageBlue, kirbybruno, wahine, Massapequa Dem, Amayi, poorbuster, the national gadfly, commongood, slagrock, FarWestGirl, twindad9, BlueHead, Fickle, Liberal OC Woman, Nicci August, trs, Maribeth, MartyA, SERMCAP, page394, Greek Goddess, Haf2Read, Cinnamon Rollover, BicycleDave, MKHector, matthewj, ECEarl, Laurie Gator, cbr, sjr1, merrily1000, Vtdblue, BlackBandFedora, The Rational Hatter, kurios

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site