Skip to main content

Yesterday was trashday and I'm blaming the great Green diaries here making me wonder just how much trash I've accumulated since birth. I'm hoping my individual contribution can't be measured in Yottabytes. Now imagine every keystroke, every tweet tweeted, every satellite satelliting, being saved on big computers.

On a remote edge of Utah's dry and arid high desert, where temperatures often zoom past 100 degrees, hard-hatted construction workers with top-secret clearances are preparing to build what may become America's equivalent of Jorge Luis Borges's "Library of Babel," a place where the collection of information is both infinite and at the same time monstrous, where the entire world's knowledge is stored, but not a single word is understood. At a million square feet, the mammoth $2 billion structure will be one-third larger than the US Capitol and will use the same amount of energy as every house in Salt Lake City combined.

It's getting so big that we're about to hit the Yottabyte. James Bamford reviewing The Secret Sentry: The Untold History of the National Security Agency by Matthew M. Aid in the November 5th NYT Review of Books (Who's in Big Brother's Database?) writes, "numbers beyond Yottabytes haven't yet been named":

Just how much information will be stored in these windowless cybertemples? A clue comes from a recent report prepared by the MITRE Corporation, a Pentagon think tank. "As the sensors associated with the various surveillance missions improve," says the report, referring to a variety of technical collection methods, "the data volumes are increasing with a projection that sensor data volume could potentially increase to the level of Yottabytes (1024 Bytes) by 2015."

h/t to Meteor Blades who mentioned John McCreary's NightWatch ("an executive level intelligence recap drawn from domestic and international reporting" source) where I found this. Operation Strike of the Sword began in July; checking 6 July 2009 I get slightly discombobulated— McCreary doesn't use the caplock key and I've never heard of a Yottabyte before. Sorry, but I thought it was snark at first:

Extrapolating from current trends, according to a report by the JASON defense analysis group and just recently reported by Stephen Aftergood, data production could hypothetically reach the Yottabyte range by 2015.

McCreary didn't provide a link, but Steven Aftergood blogs at Secrecy News (from the Federation of American Scientists), where I found Yottabytes and the Data Analysis Challenge (July 6th, 2009). What got me was the PDF report from the JASON defense advisory panel (which mentions Yottabytes) was obtained by Secrecy News via Freedom of Information Act request. The "December 2008 JASON report was initially withheld from public access". What makes this more interesting (if not worse) is that Matthew M. Aid previously reported the disappearance of Yottabytes of information:

Beginning in the fall of 1999, and continuing unabated for the past seven years, at least six government agencies, including the Central Intelligence Agency (CIA), the Defense Intelligence Agency (DIA), the Defense Department, the military services, and the Department of Justice, have been secretly engaged in a wide-ranging historical document reclassification program at the principal National Archives and Records Administration (NARA) research facility at College Park, Maryland, as well as at the presidential libraries run by NARA.

But it's not just Yottabytes being collected or disappearing, it's Yottabytes of information that can't be analyzed, which appears to be McCreary's concern: who does the analysis?

The challenge of performing even superficial analysis of the constantly exploding volume of collected intelligence data is not a new problem. NSA’s brilliant and only NIO for Warning, David Y McManus, commented over 25 years ago that information could be collected and distributed faster than it could be evaluated. Human analysts remained the weakest and slowest link in the intelligence process, he lamented, because no tools existed in 1983 to facilitate and accelerate analysis.

What's a NIO for Warning?

from A Consumer's Guide to Intelligence

{Page 15} At the core of the NIC [National Intelligence Council] are the National Intelligence Officers (NIOs). The NIOs supervise the production of National Intelligence Estimates, other NIC publications, manage the Intelligence Needs Process, provide briefings to senior policymakers, and focus Intelligence Community Collection and analytic resources on priority issues. Currently, there are 12 NIOs who come from various intelligence agencies as well as academia and the private sector. The NIOs are organized by region—Africa, East Asia, Europe, Latin America, Near East and South Asia, Russia, and Eurasia; and functions—Economics, General Purpose Forces, Global and Multilateral Issues, Science and Technology, Strategic Programs and Nuclear Proliferation, and Warning.

{Page 27} The National Warning Staff (NWS) is an interagency body serving under the National Intelligence Officer for Warning. The NWS assists the NIO for Warning in various functions, including identifying warning issues, and advising the IC on warning methodology, traning and research.

{Page 56} [The National Intelligence Estimate (NIE)] are the DCI's most authoritative written judgments concenring national security issues. [Iraq's Continuing Programs for Weapons of Mass Destruction]

{Page 53} [The Director of Central Intelligence (DCI) is] the primary adviser to the President and National Secuirty Council on national foreign intelligence, appointed by the President with the consent of the Senate; head of the IC and responsible for the development and execution of the National Foreign Intelligence Program; Director of the Central Intelligence Agency.

see also: The National Warning System: Striving for an Elusive Goal, Warning and Iraq's Invasion of Kuwait: A Retrospective Look & The Mission to Warn: Disaster Looms

Highly recommend Bamford's review outlining Aid's book discussing "automated surveillance on steroids", the UK scrapping its "big brother database" after public outcry, various intelligence hits and misses, how intelligence gets used and abused, TURBULENCE, and what Aid calls NSA's biggest problem: electrical power—"The issue is critical because at the NSA, electrical power is political power". Here's one more snip:

Based on the NSA's history of often being on the wrong end of a surprise and a tendency to mistakenly get the country into, rather than out of, wars, it seems to have a rather disastrous cost-benefit ratio. Were it a corporation, it would likely have gone belly-up years ago. The September 11 attacks are a case in point. For more than a year and a half the NSA was eavesdropping on two of the lead hijackers, knowing they had been sent by bin Laden, while they were in the US preparing for the attacks. The terrorists even chose as their command center a motel in Laurel, Maryland, almost within eyesight of the director's office. Yet the agency never once sought an easy-to-obtain FISA warrant to pinpoint their locations, or even informed the CIA or FBI of their presence.

But pulling the plug, or even allowing the lights to dim, seems unlikely given President Obama's hawkish policies in Afghanistan. However, if the war there turns out to be the train wreck many predict, then Obama may decide to take a much closer look at the spy world's most lavish spender. It is a prospect that has some in the Library of Babel very nervous. "It was a great ride while it lasted," said one.

Originally posted to northanger on Wed Oct 14, 2009 at 01:16 PM PDT.

Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags


More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site