There's a diary right now on the rec list about an NSA wonk providing a first-hand revelation on KO that they are watching everything.
Everything.
Everything has to be stored. Somewhere. And the funny thing about storing all the 'everything' is that it has to be retrieved at some point. You know, like searching for it — and actually finding it. Whatever 'it' might be.
Usually, the hurdles to finding something increase dramatically as the knowledge base grows, such that the more you have to wade through, the harder it is to see what you're looking for.
The general argument against illegal data gathering in search of terrorists is that you can't effectively find the bad guys simply by increasing the size of the haystack.
That's pretty much true. Unless you've got a fractal database.
All the geeks over the fence.
Fractal data technology isn't new. It's been around since the '80s and has been used to burn CDs in addition to compressing and de-compressing images.
A fractal is essentially any geometric pattern that repeats inward and outward upon itself. Forever, like an unending snowflake.
No matter how many times you zoom in, there are more tiny snowflakes. This is a bit simplistic because in a fractal database, the data isn't so structured and neatly organized as this. But the nature of the geometry makes it structured. Fractal technology creates order from chaos. The very same chaos that occurs from dumping literally a centillion bits of bytes into a database someplace deep in the heart of the NSA. We're talking many centillion terabytes of data. About you and me. Your kids. Your friends and what you do late at night when you think no one is watching. They're watching everything.
Until recently its applications have been limited. The real power behind fractal geometry as it applies to data has been pretty elusive.
But as with all things technology, Moore's Law ensured it was only a matter of time before it became powerful enough, and cheap enough, where someone figured out how to make it work and money from it — or how to effectively spy on people.
With a fractal database, someone simply has to dump data in. Then the magic happens.
That fractally little thing knows stuff about you. And it doesn't speak in any sort of code. It speaks in normal words that reveal who you are and what you do.
It looks at you and it knows you travel. And where. It knows who your friends are through your phone calls and your Facebook profile. It knows what you spend and how you spend it. It can search in upon itself again and again, in mere nanoseconds, finding patterns and themes about you like the most complex mashup ever seen, combining the web sites you visit, the phone calls you make, where you work, your donations, your grocery purchases, and the kind of underwear you buy.
It can find patterns of behavior that are similar to others. It links you to others you don't know.
And the government is very hungry for more data about you.
Friday, January 20, 2006
The Justice Department said yesterday that it subpoenaed four major Internet companies for data on what people search for on the Web as part of an eight-year battle over a federal law designed to shield children from online pornography.
They don't have to find anything specific to begin. They just pour it all in the well and let the data connect itself to other pieces of data. After the patterns emerge is when they find the specific stuff:
New York Gov. Eliot Spitzer's role in a prostitution scandal grew out of a public corruption inquiry triggered by his movement of cash
Homeland Security Investigates Bill Payers: They paid down some debt. The balance on their JCPenney Platinum MasterCard had gotten to an unhealthy level. So they sent in a large payment, a check for $6,522.
And an alarm went off. The Soehnges' behavior was found questionable.
Fractals allow their overseers to peer into every intersection of every facet of your life. The connections are all over the place and now the technology exists to mine it.
So the moral of the story isn't what we knew all along – that spying on Americans is illegal, but rather the damage that can be caused by amassing all that data. It's not that they know you post here or that you buy stuff at Target. It's that they're profiling you based on all this data and then connecting you to terrorists or exposing your hooker habit for political gain.
The fact is that the technology exists to mine the data where it didn't exist before.
So don't look now, but they're looking at you.