What if you could estimate not what but when big changes were afoot - in consumer buying preferences, in technological innovation, in politics (our favorite topic here in the Great Orange City)?
I don't mean writing a book with a few hundred footnotes strung together around four or five cadged ideas pulled from the zeitgeist. If you want secular preaching there's always a bookstore. I would advise a used bookstore - and buy the old stuff people actually kept for seasoning a decade or two. It's probably better by a magnitude or two than the new stuff.
What I mean is being able to derive the interactions of a modest number of trends - say, something as elementary as stock data, map it to news headline counts at the ends of temporal line segments, weight them, relate them, and give you not only a scientific prediction on which way the stock markets will turn on a given day but why.
That'd be a nice trick, right?
What would it take to get such a project going?
Data, not necessarily a lot of it, just a lot of consistent quality over time. Predictive modeling is all the rage in financial services these days. The problem from where I am sitting: it's trying to go in the one direction that's flustered very, very smart people with rich backers for decades. It's trying to be even more complicated than the complicated models companies like Long Term Capital had going.. until those models killed those companies off. Oh - did I say LTC? I meant Enron. Did I say Enron? I mean... Bear Stearns. Lehman. MBIA. Ambac. Etc. Etc. Etc.
My counsel: Go simpler.
However, my interest is not so much in helping such folks. They have all the help they need. They were even told the cupola model responsible for justifying the all-in approach to subprime mortgage lending was a bad idea.. and they did not listen.
I'm more concerned for the schmoes who have no research teams, no trading desks, no training to even know what's trained against them.
Here's how it goes:
You can with simple stock market time series data and rudimentary skills at Google have a good handle on how to cut through the crap - both elsewhere and here (yes, Virginia, there are hyper-clauses even at DKOS) on all topics.
For example: Let's take one of the most uncontroversial events of the past decade, the day that a few guys on a few jet airplanes decided to crash-land them into heavily populated buildings, two of them in Lower Manhattan.
For obvious reasons the markets were closed that day, and for a while after.
And there was understandably a LOT of noise in the trading behavior for a while afterward.
Eventually, a pattern emerged - if you bothered to look for it.
A very strong one.
A very (in hindsight) we-really-should-have-seen-this coming one.
Not too long after 9/11, the strongest R-squared linear regression on S&P500 data reached all the way back to a two week range around October 31, 1999.
The night EgyptAir Flight 990 took a dive into the Atlantic Ocean, all lives lost.
It was a loud and clear signal - a new kind of hijacking that had no intending of taking control of a plane and redirecting it for transportation and publicity and demonstrations of power.
It was all about accomplishing all three by turning an airplane into a weapon.
...
I can repeat the same exercise with almost any event you can name. To quote Mister Universe from "Serenity" - You can't stop the signal, Mal. The signal is everywhere.
What's astonishing is how easy it is to pluck this context from the noise - but no one pays for simple.
They pay for the interpreters of the mutterings of the crazed oracle at the temple.
I'm working on a way to save you from making that trip - to save all of you from having to do so.
This being my answer to a by-the-way comment that apparently I've not been doing much lately.
My assurances: I'm very active.