Skip to main content

What if you could estimate not what but when big changes were afoot - in consumer buying preferences, in technological innovation, in politics (our favorite topic here in the Great Orange City)?

I don't mean writing a book with a few hundred footnotes strung together around four or five cadged ideas pulled from the zeitgeist. If you want secular preaching there's always a bookstore. I would advise a used bookstore - and buy the old stuff people actually kept for seasoning a decade or two. It's probably better by a magnitude or two than the new stuff.

What I mean is being able to derive the interactions of a modest number of trends - say, something as elementary as stock data, map it to news headline counts at the ends of temporal line segments, weight them, relate them, and give you not only a scientific prediction on which way the stock markets will turn on a given day but why.

That'd be a nice trick, right?

What would it take to get such a project going?

Data, not necessarily a lot of it, just a lot of consistent quality over time. Predictive modeling is all the rage in financial services these days. The problem from where I am sitting: it's trying to go in the one direction that's flustered very, very smart  people with rich backers for decades. It's trying to be even more complicated than the complicated models companies like Long Term Capital had going.. until those models killed those companies off. Oh - did I say LTC? I meant Enron. Did I say Enron? I mean... Bear Stearns. Lehman. MBIA. Ambac. Etc. Etc. Etc.

My counsel: Go simpler.

However, my interest is not so much in helping such folks. They have all the help they need. They were even told the cupola model responsible for justifying the all-in approach to subprime mortgage lending was a bad idea.. and they did not listen.

I'm more concerned for the schmoes who have no research teams, no trading desks, no training to even know what's trained against them.

Here's how it goes:

You can with simple stock market time series data and rudimentary skills at Google have a good handle on how to cut through the crap - both elsewhere and here (yes, Virginia, there are hyper-clauses even at DKOS) on all topics.

For example: Let's take one of the most uncontroversial events of the past decade, the day that a few guys on a few jet airplanes decided to crash-land them into heavily populated buildings, two of them in Lower Manhattan.

For obvious reasons the markets were closed that day, and for a while after.

And there was understandably a LOT of noise in the trading behavior for a while afterward.

Eventually, a pattern emerged - if you bothered to look for it.

A very strong one.

A very (in hindsight) we-really-should-have-seen-this coming one.

Not too long after 9/11, the strongest R-squared linear regression on S&P500 data reached all the way back to a two week range around October 31, 1999.

The night EgyptAir Flight 990 took a dive into the Atlantic Ocean, all lives lost.

It was a loud and clear signal - a new kind of hijacking that had no intending of taking control of a plane and redirecting it  for transportation and publicity and demonstrations of power.

It was all about accomplishing all three by turning an airplane into a weapon.

...

I can repeat the same exercise with almost any event you can name. To quote Mister Universe from "Serenity" - You can't stop the signal, Mal. The signal is everywhere.

What's astonishing is how easy it is to pluck this context from the noise - but no one pays for simple.

They pay for the interpreters of the mutterings of the crazed oracle at the temple.

I'm working on a way to save you from making that trip - to save all of you from having to do so.

This being my answer to a by-the-way comment that apparently I've not been doing much lately.

My assurances: I'm very active.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Isac asimov delved into "psychohistory" FOUNDATION (6+ / 0-)

    series and the problems with such

  •  Okay, I'm dumb. What does this mean? (2+ / 0-)
    Recommended by:
    where4art, The Angry Architect

    I'm not sure what you're getting at here. You can use googleable events to predict stock market activity? Or the other way around? And how do you choose the events?

    •  On a given day people make decisions (6+ / 0-)

      They have different ideas in their minds about what's going to happen on a given day to their money.

      The media has a big role to play in this.

      All stock data has in embedded in it not just one but many trends that start in the past at different times.

      Different "stories"  associated with prevalent influences rise and fall over time - some are spikes and last a day or so. Others linger. Others are deep foundations.

      You can obtain a pretty good idea what the story is about by looking at news archives at or around the estimated "start" date.

      You also have to eliminate groupings around a given time - that way you don't "double dip" the same event.

      If markets were truly efficient, these patterns would be random from day to day.

      They're not.

      There is persistency and there is if not causation then introduced bias to market decisions.

      The media headlines are both participants and data in this analysis.

      That's the beauty of it - no one will stop posting headlines, and the headlines do have demonstrable, actionable effect on markets.

      In other words, it can be traded on.

      I seek a world where lying in print and on the airwaves and on the internet produces no profitable result.

      And I think I've come across a very easily teachable way to do so.

  •  I love you for this sentence: (3+ / 0-)
    If you want secular preaching there's always a bookstore. I would advise a used bookstore - and buy the old stuff people actually kept for seasoning a decade or two. It's probably better by a magnitude or two than the new stuff.
    There's much we can learn from old books.

    If you recommend a book, and provide an Amazon link, you are never, ever allowed to bitch about book stores closing up.

    by SeattleTammy on Sun May 06, 2012 at 06:59:25 PM PDT

  •  This sounds like "event study" analyses (5+ / 0-)

    For about twenty years, published work on the movements of stock prices in response to or in anticipation of various events have been appearing in the academic literature.  (Some of it, such as studies of movements of various stock prices in the days just before an election or a regulatory change, or in advance of a covert operation (the 1953 coup in Guatemala) have a lot of political implications.  You are a bit vague about describing what you are doing, but you seem to be talking about performing such a study.  While in some sense they are indeed "simple," I admit to being skeptical that this is a domain for "barefoot doctors" with no statistical background at all.

    •  I'm strictly retrospective (5+ / 0-)

      One can gain a LOT of valuable information on knowing with greater accuracy what HAS happened.

      Insofar as paradigm shift prediction is concerned, perhaps it is fairer to say I am seeing to predict when extant paradigms no longer are in force a bit more precisely AND quickly.

      Something I learned at UNC back in the day - that treating social sciences as a close cousin to physical sciences is asking to get disappointed.

    •  event studies of that kind - political economy (3+ / 0-)
      Recommended by:
      G2geek, kaliope, semiot

      The markets despite their rep are rarely useful for prediction.

      What they do provide is explanation as to what is being done to countries and countrymen alike.

      I had a prof explain quite eloquently in words and in economics notation what tariff protectonism/free trade says about not just countries but political coalition formation within said countries.

      And in hindsight. And it was very useful information.

      His name was Timothy McKeown. A name more people should know.

      •  "the signal is everywhere." (3+ / 0-)
        Recommended by:
        kaliope, susakinovember, cskendrick

        Imagine you have access to communications traffic data: the "to/from/time" information on every email and phone call to and from individuals in a defined area (geographic or community-of-interest such as "stock traders" or some other).

        Now consider that you're interested in identifying certain types of actors in that population: actors who for whatever reason are highly capable of minimizing their own communications footprint.  

        I believe that it should be possible to identify those actors by watching for subtle changes in the communications patterns of others who are adjacent to them.  By "others" I do not mean others who are involved in the actions for which the actors are being sought, but rather, un-involved others such as the neighbors of a terrorist cell.   And by "adjacent" I mean, not just geographically but in other ways that can be operationalized for purposes of measurement.

        Hypothetical case: the 9/11 hijackers had neighbors.  Now take all the NSA traffic data on the communications activities of those neighbors, and it should be possible to find subtle changes that occurred after the hijackers moved into the neighborhood.  

        The problem is that the signal is confounded by noise: for example prejudice against foreigners or whatever.  None the less, with sufficient raw data it should be possible to tease out the signal whereby the neighbors were reacting to something they perceived about the hijackers in their neighborhood.  

        I believe this type of analysis can be developed and utilized, not only to spot terrorist cells and suchlike, but to spot other types of "signal" in defined areas.  

        Over time, it should be possible to ascertain a number of different categories of types of subtle shifts in communications behavior.

        Thoughts?

        "Minus two votes for the Democrat" equals "plus one vote for the Republican." Arithmetic doesn't care about your feelings.

        by G2geek on Sun May 06, 2012 at 10:12:47 PM PDT

        [ Parent ]

        •  I think using mass data methods to predict (0+ / 0-)

          specific behaviors is far less rewarding than using them to describe a cluster of anomalous reactions to something anomalous... like noting gravitational lenses around dark matter clusters in the distant sky.

          So I guess, yeah. I'm down with this.

          And having a one-day/same-day lag on "what the neighbors think" the (actual) terrorists next door are up to is a lot better than a two-day lag.

  •  I don't know why folks have a hard time (3+ / 0-)
    Recommended by:
    kaliope, cskendrick, semiot

    ...with predictions about matters economic. Or even matters political.

    It's not like the patterns aren't repeating with absolute precision.

    It's not like the underlying fundamentals are in the least obscure.

    It's not like the trends have wavered one degree in incline since the late 70s.

    It must be a neurotransmitter disorder.


    Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that "my ignorance is just as good as your knowledge." ― Isaac Asimov

    by Pluto on Sun May 06, 2012 at 11:01:21 PM PDT

    •  The chief weakness of modeling is model limits (1+ / 0-)
      Recommended by:
      Pluto

      As in not recognizing that models have them, that

      1. the patterns do stop (women not getting to vote, men/women not getting to drink beer)
      2.  that fundamental decision-maker/breakers have eras (religious orientation of one's monarch = conversion behavior else beheading behavior)
      3. That those eras can cycle back after fashion
      4. That, odd you should say it, the long run baseline trend for markets this week dates back to 1974. :)

      •  To explain last snark a bit.. (0+ / 0-)

        That the patterns do not always change QUICKLY.

        •  I've been working on this one for some time: (1+ / 0-)
          Recommended by:
          cskendrick
          4. That, odd you should say it, the long run baseline trend for markets this week dates back to 1974. :)
          I'm not sure what you are looking at -- but I had mistakenly marked the trend's beginning at the Reagan deregulation policies. But now i see that it was Nixon taking the US Dollar off the gold standard that triggered the mathematically precise dollar debasement and all that that brings with it.

          Not all patterns repeat in a human lifespan. The Chinese enacted and repealed prohibition 41 times over thousands of years because they kept forgetting that it didn't work. I imagine they are about due to circle that drain again.


          Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that "my ignorance is just as good as your knowledge." ― Isaac Asimov

          by Pluto on Mon May 07, 2012 at 01:08:18 PM PDT

          [ Parent ]

          •  Not sure if gold standard is it though... (1+ / 0-)
            Recommended by:
            Pluto

            as that was in 1971... though late 1974 was the end of the crash that followed the collapse of Bretton Woods.

            What might be more telling is who that crash hurt more - a country that had a REALLY bad day today in the stock markets:

            The UK.

            There may be yet another big story from across the Atlantic that the conservatives cannot just explain away as socialism run amok, as it is happening thanks to THEIR kindred's policies in London.

            •  There are many factors that would (1+ / 0-)
              Recommended by:
              cskendrick

              ...come into play following the gold standard incident. It was necessary to immediately dollarize the world with all speed. Nations were still buying commodities with their own currencies, despite Bretton Woods. Thus, the petrodollar had to be formed, and this required a serious deal with the Saudis to make it so and make it stick throughout the newly formed OPEC.

              And, stick it did until that wacky Saddam announced he was going to sell for Euros. He was as good as dead three months later. Same thing for Gaddafi when he announced oil for gold Dinars. Three months to live.

              Speaking as someone in the currency exchange business, three years sounds about right to shake things out and for the trend to secure an unshakable grip on the dollar. Especially when consumer debt began to be monetized for the first time and the future became clear.

              As for the UK, they have been dead men walking for awhile. Any nation that attacks Afghanistan essentially collapses. That pattern has long been established. I wouldn't bet against it. It's pretty much a slam dunk.


              Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that "my ignorance is just as good as your knowledge." ― Isaac Asimov

              by Pluto on Mon May 07, 2012 at 02:18:45 PM PDT

              [ Parent ]

              •  Well perhaps we can tighten it down (1+ / 0-)
                Recommended by:
                Pluto

                The precise candidate date is August 14, 1974.

                Oh, my.

                That is the date Congress authorized U.S. citizens to own gold for first time in 40 years...

                •  Nice (1+ / 0-)
                  Recommended by:
                  cskendrick


                  Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that "my ignorance is just as good as your knowledge." ― Isaac Asimov

                  by Pluto on Mon May 07, 2012 at 07:34:12 PM PDT

                  [ Parent ]

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site