I was happy to see the headlines this morning in the New York Times (For First Time, AIDS Vaccine Shows Some Success in Trials), the Washington Post (AIDS Vaccine Experiment Yields Unanticipated Positive Results) even my hometown paper the Montreal Gazette (AIDS breakthrough as vaccine cuts infections for first time).
As a scientist, I always scan these articles for the statistics. The statistics can tell you the difference between "WOW! This changes everything!" and "Huh. Interesting. We should look into that more." Turns out, this is another example of "Huh. Interesting. We should look into that more" being blown up into "WOW! This changes everything!". I wouldn't bet a night's winnings at the craps table on this result.
The statistics are these: between the placebo group and the group receiving the vaccination, there were 125 total infections. Of those, 74 occurred in the placebo group, while 51 occurred in the group receiving the vaccine.
The question to ask: assuming the vaccination offered no protection whatsoever, and you ran this experiment an infinite number of times, how often would you expect 51 (or fewer) infections in the vaccinated group to be infected?
The answer: 2.4% of the time.
Now, that might seem like a small number. It means that, if the vaccination offered no protection whatsoever, there's only a 2.4% chance that you'd get an experimental result that would look this (falsely) promising. In science, that's not that impressive -- it's "Huh. Interesting. We should look into that more."
Why such an unenthusiastic response? Because of the problem of many researchers working on the same problem -- while this was a very large trial, there are certainly many other AIDS trials going on, and many of those trials are finding that their proposed remedy has no effect on the virus. Those failed trials have to be included in the calculation to determine the true statistical significance of the result.
It's like you and 4 of your friends go to Vegas, and each of you hit the craps tables, determined to win on the first roll by hitting 7 or 11 -- which has a probability of occuring of only 22.2%. You all line up at the table, you all take a shot. Chances aren't great that you yourself will win; but chances are very good that someone in your group will. Here, "winning at Vegas" means the same thing as "getting a positive result for a vaccine trial, when actually the vaccine is no different from the placebo."
Another example: let's say there are 30 different groups testing for cures to AIDS (the true number is certainly more than this). Let's say they all complete their studies, and you ask each of them to reproduce the calculation above ("assuming the approach offered no protection, how often would you expect the outcome you obtained, or better?") Would any of them say, "I got a result which says my outcome would happen <2.4% of the time"?) Chances are, yes, someone would say that - and not because they'd discovered a cure, only because if you run an experiment many times, the probability of a falsely positive outcome increases with the number of times you run that experiment.</p>
Certainly, this experiment was designed to test for a 100% effective vaccine. Let's say that only 10 people had been infected in the group receiving the vaccine, while 74 had been infected in the placebo group. The probability of that outcome, assuming the vaccination offered no protection at all, is less than 1 in one hundred million -- well above the probability of hitting the Powerball jackpot. That would mean, it is highly unlikely that the vaccine offered no protection, and far more likely that it is offering protection. Even if there were 300 experiments going on just like this one, it's still highly unlikely that any one of them would hit a result this good (how unlikely? it would occur less than 0.0003% of the time).
In other words: I wouldn't recommend selling all you own -- or even a night's winnings at the craps table -- to buy the stock of the companies producing the vaccine. I wouldn't recommend ending other avenues of research. I wouldn't even walk around with a case of cautious optimism, because this is just not that unlikely an outcome, given the environment of many workers attacking the same problem.
But, you can say, "Huh. Interesting. We should look into that more."