What are the attributes of recommended diaries, compared to those that fail to make the list?
To shed some statistical light on this question, I analyzed 471 consecutive diaries, isolating the quantifiable factors that were associated with making the recommended list.
Overview
I examined every diary from two consecutive weekdays (picked at random) in June 2007 – June 13 and 14. I coded each diary along several dimensions, including topic, format, diarist reputation, and the apparent effort put into it. I’ll present the basic findings, followed by a description of the categories used in each chart. The statistical jargon is bracketed; everything else is presented in a straightforward manner.
Baseline stats:
Diaries analyzed: 471 (several were excluded due to missing data)
Median number of recommendations: 9
Diaries that made the recommended list: 33 (7.0%)
Which diary topics are most likely to be recommended?
Topic | Number | Median # of Recs | % Making the Rec List |
Immigration | 22 | 7.5 | 0 |
Israel/Palestine | 14 | 9 | 7.1 |
Iraq | 61 | 7 | 6.6 |
Impeachment | 14 | 10.5 | 7.1 |
A. Gonzales | 21 | 17 | 19.0* |
Healthcare | 19 | 15 | 15.8 |
Elections | 104 | 8 | 5.8 |
Plame/Libby | 13 | 14 | 7.7 |
[Asterisk indicates a statistically significant (p<.05) difference on a chi-square test of the 2x2 crosstab: "recommended or not" and "(topic) or not"]</p>
Perhaps the most fascinating finding is that, while immigration was the hottest topic in the mainstream media that week, not one of the 22 diaries with an immigration tag made the list. I guess it’s no surprise that we don’t let Lou Dobbs and talk radio set our agenda. Overall, though, the topic doesn’t seem to matter, at least on a day in which no huge story is breaking.
Note: The "topic" of the diary was determined by the reading the tags; thus, several diaries fit more than one topic. This is a blunt method of coding the topics, and it most likely exaggerates the number of diaries that actually focus on the topic. But it is more objective than reading the diaries and trying to shoehorn them arbitrarily into one or more categories. Only the most frequently occurring topics during those two days were included.
What types of diaries are most likely to make the list?
I created a typology of diaries, based on my 3 years of reading the site. I had specific, concrete decision rules for placing each diary into a type, which I’d be happy to discuss in the comments.
Type | Number | Median # of Recs | % Making the Rec List |
General Commentary | 191 | 7 | 3.1* |
Breaking News from MSM, Minimal Commentary | 24 | 13 | 12.5 |
Breaking News from MSM, Extensive Commentary | 68 | 13 | 11.8 |
Breaking News from Alternative Source | 34 | 14.5 | 8.8 |
Call to Action | 19 | 11 | 10.5 |
Candidate Advocacy: Edwards | 7 | 33 | 28.6* |
Candidate Advocacy: Non-Edwards Presidential | 15 | 9 | 6.7 |
Candidate Advocacy: Non-Presidential | 13 | 8 | 0 |
Original Report | 23 | 17 | 21.7* |
Media Criticism | 17 | 7 | 0 |
Promotional | 23 | 6 | 0 |
Meta/Community | 9 | 20 | 22.2 |
[Asterisk indicates a statistically significant (p<.05) difference on a chi-square test of the 2x2 crosstab: "recommended or not" and "(type) or not"]</p>
"General commentary" is a generic category for diaries that don’t fit another type. They are often (though not always) short and poorly sourced ("Yet Another Take on Larry Craig," etc.). It thus speaks highly of the site’s discriminating tastes that such diaries have a much lower probability of making the list than those in the more concrete, purposive categories. Breaking news – a diary touting a news story less than 24 hours old – fared slightly better than average. Original reports – diaries that do original, primary-source research (i.e., drational) or that consist of first-person accounts of events attended by the diarist – are far more likely than most types to make the list. Again, that seems to speak well of our ability to recognize value-added content.
For obvious reasons I had to separate the presidential advocacy diaries into Edwards and non-Edwards. Guess which ones were recommended more often? On the other hand, non-presidential candidate-advocacy diaries struck out completely, as did media criticism and promotional diaries (which drew attention to or, reviewed, a movie, TV show, or consumer product). Obviously, media critics (such as KingOneEye) have made the rec list – but none did during the period of analysis.
What are the attributes of recommended and non-recommended diaries, on dimensions such as effort and diarist reputation? (Numbers in cells are averages)
| Recommended | Not Recommended |
Total Links | 5.5 | 4.2 |
Link Diversity | 2.4 | 2.0 |
# of pics/videos embedded | 0.8 | 0.5 |
User ID# | 58428 | 69209 |
Diaries by User | 20.0 | 12.5* |
Hot Diaries by User | 6.4 | 1.9* |
Hot/Total Diaries Ratio | 0.3 | 0.1* |
Misspellings (per 1000 words) | 1.4 | 2.2 |
Word Count | 987 | 693* |
Flesch Reading Ease score | 52.6 | 54.9 |
[Asterisk indicates statistical significance (p<.05) on a difference of means test.]</p>
Is a well-sourced diary privileged over an off-the-cuff rant? Not really. Recommended diaries had slightly more links, and slightly greater link diversity (measured as the number of distinct types of links – i.e., blog, newspaper, television website, academic/scholarly, reference, government, etc.) than non-rec diaries. But the difference was not significant. Also, recommended diaries have fewer spelling errors, but not significantly so. One measure of effort that fared better was the word count: Recommended diaries are quite a bit longer, on average, than non-recs. Finally, recommended and non-rec diaries were written at about the same grade level.
Diarist reputation appears to be crucial, at least without controlling for any other factors. "Diaries by user" refers to the number of diaries written by the diarist from April 1 through June 12, 2007. "Hot diaries" are defined as those with at least 100 comments (yes, I know it’s not the best measure – but it was the most feasible one). The average recommended diarist had produced almost twice as many diaries – and almost four times as many hot diaries – over the previous two and a half months.
Does reputation alone lead to automatic recommendations? Or do established diarists make the list because they produce better diaries? Only an analysis that measures the effect of each variable, controlling for all others, can shed light on this question. [I ran a binary logistic regression, using STATA’s maximum likelihood algorithm, for the dependent variable "made the recommended list or not."]
The results are a bit messy, so I’ll simply indicate whether the variables attained statistical significance. That is, holding all other variables constant, did a variable increase (+) or decrease (-) the likelihood of a diary making the list? Or did it have no effect? (n)
Breaking News from MSM, Minimal Commentary | + |
Breaking News from MSM, Extensive Commentary | + |
Breaking News from Alternative Source | n |
Call to Action | n |
Candidate Advocacy: Edwards | + |
Candidate Advocacy: Non-Edwards Presidential | n |
Original Report | + |
Meta/Community | + |
A. Gonzales | n |
Healthcare | n |
Total Links | n |
Link Diversity | n |
Pictures/Video Embedded | n |
User ID# | n |
Diaries by User | + |
Hot/Total Diaries Ratio | + |
Diarist is Government Official | + |
Misspellings (per 1000 words) | n |
Word Count | + |
Reading Ease | n |
Basically, the same variables are significant as in the prior analysis. One exception is that breaking news from a mainstream source makes a difference here. Note: several variables were excluded from this model for nitpicky statistical reasons.
Implications
Overall, I would argue that the Daily Kos community does a good job with what I call "distributed gatekeeping" – that is, enabling the cream to rise to the top through community consensus, rather than through top-down journalistic editing. Though I would like to have seen more impact from some of the "effort" variables, the diary types that tend to be of higher quality than typical blog rants are apparently more likely to be recommended than the catchall "general commentary" diaries. Also, though some might interpret the power of reputation as undermining the egalitarian nature of the site, I disagree. In the blogosphere at large, the dominance of A-listers appears to have created high barriers to entry. I could start a blog tomorrow – but will Josh, Glenn, and Duncan link to me? On the other hand, all Daily Kos diaries start on the same playing field – the highly trafficked "recent diaries" list. As a result, first-time or infrequent diarists often make the list solely on the merits of their diary. Still, this study provides at least some empirical fodder for those who complain that certain diarists gain automatic entry to the rec list.
The Daily Kos diaries ecosystem is an amazing experiment in distributed gatekeeping. Though it still has some kinks to work out, the community comes closer than any other Internet forum to providing the much-needed middle ground between the top-down gatekeeping of the traditional press and the "any jackass with a modem can publish an opinion" chaos of the Internet. If anything, I think it undersells its potential. Casual visitors often miss the recommended list amid the clutter of the homepage. Given that it results from the collective editorial judgment of this sophisticated community, I’d love to see the rec list distinguished more clearly from the rest of the homepage, the same way traditional news clearly signifies its editorial judgment by placing stories above the fold of a newspaper or at the top of a newscast.
Limitations
I make no claim that this is any sort of definitive study of recommendation patterns. It’s merely what social scientists call "suggestive." First, as noted above, the attributes that make a diary desirable are extremely difficult to quantify. Most of these measures are, at best, proxies for fuzzy concepts such as "effort" and "reputation." Also, although 471 is by no means a small sample, the results would be more robust with a sample that is double or triple the size of the current one, and that spans a longer time period. Unfortunately, each diary takes about 10 minutes to analyze, and I have no budget for research assistants. Let’s see you do 1000 of them :) In short, while there are obvious limits to this methodology, I think it serves as a nice complement to the first-person narratives from which diary recommendation is typically discussed.
Adapted from a paper presented at the 2007 American Political Science Association annual conference.
--Adam S.
EDIT: Woo Hoo, my first Rec Diary! Thank you for the thought-provoking comments and questions. Somehow I'm not surprised that the community would provide such sophisticated feedback.