Do duplicate diaries bug me? Yes they do, both as an author and a reader.
I've developed an automated method to flag diaries as possible duplicates, based on counting the number of identical stemmed words remaining in "box one" text after removing stop words. This simple approach seems to work. An example is shown below the fold, consisting of the candidate duplicates from 500 sequential diaries collected recently.
If this check, or some functional equivalent, could be added to the diary preview it would help authors avoid posting duplicates as well as direct them to previous diaries on the same subject.
Candidate duplicate diaries from 500 diaries collected up to 2 PM PST November 13, 2004. Each group of duplicates is numbered, and headed by comments (not generated automatically!).
- These diaries have the exact same box one text, so the method flags them.
Billmon Series #4: Slipping by paradox Fri Nov 12th, 2004 at 09:57:42 PST (1697 words)
Billmon Series #3: Caroline by paradox Fri Nov 12th, 2004 at 09:57:24 PST (1642 words)
Billmon Series #2: Detective Radjik by paradox Fri Nov 12th, 2004 at 09:48:04 PST (1346 words)
Billmon Series #1: Claudette by paradox Fri Nov 12th, 2004 at 09:47:56 PST (1262 words)
- These diaries appear to all be on the same subject - method works.
Turmoil at the CIA by Gingerandspice Sat Nov 13th, 2004 at 13:19:08 PST (1135 words)
WaPo: CIA said to be in turmoil by rukkus Sat Nov 13th, 2004 at 09:06:05 PST (166 words)
Goss to senior CIA staff: Go Cheney Yourself (WaPo) by DC Pol Sci Sat Nov 13th, 2004 at 07:41:46 PST (199 words)
CIA FALLING APART: Deputy Director Resigns! by georgia10 Sat Nov 13th, 2004 at 04:43:01 PST (971 words)
- A repost.
Another Dean Diary by Redst3 Sat Nov 13th, 2004 at 03:48:42 PST (391 words)
Another Dean Diary by Redst3 Sat Nov 13th, 2004 at 03:37:12 PST (382 words)
- I'd call these duplicates, the authors might not agree.
Mocking Arafart's Death by georgia10 Fri Nov 12th, 2004 at 08:29:58 PST (189 words)
FIRE IMUS ON MSNBC! by edrie Fri Nov 12th, 2004 at 04:49:30 PST (213 words)
- A repost?
A Statistical Perspective on American Power by Ptah Fri Nov 12th, 2004 at 18:04:52 PST (120 words)
A statistical perspective by Ptah Fri Nov 12th, 2004 at 17:57:18 PST (115 words)
- A repost.
RECOUNT OHIO:Cobb & Badnerik Fundraising Drive by Brian Nowhere Fri Nov 12th, 2004 at 11:26:27 PST (243 words)
RECOUNT OHIO:Cobb & Badnarik Fundraising Drive:Reposted for Freshness by Brian Nowhere Sat Nov 13th, 2004 at 09:36:13 PST (342 words)
- Diaries on the same story.
Election Handed to republican by "Computer Glitch" by seamus Fri Nov 12th, 2004 at 11:06:25 PST (147 words)
Computer Glitch Changes Local Election Result In Favor Of Democratics by Newdemo12 Fri Nov 12th, 2004 at 10:32:08 PST (263 words)
- If a duplicate detector had been available, the author would have known the answer is yes.
Indiana is screwed up! has this been posted? by drummer55 Thu Nov 11th, 2004 at 23:24:01 PST (301 words)
Indiana machines count Dem votes as Libertarian by NewWay4NewDay Fri Nov 12th, 2004 at 19:45:13 PST (266 words)
- Duplicate post by the same author.
My God is bigger than their god by JollyBuddah Fri Nov 12th, 2004 at 01:44:45 PST (356 words)
My God is bigger than their god by JollyBuddah Fri Nov 12th, 2004 at 01:43:54 PST (356 words)
- Duplicate subject (mostly text of Kerry's speech).
The concession Kerry should have made by Spud1 Fri Nov 12th, 2004 at 12:03:12 PST (919 words)
The Best Concession Speech Ever by spartan68 Fri Nov 12th, 2004 at 07:23:15 PST (1024 words)
The method seems to work pretty well ... no false positives.