Searching Daily Kos
Quick Search Tips
Search: enter words you want to find in the target collection. By default only targets containing all the words will be returned. Capitalization is ignored. If you want to find several words in a precise order (a phrase) put double quotes around them as in "this is a phrase". Single quotes are not permitted, and are removed. Although "AND" is assumed by default, "OR" and "NOT" are supported, as follows.
Multiple query words are by default understood to be joined by "AND". Certain characters have to be "escaped" by placing a back slash in front of them; to find "/", use "\/", to find "=", use "\=". Complex queries can be constructed using parentheses and restricted queries. Restricted queries: limiting the parts of the document to be searched. The available options are described here. For both stories and comments (documents):
For stories only:
For comments only:
Find: In order to choose what kind of documents to find, use the pull down menu to select one of the following. Each option is identified as either a database search, meaning that the primary database is searched using Scoop directly, or index search, meaning that a specially constructed index of words contained in documents is searched using swish-e. There are three different indices, one for diaries (aka user diaries), one for stories (aka front page stories), and one for comments. User diaries that are elevated to the front page are found in both story and diary indices.
From: and To: pull down menus restrict the time interval of the search. The default interval is from one day ago to most recent. Most recent may be as much as 5 minutes out of date. Sort by: The order of presentation of results. Not all choices are relevant to all searches. Story and Diary Search sort choices.
Comment search sort choices shared with stories and diaries are "Relevance", "Title", and "Time", which are as described above. Those specific to comment are "Cid", "Nrec", and "Ntroll".
More example queries
For comments only Everything past the "#" is a comment, not part of the query!
Searching Daily Kos with election tags (etags)You can now search using election tags as follows. etag=CT-SEN Etags with embedded spaces should be quoted. etag="CA-LT. GOV" You can only use one etag per search. Good election tags have a two letter postal state abbreviation hyphen seperated from either a house district number, or a statewide office abbreviation (SEN, GOV, Lt.GOV, SOS, etc). There is one exception, etag="2006 ELECTION" also works. Good etags are ones we already know about by finding them in the tag list of an existing diary or by appearing in one of the election roundup diaries. The etag is used to build a more complicated query that looks for the tag in tags, title, or body text (stories/diaries) or in subject and comment text (comments). Certain meta dairies are excluded, including open threads, jotter's daily lists, and sidinny's election roundup, and the top comments diaries.
Software used.The search facility currently installed is based on the swish-e search engine: it is fast (compiled C code), open source, and flexible (you can feed it documents out of a database using your own custom preproccessor). It is intended for document collections numbering a million or less, which will do for Daily Kos with under half a million documents (excluding comments). In addition, swish-e offers many of features most desired, including search of phrases and search within configurable document areas (title, author, tags, URLs). What is indexed. All Daily Kos documents from the beginning to within 5 minutes of the present moment are indexed. Older documents are stored in static yearly indices while more recent documents are indexed and reindexed, frequently at first, then less frequently as they age, until they roll over into a growing, static, permenant yearly index. The right mix of number of index files and frequency of update to obtain the optimal combination of ease of maintenance and speed of search may require further experimentation. At present three different series of indices are maintained, corresponding to (in order of increasing total number of documents) front page stories, user diaries, and comments. For 2005 the number of stories, diaries, and comments was 5097, 88943, and 693224, respectively, containing 56993, 371578, and 2945916 distinct words. Design Considerations. The utility of search to a community of users is delivered not simply by the results of the text search, but most critically by the ordering of the results such that highest "value" hits are displayed first. (Most search researchers know that no one looks beyond about the first handful of results; almost no one looks beyond the first page.) The key to arranging for best to come first is to understand, have access to, and use the meta data associated with the text documents searched. Meta Data. At Daily Kos, we have a number of sorts of meta data that are relevant to documents. First among these is the time of posting. What is the latest? When did they say that? Who was first? That sort of thing. So the one thing search has to get right is time, in the sense of being up to date, that is including the latest diaries, as well as in the sense of permitting searches to be organized by reference to the present time. Making it up to date. Because recommendations are only open for 24 hours after a diary is published, it is important that search indices be refreshed sufficiently often during that interval to provide reasonably up to date information on total recommendations and comments. Currently, diaries up to 6 hours old are refreshed every 5 minutes, and diaries 6 to 24 hours old are refreshed every 15 minutes. More Meta Data. Other important meta data associated with stories and diaries includes comments and recommendations. The number of recommendations offers insight, imperfect, but useful, into the degree of approval a diary has received from the Daily Kos community. The number of comments indicates the size of the discussion engendered. The two numbers together give an interesting insight into a diaries history at Daily Kos. Some user diaries are promoted to the front page, and some diarists ("front pagers") are allowed to post directly to the front page. The latter are not open for recommendation. In order to harmonize listings from searches which include both front page stories and user diaries, a plausible value for number of recommendations (83) is assigned to front page stories, but in lists, this number is indicated by "*" to avoid placing too much emphasis on this made up number. The search query. Search queries can contain individual words, all of which must be found in a text document to constitute a hit. Queries can also contain phrases, indicated by double quotes around the phrase. All words are converted to lower case while searching, so letter case is of no consequence. In addition, queries can include boolean terms, such as "and", "or", and "not" which can be used to construct more complex and perhaps more informative results. By default "and" is assumed between individual words in a query. Finally, queries or parts of queries can be restricted to only search within parts of a document or document metadata. The parts available are shown above, along with examples. Certain characters have special meaning to the search engine (open and close paren, double quote, equals sign). Only a restricted list of characters are expected to occur in a word. For the indices here at daily kos, the word characters are as follows.
All other characters (non word, non special) are silently replaced by spaces while searching. For instance searching with "Jerome a Paris", "Jerome.a.Paris", or "Jerome,a,Paris" all give the same result. This is important in certain cases, for example searching for the url http://riverbendblogspot.com is accomplished with the query site=("riverbend.blogspot.com"). The reasonable looking but incorrect query site=riverbendblog.blogspot.com doesn't work as expected because in the absence of the double quotes and parentheses, replacement of "." with " " and the default boolean AND understood between search words makes it equivalent to site=riverbendblog AND blogspot AND com, which returns as hits only documents with riverbendblog in a url, and blogspot and com in the body text. More on Search
If you are intriqued by search and would
like to know more about the basics of how it is done, this is a
good introduction.
On Search. Basic basics. by Tim Bray. Since I've emphasized the importance of meta data, I'll also point to another article from the same series, On Search. SearchMeta, with some further thoughts on things meta. While I'm on the subject of underlying technologies, let me end this section by pointing to an article by Tim Bray on Ajax, the other new addition here at Daily Kos. In it, he also gives credit to Jesse James Garrett for the Ajax acronym in the process of thoroughly describing it.
Daily Kos is an amazing place, community, phenomenon. It has been my
privilege to have the opportunity to contribute.
-- jotter |
Recommended
- I make ZERO bones about the fact that I think the NYT is doing more harm to our democracy than any other news organization, and I include Fox News on that list. You, dear reader, who I would have to assume is a left-leaning voter that frequents...38 comments 116 Recs
- — — I made the decision to listen to the SCOTUS hearing yesterday. I had lots of emotions in reaction to it, but the overriding one had to do with a sense of dread. Followed by a nagging thought: “Is there any common sense left on the bench that Trump...204 comments 565 Recs
- Kristi Noem is the Governor of South Dakota. A leading candidate for Trump VP. And a woman who takes pleasure in killing things. To promote her candidacy, she will publish her obligatory autobiography, “No Going Back: The Truth on What’s Wrong with...394 comments 245 Recs
- Many Americans are confused by the spectacle they heard (we couldn’t “witness” it because Republicans on the Supreme Court won’t allow their proceedings to be televised) yesterday as an attorney for Donald Trump, at least three different times in...98 comments 272 Recs
- Ukraine sets fire to a Ministry of Defense helicopter in Moscow. Whoa. That’s close. The Russian who fell off before they hit the mine is thanking his lucky stars. Housekeeping is going to be pissed at the mess. Russian soldier discovers an effective...6 comments 43 Recs
- Reposted from Substack This week MAGA Media outlets have breathlessly declared that they’ve found “smoking gun” evidence that proves that the Biden Administration was in communication with ...16 comments 139 Recs
- All our fears to the contrary, Trump actually has little real chance of winning the upcoming election. The key is demographics. There are, of course, a number of factors that influence which voters vote for which candidates, and these factors can...79 comments 94 Recs
- If the Supreme Court thinks former presidents have, or do not have, some or absolute immunity against criminal prosecution, why not just say so now? Why would it take months? Do they have something better to do? Are there other priorities on their...304 comments 626 Recs
- Expect the Senate race to be close this fall—unless Wisconsinites find out what Eric Hovde thinks of them. Are you a farmer? Eric Hovde thinks your retirement age should be pushed up to 72 because after all, “you’re largely driving around on a tractor”...24 comments 143 Recs
- I am channeling my inner Andy Rooney for this VERY short diary, and am delighted that the Image Library is here to help yet again. Did ya ever notice how, right after the Democrats pull off something absolutely remarkable, this strange thing happens...417 comments 793 Recs
- Never in my long life have I been as enraged and sickened about American politics as I am now. I am close to utter despair at what I’m seeing. My beloved country is becoming an insane asylum, filled with people who want to see the American experiment...133 comments 287 Recs
- Happy Freidai, Peeps! Some of us donât have to sweat out a workweek, but even so, sometimes Freidai makes us want to dance. But what KIND of dance? But FIRST: A gentle reminder of how we do ...77 comments 79 Recs
Recent Diaries
- The language police have become mall cops. It’s been a week with police riots at colleges across the US: Emory, GOP photo-ops in NY, and everywhere there are Clint Eastwood�central administrators ...1 comments 0 Recs
- Got on of these in my mailbox the other day. It was addressed to somebody up my street I didn’t know, so I just put it back in my mailbox with the flag up for my mail carrier. Today, I got one addressed to me. This is some kind of weird anti-Catholic...1 comments 0 Recs
- Day 4 of the Trump criminal trial saw three different witnesses take the stand. -The judge rescheduled Trump’s contempt hearing from next Wednesday to Thursday. Still no ruling on the previously pending contempt motion. -Testimony continued with more...3 comments 11 Recs
- Commentary by Black Kos editor JoanMar I thought of Barack Obama as a once-in-a-generation politician. He epitomized everything I wanted to see in someone who’d seek public office. His speeches often moved me to tears; he was/is that inspiring. But...59 comments 18 Recs
- Ukraine sets fire to a Ministry of Defense helicopter in Moscow. Whoa. That’s close. The Russian who fell off before they hit the mine is thanking his lucky stars. Housekeeping is going to be pissed at the mess. Russian soldier discovers an effective...6 comments 43 Recs
- April 27, 2024 marks the 30th anniversary of the end of legalized racial oppression in South Africa — known officially as “apartheid.” However, most people still don’t know the name of an important anti-apartheid hero — a man named Dimitri Tsafendas;...1 comments 2 Recs
- Donald calls fascists "little peanuts", denies affairs, and asserts absolute immunity, whereas Kristi Noem, possible VP, exercises the lethargic ethics of animal abuse. Mitt Romney: "I strapped a dog to the roof of my car" Kristi Noem: "hold my beer" —...6 comments 10 Recs
- I make ZERO bones about the fact that I think the NYT is doing more harm to our democracy than any other news organization, and I include Fox News on that list. You, dear reader, who I would have to assume is a left-leaning voter that frequents...38 comments 116 Recs
- The demented drivel that emanates from Fox News on any given day is relentlessly ridiculous. They find fault with President Biden and Democrats on everything from the atrocity of eating an ice cream cone to the flagrantly false fear mongering about...20 comments 24 Recs
- Greetings and welcome to another of Marko's little hairballs. It's almost all fluff. There might be a hint of something seriously phlegmy between the pictures. All of it is just an excuse for ...23 comments 11 Recs
- It is this type of Republican that will make Democrats win big in 2024. We must engage them. This therapist uses her experience with the trauma we can extrapolate to our politics. This Republican is the 2024 template. Watch Politics Done Right T.V....1 comments 0 Recs
- Super-patriotic GOP candidate forgets words to the Pledge of Allegiance Watch him pledge his allegiance to … um … uh … Delusional Senate Republicans still believe they can control Trump What’s that definition of insanity again? Cartoon: A prickly...1 comments 3 Recs
- Main points: The startling thing about what the Supreme Court has done to help Trump avoid being tried for his most serious crimes against the Constitution before the election (if ever) is this: THE SUPREME COURT HAS SIDED WITH THE CRIMINAL AGAINST THE...1 comments 3 Recs
- Here in the Wolverine State, we’ve seen a remarkable turnaround. In 2016, we had a Republican Governor, Republican Attorney General, Republican Secretary of State, Republican-held Supreme Court, Republican-controlled State House and...1 comments 2 Recs
- Ironically, those who fail to learn from past mistakes are destined to repeat them—this overused and self-explanatory old saying has been repeated in various ways over the years. The Democratic National Convention will be held in Chicago from August...91 comments 8 Recs
- All our fears to the contrary, Trump actually has little real chance of winning the upcoming election. The key is demographics. There are, of course, a number of factors that influence which voters vote for which candidates, and these factors can...79 comments 94 Recs
- We can be sure of this well documented thing our “founding fathers” absolutely didn’t want to have a king It’s clear from their conversations and what they wrote they wanted a democracy and the people’s right to vote. But Trump sees things differently,...2 comments 14 Recs
- Yesterday the Supreme Court was hearing oral arguments on the Immunity case. Hypothetically? Theoretically! Really? lawyers and even SCOTUS judges along with my hero Rachael Maddow used these words today in reference to what giving Donald Trump...3 comments 6 Recs
- When a justice is nominated to the Supreme Court they are first confirmed by the Senate. During the confirmation process they frequently answer a Senator’s question on how they would rule on specific kinds of cases with “I can’t answer that because it...12 comments 29 Recs
- Happy Freidai, Peeps! Some of us donât have to sweat out a workweek, but even so, sometimes Freidai makes us want to dance. But what KIND of dance? But FIRST: A gentle reminder of how we do ...77 comments 79 Recs