Skip to main content

All right, we're busy coming up with the design requirements for what DK4 will have to do. But one of the biggest early decisions will be this: how should the brand-spanking-new backend be built? What language? What frameworks? What tools?

We've talked over various options, but we're going to throw it open to the community for discussion too.  Nothing can get as heated or as obscure as quickly as a bunch of techies talking tech. You think your Thanksgivings with GOP relatives get bad, try wandering into a room full of programmers and saying "Java sucks!" or "I just got a 2.18% speed increase in my data archiving cronjobs by converting them from Ruby to Python."

So a good old-fashioned technology free-for-all might be fun, here. More to the point, I'll have real-world consequences -- we're going to be influenced by your arguments.

If you don't know what the heck I'm talking about below, please ignore this thread. This one's just for the techies.

Language: Perl, Ruby, or Python?

Java is right out, because I hate it and that much friggin' typing screws with my carpal tunnel issues. PHP is right out, because of security and other considerations. The only three languages seriously being considered are Perl, Ruby, and Python.

As it turns out, people have strong preferences towards Perl, Ruby, or Python. And I mean intense feelings about it. This is probably because all three languages suck, as do all their language predecessors, but each language has a different philosophy about where it's important to suck the least.

So which should we use? That ought to be a good flame war, right there, but there's something just as important, and that's the web framework itself...


Core web framework is dictated by language choice, and vice versa. If you're using Ruby, you're almost certainly using Ruby on Rails. If you're using Python, you might be using Django or Zope. If you're using Perl, you're using mod_perl, maybe with one of the higher level frameworks available from CPAN that build on top of it (Catalyst, Mason, etc.)

So, which is better? Anyone use more than one of them and have a real-world opinion?

To prime the pump, I'll say I'm nervous about trying to do this site on Rails: I have little information on how well it will scale, and I'm not certain how many of their most-hyped features we'd just be immediately working around anyway (like their data abstraction stuff). These two concerns also apply to dozens upon dozens of other framework engines -- pretty much all of them, in fact, to one extent or another.

Note that we've actually got rather specific technical requirements, here, and they revolve almost exclusively around speed. The number of primary "pages" on Daily Kos is very few -- the frontpage, story page, etc. But they're always in memory, they're always changing, and they get hammered hard. Optimizing that behavior is far more important than prototyping them quickly.

Now, here's the rub. All of this might be moot, or at least secondary, if by some (remote) miracle we decided to use an existing...

Content Management System?

Is there an existing content management system powerful enough to use either out of the box, or with modifications? And one that can be so easily modified that we can be confident we'd still want to be using it a few years from now, with whatever new features we might dream up?

The entire reason we're not going to be going with Scoop with this one isn't because of anything Scoop inherently does wrong: it's because the site has already begun to diverge from what Scoop was originally intended for, and as it turns out, there's large chunks of it that now don't have much to do with Scoop at all.

I'll be honest on this one. I don't see much value in using an existing CMS, because this is a very specialized application, and because the amount of customization and new features we expect to spec into DK4 (including a lot of AJAX-based code, etc.) means that we'd be ripping up as much of a CMS, percentage-wise, as we would save. And that'd put us right back where we are with Scoop now -- using a tool designed for one thing, but slowly frankensteined into something else. But if you've got one in mind that has remarkable, amazing, awe-inspiring features already built in, then tell us about it.

So there you go. Suppose you're in charge of how the backend of Daily Kos 4.0 gets built. How would you do it?

Originally posted to Daily Kos on Wed Jan 03, 2007 at 03:34 PM PST.



Your favorite Web development language?

41%1122 votes
29%801 votes
29%806 votes

| 2730 votes | Vote | Results

Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags


More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Just get rid of these ridiculous tiny margins (3+ / 0-)

    and make a search engine that searches a lot better, and I'll be very very happy.

    "Think. It ain't illegal yet." - George Clinton

    by jbeach on Wed Jan 03, 2007 at 03:32:33 PM PST

    •  Searching is more key than you realize (5+ / 0-)
      Recommended by:
      jbeach, maop, 4jkb4ia, jcrit, Shapeshifter

      I'm not a language comparison guru, but conventional wisdom is that perl is the regex champ. If that's true, then it would seem to be a big plus given how many tasks in CMS revolve around regexes (regices?).

      On the other hand, perl is the least structured, so there would be a greater onus on the developer to write it in a disciplined manner vis a vis documentation and structure.

      Perl also has the biggest (I believe) library to draw upon, but the least consolidated web framework (compared to RoR and Zope). This lack of consolidation from the development community may be of no obstacle if so much of the CMS is being custom built from scratch and the framework is just needed to handle the meat and potatoes, e.g. form submission, database interaction (efficiency of this piece is a biggie though), file upload and parsing, etc.

      And if I laugh at any mortal thing, 'T is that I may not weep;

      by splinterbrain on Wed Jan 03, 2007 at 03:46:55 PM PST

      [ Parent ]

      •  Python has regexes too (1+ / 0-)
        Recommended by:

        They're just more tightly integrated into Perl than Python. You don't have to import a library to use them in Perl, for instance.

        I've never seen a regex problem that I couldn't solve in either language. Sometimes with a bit of work because I'm more familiar with Perl's regexes than Python's, but that's just a matter of familiarity.

      •  Regexes in Perl/Python/Ruby/etc are irrelevant (5+ / 0-)
        Recommended by:
        nepolon, maop, montpellier, unfounded, Croaky

        Regex's can be very useful in some cases (such as scanning diverse text file formats in scripts that manage Unix systems). However, in a typical n-tier architecture, the real work of indexing and searching data is done by a DBMS, and the regex implementations in Perl/Python/Ruby/etc are irrelevant.

        I've only seen one webapp that made heavy use of regex's... and I never, ever want to touch that code again.

        •  A Good Site (0+ / 0-)

          doesn't use either regex or the DBMS.  You need something like Lucene, the industry standard search engine system that powers most enterprise websites.  It gives you proper support for stop words (filters for commonly used words like and, or, am, to, it), stemming filters (search for "walking" matches "walk", "walked", "walking"), etc.

          Plus using a database for search places an unnecessary load on the DBMS which should be used for accessing data, rather than executing search queries.

          Ruby has a port for Lucene: Ferret.

          •  Not Really... (0+ / 0-)
            Lucene is designed to index and search static content. Think of a directory with 10,000 pages that need searched.

            Everything you mentioned: filtering common words, stemming, logical operators, etc, can all be done very well using an RDBMS. In fact, saying "a good site doesn't use DBMS" is a little silly.

            You say that a DB should be used for "accessing data" not "executing queries." How, exactly, do you think the data gets from the DB to the user? Executing a query is the only way to do that. And running a query like this:

            select * from table where id = 12345

            isn't much easier for the DB than

            select * from table where someField = "x" and someOther = "y" and someElse <> "p"

            Not to mention, all RDBMS's worth their salt have built in full-text-indexing services.

            •  Actually (1+ / 0-)
              Recommended by:

              Lucene, Swish-e, etc. are pretty good for dynamic sites; it's easy to set up their indexing processes as cron jobs which run multiple times a day, so the index stays relatively up-to-date and you get the benefit of using real, dedicated search code.

              •  Of course. (2+ / 0-)
                Recommended by:
                Croaky, figleef

                Of course this is possible: all pages--dynamic or otherwise--end up as HTML.

                But you really think this is the best solution?

                1. A crawler is written/bought/downloaded. It crawls the site. (Lucene is not a crawler itself)
                1. A link to a dynamic page is followed.
                1. DB queries get run, page gets assembled, HTML is served.
                1. Lucene is invoked to index it
                1. Lucene updates its index (a 'document based' database)
                1. Someone searches for "Bush Sucks"
                1. Lucene DB is queried, matches for Bush Sucks are returned as links.
                1. Link is followed, dynamic page is served to the user.

                And this is somehow better than:

                1. Every time a comment is added to the DB, the comment is indexed by the DBs full text service
                1. Someone searches "bush sucks"
                1. The DB query is run against the "comment" field, the full-text service is invoked automatically by the DBMS, results are returned.
                1. Link is followed, dynamic page is served to the user.
                •  It's very, very, very much better (2+ / 0-)
                  Recommended by:
                  klondike, jfm

                  Even with fulltext indexing, searching directly against the DB which stores your content will be slower than searching against a dedicated, optimized search store, and will drive up your number of database queries, which hurts your performance even more.

                  Compound search queries (such as "bush sucks" OR "i hate bush" AND "contribute to democrat campaign") make the situation even worse; dedicated search systems are optimized to handle them, but doing it in straight SQL against your content database means you're constructing and running lots of potentially complex queries (and, worse, queries which by their nature can't be pre-compiled using PREPARE), thus dragging your performance down even further.

                  Additionally, using fulltext indexing means that every DB write hits you for an INSERT or UPDATE and a re-generation of any applicable indexes. On a read-optimized setup which occasionally has flurries of DB writes due to explosions of diaries and comments after major events, that wouldn't be any fun at all.

                  •  for the specific case of dkos (1+ / 0-)
                    Recommended by:

                    and compulsive readers (not that we know any of those), where real-time updating is key, can you picture using a swish-e search engine when there's a window between the creation of content and the visibility of that content in the search-space?  

                    For example: I see a diary in the recent diaries list, I expect to catch it if I search on tags that include that diary.

                    Would you drive the swishe update process asynchronously by the availability of new content, or setup a periodic update, and would you not eventually scale to a point where index times grew so long that you'd have to violate requirements like the example above?

                    •  Surely this is an empirical question? (0+ / 0-)

                      Probably both approaches should be prototyped and profiled, right?

                      •  which always means a tough decision (1+ / 0-)
                        Recommended by:

                        on the managerial side, because questions of architecture can always be answered empirically given time and money.  kos has neither.  I'm guessing dk isn't wildly profitable, and I'm sure kos wants dk4 in time to have an effect on the '08 primaries.

                        I'm a startup guy, so the projects I work on typically lack either time (usually) or money (occasionally) as well.  Deciding which architectural questions need to be prototyped and profiled is always a bitch, and I almost never come away satisfied that I 'got it all'.

                    •  It's a tradeoff, really (1+ / 0-)
                      Recommended by:

                      In the case of tagging, I don't see an issue -- fetching "everything with tag 'foo'" is not what I really mean by search, and is usually a straightforward database query. "Search" means things like "find me anything which has the text 'foo' in its title or contents", and is better done by dedicated, optimized search facilities than by constructing complex queries which join all your tables and look at all their text fields. There still will be a lag between something going into the DB and something becoming accessible by search, but it's unavoidable; you can either kill your DB setup trying to have up-to-the-minute search, or you can have performance with a slight delay in getting fresh results.

                      •  or you could delay (0+ / 0-)

                        something becoming visible until it's indexed.  There are ways, certainly. With the penetration of XML into every corner of the industry it's certainly tempting to use xml flat files for storage if possible.

                        I'm kicking myself for not trying swish-e on the last project, but there were reasons, all of them bad.

                    •  just reviewed this thread (1+ / 0-)
                      Recommended by:

                      hope you come back and see this.

                      Actually I have no trouble picturing using swish-e here.  Because I did.  And the reason a new search facility was needed?  The built in search (sql driven) was broken; it just didn't work anymore due to the size of the database.

                      The problem of keeping everything up to date was solved by having multiple indices of varying ages, and rebuilding them less often the older they got.

                      For diaries and stories, everything up to six hours old is rebuilt every 5 minutes, everything up to a day old is rebuilt every hour, everything up to  three days old is rebuilt every six hours, everything 3 to 30 days old is rebuilt once a day, and at 30 days old (after which no further changes are allowed) everything is merged into a growing static index.  The schedule is different for comments, but the approach is similar.

                      I've gotten very few complaints about synchronization between what is published and what is indexed.

                      The speed of swish-e, along with the ability to specify multiple indices in a single search, made this approach possible.  A single server handles all indexing and query service, and has never been stressed due to query traffic (although there were some problems with indexing before I sorted out the optimal size for the growing static indices - merging is slow).

                      The main benefit of building indices is the abilty to roll in site specific meta data.  Without these, the search results would be much less useful.

                  •  Wow. I really disagree. (0+ / 0-)

                    First, "compound search queries" are hardly a DB bottleneck. It's just result filtering. Go run one of these queries in Query Analyzer and look at the profile. Very low cost.

                    Second, no matter whether you use Lucene or not you're DB is going to have indexes. These indexes are going to need to be updated whether or not your running search queries against the DB.

                    Third, You do realize that all that a dedicated search box is, is another DB Server?

                    Fourth, using your system there will either be a delay between DB update and search indexing OR there will be a huge overhead created by Lucene re-indexing in perpatuity.

                    Fifth, have you ever used full text indexing in SQL Server? There is no reason to think that it's any slower than a dedicated search product. In fact, I'd say it's likely that SQL Server is faster and more robust than an OSS search product.

                    Sixth, it's better to maintain one DB cluster than two (which is what you'd need in your system: production & search, both of which would probably need at least 2 boxes). One cluster would be cheaper in both hardware terms and administration.

                    Seventh, delayed writes solve your index building problem, even though I disagree that it'd be a problem at all.

                    •  A bit more detail (3+ / 0-)
                      Recommended by:
                      Hunter, klondike, jfm

                      The problem with compound, as pertains to search, is something I explained parenthetically: with normal queries you can use PREPARE with placeholders, and the DB will cache the compiled form of the query for you; then when you actually execute it and bind values into the placeholders, you get better performance (how much better varies, but on some DBs -- particularly Oracle -- it's an extremely significant jump).

                      Except you can't do that with search. Each query has to be constructed dynamically because you don't know exactly what's going to go into it; maybe User A wants to search diaries and comments, User B only wants to search diaries and User C only wants to search comments. All three have different combinations of "and" and "or" clauses in their search terms. There's no easy way to have optimized prepared statements for these cases, which means you're taking a performance hit compared to your normal, repetitive "fetch story 12345 and associated comments" style of query (which is easy to store as a prepared statement).

                      On indexing: yes, you're taking a hit either way, as I thought I'd made clear. Personally, I'd rather take the hit from a process running at known intervals than deal with what happens when a flurry of writes hit a read-optimized database. Also, moving to dedicated search means that your indexing process doesn't write to your DB; all it has to do is read.

                      Dedicated search doesn't mean "relational database". Often, you're querying highly-optimized flat files. Break free of the notion that everything has to be in an RDBMS at all times, and you'll see that there are some useful specialized tricks you can pull off.

                      Which is why good dedicated search is going to beat building the queries in SQL; dedicated search is optimized to the particular needs of search, but general-purpose relational databases can't specialize to the same extent and so will necessarily fall behind in the long run.

                      •  A bit more on indexing (1+ / 0-)
                        Recommended by:

                        One big misconception I have seen here about Lucene is the spider or batch job requirement. That is not the only way to use lucene. It is, in fact, plenty easy to send off your objects to lucene for indexing at the same time you save them to the database. Do it via a remote call to a lucene server (site seems big enough to warrant it), thread lucene a little, your users will not even notice the difference (and whatever you save will be indexed in a matter of seconds). That is exactly how I use lucene, it works wonderfully.

                        Most database searches for a site of this type will not make use of normal database indexes. Why? Wildcards! When a user wants to see articles with the word "clinton" in them, your query is going to have '%clinton%' in there as a parameter. Databases do sequential scans on wildcards. Why? Because they haven't tokenized the strings out to know exactly what is in each article. One could try solving by making an index table that relates words to articles (some php bulletin boards do it!), but now they are recreating a much slower, less featured, lucene. Why do that?

                        Compared to a database, lucene stores all that information in such a way that it can perform a quick bitwise search across its index for the term. It will pull up those thousands of articles, comments, etc. about clinton in no time. Here, lucene is able to use its index even when wildcards exist. In fact, if you store the subject & the first few lines of the article in lucene, you could get away with not even hitting the database on a search! That's going to improve site performance a bunch (let the db stay busy pulling down all those articles & comments when the users want them).

                        Beyond all that, lucene has a great many features you just can't do easily in a database. Take a look here.

                        Lucene is what I use & know, but just about any full text searching system is going to be vastly superior to database queries on a site this size with searching of this manner.

                        •  that was the obvious issue I saw with (0+ / 0-)

                          lucene/swish-e - the potential window between saving and indexing.  Outstanding comment and I hope Hunter is still listening.  I think this potentially has major benefits for dkos.  

                        •  tsearch2 for PostgreSQL does full-text indexing (1+ / 0-)
                          Recommended by:

                          And it works quite well. I've installed it and even played with the code a bit.  It's quite fast, supports complex queries (uses multiple indexes and allows boolean logic AND/OR/NOT), and appears as a condition in a standard SQL query.  It's essentially a GiST index with some custom procedures that you load into the database.  I don't profess to know much about Lucene for comparison purposes, but tsearch2 is an example of indexing in an SQL database that does exactly what you suggest cannot be done.

                          •  but turn that response inside out (1+ / 0-)
                            Recommended by:

                            and ask yourself what you get from storing lumps of markup-language in an RDB that makes it the right solution, not just doable?  

                            A relational database's raison d'etre is relational integrity and I'd suggest that once you're talking about how to physically store and search lumps of markup you're outside the realm of things that RDB's were designed to do.  Any RDB solution to that problem merits consideration only on an even footing with things like lucene, not with a presumption of superiority or even adequacy.  

                            I suspect that if I dug deep in the dkos requirements, I would see a very bright line between the obviously relational data in dkos (hotlists, subscriptions, account info) and the lumps of markup that make up the text of the site and no compelling reason not to apply the best tool on each side of that line.

                          •  what you get is more complex queries (1+ / 0-)
                            Recommended by:

                            I agree with what you're saying about using the right tools for the right tasks.  The obvious benefit that I see is the ability to use specialized full-text indexes in combination with relational data. For example, how else would you do a search for words A, B, C in the diaries of users X, Y, Z if your tools are isolated?

                            To the extent that Lucene supports fields, it also is trying to do some minimal db-like organization, and so they are trying to achieve the same goal of data access and search.  They should be compared on the merits (speed, flexibility, scalability, compatibility, etc.).  I don't think you can write off either without addressing which is better for this site's configuration and needs.

                          •  that's the implicit argument of the SQL fans (0+ / 0-)

                            the presumably tighter integration using an RDB, and the presumbaly clunkier integration using an external tool.

                          •  I would consider tsearch2 (1+ / 0-)
                            Recommended by:

                            A viable indexing option in the category of "any full text indexing." I believe mysql has a few full text addons as well.

                            It is an addon to a db, not a standard db thing though, so I don't consider it part of the sql query solution some were suggesting here. :)

                            I don't have experience with tsearch2 to compare it with lucene. However, it may need to run on the same machine as postgresql (which I think it does from some cursory overview I gave it a while back, apologies if I am wrong). One should be careful to not overload a single machine. For a site of this size, I would actually heavily recommend running separate db & indexing servers. Or at least leave that option open.

                            That said, tsearch2 would still be much better than straight sql-standard searching.

                        •  I know you can do that... (0+ / 0-)

                          But in such a performance-critical situation I do wonder about the impact. Adding even a tiny amount of overhead per request, when that overhead could be jobbed out to a process running at predictable times, is something to weigh carefully before making a decision.

                          That said, I'm glad to see someone else understands that search-optimized systems are a good idea for this sort of thing :)

      •  CW? (0+ / 0-)

        There are Regex implementations in basically every modern language. Regular Expression engines are not difficult to write. Perl holds no upper hand on regular expressions. It does, however, have special features that other implementations didn't always have. This is why you can find a "perl compatible" reg-ex package for basically every mainstream language.

        It just so happens that Perl was first. That's the only 'upper hand' they have.

        •  I should add (0+ / 0-)

          That regular expressions have basically nothing to do with executing search queries.

          RegEx's are used for things like text processing and manipulation. Like "extract the customer ID from these 100,000 flat text files"

          Using a Reg-Ex to search a site like DKos would be a hilarious folly.

          •  Regex is so 90's (4+ / 0-)
            Recommended by:
            Curt Matlock, montpellier, Uniter, maddogg

            Definitely.  The reality is that when people were hacking together websites from the ground up back in the late 90's, perl/CGI was the way to do it because most of us were working off flat text files, etc.  I've hacked together more than a few sites that way and I'm glad I haven't had to do it in a very long time.

            Now, having said that, I think throwing Java out from the get go is a bad choice.  I've been writing Java code for going on 10 years now and I like Java for two major reasons:

            1. It's structure leads to code that is cleaner and easier to maintain.  If you're just going to write a quick site, yeah it's a lot of overhead.  If you're going to put together something big and robust that multiple people will work on in the future, it's very handy
            2. There are a lot of excellent frameworks and 3rd party libraries that really do a lot to simplify your work.  Off hand, consider Struts, Hibernate, and my new recent favorite, DWR, which makes Ajax coding a crazy simple.  

            I know I'm wasting my breath here, but I think it's kind of silly to throw out Java out of hand because it involves a lot of typing.  While arguably Java might take more work up front (arguably because of my commentary on the various frameworks), it will make for a system that's easier to manage in the long term, IMHO.

      •  absurd (0+ / 0-)

        To perform Regex evals in Perl/Python/Ruby, you have to load data into memory.  You cannot load the DKox DB into memory.


        Why does Shrub never talk about his first wife, Reality? He divorced her with prejudice and all of his alimony checks to her have bounced.

        by nepolon on Thu Jan 04, 2007 at 11:32:34 AM PST

        [ Parent ]

    •  Can I bitch about the margins some more? (2+ / 0-)
      Recommended by:
      mik, kurt

      I mean, look at either side of this page. All that wasted space that means users have to scroll down further to read the same amount of text. There's no point to it, it doesn't look good, it sucks, it goes contrary to the whole concept of enabling easy flow of information, but besides that it's awesome.

      "Think. It ain't illegal yet." - George Clinton

      by jbeach on Wed Jan 03, 2007 at 04:01:43 PM PST

      [ Parent ]

      •  This is why you don't leave UI design to amateurs (10+ / 0-)

        Pick up any (English or Romance language) newspaper you care to name.  Look at its layout.  Do you see text written in a single column as broad as the page is wide?  I didn't think so.  You see text broken into four, five, or even six columns on the page.  In a print or magazine, with a narrower standard page, you'll see two or three columns, maybe four.

        That's not just so pictures can be more easily interlaced into the text, although that's a nice side benefit.  Professional print media lay out text in thin columns because people read text better that way.  Laying out a webpage in thin columns makes it more readable, not less.

        this message is intended to inform. any annoyance, abuse, threat, or harassment is solely in the perception of the reader, not the intention of the poster.

        by horsewithnoname on Wed Jan 03, 2007 at 04:17:25 PM PST

        [ Parent ]

        •  Ahem - Print design is not web design (9+ / 0-)

          So your comment is off to begin with.

          In fact, that comment shows why you don't leave WEB design to PRINT designers. In the web field, print designers become - dare I say it - amateurs. And have a very hard time admitting their lack of knowledge.

          Let's look at a couple websites that are known for their web-only content:

 - has margins for the text, because on either side of the text there's STUFF THERE. Links, polls, etc.

          Not like I love this site for its content, that's why I'm here - but on our site, we have margins in addition to the stuff on either side of the text. Again, for no good reason.

          On Slashdot, even the columns go away when you reach an interior page  -

          See? No margins on the right side, now. So people can read more of the information without scrolling. That's because the web, unlike newspapers, is read on a different interface, and senselessly introducing extra scolling because an interface issue.

          But what does Slashdot know? They've only been one of the most-visited websites on the Internet since 1998 or so.

          Another very popular web-only site -

          Notice the lack of useless margins?

          Finally, I'll note that your rude assumption I'm an amateur is incorrect - I've been doing web work for a living since 1996. All phases, design to programming to graphics production.

          "Think. It ain't illegal yet." - George Clinton

          by jbeach on Wed Jan 03, 2007 at 04:36:44 PM PST

          [ Parent ]

          •  The fact that a site has web only content (5+ / 0-)

            ...does not mean that it's a good example to emulate. slashdot is arguably one of the most miserable sites extent so far as usability standards are concerned.

            Print design cues do matter here, because print media long ago discovered that people can more easily read in narrow columns. The eye tracks better from line to line when both ends of that line are immediately observable. If the other end of that line is on the other side of the screen, it's very easy to lose track of which line you were reading when you scan back to the left to begin the next line.

            Highly usable online media sites give preeminence and priority to readability, and then wrap all their stuff around it... hence the "lack of useless margins" you see.

            I, like you, have been making a (rather successful) living as a web architect since the commercialization of the web, and a usability wonk and human/computer interface pro long before that. And while I won't stoop to characterize you as an amateur, I will suggest that you're a generalist; and issues of usability and user experience are better left to specialists. ;)

            "If you're looking for friends when you need's too late." -- Mark Twain

            by Riddle on Wed Jan 03, 2007 at 09:36:21 PM PST

            [ Parent ]

            •  I agree, (0+ / 0-)

              when two or more Kossacks are really going at it, such as in hidden comment troll and troll hunter interchanges, where they are both up against the right wall, it is easy to read, for exactly the reason you say.  You cannot lose track of what line you are on.

              However, I sometimes hate newspaper columns when they justify the crap out of words, and spread some words way out to touch both margins.

              But we do not have that problem here.  Very pretty print.

        •  Break it into separate columns, then (2+ / 0-)
          Recommended by:
          kurt, BruceMcF

          like does.

          You can still have the text in tighter colums, without wasting all those pages-worth of space on either side of the text.

          They still do the useless-margin thing somewhat, but much less, and it's much less obnoxious because of it.

          personally I think breaking text into columns on a website is distracting and annoying, because people read computers differently than they do newspapers. Including folding the newspapers to blank out other columns.

          "Think. It ain't illegal yet." - George Clinton

          by jbeach on Wed Jan 03, 2007 at 04:42:46 PM PST

          [ Parent ]

          •  How many BOOKS have multiple columns? (2+ / 0-)
            Recommended by:
            jbeach, kurt
            Newspapers have multiple columns because they're usually on very wide pages.  It's best to have around 10 words per line.
            •  Wide books have multiple columns (0+ / 0-)

              Textbooks often have two to three columns.  Bound academic journals likewise.  Coffee-table books and similar oversize books, two to four.  Cheap paperbacks are single-column only because the size of the printed area on the page is constrained to keep the single column about a dozen words wide.

              this message is intended to inform. any annoyance, abuse, threat, or harassment is solely in the perception of the reader, not the intention of the poster.

              by horsewithnoname on Thu Jan 04, 2007 at 09:22:48 AM PST

              [ Parent ]

          •  small type in a wide column is unreadible (3+ / 0-)

            All your examples have columns about as wide as the one here. They just cram more stuff in the side columns, but I take our comment to be about the middle column. In fact, if you scroll down on the side stuff disappears and you have a narrow column of text with miles of white space on the sides.

            I don't mind the extra white space on he side of teh main column here, it just makes it easier to read.

        •  That's not why newspapers use skinny columns... (2+ / 0-)
          Recommended by:
          Luetta, bigjacbigjacbigjac

          Print and web design is different and one way they differ is that lots of little columns are difficult to read on a web page and creates needless scrolling and eye placement. Most professional web designers that I have worked with or read on the web agree with this principle--it's considered fundamental in these parts. Further, I completely agree with jbeach on this issue.

          You have made a false assumption about the origins of newspaper layouts, the skinny columns were not created because it was easy to read! It had to do with typeset holder length (racks that held the type), a desire to have variety of subjects presented above the fold, and a few other obscure reasons. But that's a whole different discussion which I will leave to others.

          While Daily Kos has prioritized many site issues over strictly design considerations, it is truly a beautifully designed site! It's clean, simple, and intelligently designed (it's a little that).

          The only design issue I have is that it bugs me is that the banner ads at the top don't go all the way to the left or right of the screen (they centered and just hover on top). I'd like to see them inserted into a frame or something (an orange rectangle?) so that they are integrated into the masthead or otherwise anchored.

        •  horsewithnoname is right (2+ / 0-)
          Recommended by:
          montpellier, bigjacbigjacbigjac

          when you're dealing with a giant mass of print, wide margins make it easier to lose your place from one line to the next, and a lot (lot) harder to speed-read.

          I'd rather have to scroll down (oh god, it hurts) than decrease my information intake rate (that is what jbeach was originally talking about). When I read blogs I open a sidebar or narrow the window to make it more column-like.

          Plus narrow it encourages.



          I tell you truly, whatever you did for the least of these brothers of mine, you did for me. -- Matthew 25:40

          by mSnook on Thu Jan 04, 2007 at 01:00:09 AM PST

          [ Parent ]

        •  Hahaha (1+ / 0-)
          Recommended by:

          and this is why you don't leave web UI design to print designers.  Imagine a webpage laid out like a newspaper.  Talk about scrolling hell!

        •  No, no, no. (2+ / 0-)
          Recommended by:
          jbeach, Bill Evans at Mariposa

          Have you ever tried reading IRS instruction PDFs on screen?  They use a multiple-column layout, and you wind up spending more time scrolling up and down and trying to figure out where you left off than you spend actually reading.

          Columns are good for print, but not for display.

      •  Wrong (1+ / 0-)
        Recommended by:

        I'll take readable (NARROW) text, thanks.

        •  Fine, have that option if you like it. (3+ / 0-)
          Recommended by:
          nasarius, jcrit, kurt

          I'd be fine with that as an option that could be set.

          Skins, even, would be really really cool. user-defined, uploadable templates would be an amazing thing. That way we could all be as right as we want.

          But this current lack-of-choice between narrow margins, and slightly-less-narrow margins, is IMHO for crap.

          "Think. It ain't illegal yet." - George Clinton

          by jbeach on Wed Jan 03, 2007 at 04:38:55 PM PST

          [ Parent ]

          •  Skins, yes, skins!!! (0+ / 0-)

            I have never understood why Web sites restrict the reader to one or two layouts. With CSS and similar techniques it is so frickin' easy to empower the user to make these kinds of decisions - margins vs. no margins, wide vs. narrow, single vs. multiple column, dark text on light background vs. light text on dark.

            Give the (layout) power to the people! ;-)

      •  Margins should be dynamic. (6+ / 0-)

        I consider that more important than having them match the width of the banner image :)

        I currently run 1600x1200 resolution and will at some point in the future buy a shiny new lcd monitor with 1920x1200 resolution and will then run that. I am going to have a different margin preference than someone on a corporate machine at 800x600 or--worse--640x480 resolution. (Do such unfortunate beings still exist in this new, neon age?)

        If not actually dynamic, there should at lest be an option to set your "DKos resolution". That is, set what size of DKos you want rendered with a decent selection to choose from. (Ie, a dropdown box with 640x480, 800x60, 1024x768, 1280x1024, 1600x1200 selectable and then styles that will display DKos with those a display area of that size in mind.)

        The Shapeshifter's Blog -- Politics, Philosophy, and Madness!

        by Shapeshifter on Wed Jan 03, 2007 at 05:52:36 PM PST

        [ Parent ]

        •  Margins (0+ / 0-)

          I agree with the width arguments.  wide columns are difficult to read.  However, since you are writing from scratch, you might consider especially with comment threads, putting "relevant" material links or links with snippets to the right of articles.  (This is a google philosophy type argument, in the sense of assume that processor power etc will be cheaper in the near future and design for what the cost it will be when your app is in late maturity.)   Two examples,

          1. right click in windows usually has choices that are useful depending on the object, make the margins have links to those useful things for whatever is the subject in the main column. make this tailorable in individual users profiles, this will give you measurements on how successful/widely used various option are. look at how gmail provides opening phrases of messages, so that people can tell if they want to read it.
          1.  Allow people to make "correlations" of this link is like that link.  (see also )  that would allow people to link up stories/diaries on similar topics, or diaries in a series.

          Marc In order to understand recursion, you must first understand recursion

          by msobel on Thu Jan 04, 2007 at 07:39:39 AM PST

          [ Parent ]

      •  Margins (0+ / 0-)

        What you ask for is non-trivial.

    •  I think your choices suck! (2+ / 0-)
      Recommended by:
      Neo Leftist, Arclite

      As much as everyone bitches about windows, its got billions of dollars behind its success.  As for scaling, nothing beats .Net 2.0 (and 3.0 is right around the corner).  Speed, I'll admit its a little slow on the draw, but just a little. What it lacks in raw speed, it makes up for in efficiency.  As for security, you don't get better than a windows server.  You have that break down its cause you coded wrong.

      As for existing CMS's, because I'm partial to .Net, I like DNN.  But if you want an opinion aboutt he next best, its E107, which is PHP.  Security is always an issue with PHP, but then it depends alot on how you code.

      Ruby is a great choice though if you want alot of coding to do... and for for a site like this you will. But don't worry, I wont tell you that you can code the same results, better, and with half the lines with a .Net framework. Ha!

      Just cause Microsoft did suck, and also because its terribly corporate, doesn't mean they suck now.  they are getting their shit together.  You should think about the benefits of a windows solution.

      •  Microsoft / DNN etc (2+ / 0-)
        Recommended by:
        mimi9, Soul

        I was wondering when someone would jump in with .NET.  I use it and program with it.

        I also have several internal and Internet sites running on DNN and I gotta say, the speed thing is a major issue on the sites I work with.  Would probably be a disaster on DKOS.

        •  I don't think so... (0+ / 0-)

          The new version of DNN uses compression now, and that increases speed dramatically. Plus they have new filters for condensing code itself, such as the whitespace filter.

          I think with a site this size, the designers of DNN would probably chip in on a project like this. I think the next version of .Net will solve alot of the speed problems (which are becuase of all the security and efficiency built in).

          •  .NET is fast, DNN is nice, but... DNN is SLOW. (0+ / 0-)

            I think .Net 2.0 is a great way to go.  It's blazing fast and scales well.  

            I am also a DNN fan, but IMHO this site is not a good application for DNN.  DNN is a great all-purpose portal framework, but it's simply not fast enough, and would be quickly buried by the kind of volume the DKOS site gets.

            •  Alot of that depends... (0+ / 0-)

              Not so much on DNN, but on the servers that run it. I wouldn't undersell DNN. But you're probably right about it not being right for DKOS. Its core has all sorts of things that aren't really necessary. I think DNN can be really fast, but its weight is probably more than is needed here. But .Net is really the way to go with a site like this.

        •  I'm no longer using DNN (0+ / 0-)

          It offered something nice back with .NET 1.1, mainly a nice easy CMS type environment where you could maintain a standard page layout and add content.

          But with the introduction of Master pages in .NET 2.0, there is very little point.  It adds a lot of overhead.

          Although it is also nice in that it does a great deal for you.  The big problem is, unless you are building a specific type of website the navigation patterns are kind of horked.

      •  .net more consise than Ruby? (0+ / 0-)

        Is that so? Got some code comparisons to back it up? Last i checked, you have to get into obscure functional languages to beat Ruby.

        (If you're just saying ".net's frameworks are more extensive than Ruby's frameworks then this is not necessarily going to be true for DK4, although it might be. I think it is a little less straightforward than you have put it, however.)

        The Shapeshifter's Blog -- Politics, Philosophy, and Madness!

        by Shapeshifter on Wed Jan 03, 2007 at 06:45:40 PM PST

        [ Parent ]

        •  It is less straightforward... (0+ / 0-)

          I was referring to the shear mass of prewritten code in .net as opposed to Ruby's, which means that there is more to code in Ruby ultimately. As for conciseness, that's a tough one. Ruby is definitely more concise that Perl, but I think that in most cases .Net runs cleaner.

          But then, I really don't know. I haven't had much practice in anything other than .Net and PHP. So I may be wrong about the comparisons. Ultimately, in the long run though, and for many reasons, think that a .Net is better.

        •  Concise? No. (1+ / 0-)
          Recommended by:

          I've looked at Ruby, and I'm not sure I quite see the appeal for it.  There's some nice stuff in Rails, mainly with the data access and scaffolding.  But a lot of the "cool" things people speak of in RoR we've had in .NET for years with the databound controls.

          I've been experimenting with a project called Subsonic for ASP.NET which is an attempt to implement much of the Rails data access and scaffolding for .NET.  It's nice in many ways, and I think it's going to work exceptionally well for the project I have at home.  But in my other role, I like well defined objects and that's not the solution.  That's where you get into ORM wrappers such as NHibernate(which is a .NET implementation of the Java Hibernate) and so forth.  There you can map your objects to a database for persistence, so your database models your app, not the other way around like RoR.

          I'm not a fan of languages with concise syntax just for the sake of it.  I don't find it to be all that readable.  This is perhaps why I'm also not a fan of lisp based or functional based languages.  I recognize that these have advantages for certain circumstances, but if it's difficult to read it's difficult to maintain.  Now, understand, I work under the principle of "I don't want to be doing this shit forever", so I try to build solutions I can hand off to someone else so i can move on to another job.

      •  And I, yours (0+ / 0-)

        As for scaling, nothing beats .Net 2.0 (and 3.0 is right around the corner).  Speed, I'll admit its a little slow on the draw, but just a little.

        Aren't these two statements more than a bit contradictory of each other?


        "If you want to live like a Republican, then you better vote for a Demcrat" - Harry S. Truman

        by grndrush on Wed Jan 03, 2007 at 11:24:31 PM PST

        [ Parent ]

      •  Wow you have no idea what you're talking about (3+ / 0-)
        Recommended by:
        jbeach, montpellier, jfm

        Microsoft is a scourge upon the Web as we know it.  No website and I mean NO WEBSITE!! worth anything out there uses Microsoft crap!

        And "Ruby is a great choice though if you want a lot of coding to do"?  Spoken like a typical person who has never tried programming in Ruby.  Ruby's a metaprogrammer's paradise and the level of productivity you get out of Ruby programmer will run circles around a Microsoft .Netter.

        •  almost true (0+ / 0-)

          ... but, the C# talent in the marketplace has changed that dramatically.

          When consulting on .net programming hires, I attempt to sway the language of choice away from VB.NET to C# (not always successfully, mind you).

          The typical thought is that there are a ton of VB programmers, so there is a lower cost factor for talent ... but, cheaper is not always better.

          While i am not a huge Microsoft corporate fan, they did provide the industry a kind of double-edge sword many years back. VB, from its onset, really kicked into high gear the number of people getting interested in coding ... that was the good ... the bad is that the skill set of a good chunk (not all) of them is amature at best.

          C# is OO and its syntax is derived from C\C++\Java.  It was designed from the ground up by Hejlsberg (the original Borland Delphi & Turbo Pascal constructor), that alone provides me with a comfort level & respect for the language.

          In the market, we see a much more solid set of C# programmers with good foundations in OO, many crossovers from C\C++ & Java.  They are typically very productive.

          It really is all about the underlying skill set of the programmer, not so much the language.  A solid programmer should be able to pick up any syntax & roll with it.

          I often like to make a musical analogy on this topic, put a great drummer in front of any set of traps & he\she can rock the house ... it is all about the rudiments, not just the tools.

          There are both leaders & followers in this world ... which will you be? --- My Mom.

          by DBW21 on Thu Jan 04, 2007 at 04:26:14 AM PST

          [ Parent ]

        •  Ruby is a cheap clone of Perl... (0+ / 0-)

          And your attitude about Microsoft is naive and typical of a programmer stuck in the old ways and incapable of creating solutions worth a damn. "Metaprogrammer," what the hell is that? You must be one of those programmers that learned from "Ruby for Dummies" on Amazon's worst seller list.

      •  not a programmer really, however (2+ / 0-)
        Recommended by:
        jbeach, jfm

        this is an outrageously inaccurate statement

        As for security, you don't get better than a windows server.

        I wont get into the "which os is more secure?" flamewar (though I personally prefer LInux or Unix) - because like a car, the driver is probably the most important factor in security. Yeah, a volvo has more safety features, but if you smash it into a wall at 50 mph you are probably still gonna kill yourself.

        A more critical factor is: how well is the machine adminstered? have decent measures been taken?

        •  I agree... (1+ / 0-)
          Recommended by:

          It mainly depends on the driver... I just think the latest windows servers are more secure (out the box) for those drivers that cannot drive properly. I don't discredit the utility of a unix server either. I guess its a matter of taste.

          •  And I totally disagree with the assessment (2+ / 0-)
            Recommended by:
            jbeach, jfm

            of windows. Having had many more  hacking experiences with windows servers than Linux in the past few years (all last few years hacks were win 2003 & win 2000, though we have equal numbers of both win and li/unix) I totally disagree.

            In the past I would have said Linux wins hands down.

            The problem is "out of box" is a useless metric. a production server should never be "out of box" -

            but actually this is all irrelevant, because kos will never run windoze.

            •  You're right.. (0+ / 0-)

              It will never run on Windows... and I agree that Unix was better in the past. I'm really starting to think that Microsoft has got the hint though. I also agree with your complaint about OUT THE BOX. Any server must be fined tuned to a variety of things.

              Don't get me wrong, I still am not sold completely on Microsoft getting their shit straight. They've made alot of improvements. Hell, look at Explorer 2007. They actually made their browser compatable with Firefox (although I don't see myself using anything but Firefox).

              My only claim here is that a windows solutions should not be discounted for DKOS simply on the basis of "Microsoft Sucks." The better question is does it suck now? Will it suck two years from now in comparison with Unix? A good developer will examine all solutions. I don't see a serious examination of that here.

      •  quote of the day (2+ / 0-)
        Recommended by:
        jbeach, montpellier

        As for security, you don't get better than a windows server.

        This made tea (Prince of Wales, cream, 2 sugars) come out of my nose.

    •  ruby won't scale, perl is too ugly, use python (3+ / 0-)
      Recommended by:
      Jer, jotter, jfm

      You should definitely use Python and the Django framework.

      You are right to worry about Ruby and scalability. Django, on the other hand, runs parts of the Washington Post's web site. It can scale. And it can do everything Ruby on Rails can do, with greater flexibility.

      And Python has less 'black magic' than Ruby or Perl. If you have to bring in extra web developers, its easy to get them up to speed.

      One caveat... Python has a global interpreter lock. This means that you might have issues if you try to do multithreaded applications outside of the Django or Twisted framework.

      But if you try to implement that kind of architecture, you'd better be using Java or .NET. You're beyond what scripting languages can reasonably provide.

      Of COURSE you can trust the government! Just ask the Indians.

      by bex on Wed Jan 03, 2007 at 07:33:46 PM PST

      [ Parent ]

      •  def twocents(ruby,python): (1+ / 0-)
        Recommended by:

        I just recently redid my homepage using Ruby on Rails.  It only took me about a day from knowing nothing to having a working prototype.  That said, the Ruby language itself is not known for speed.  I don't think this will matter for my little personal site but it would kill dKos.  See this python/ruby benchmark for comparison.  That site is good for language comparisons in general.

        Python is closer to what kos will need and there's a lot of fascinating work being done on the language core notably PyPy which promises to have applications outside the Python world too.  Perl is faster still but the code is notorious for being unmaintainable.  Avoid Java like hot flaming death.  Strong typing is silly for RAD IMHO.

        Then did he raise on high the Holy Hand Grenade of Antioch, saying, "Bless this, O Lord, that with it thou mayst blow thine enemies to tiny bits, in thy mercy."

        by Event Horizon on Thu Jan 04, 2007 at 12:16:49 AM PST

        [ Parent ]

        •  Use Memcache (1+ / 0-)
          Recommended by:
          Event Horizon

          Memcache is what Livejournal developed to store cached objects in memory for fast access and reduction in application and database load.

          The only people who say Ruby won't scale are people who have never developed a site in Ruby.  There are more than enough examples of big, heavy traffic websites out there running off Ruby that have no problem whatsoever handling their load.

          Not to mention the fact that a new virtual machine for Ruby has been developed and was just merged into the trunk of the core.  It should be a matter of months now before it comes out in the next stable release.  With this change, Ruby now benchmarks at or better than Python.  It's now 2-10 times faster than the current stable version.

          •  Because (3+ / 0-)
            Recommended by:
            Event Horizon, Soul, jfm
            Because it adds insane amounts of processor load, overhead and complexity to the server.

            Because it's terrible to debug when there's a problem in your complex web application.

            And for all this cost, it's not even an especially good language.  It forces very strict object orientation without really giving you the OO features of, say, Python.

          •  Productivity (2+ / 0-)
            Recommended by:
            klondike, jfm

            I know the arguments about scalability and so on.  cjohnson makes a good case in this thread.  In practice the language is too strict about typing and the compile phase slows you down.  The CLASSPATH insanity is also pernicious.  I worked in a Java shop until a couple years ago.  We all thought Java was great and it was compared to the ad hoc assemblage of poorly written Perl scripts we were using.  In practice we got very little done with it due to all the admin overhead and the issues I just mentioned.  It also ended up costing a fortune in IT salaries and support calls to Sun. I don't know if Markos really has the dough for that.  In the end they junked Java for .NET which turned into a boondoggle of a slightly different kind but for the basic reasons.

            Then did he raise on high the Holy Hand Grenade of Antioch, saying, "Bless this, O Lord, that with it thou mayst blow thine enemies to tiny bits, in thy mercy."

            by Event Horizon on Thu Jan 04, 2007 at 01:57:17 PM PST

            [ Parent ]

      •  Perl is beautiful! (2+ / 0-)
        Recommended by:
        EightBall, montpellier

        Perl's not ugly, at least once you understand it. The biggest problem with perl is that it's sooo easy for someone who knows another language, or even just how to use the cli, to hop right in and get real work done. This means you get lots of people writing c programs in perl, or bash scripts in perl, etc.  And that's a great thing, because those people got their work done.  Perl is the people's language! Non-l33ts welcome.

        However, it's always better to write perl programs when you're using perl. No, c programmers won't understand the idioms as well, but perl programmers will be much more efficient.  

        Of course the argument from python people is that there is only one right way to do it. It's just not true, and I've waded through tons of crappy python programs where a programmer didn't follow the advice I gave about perl earlier: write in the language you're using. It is entirely possibly to write shitty code in any language.  Don't let fear of punctuation keep you from the joys of perl.

        After that sermon, my vote would go for mod_perl, probably with mason on top. I've never used mason on a high traffic site, but I've worked on mod_perl sites busier than kos. It will merrily handle what you throw at it.

      •  Ruby scalability, Mongrel (0+ / 0-)

        Scalability in Ruby was more of a issue a few months ago than it is now. If you use the new Mongrel HTTP library, rather than the FastCGI that you would have to use a few months ago, you'll be better off.

        Python does have a good track record, but you have to like the language. I found switching from Java and Perl to Ruby to be a very comfortable transition.

  •  Sorry, but... (19+ / 0-)

    ... if your technical level is "Java sucks", I'm afraid I have nothing to add. Continue to use any of those other antiquated technologies that are within your powers.

    Java isn't my favorite language over the last 25 years either, but I long ago learned that "religious fanatics" whose opinions boil down to "xyz sucks" can be dismissed from technical discussions.

    •  It's true; big, big sites use Java and Struts (7+ / 0-)

      And do just fine. People know what they like, I'm sure. Just sayin'...

      "Think. It ain't illegal yet." - George Clinton

      by jbeach on Wed Jan 03, 2007 at 03:40:30 PM PST

      [ Parent ]

      •  SoapBlox is a whole blog framework (3+ / 0-)
        Recommended by:
        bolson, MarkInSanFran, johne

        built on Java and Struts!

        I love Java.

        Don't start a blog, build a community with SoapBlox - the NEW blog framework.

        by pacified on Wed Jan 03, 2007 at 03:41:32 PM PST

        [ Parent ]

      •  There are many solutions, but... (4+ / 0-)
        Recommended by:
        SWicklund, ferg, cjohnson, joanneleon

        ... writing a huge thing like the new DK backend with Python or Perl?

        You couldn't pay me enough to take on a nightmare like that.

      •  True, Java sucks (1+ / 0-)
        Recommended by:

        I am a C++ guy, and have hated Java all its life. But when it comes to web programming, I think Java probably offers the best options. I don't like scripts - or anything that's not compiled, in general.

      •  beg to differ (6+ / 0-)

        My reading/podcast-listening indicates that the big sites Google,Amazon,Yahoo all eschew J2EE in favor of LAMP solutions with home-brew session management.  I could be wrong, do you have other information?  

        I wrote one of the first Java books ever and love the language to death, but, imho, J2EE is not the best framework for building scaleable web sites today.  

        In fact, the language is unimportant - it's the architecture that counts and imho, the architecture should be LAMP, or some variation of LAMP that meets the criteria of:

        infinite, inexpensive scaleability
        ease of maintenance
        ease of modification

        imho, java struts doesn't meet any of those criteria, and in particular shuts out a whole class of potential site developers who can't handle struts wierdness

        •  J2EE is overblown, but some of the patterns (1+ / 0-)
          Recommended by:

          are useful by themselves. E.g. Filters, Front controller.  It is always pick and choose, throw out the extraneous and tune what you need, no matter what language/framework/toolkit you use.  If there was one solution that handled all the needs, everyone would be using it.

          9/11 was a Faith-Based Initiative - Bill Maher

          by glazeone on Wed Jan 03, 2007 at 04:42:54 PM PST

          [ Parent ]

          •  some of the patterns are fine (8+ / 0-)

            it's the implementation that blows.

            The problem with J2EE is that you can't just buy a little of it.  If you deploy to a J2EE server you've committed to a boatload of configuration just to deploy a single JSP page that adds 2 integers and displays the result.  It's as bad now in J2EE as it ever was in any Microsoft server, where doing any little thing dragged in so many god-forsaken bits of microsoft office, microsoft back office, IE4,5,6,7 and FrontPage that you ended up wishing you'd written it all in assembler instead.

            The best discussion I ever heard of this boiled down to a single bit of advice.  Dump the middleware entirely.  Leave all persistence and concurrency to the database server.  Use frameworks that can invoke standard database backends and that then encompass almost all their value-add functionality as standalone bits of code in the actual page (i.e. JSPs making stored proc calls).

            Manage user sessions in a standalone, home grown session manager and only share session primary key across server boundaries (through cookies or whatever) to maintain the hardware hot-pluggability of the LAMP architecture.

            •  You've pretty much nailed (4+ / 0-)

              what frustrates me about J2EE. It's terribly all or nothing -- and the learning curve alone, on all of those components you're sucking in for even the most simple of tasks, can be maddening.

              More than that, though, it's a bit of a personal grudge, with me: I've had nothing but bad experiences with it. Ever. My professional history has been punctuated, on multiple occasions, of some boss somewhere telling the entire IT group that they had to use J2EE instead of whatever else they were using at the time, because Java Was Better.

              End result each time?  They had to hire LITERALLY five times as many programmers to write and maintain the thing. And even after they did, the "better" java apps kept crashing, to the extent that cronjobs had to be used to restart the damn processes automatically every time they went down.

              So my frustration with Java is based on that history.  C, Perl, Python, Ruby, C++, Smalltalk -- I don't care, I'll use whatever has the best tools for the job at hand. But in both Java and Visual Basic (shudder, don't ask), I've never been burned so badly, so predictably, and so often.

              I swear, after all that, Java has often seemed to me to be little more than a transparent reason for IT bosses to have a bigger tech department. Might not be fair to the language; then again, maybe it is.

              •  "We should code it all in Visual Basic" (5+ / 0-)
                Said the big boss,

                "Because I've been looking on, and Visual Basic programmers have the lowest salary range. We'll save a ton of money!"

                I only wish I were making this up.

                Fry, don't be a hero! It's not covered by our health plan!

                by elfling on Wed Jan 03, 2007 at 06:36:33 PM PST

                [ Parent ]

                •  Nah, outsource overseas (0+ / 0-)

                  there's never any problem with time zones and spoken languages.

                  •  don't laugh - when you're a manager (2+ / 0-)
                    Recommended by:
                    esquimaux, bigjacbigjacbigjac

                    having all the programmers asleep when you're awake and speaking a language you don't understand can be a very attractive proposition.  In fact, it's more like a literal realization of the actual situation.  Programmers might as well be asleep for all the work they seem to do when watched, and they might as well speak another language for all the comprehension they display of even the most clearly stated instructions.

                    What's the difference between a herd of cats and a herd of offshore programmers.  Neither one understands what you want, but at least the offshore programmers will all wander off in the same direction.

              •  You weren't burned by Java (2+ / 0-)
                Recommended by:
                Hunter, bolson

                you were burned by an idiot manager who didn't pick the right toolset for the situation - not just the project, but the staff and the legacy situation as well.

                I had a new boss (a loooong time ago) who came in and wanted to change my project from VMS and VAX BASIC to unix and c. Problem was, we were just about done and the project was mission-critical to the company. I quit and came back the next day as a consultant, making twice the money. He hired a replacement software manager (the job I had until I quit) who knew both VMS and unix, and that guy proceeded to decide that he would keep everything in VMS, but migrate to c one small module at a time, with additional staff to do it. Talk about winning the battle but losing the war :-)

                Come see TV from the reality-based community at

                by MarkInSanFran on Wed Jan 03, 2007 at 07:02:01 PM PST

                [ Parent ]

                •  I have to take issue (3+ / 0-)
                  Recommended by:
                  Hunter, MackInTheBox, jfm

                  either that or I've had too much coffee.  

                  We're not talking about the language which is God's gift to programming imho, but the J2EE framework.  If you look at any of the app servers, jboss for example, there are too freakin many teeny little properties files just to get a single JSP page fired up that adds 2 numbers and displays the results.  web.xml, server.xml,,,, login-config.xml and that's just one directory of a project I'm doing right now.  God help you if you get into a classpath issue.  You'll end up as one of those weird suicide statistics - "he seemed happy, didn't leave a note, just shot himself at his desk - bled all over the project plan".  Same thing if you have jar conflicts (xerces anyone?) or a missing jar.

                  Nope. Anyone contemplating a J2EE project needs first, and foremost to hire a J2EE app server guru.  I have one, so I've postponed my own suicide, but when he leaves I'm right behind him.

              •  Haven't you heard of Ada? (5+ / 0-)

                The best language ever, for anything, on any hardware. Our own DoD says so.

                Wish I could find the paper I wrote about 25 years ago on a bunch of new statements that should be added to programming languages. The one I remember:

                The IF ONLY statement: used to express the hoped-for results of a computation.


                    IF ONLY Array A now contained the solution matrix

                News is what they don't want you to know. Everything else is publicity. --Bill Moyers

                by RobLewis on Wed Jan 03, 2007 at 07:08:14 PM PST

                [ Parent ]

        •  Wrong how? You mean, big sites don't use Java? (2+ / 0-)
          Recommended by:
          zericm, Curt Matlock

          We do where I work here - and other associated sites. Ridiculous amounts of hits per day. On Java and Struts.

          Whether it's the best is another discussion - but it's very much what we use, and it's a very big set of sites that use it.

          "Think. It ain't illegal yet." - George Clinton

          by jbeach on Wed Jan 03, 2007 at 04:54:22 PM PST

          [ Parent ]

          •  I was very surprised by this too ... (0+ / 0-)

            but I heard it with my own ears.  I forget which podcast it was, but the source was impeccable - a Google guy as I recall.  Went down the list of the top 5 sites and they were all LAMP.  No J2EE.  In fact, the Google guy was a bit defensive - "well, yeah, there's some Java in Google" but it was clear that it ain't much.  In any case, it'd be easy enough to prove - hit a few Google, Yahoo and Amazon pages and see what kind of server responds.  That should tell all.

            I buy the big set of sites for J2EE (somebody's buying all those books) - I've sold J2EE solutions to them.  It's just not the very top tier of high-volume sites.  It's the enterprise market, not the top end of the b2c market.

            Again, I was surprised, but it makes a lot of sense.  Not long ago, you were getting reamed really royally for websphere, weblogic, and jdbc licenses for the "commercial" database servers and your options were really limited.  I suspect that the top sites set out to win a money battle with the expensive middleware and database vendors and ended up discovering something important about technology as a side effect.

        •  You're on to a different issue - and it's key (2+ / 0-)
          Recommended by:
          klondike, jfm

          How much independent design and maintenance will there be for DK4?  The biggest benefit of Java and especially of J2EE is the complex libraries it provides for you (caching, transactions, distributed objects, etc.).  Ruby, Python and Perl are all relatively new to have the depth of libraries that Java has.

          What I find most salient about your point is that the biggies largely chose to build over "buy" with existing libraries and frameworks.  I think this is probably a fairly salient comparison for DK.  You can always get an app that performs better at a small set of specific tasks by building for exactly those tasks, but then you have to live with what you built.  But the biggies also are able to attract and pay for the deepest levels of knowledge for each one of those pieces (Yahoo apparently has a lot of the world's best FreeBSD geeks).  I think the language choice needs to follow from the choices about platform, community and maintainability, not lead them.

          I obviously don't know the details of the development plan, but I'd worry most about keeping the platform running over 5 or 10 years.  Even if it doesn't make sense to go with a CMS technically, an open source CMS tends to come with a community of devs who at least remain interested for a while.

          Just tossing my couple pennies in there.

          •  shiny pennies they are, but ... (1+ / 0-)
            Recommended by:

            to expand further ... what I got out of the discussion, was not that the big guys built their own J2EE servers because they had the programmer bandwidth to maintain them.  Exactly the opposite.  They looked at J2EE (et al) and determined that it was a bad architectural choice, especially EJBs, and that all they really needed from the app server was session management.  Implement session management from scratch, push persistence back to the database, push UI logic out to the client, eliminate interdependence of web server instances and voila, you have infinite scalability via commodity hardware.

            A guy from chimed in on another comment thread and the way they do it is content management via J2EE/struts, but that system just feeds an Akamai distribution network.  As far as I'm concerned, that's the way to do J2EE scalability - by not trying to scale J2EE.

            I would argue that dkos is unique, state of the art, pushing the envelope in all sorts of directions and is a perfect candidate for leadership of an OSS project.  Get the maintenance power of OSS, but get what dkos needs in the process.

            I took a quick look at CMS's a couple of years ago and they were crappe.  Not applicable to this problem at all.  All aimed at either enterprises or wiki apps.

        •  A critical evaluation (0+ / 0-)

          Fact: Google relies heavily on server-side Java, Python, PERL, C, and more.  They use a bit of everything, based on the nature of the task at hand.

          Fact: Amazon's architecture is crap, and any web developer there will very strongly advise you to NOT emulate their model.

          Fact: MySpace runs on ColdFusion... LAMP is not magic.

          You imply Java is not cheaply scalable, difficult to maintain, and difficult to modify.  I disagree on all three points.  Your invocation of J2EE as "The way Java is used for the web" says you are not current on Java web technologies, and

          Why?  Your arguments are unsupported?  Why are Java Servlets less scalable inherently than Perl/PHP?

          Why is it "easier" to modify?

          Why is it "easier" to maintain?

          Why do you assume the use of Struts?  Struts is just ONE possible framework.

          Without intimate knowlege of the current internals, I would suggest that because DKos is composed of very dynamic page content, most elements are infrequently modified, and very little user session data is strictly required for the existing functionality.

          In my mind, the most critical decisions left are:

          • how can we partially compile the site logic to eliminate redundant parsing cycles on each request and
          • which well-tested, robust, scalable caching library to leverage

          If the selected language does not have an answer for both, it should be immediately discounted.

          Finally, it really really depends on who the developers are, and what they know.  If Hunter were to invite me to participate in the project, I would want to work in a language I was both comfortable in, and felt would allow me to deliver the best functional result in.  

          Why does Shrub never talk about his first wife, Reality? He divorced her with prejudice and all of his alimony checks to her have bounced.

          by nepolon on Thu Jan 04, 2007 at 12:11:37 PM PST

          [ Parent ]

    •  No one (0+ / 0-)

      said it "sucks".

    •  Hunter (0+ / 0-)

      said java was out, not because he hates it, but because, "that much friggin' typing screws with my carpal tunnel issues".

    •  I could give you long, drawn out reasons (8+ / 0-)

      why I believe Java is inappropriate or simply wasteful for most kinds of web development, but in the end it'd boil down to simply "it takes too long to write very simple things, the language is IMO annoyingly un-compact, and that, over the years, drives me nuts in all excess to the benefits of the language." I'm willing to use whatever tools work, but Java drains my patience faster than any other language out there, (with) the (possible) ((exception of) (Lisp)), which many people either love or hate with a magma-infused passion.

      All other language choices suck too, of course. They just suck in other ways that some programmers are more willing to live with. (Python's "meaningful whitespace" for blocks syntax... I mean, ugh. Just ugh... Oh, the technology of the freakin' bracket is too hard to understand! Let's make your program break if you hit "TAB" the wrong number of times!)

      •  programmers vs scripters (4+ / 0-)
        Recommended by:
        zericm, MarkInSanFran, nepolon, Capn Guts

        You said yourself there's a small number of actual pages to be done, what was important was execution speed.  Java is more suited towards that than python, ruby or for christ's sake, perl.  You ruled out PHP for security but left in Perl?  You know those languages in a typical environment recompile every page for every single hit, right?

        In java you're in an application environment instead of a scripting environment, you can do singletons, shared caches across page views, etc.  For a low-traffic site of dailykos's type I'd say it was the wrong choice because it's not complex enough to justify a complex architecture.  But given how much you'll have to optimize, Java in a fast container should at least be in the running

      •  Hunter, what is your preference? (0+ / 0-)

        My recommendations thus far have been based on my educated guesses about the site's needs and the management and programming styles of you and ct.  For what it's worth I agree with you about Java (even though I'd support it as a viable option).

        But at this point, with which are you the most comfortable working?

        •  As of this moment, I'm mixed. (1+ / 0-)
          Recommended by:

          Of the three languages, I far prefer perl, even with the awkward syntactic cruftiness of OO in perl.  Of the frameworks, I'm more impressed with Django over the others, Rails and mod_perl/Catalyst/whatever included. Rails itself has good and bad points: I haven't made up my mind entirely about it yet, but I'm skeptical about their particular implementation of MVC, so far.

          There are of course other options -- many other options. Not an easy choice, which is why we're soliciting other opinions too. If there's a killer framework out there, that'll certainly make the decision on language for us, because the three languages are not dissimilar enough to be dealbreakers.

          •  Oh, and Ruby vs Python (0+ / 0-)

            is a wash, in my book. Like parts of Python better, but can't stand the meaningful whitespace stuff.  Generally like Ruby's version of OO, but you can do the same thing in any of the other languages with very, very few exceptions.

            •  tabs vs brackets shouldn't be (0+ / 0-)

              the deciding factor.  Once you get past the decision to use python for a project and get into implementation the whitespace thing should just go away.  You'll wonder what the heck you were thinking within a week or two, I promise.

              There are lots of more important differences.  

              I really like this run down of one coder's summary of ruby vs python.

              full disclosure: I don't even consider anything but python anymore.

              Be sure to check out Pycon for a flavor of what's up with the python world.

            •  The whitespace stuff (2+ / 0-)
              Recommended by:
              jfm, Bill Evans at Mariposa

              The wonderful use of indentation for scoping is exactly what drew me to Python 15 years ago and it has been my language of choice ever since.  In that time I've seen very few bugs caused by mis-indentation, other than short bursts of confusion caused by mixing spaces and tabs in the same file, and that only lasts until the programmers agree on an indentation style (1 tab or 4 spaces) and set up their text editors accordingly.
              Anticipating the next cagematch: tabs.
              Anticipating the one after that: not CVS.

          •  Interesting (1+ / 0-)
            Recommended by:

            I agree with you on the languages; OO Perl can be really elegant if you use it correctly and intelligently (and I believe many of its critics just haven't done so).

            Personally I find frameworks overrated. I've found simple but robust solutions using simple CGI scripts or mod_perl handlers and a simple templating system like HTML::Template (tweaked for usability and efficiency). Catalyst seems overengineered and restrictive to me though admittedly I've only scratched its surface; I think my preference would be Apache::PageKit, which I find to be more intuitive.

            My larger point, however, is that if you're most comfortable with Perl and are responsible for DK4, and especially if there's already a large Perl codebase, then I would probably recommend sticking with that (and enlisting the help of willing contributors like myself).

            The people who are pushing complex Java solutions, while credible, may be misunderstanding the particular needs and resources of this site.

          •  Django is a beautiful framework (0+ / 0-)

            Somehow, the Django developers have managed to hit the perfect balance between providing lots of useful functionality (form managers, QuerySets, pluggable authentication) and keeping it simple (minimal configuration required, easy-to-understand codepaths).  The Django codebase is almost all understandable to any moderately-skilled Python programmer, to the point that it's feasible to rip out and replace bits of its innards if you need to.

          •  Rails persistence (0+ / 0-)

            will not serve you well.  It is too chatty, doesn't compile to bytecode yet, and you face potential deployment challenges as the fcgi libraries available to deliver Ruby are buggy.  That can introduce a slow one-time deployment headache as the team learns the magic dance to avoid memory leaks in fcgis.  Some folks have never encountered these memory leaks.  I did.

            Why does Shrub never talk about his first wife, Reality? He divorced her with prejudice and all of his alimony checks to her have bounced.

            by nepolon on Thu Jan 04, 2007 at 12:23:10 PM PST

            [ Parent ]

            •  FCGI is so 2006 (0+ / 0-)

              The only big Rails apps that use FCGI still are the ones that haven't moved over to running on Mongrel yet. In six months, I doubt there will be any to speak of. Mongrel is fast, secure, and easy to work with. And it's an HTTP pipe so you can easily run it on the same box as your web server or SLB, or on a different one, and it has great support for clustering.

              Rails persistence layer (ActiveRecord) has some drawbacks, but so do all of the other persistence layers too.  AR is a pretty nice ORM, and it is particularly well-suited to managing the kind of schemas that I would guess DK would use.

              "Instead of asking what you could do, you ought to have been asking what needs to be done."

              by khaavren on Thu Jan 04, 2007 at 10:42:59 PM PST

              [ Parent ]

              •  It's still January! (0+ / 0-)

                I wouldn't bank DKos' uptime on Mongrel yet.  YMMV.

                ActiveRecord, was still too chatty in 1.1.  It was chatty because it expected the schema to change, even in produciton mode.  That is stupid for a site that uses few tables that don't change often.  I maintain that you still need the bytecode compilation, too.

                Why does Shrub never talk about his first wife, Reality? He divorced her with prejudice and all of his alimony checks to her have bounced.

                by nepolon on Fri Jan 05, 2007 at 11:12:40 AM PST

                [ Parent ]

      •  Lisp! (5+ / 0-)
        Recommended by:
        gong, mik, Shapeshifter, Boreal Ecologist, jfm

        Well, if I were programming it, it would all be in Lisp. Yep, magma-infused passion and all! (I'm one of the lucky few who gets to program in Lisp for a living, dontcha know)

        Guess that's why I'm not on the team!  

        •  Hyper Meta Alt Ctl Shift to you, pal! (0+ / 0-)

          Yes, I actually once did work on a Lisp machine. But my memory is not good enough to hold that many control sequences. I use gcc/gdb/emacs for everything, myself. Dated, but works for me.

          Totally irrelevent to the Q at hand, I admit.

          It's a lousy world, Sir Magnus. A few happy fish will make it better. (Le Carré)

          by Boreal Ecologist on Wed Jan 03, 2007 at 06:55:00 PM PST

          [ Parent ]

          •  Not quite old enough to have worked on a (3+ / 0-)
            Recommended by:
            gong, Boreal Ecologist, jfm

            Lisp machine.  I use emacs for everything.  But our website serves up biological data on a couple hundred organisms, complete w/ graphics generated on the fly, and it's all written in Lisp (even the web server -- there is decent Lisp web server software out there nowadays, but back in '95 when I first wrote this stuff, there wasn't, so I had to write my own).

            Plus, I've heard Paul Graham give his talk about the advantages Lisp gave him when he wrote what later became Yahoo!Stores.  I believe Orbitz is Lisp, too.

            So I'm kidding in the sense that I don't expect DK4 to be written in Lisp, but it's not a totally unreasonable way to go.

            •  what kind organisms/data.... (0+ / 0-)

              ...may I enquire to ask? Reason is, I'm involved in some ecological projects that might benefit from a glimpse at the content and structure of those web pages you mention. You can contact me through my profile if you are willing.

              It's a lousy world, Sir Magnus. A few happy fish will make it better. (Le Carré)

              by Boreal Ecologist on Wed Jan 03, 2007 at 11:20:26 PM PST

              [ Parent ]

            •  Get SLIME-ed (0+ / 0-)

              Get the SLIME extension for emacs if you haven't already.  Makes Lisp so much more fun.

              Then did he raise on high the Holy Hand Grenade of Antioch, saying, "Bless this, O Lord, that with it thou mayst blow thine enemies to tiny bits, in thy mercy."

              by Event Horizon on Thu Jan 04, 2007 at 01:03:37 AM PST

              [ Parent ]

        •  Nah, do the whole thing in APL (0+ / 0-)

          Now I am really dating myself.

          Come see TV from the reality-based community at

          by MarkInSanFran on Wed Jan 03, 2007 at 07:14:50 PM PST

          [ Parent ]

        •  Heh (3+ / 0-)
          Recommended by:
          brillig, ppluto, Boreal Ecologist

          Only do a bit of jscheme nowadays myself - but nothing since has compared favorably to programming on Symbolics machines... Modern java (and, I'm told, C#) IDEs are finally getting there... after what, 25 years?  sigh.

          Yah - I've come to think that strong typing has its place, but real closures, macros (I pine for lisp macros), and a full oo system... sigh.

          Never wear your best trousers when you go out to fight for freedom and truth. -- Henrik Ibsen

          by mik on Wed Jan 03, 2007 at 07:57:22 PM PST

          [ Parent ]

          •  strong typing for weak memories :-) n/t (1+ / 0-)
            Recommended by:

            It's a lousy world, Sir Magnus. A few happy fish will make it better. (Le Carré)

            by Boreal Ecologist on Wed Jan 03, 2007 at 08:09:20 PM PST

            [ Parent ]

            •  Nah (0+ / 0-)

              more for optimization hinting... My last few years of commonlisp hacking included many man-months spent hand-optimizing the critical paths of a large system - the resulting code was approximately as type-declared as an equivalent Java (java2) system... but it ran a heck of a lot faster.

              With a modern java-like (maybe "self-like" is more accurate) runtime compiler, you can maybe do a lot better with weakly-typed code than without, but sometimes you need to get closer to the silicon for performance purposes, ugly as it is.  Dunno if the remaining lisp vendors are able to keep up on this sort of technology - it looks to me like most of these folks are working for Sun nowadays...

              I do not like scheme's binding of a single value to a single symbol - it is way too easy in large systems to have someone lazily declare a function with a name that someone else lazily sets to a value...

              Never wear your best trousers when you go out to fight for freedom and truth. -- Henrik Ibsen

              by mik on Thu Jan 04, 2007 at 07:03:50 AM PST

              [ Parent ]

              •  it was a quote... (0+ / 0-)

                ...from some long ago hacker screed.

                Personally, although I don't use C++, first thing I do when developing code is figure out what the basic data structure is, and then write code to make one, delete one, read one and write one. After that, eveything is pretty easy.

                But the applications under discussion in this thread are completely opaque to me, other than the references to database server back ends.

                It's a lousy world, Sir Magnus. A few happy fish will make it better. (Le Carré)

                by Boreal Ecologist on Thu Jan 04, 2007 at 09:30:32 AM PST

                [ Parent ]

        •  (((((((((LISP)))))))))))))))) (2+ / 0-)
          Recommended by:
          TalkieToaster, klondike

          (((((I did some Lisp (programming) once too))))))))

          (((((Didn't really like it)))))))

        •  Lisp! (1+ / 0-)
          Recommended by:

          (I'm one of the lucky few who gets to program in Lisp for a living, dontcha know)

          I envy you.

      •  It's ok to admit... (32+ / 0-)

        your team doesn't know how to use Java effectively. There's a lot of crap architectures for Java out there and a lot of false starts, it's hard to know what to use without actually working on large Java projects.

        I have to agree with unfounded, you're approaching this like a scripting job instead of a software engineering project.

        As noted elsewhere, we really need a set of requirements to make any serious recommendations. It's hard not to waste your time if we don't know what requirements need to be met.

        First of all, you guys obviously need Open Source solutions, both from a budget and customization / optimization perspective. Given what I know from using the site and not really having thought a lot about what DK4's requirements are, I'd use PostgreSQL for the database. It's the closest thing to Oracle that you're going to find Open Source, and a site like this really needs that kind of power. Postgres is fast, clusterable, feature rich, ACID compliant, SQL 92 compliant and has built in GIS functionality. Not to mention, it's stored procedure facilities rock.

        Given the inherent uses of geographic knowledge for a political community site, the GIS capabilities of Postgres alone would seal the deal for me.

        I'd also use Java. Full on J2EE with Hibernate, Data Access Factories, Stateless Session Beans which send views to the web tier as Data Transfer Objects. The web tier should be done with Tiles and JSF, use the Apache implementation. The easiest way to do this, especially if you're coming from a perl/python two-tier background is Spring/Hibernate on JBoss.

        Scalability is built right in, there are currently large applications built on this architecture that would compete if not surpass DK's requirements.

        Your comments about verbosity are not unfounded. Java is verbose, I like it that way, Java is meant to self document and the patterns which have survived the real world keep information about the application well organized and easy to maintain.

        The verbosity is dealt with by using tools, I usually code plain old java objects by declaring their fields based on the business object model docs and then have eclipse generate the accessor methods.

        After adding Hibernate Annotations and/or Spring Attributes, your DDL for the database is generated during the build. Customization can be added through XML configuration for named queries. Stored procedure mapping can be configured using the annotations. You can also create your database and then generate pojos from it, but I find that optimization is easier when generating the DDL with Hibernate. Hibernate automatically generates SQL queries optimized for your particular database and customized to your caching and transaction requirements. In JBoss, it uses clustered pools of database sessions and an object cache to minimize calls to the database.

        The Data Access Factories are simply abstracted queries using Hibernate and the Business Object Model. The methods all return the pojos created for Hibernate. Both Spring and the numerous frameworks I've written and used use Stateless Session Beans for the business logic. These can be exposed to a web front end via XML-RPC, SOAP, RMI, CORBA or anything else you fancy. JBoss and other J2EE containers handle this via both XML meta-data and programmatic configuration.

        Again, most of these configurations are handled through annotations or other Javadoc like markup that generates the actual meta-data files during the build. It also generates most of the files you encounter when reading the Hello World EJB 1-2 apps. I'd have carpal tunnel too if I had to write all that crap. EJB 3 actually gets rid of most of those files and has the necessary glue code generated by the container during deployment, which integrates much of the work tools like XDoclet used to do into the platform.

        This allows you to stay in a Java environment, you don't need to switch to an XML editor or keep this information about your application in two separate files. The information needed for that aspect of the application is contained with the source code that controls that aspect.

        You should also look at the Apache project Lucene. It’s a Java search engine implementation which integrates seamlessly into the environment I’m describing. You should be able to quickly implement features like tagging with it’s toolkit.

        For the web tier, I'd use JSF/Tiles, you could use the Spring Framework for this, which would also provide the backend framework I've described. The JSF / Tiles work should be fairly simple, I would only have it deal with authentication, navigation and session maintenance. The actual UI should be completely abstracted. You need to be able to separate the bulk of UI consideration into plain markup. XML JSP actually accomplishes this and works with tools that web design people use like Dreamweaver.

        The JSPs should be simple and deal mainly with layout, the style information should all be in CSS with the JSP only displaying the correct classes for given tags. Avoid doing too much with JSP Tag Libraries, if it can't be done with the standard tag libraries, you're probably better of dealing with it in the Controller.

        The AJAX functionality should be delivered using the recently Open Sourced Google Web Toolkit. It's what Google used to build the AJAX server side and javascript for their web applications like GMail. You write AJAX widgets in Java using the GWT API, these are then compiled into Javascript that can be cached. The widgets are rendered by the web browser using the Javascript and attached to named tags in the page. Google even Open Sourced their oh-so magnificent debugger, thus bringing sanity to the world of Javascript. Seriously, in the 11 years I've been working with Javascript, GWT is the first thing that's made me excited about it.

        Just use the Tomcat web container in JBoss and leave the entire architecture in a single instance of JBoss. Front end Tomcat with Apache for static content and security. Your entire application should consist of a set of config files for JBoss/Tomcat and Apache, the generated DDL for Postgres and a single .ear file for deployment. You should be able to add a new node to your cluster within minutes.

        You glue everything together with Ant, Maven is still a time sink for most dev environments. Ant is like make, but more Java oriented. Your ant build should produce a binary distribution for the server and should also be capable of automated server configuration.

        The reason for using this architecture, which I'm sure you've found several areas that appear bloated... snark, is that it is superior to others in it's ability to adapt to new functional requirements, changes to existing requirements, performance tuning and optimization. I'm sure that any of the architectures presented here are capable of delivering this project, however, none of them are as agile as this one while not jeopardizing security, scalability or previous optimization efforts. What is typically considered bloat in Java are the infrastructure requirements to separate the actual business logic from the environment it runs in. This allows for greater division of labor, thus lowering maintenance costs and is the primary reason it dominates the Insurance and Financial services industries.

        Java is also far better at XP/Agile methodologies. The Unit Test facilities are superb. JUnit4 in Eclipse is the easiest thing I've seen for ensuring that your new development doesn't break old development and that your code actually does what you want it to do. Also remember, put everything in SubVersion. Much easier to maintain than CVS, but a bit more problematic to install on some Linux distros. Do not even consider dev platforms that do not allow you to easily develop, run and maintain a large number of unit tests.

        Really, best practice is to adopt a similar dev environment to what the JBoss Group uses. They've gotten the Open Source / distributed development team thing down pat. They implement new changes over a wide range of products with amazing speed, quite frankly the Open Source Java community can make the Perl/Python community look like a bunch of snails at times. A good trouble ticket / bug tracking system should be part of your dev environment, there are a number of good ones out there. You want to be able to recreate the dev environment quickly, track project progress centrally and have an automated build and test environment that can run nightly and expose errors quickly.

        I think that since DK is going to be around for years and is attempting to rapidly expand in both functionality and entirely new areas, such as the publishing venture. I think if you look at the range of possible functionality from publishing / CMS, geographical information and community features that could be useful to the goals of DK, an environment that allows the business logic and presentation more liquidity within the system makes sense. It is probably impossible for even yourself or kos to predict what DK will need to do in 3 years, being able to quickly retool the organization’s basic functionality or presentation could become vital.

        Without requirements, it’s impossible to present any arguments beyond experience with past projects, which may or may not match your needs. The most important aspect I believe is that the technical requirements are driven by a long term view of the DK website. My arguments for this particular set of architecture choices is based on long term development efforts for software projects that represent core business functionality and are customer facing, which is in essence what the website is to this organization. I think that if you poll most people here with a lot of experience in large projects, you’ll see the same items prioritized.

        Sorry this is so long everyone, but that’s what you get when you ask techies tech questions!

        •  Wow, someone who actually *knows* something! (4+ / 0-)
          Recommended by:
          stevelu, Legolas, Capn Guts, klondike

          I wish I coule give you 65536 rec's for that; thanks for making it so detailed. This is a software engineering job, not a quick-and-dirty hack. I was in software for 20 years and I have done both.

          Even if each and every one of your recommendations is not perfect, the approach you are taking is. You rock!

          Come see TV from the reality-based community at

          by MarkInSanFran on Wed Jan 03, 2007 at 07:20:15 PM PST

          [ Parent ]

          •  Please pay attention to this guy (8+ / 0-)

            I want to second what MarkInSanFran said; everything cjohnson said about process is spot-on, and that's what Kos needs to focus on right now.  This site is too important to the fate of civilization, to roll the dice on a hack.  Sorry to be all frumpy and corporate like, but that's reality.  Now, I'm not current enough with Enterprise-scale mega-projects to know if cjohnson really knows his tech or just reads a lot and talks out of his butt, but it sounds convincing and is 100% consistent with everything I know.  I would take what he says seriously, but validate all of it with a diverse group of developers who all have bona-fide large-scale web app development experience.  I would help you find those guys if you need the help.

            Frankly, while I'm a huge fan of Hunter's political writings and occasionally amazing swat-team leadership on fast-moving political issues, I can see no justification for Kos letting someone, however trustworthy, mess with his brand in that fashion.  No one is going to stop reading this site because it's not hacked together out of unintelligible spaghetti by two guys with really long hair, strung out on Mountain Dew in the middle of the night.  Kos needs to think like Google.  Even the smartest software geniuses in the business need good software engineering practices, if you want high probability of success.  Imagine what would would happen if Google all of a sudden stopped being wicked-fast and ultra-reliable, because they revved the product with such a cavalier attitude as this.

            One thing that is as sure about the leading progressive blog as it is about any widely-used software-based platform, be it Google, iTunes, Mozilla/Firefox, Amazon, ... whatever: you don't get multiple chances; it sucks, you're dead.  Kos needs and deserves the same predictability and reliability in a software project as any of those.  I don't doubt Hunter's talent, but his trademark sarcastic style, in a discussion of this particular topic, should scare the shit out of anyone who knows a lot about software engineering.  I'm not talking about pointy-haired IT bosses.   Good software engineering is really hard, which is the main reason most large projects turn out somewhere between awful and disastrous, including it seems all the Java projects Hunter has had the misfortune to be involved in.  He is not alone.  However, the simple fact that an apparent majority of people doing large projects in Java don't really know what they're doing and misuse the tools is not, in itself, a reason to rule it out.  I tend to agree with cjohnson that, despite the standard disclaimer about not weighing in on solutions without knowing all the requirements, it is likely that Java-based frameworks and tools should be in the running just because of scalability and maintenance concerns.  Without meaning any personal affront to Hunter, who I still think must be a great guy, if he's not being entirely tongue-in-cheek in this article about what he likes and dislikes about programming languages, then he's not qualified to choose an architecture.  And if he is being entirely tongue-in-cheek, what is the point of scaring those of us who actually know how this stuff works in real life?

            How do I know?  Like MarkInSanFran, I was involved in software development and systems integration for over 20 years, and took part in some really amazing and some really awful projects.  I've unfortunately seen some of the brightest coders in the industry, some with huge insider mojo, waste as much as ten million dollars on developing a pile of useless, unredeemable crap, because they took the "Just trust me; I know what I'm doing" approach, and ignored best practices.   Myself, I was the lead software architect on a mortgage funding network for a Fortune-50 company, used by thousands of their customers back in the nineties, before the Internet was sufficiently available.  We were one of the few companies Sun talked to the most when they developed Java.  For the first few years it was at least as awful as Hunter believes it is, but that was at least six or seven years ago.  For the past few years it's been a whole different story, and Java has really come into its own.  Sun has opened it up, and the Open Source community has embraced it, and that pretty much clinches it for me.

            There's a middle ground that Hunter's probably not aware of, since most people aren't.  Apple has a redheaded step-child product they don't talk about much, called WebObjects Framework.  It's a phenomenal toolset for quickly developing complex and scalable, Java-based application servers, for people who don't like to type.  You could easily adopt this technology -- you don't need to run it on Mac hardware, though the fact that Kos is an Apple fan is helpful.  People routinely deploy the WO Framework on Linux, Solaris, even Windows.  Folks that haven't used it tend to be very dismissive of it -- the pure J2EE or nothing story -- but it could very well be the right tool set for DK.  It's all Java, and complies with most of the Java architecture standards, though it does an amazing job of hiding as much of it as you want it to.  It's quick to come up on and scales really well.  It's lighter than most (all?) J2EE containers, and the Java architecture thingies that it lacks are things DK almost certainly doesn't need.  Assuming it still powers the iTunes Music Store (I believe it does) and the Apple Store, it can't be too lame, can it?  I was quite involved with the early developers of the product, back when it was NeXT Software, before they took over Apple Computer.  For many years it powered Dell Computer's online store.  If anyone here wants to know more, I'd be happy to enlighten them.

            This is a long post.  I haven't even started to scratch the surface.   This stuff is actually very hard to get right.  Unless Kos can count on remaining the luckiest guy in the world, he really needs to pick the right framework, focus on good, sound process, and make sure he has a team lead with a successful track record on the chosen platform.  Don't let the programmer pick which language she likes best.  The programming language is mostly incidental, apart from the critical need for it to be really fast and scale really well.  Most importantly, Kos needs to make sure he doesn't have people hacking together a custom architecture that mimics what's already available off the shelf.  That's the path to disaster.  "Well," you might say, "isn't that how all the best-known sites started out?"  True enough, but then you wouldn't know about all the much more numerous little-known sites that cratered.  Dot-bombs, anyone?  DK is a going concern, and a brand beyond all valuation.  That's what really matters.

            •  WebObjects (0+ / 0-)

              You are correct in stating that WebObjects power iTMS, which more than proves its ability to scale.

              While Apple hasn't done a whole lot new with the framework in recent years, the open source community has built a great set of libraries and tools on top of the Apple provided framework. Project WONDER and WOLIPS(an Eclipse IDE mod) together make a pretty kick ass development environment in my opinion (I have worked with both php and ruby on rails).

              The combination of massive scalability, nice ajax support (with the WONDER Ajax framework) and joyful development should make WebObjects a serious candidate for DK4.

        •  Very good arguments for JAVA (0+ / 0-)

          Your analyses is spot on! You obviously have done very large, very scalable projects using JAVA and various JAVA platforms! I suppressed comments on JAVA because:

          • I thought Hunter did not want to hear it.
          • I know there are others, (such as you) that are much more proficient with JAVA and JAVA solutions than I am!

          No more gooper LITE!

          by krwada on Wed Jan 03, 2007 at 08:01:30 PM PST

          [ Parent ]

        •  That's a highly useful comment (2+ / 0-)
          Recommended by:
          souvarine, jfm
          and you obviously know Java...

          But still, I can't help but notice how verbose your comment is and what a steep learning curve it has for those of us who aren't currently coding in Java. :-)

          Off to Google, to upgrade my buzzword compliance!

          Fry, don't be a hero! It's not covered by our health plan!

          by elfling on Wed Jan 03, 2007 at 09:48:07 PM PST

          [ Parent ]

        •  buzzword bingo (2+ / 0-)
          Recommended by:
          londubh, klondike

          And the wrong advice for scalable websites. There is a good presentation on how eBay scales at addsimplicity that responds to many of your points. My personal favorite is ACID and stored procedures, their advice is to store no business logic in the database (no stored procedures) and move referential integrity to the application where it can be optimized. Avoiding transactions has been web gospel for a decade and is much of the reason MySQL is so popular. A well architected REST application has no need for ACID.

          Unit tests, smoke testing, version control and bug tracking are points well taken but they have all been part of the perl culture since 1998 (CPAN/Bugzilla). Leave it to the Java people to think they invented 'Agile' development.

          My biggest issue with Java, and perhaps someone can enlighten me, is the difficulty of optimizing. It is apparently very difficult to dig through layers of jar files to fix or replace brain-dead code in someone else's class library. The Java people I work with are reluctant to dig under the hood and prefer to throw hardware at performance problems. At least in perl you can rip out a slow module and replace it with C when you really need speed.

          In any case my advice would be perl or python and stick to MVC. Both are well supported under apache and easy to tune. Python's XML support is better, if DK4 wants to go that way. A good OSS XML CMS would be interesting. Stay away from frameworks like Mason (un-MVC, also the reason to avoid PHP). Also stay away from 4GLish DB frameworks like RoR, they shove too much of the model into the DB which is hard to scale.

          •  buzzword experience (4+ / 0-)
            Recommended by:
            neodem, neele, corncam, Legolas

            Time to update your skillset. First of all, nothing happens in an environment where data integrity is needed without transactions. Secondly, you will wind up writing transaction functionality into your application if it is complex enough. Either use industry standard methods of handling transactions or take on the cost of maintaining your homebrewed solution.

            MySQL is popular cause most people on the internets just need to build something quick. First to market has been far more successful than elegant software on the internet. MySQL got a lot of boost from inexperienced developers like the Slashdot crew who had no experience with software scalability in general. Postgres has always been a superior database feature wise and has outperformed MySQL in a number of very important aspects over the past few years. I cannot remember the last project I was on where anybody seriously considered MySQL.

            MySQL's claim to fame has more to do with the contents of HOWTO documents and the fact that it's command line will take just about any mish-mash of SQL dialects you throw at it. Developers really don't like to learn how to format dates on new DB systems. Especially if they're throwing some crap together, which is what most LAMP projects started out as.

            The only reason for the recommendations of moving stuff out of the DB is that the DB their suggesting can't handle it. It's a limit of the implementation, not relational databases or distributed systems in general.

            Microsoft has been recommending moving everything to stored procedures for the last decade, especially for web applications. Oracle and DB2 also recommend stored procedures in a number of scenarios. Your recommendations sound like the babble from the high priests of LAMP development for the past 8 years or so. LAMP hasn't provided elegant solutions for any of these problems. CPAN is inadequate, I've been using it since 96. Bugzilla is POS in and of itself. The communities for both these efforts have ignored a lot of issues that relate to ease of integration and quality control for years. Those issues make the total cost of ownership for these technologies excessive in most organizations.

            Perl is one of the most diffcult to integrate platforms. CPAN is full of poorly documented minefields. Much of this is simply accepted by the community under the auspice that, well "it's open source" or part of Perl's "write-only" feature. Sorry, it's not good enough for getting the job done and most of the time it's just being too clever for your own good. Java libraries are, in general, better documented, easier to maintain and a hell of a lot easier to integrate with other Java libraries and existing development efforts. There's no worrying about whether this library was written in OO Java vs. normal Java or anything like that. I can typically pull an ancient Java 1.1 library and use it with the latest and greatest OSS libs compiled for Java 5.

            Agile development has only been a repeatable success in the Java world. The Java world has produced 10 times the number of software engineering quality control applications as the Perl or Python communities. Just Unit testing alone, there are a number of commercial solutions, automated reporting add ons, contract tests, etc. Add in bug tracking, and centralized project reporting and you're talking about 30 solutions from the Open Source community alone. Java developers, in general, take software engineering a lot more seriously than Perl or Python developers and it shows in the tools and dominant development patterns.

            And if MVC or MVP is what everyone wants, why would you use anything but Java? Velocity was the last major Java web framework that wasn't based around these patterns and it's been on the decline for years. These types of patterns have been matured extensively in the Java world. If you pick your specific component architecture and which bits of J2EE you need based on the patterns that your application requirements dictate, determining which TLA's and tools you'll need is easy.

            As far as optimizing, there's usually a better solution in the API by extending or overriding the object model, or just use Open Source and write a patch. You're not supposed to go mangling jars if avoidable, then you have to maintain a custom version of some jar file.

            I've been part of the OSS community for at least 10 years now and I’ve been watching the technical development of many large scale sites since they were quite literally being run out of dorm rooms and basements. My /. uid is low enough to be worth money, I’ve watched all of these technologies grow and get hashed out. I've worked on projects with all of them, and I've bet my job on a few of them. I can tell you that the sum of my experience is that proper software development is rarely found outside of C/C++ and Java shops. Perl, Python and PHP shops are typically the worst with MS and COBOL shops falling somewhere in the middle. The worst code kludges and hacks that I've ever had the misfortune of having to deal with have come from people who mistook a scripting language for an applications development environment. These tools have their place, primarily for system administration centered functionality and prototyping, but successful use for large scale software development tends to be the exception and not the rule. Java has been designed with software engineering best practices in mind. It is not the raw execution speed of Java, it's terseness or lack thereof, nor is J2EE easiest way to write Hello World. None of those things are priorities for good software engineering practices. Good software delivers functionality with lower total cost of ownership. J2EE is meant for applications which require enterprise class infrastructure such as transactions and clustering. A Hello World app in J2EE may take almost as long as any simple business service to write, but it will benefit from the J2EE infrastructure just the same and be a clustered, scalable Hello World with transactional integrity. So really, you don’t understand how to use J2EE unless you see that your business service is now as easy to write as Hello World.

            Oh, and don't start talking XML development until you've played with JAXB. That's another area where Java's maturity really shines. That and Hibernate integrates with XML data nicely as well. In fact, XML data sources are built right into the EJB3 spec currently supported by Hibernate, Spring and JBoss.

          •  sorry (0+ / 0-)

            but the Java devs you work with are not very good, are they?

            Do you compare good PERL devs to poor Java devs and call it fair?

            Your snark about Java people thinking they invented Agile is unfounded, and frankly that sort of attitude poisons the well of discussion.  

            Why does Shrub never talk about his first wife, Reality? He divorced her with prejudice and all of his alimony checks to her have bounced.

            by nepolon on Thu Jan 04, 2007 at 12:32:09 PM PST

            [ Parent ]

            •  java devs (0+ / 0-)

              I don't know if they are good Java devs, but they are the best I have worked with. Most of the Java groups I have worked with either drove their company into the ground (Ars Digita) or were fired after their efforts collapsed. At least this group is succeeding, if at a high hardware cost.

              This group is mostly Amazon alums, I think they are pretty good. What they are doing is impressive (hundreds of millions of records, billions of transactions), but their processes live up to Java's reputation as a resource hog. A previous incarnation of the system in perl was faster on older hardware. I've inserted bits of python into their process and dramatically improved performance.

              I will grant that the Java OSS community has gotten much better in recent years, and projects like Lucene and Tomcat produce very good software. But my question was about optimization, and cjohnson gave the answer I've gotten before, don't look under the hood. And that's fine, I'll keep putting in perl or python (wrapped around C libraries) to clear out performance bottlenecks.

              cjohnson's responses on transactions and stored procedures suggest to me that he is trying to solve the wrong problem. A blog like dailykos does not require ACID level data integrity, it requires fast response times so that it can handle peak loads. Having individual transactions fail gracefully is better than having the whole site slow to a crawl.

              I think the snark was justified given the scripting versus software engineering crack. Good perl developers have been using solid software engineering practices since before Java existed, and bad Java developers are worse than an SA hacking on perl scripts. It is easy to find bad developers in any language.

              •  quality devs in any language (1+ / 0-)
                Recommended by:

                exist.  Finding them is the challenge.  I won't trash your devs, but I have to hope they learned their best practices somewhere besides Amazon! :*)

                cjohnson's answer (para: don't crack the jars) is generally the "correct" answer.  If there is a performance bottleneck in the code you want to call, you probably need to investigate alternatives.  Sounds like you are doing that, so it sounds like you are on the right track already.  Sometimes the best alternative is to tune the lib and contribute the fix back.  Sometimes not.  Fear isn't a good excuse, though.  

                We had a similar challenge a couple years back where we inherited a system that had issues.  It imported large XML files, then transformed them using a very rich and flexible XSLT.  It was taking too long.  We discovered it was relying on a DOM parser instead of a SAX parser, and it wasn't caching the template which was widely reused over a number of such large XMLs.

                We changed the parsing to use SAX and cached the XSLT template and cut the median evaluation time by 80%.  That's a pure-Java solution.  Sometimes it's just easier to fall back on JNI, but you assume a possible maintenance risk in doing so.  Sometimes assuming that risk will be the right answer.

                Why does Shrub never talk about his first wife, Reality? He divorced her with prejudice and all of his alimony checks to her have bounced.

                by nepolon on Fri Jan 05, 2007 at 11:04:29 AM PST

                [ Parent ]

        •  I was right with you until (1+ / 0-)
          Recommended by:

          you said cluster which I take to mean distributed objects shared across multiple jboss instances.  Correct me if I'm talking nonsense.  EJB3 and annotations - sure.  Not my cup of tea but perfectly reasonable.  But distributed objects, forget it.  To me that says you're handling concurrency in the object-relational-modelling layer, not in the database and it also says that you're really jumping through hoops to handle a single node failure, and you're in a bleeding-edge configuration of the app server.  True?

          •  Clustering is out of the box. (4+ / 0-)
            Recommended by:
            neodem, enigmaboy, Flywheel, klondike

            JBoss clusters the objects in the container across the network with a on switch for a simple cluster and numerous options for others. The other frameworks latch on to this capability. The idea is really global caching and session failover in the application layer. Database concurrency is handled through transactions and clustering the databases. You can use versioning in Hibernate and Toplink, which uses timestamp columns in your tables. The basic setup is to treat the database as it's own system. It handles it's concurrency, and transactions are for communicating that. The ORM tool can either utilize the databases capabilities or explicitly manage concurrency.

            I don't see where you're getting the picture that the ORM handling object concurrency in the business model is problematic. It's just as easy as EJB3's or any other object. Other application servers are more difficult, JBoss takes advantage of their AOP design to provide infrastructure like this at almost zero cost to the developer.

            •  obviously I'm bringing history to the table (2+ / 0-)
              Recommended by:
              enigmaboy, Flywheel

              though in my defense you have to admit that pre-EJB3 which isn't that old (1 year?) EJBs were a huge pain in the ass.  You needed addons to get the AOP.  You saved a million lines of code by writing a million lines of configuration.

              Also in my defense, you know what you're talking about and I don't ;)  In any case we've gone way beyond what the diarist is willing to consider but I like the subject.  

              What I'm getting at with object concurrency is just this - why?  If you can handle concurrency in the DB, and go straight at it with things like recordsets or stored procs then who needs an app server cluster?  To me it seems like the entire requirement for the complexity of the app server cluster is generated by the management of concurrence in the distributed business object layer.

              I'm certainly not the app server guru, but we're a shop that deploys into jboss and we couldn't do it without a dedicated jboss guru.  When I see a tool that requires its own guru, that tool worries me.

              •  complexity (3+ / 0-)
                Recommended by:
                Legolas, Flywheel, klondike

                EJB3 is new, but it incorporates many of the processes developed through XDoclet, which is how us J2EE developers managed never to write those configuration files or the glue code for EJBs. It's also how we got SOAP interfaces to our EJBs by just changing the build script and annotating our existing business interfaces.

                The complexities of J2EE that were previously dealt with by using tools have been rolled into JEE5. Much of the complexity is finally offloaded to the container, where it belongs.

                J2EE requires a guru because it requires knowledge about enterprise infrastructure. I've worked with JBoss, WebLogic and WebSphere extensively. I've ported apps between each of these platforms, from numerous frameworks and from ORM tool to ORM tools. If you understand the infrastructure requirements of distributed applications and the patterns that are required, it's not that hard to move from one to the other. Nor would it be terribly challenging to migrate these patterns into tools for other development environments, it's just that Java has mature ones now.

                The requisite knowledge is the same as one would need for mastering other distrubuted interoperable environments like SOAP and CORBA. If you don't know how to work in these, your application will suffer regardless of the basic architecture and language choices. The joke at our office is that SOA is really "Same Old Architecture".

                As far as the need for an app cluster, many of the projects I've worked on required XA data sources and shared a data in a database with a number of applications. Managing concurrency here has to be handled at the application server. Simpler applications that can have thier concurrency managed in the database have the advantage of a more direct pipe from the view to the data. You can take shortcuts here like simply using a round-robin apache front end to non-clustered app servers pointed at the same database server.

                Again, it all depends on your app's requirements.

                •  yes, requirements rule (1+ / 0-)
                  Recommended by:

                  and I've used xdoclet (we just migrated to ejb3 in one app) but I think xdoclet actually helps make the point that many have made here - that j2ee is perhaps too big a hammer for many apps.  Hopefully Hunter's gotten an appropriate taste of the app server wars.

        •  My only quibble (0+ / 0-)

          is that Maven2 is easier and more efficient than Ant build scripts.  It's time to go give it a chance again.  

          I could parade the bennies for you, but the bottom line is that if I was starting ANY new project in Java I would do it with Maven2 without any hesitation.

          Why does Shrub never talk about his first wife, Reality? He divorced her with prejudice and all of his alimony checks to her have bounced.

          by nepolon on Thu Jan 04, 2007 at 12:39:43 PM PST

          [ Parent ]

          •  really? actually tried to use it in a prod env? (1+ / 0-)
            Recommended by:

            It's an overly complicated bloated POS. It's not ready for prime time. I was enamoured with Maven when it first came out, conceptually at least. Getting Maven to integrate into existing applications is impossible, Maven2 doesn't fix this. Understanding the module dependencies and which property does what is unbelievably complex and requires a specialized person on the team. I know because I spent 9 months working with one of the few people in the country getting paid to sit around and figure out how Maven works and which small animals were required sacrifices to get the damn thing working.

            Maven experiences have been as legendarily frustrating as large SCSI setups were 15 years ago. Nice tech, but let's see you actually try and use it.

            I dislike Ant, it offends my sense of engineering aesthetics, but it works. Maven may be a beatiful idea, but if the build & deploy team can't get your code built from version control and they can build everyone elses, I'd blame the tool.

            Wake me up when someone actually documents Maven in a sane way. BTW, my experience was over the past summer, but I've been trying Maven on and off for 3 years.

            •  yes, in a prod env (0+ / 0-)

              but it is worth noting that there is no production role for Maven.  We use it to build, test, and package our apps, tag releases, integrate into our Continuous Integration package.  We don't rely on Maven to copy our jar, war, or ear archives to a staging or production environment.

              I dunno what you are talking about in terms of "understanding the module dependencies."  The point of dependency management is you don't care which jars are required by the libraries you directly depend on.

              Migrating an existing app to Maven is irrelevant in this context, and in the context of my recommendation above ("I was starting ANY new project in Java").  Migration experiences can vary.  A lot of teams have managed to do it

              If you try to make it behave like Ant, it won't work. It isn't a replacement for Ant, it is a different kind of tool.  Hand-compiling from source with autoconf and installing with dpkg / apt-get are different methodologies.  That is the analogy to Ant and Maven.  

              I'd suggest your team's frustration with documentation is understandable but I started with Maven about 8 months ago, and we use it everywhere we can (We don't use it for non Java work) except for one 7yo web app we maintain.  Putting "Better builds with Maven" (free PDF book, avail online) in front of someone and starting a new project each to get to know it, we spread the word.

              Clearly the nasty licenses on some of the javax.* jars creates some challenges.  Seting up a proxy solves those problems.  Third party proxies are easier to set up.

              Why does Shrub never talk about his first wife, Reality? He divorced her with prejudice and all of his alimony checks to her have bounced.

              by nepolon on Thu Jan 04, 2007 at 02:03:26 PM PST

              [ Parent ]

              •  You've made my point (0+ / 0-)

                Set up a proxy for a build tool? How do you build when the proxy is on the corporate network and your build env on your laptop is on the plane with you? Nasty licenses kill the build tool? This is insane, it's a build tool. If it can't build my project and I can't massage it to build the project the way I want it built then it is broken. I needed code compiled, XDoclet executed, some file manipulation, java2wsdl, wsdl2java, XDoclet again, some more file manipulation and then jar, war and ear packaging. The packaged system was then deployed on WebSphere using the websphere CLI utils and jython scripts.

                Running the unit tests and generating some reports would have been nice, but I could live without it.

                I didn't want to design the project's file layout around Maven since I had more important tools that wanted to dictate filesystem requirements.

                Maven is an ivory tower project, it doesn't play well with others. I read the book you mentioned twice, it doesn't begin to cover the flexibility that Ant provides. Also, I'm not sure what environment you work in that allows everyone to become proficient with the build tool, but our deadlines never allow that. I tend to be the person who winds up in charge of the build architecture, and there is no way I could support my team with Maven. I had the pleasure of spending a month with a person who's position was to integrate Maven into the corporate IT shop's app deployment, they had been paid for the last year to work on Maven integration. They could not give me a clear set of instructions to move my app into their build environment. This was not some idiot who didn't know what they were doing, this was a competent engineer who had to spend a year reverse engineering Maven to figure out how to make it do the few simple things that Ant does. In the end, we had Ant build a "Mavenized" version of the source tree that had all of the source and file generation already executed. This was then passed into Maven with some simple scripts that just packaged and deployed the application. That was using custom Maven libraries for WebSphere the engineer had got working over the passed 6 months.

                No abstract promise of software aesthetics or idealized systems will ever make me waste resources like that. I will take Ant, with it's warts and herpes and all over Maven until I can quickly integrate it into existing projects and have the flexibility not to have my build system dictate technical requirements. Until that happens Maven is just vaporware. Ant may be ugly, but it gets the job done.

                •  getting irrational (0+ / 0-)

                  you are getting irrational--you betray too much emotion for a reasonable discussion.  I can drive a truck through holes in this rant, but you don't seem interested so I won't bother.  I'll just answer a few generalities:

                  As I said, if you treat Maven like Ant it won't work.  That is demonstrated by you charging you want a very particular build process rather than starting from the point of asking what you have to do and going forward from there.  

                  Sure Ant provides build flexibility, but comparing the time spent for most projects between building a build tool (which is what Ant requires) and the instant access I have and I won't go back easily.  When I need Ant's power, I run Ant targets from within Maven, but fairly soon after we just write a Maven plugin that is specialized for the job because that gives us the means to reuse that target elsewhere.

                  A DK4 project with the frameworks you outlined won't need Ant.  I know because we are actively building an EJB3 - Tiles - JSF ERP module for JBoss right now with Maven.  After a great post about technology that seems appropriate to the need, a dogmatic position on build tools seems silly.

                  As for time:  it took no more than 30 minutes in a training session I ran, then 30 minutes of individual work for my team members to begin using Maven.  Within a couple days they were teaching me to do things with it that I hadn't needed to do yet.  We wrote an archetype to bootstrap our most common app structure in two hours and can now start a new project in that structure in 2 minutes.  

                  When you try to use a new and different tool in an existing complex environment, like your team did, there will always be growing pains.  It seems the process was as much to blame as the tool, as it usually is in such cases.  

                  We seem to have different expectations from our build environments: I want my build process to be free, or nearly so (because I don't want to explain to a client those overhead costs in putting together a simple application).  I want it to be standardized so new team members can be effective faster.  I want it to be transparent to the application, because my team has enough on their minds.  I want it to provide a close match to our coding philosophy.  Maven is all of that to me.  

                  Sorry your experience was poor, but I can assure you that we have had quite a contrary and legitimate experience.  

                  Why does Shrub never talk about his first wife, Reality? He divorced her with prejudice and all of his alimony checks to her have bounced.

                  by nepolon on Fri Jan 05, 2007 at 10:49:54 AM PST

                  [ Parent ]

                  •  one last time, real slow (0+ / 0-)

                    Your equation of passion with irrationality is itself irrational, so let's ignore that and move on.

                    As I said, if you treat Maven like Ant it won't work.  That is demonstrated by you charging you want a very particular build process rather than starting from the point of asking what you have to do and going forward from there.


                    You're really not listening, I'm not trying to get Maven to work like Ant, I'm trying to get a build tool to complete the steps required by the source code to build and package the project. Ant hasn't got anything to do with the steps required other than it allows me to make those steps, Maven doesn't. It's that simple.

                    My particular build process is dictated by the project requirements, not any requirement of Ant.

                    You keep proving my point that Maven advocates are a bunch of Ivory Tower types who don't have to play in the real world. The idea that I have to change my project to conform to some build tool is ludicrous. What if the IDE we need to use makes project layout requirements of it's own. Or some other tool conflicts with those requirements. Do I have to re-engineer every project if I wanted to migrate it to Maven? As I've repeatedly stated and you keep ignoring, the steps required are because that's the only way the project gets built. If specific tools are not executed on the codebase in a specific order, the resulting package will not compile, deploy or run. Neither Ant nor Maven are in that list of tools, but only one of them seems to be able to make this happen. It is Maven's inflexibility that is the problem. Also you haven't answered the issues of network detachment, which you very well know is a problem with Maven.

                    Sure Ant provides build flexibility, but comparing the time spent for most projects between building a build tool (which is what Ant requires) and the instant access I have and I won't go back easily.  When I need Ant's power, I run Ant targets from within Maven, but fairly soon after we just write a Maven plugin that is specialized for the job because that gives us the means to reuse that target elsewhere.

                    Just listen to yourself. You claim Ant requires you to build a buildtool (whatever that means) and then point out how Maven requires you to either use Ant or write a plugin. Your going clockwise around the block instead of counter-clockwise, but you still had to go around the block. Also, where I've only got to support Ant, you have to support Maven and Ant or a custom plugin. No thanks. I've never had to write a custom Ant target for a project and I don't intend to start making projects dependent on Maven plugins.

                    A DK4 project with the frameworks you outlined won't need Ant.  I know because we are actively building an EJB3 - Tiles - JSF ERP module for JBoss right now with Maven.  After a great post about technology that seems appropriate to the need, a dogmatic position on build tools seems silly.

                    I never said it would. I said that the project I last worked on with a full time developer doing Maven work had specific technical requirements that could not be fulfilled by Maven. Specifically with the execution of tools like XDoclet and IBM's versions of Java2WSDL / WSDL2Java. I'm not the one being dogmatic, Maven wouldn't let us use these tools without writing custom plugins. You're being dogmatic by telling me I should have made a tool work that didn't freaking work. I'm not going to write and support a custom plugin for standard WebSphere utilities and XDoclet, I want XDoclet and IBM or Maven to provide that.

                    Do you seriously think that I'm going to walk up to my client and tell them I want a few days to build a custom Maven plugin to do something that could be made to work in Ant in an hour? Personally, I try not to waste the people's time who write the checks. It works out better for me that way.

                    As for time:  it took no more than 30 minutes in a training session I ran, then 30 minutes of individual work for my team members to begin using Maven.  Within a couple days they were teaching me to do things with it that I hadn't needed to do yet.  We wrote an archetype to bootstrap our most common app structure in two hours and can now start a new project in that structure in 2 minutes.

                    Why would I do this for one project? I'm not "the" IT shop for this client. I was on a team of contractors hired to make shit happen on a project that was a year behind schedule. My goal was to put this thing in production bug-free by the deadline.

                    The project structure of this application was different, due to it's particular requirements, than the next one I worked on. One had to expose our EJBs via SOAP where as the other didn't, but needed a web front end. There were similarities in the project structure, but they were anything but identical. Besides, in the web project, the lead dev for the front end decided to use some Eclipse plugin for the JSF / Tiles configuration. It had some particular requirements for file layout. Was I supposed to tell him that he couldn't use this tool cause it would break Maven?

                    I'm not sure what kind of software you write, but I do about two large build systems a year, because I do about two large projects a year and I'm always the guy who figures out how all this stuff winds up in the appserver. Maven has never made my life easier, only harder. Also, we never had a problem with "using" Maven, like typing mvn. The problem was making Maven actually do what we wanted.

                    When you try to use a new and different tool in an existing complex environment, like your team did, there will always be growing pains.  It seems the process was as much to blame as the tool, as it usually is in such cases.

                    OK, now you're just pulling stuff out of your ass. There was nothing wrong with the process. We needed our EJBs to get generated by XDoclet and then exposed via SOAP before we stuffed it into WebSphere. There is no error in that process, that is what the project required, not the build tool. Growing pains were budgeted for, having to re-engineer the project wasn't.

                    We seem to have different expectations from our build environments: I want my build process to be free, or nearly so (because I don't want to explain to a client those overhead costs in putting together a simple application).  I want it to be standardized so new team members can be effective faster.  I want it to be transparent to the application, because my team has enough on their minds.  I want it to provide a close match to our coding philosophy.  Maven is all of that to me.  

                    Sorry your experience was poor, but I can assure you that we have had quite a contrary and legitimate experience.

                    Well obviously Maven works for some people or I wouldn't have to put up with all the wide-eyed evangelicals it has spawned. I require my build environment to get the job done and be maintainable by the people that support the project after me. I build large applications that take 6 to 9 months with a team of very experienced devs. We put it in production and transition it to a support team. Making Ant do all of the different things our project requires is a matter of a couple of hours of work, and most of that is figuring out what all the requirements are that the devs have built in and dealing with the expectations of the deployment environment. I'd have to do that with Maven as well. Ant lets me finish the job faster, period. It would be nice if we could produce cookie cutter stuff for all our projects, but it doesn't work that way. If it did, we might use Maven for the unit testing and reporting stuff, as I stated previously.

                    Also, I'm not sure how you can explain to a client that you need to set up a repository for your build tool is easier than saying you need to write a build script. The repository concept is another "nice in theory, craptacular in practice" bits about Maven. Since I do a lot of corporate work, we wind up having to use proprietary stuff that the client has dictated, app servers, databases, rules engines, middleware, etc. Maven requires that we put all of these proprietary libraries in the repository. Ever tried to setup a Maven repo with all the stuff WebSphere needs? It's no where near as quick as writing a build script.

      •  Have you ever seen Steve Yegge's blog? (0+ / 0-)

        He hates Java, just ask him.

        In fact, he hates Java so much he was hired away from Amazon by Google. Or some such.

        The Shapeshifter's Blog -- Politics, Philosophy, and Madness!

        by Shapeshifter on Wed Jan 03, 2007 at 06:57:35 PM PST

        [ Parent ]

      •  Wait a sec - (2+ / 0-)
        Recommended by:
        brillig, ppluto

        you hate java because it is syntax-laden, and lisp because it is syntax-less? ;-)

        I don't know where DK's choke points are, but surely an actual compiled language would help?

        Never wear your best trousers when you go out to fight for freedom and truth. -- Henrik Ibsen

        by mik on Wed Jan 03, 2007 at 07:46:39 PM PST

        [ Parent ]

        •  Maybe he just doesn't like garbage collection n/t (1+ / 0-)
          Recommended by:
          •  But that would (1+ / 0-)
            Recommended by:

            eliminate perl, python and ruby, too.  perl and python use reference-counting for GC (read "major memory leaks" if you ever end up with circular structure).  Ruby uses a simple mark-sweep (read "slow and halty like truly ancient lisps and early java VMs").  None of them are anywhere near as efficient gc-wise as java or modern lisps.

            If you really don't like GC, you'd need to stick with low-level languages (like, say, C) or use a language which gives you a choice (C++) and stick with it (avoid libraries that use one of the major GC implementations).

            Never wear your best trousers when you go out to fight for freedom and truth. -- Henrik Ibsen

            by mik on Thu Jan 04, 2007 at 06:53:48 AM PST

            [ Parent ]

      •  Whitespace syntax (0+ / 0-)

        Python's "meaningful whitespace" for blocks syntax... I mean, ugh. Just ugh...

        Hear! Hear!

        I looked at Python, and liked parts of it, but this whitespace thing totally threw me off.   I believe in intentional overuse of brackets and parens in programming because it communicates with the reader.  

        I can't imagine fearing all the time that an accidental removal of a tab or space would suddenly change my whole program.

        Punctuation tells you what the programmer MEANT to do.  Just because a programmer knows how the compiler will handle a series of computation or logic symbols
        doesn't mean he should leave off the parens.

        'Meaningful' whitespace just requires a responsible coder to use comments to indicate the beginning and end of logic blocks.

        For this reason alone, I have avoided Zope and Python.

      •  Why whitespace is significant in Python (0+ / 0-)

        Whitespace is significant in Python because it helps you catch mistakes.  In block-structured programming languages that use brackets, it is easy for your indentation and your brackets to get out of sync with each other, especially as you make modifications to existing code.  Experience has shown that even though the compiler or interpreter reads the brackets, the programmer reads the indentation.  This can be a source of frustrating bugs that should be obvious, but prove embarrassingly hard to find.

        Python eliminates the problem by making the compiler/interpreter read the same thing that programmers generally do (in practise).  If you're used to using brackets, it will annoy you for the first few days, and then you'll forget about it (unless you mix tabs and spaces, in which case your suffering will be legendary, even in Hell).

        I agree with you on Java --- it is a perfectly logical and well-executed consequence of a few serious design mistakes.  It's better than a lot of things widely used in corporate environments (don't get me started on PL/SQL), but that doesn't make it good.  I disagree on Lisp: a big part of the reason Python is good is because its designer knew Lisp.

        •  Why significant whitespace is a bug not a feature (1+ / 0-)
          Recommended by:

          Whitespace is significant in Python because it helps you catch mistakes.

          A good editor like emacs does this for you with auto-indentation and a "status" line indicating what your closing brackets match in your code.

          To build whitespace and indentation requirements into a language is remarkably inflexible and short-sighted, since many programmers have very different styles of coding, proofreading and error-checking, but more importantly is a style-enforcement hack made at the wrong level of abstraction: style should be enforced on the user-level, not in the language definition.

          If you appreciate intelligent organization in the vein of MVC (or MVP, MVCC, whatever) then you should dismiss a concept like this prima facie.

    •  Java is what I would use ... (2+ / 0-)
      Recommended by:
      tmo, TalkieToaster

      although a worthy system for huge, scalable systems is (believe it or not) Tcl along with AOLServer. A few years back there was a very successful CMS based on it (ACS, the ArsDigita Community System) and I implemented an in-house intranet for the NY Times with it. AOLServer is rock-solid, hard to hack, and well-proven by it's namesake ..

      Back to reality :)  Java would be the sensible way to go, I'm not a huge Struts fan but which web framework you use is rather immaterial. Spring, WebWorks, others are available

      It's all about the appearance of security, stupid!

      by zeke7237 on Wed Jan 03, 2007 at 04:08:58 PM PST

      [ Parent ]

    •  Notably... (0+ / 0-)

      I've seen nobody provide a sound technical reason to not choose Java.  It's all been "java sucks" or equivalent.  No commentary about scalability, tool support, complexity, etc.  Just, "eh, I don't like to type that much," which suggests a lack of understanding of the language in the first place.

  •  You could be talking Swahili on this one (2+ / 0-)
    Recommended by:
    Wee Mama, redmcclain

    for all that I can make of it.

  •  Sorry, no comprendo, sr. (5+ / 0-)

    But I think it should come with one of those 7-11 slushy dispensers, if possible.

  •  My vote is for BASIC n/t (1+ / 0-)
    Recommended by:

    Turn Maine Blue - A SoapBlox powered community for Maine

    by Craig Burnham on Wed Jan 03, 2007 at 03:35:35 PM PST

  •  I prefer Tar Pits (3+ / 0-)

    By far the most interesting language, and one which ensures that your work will be around for a long, long time:
    Dire Wolf skulls

    More seriously, I care a lot more about functionality than about what language the site is written in.  I suggest that you start with a feature list, and then look at whether each language has particular features and libraries which will make writing that feature easy, and go from there.

  •  Don't know much about science books... (4+ / 0-)
    Don't know much about the French I took...

    But I do know that I love you...  (Awww)

    Seriously, I don't know much about markup language.  However, I would like to see some improvements in the search engine, and maybe a quicker load time for the comments...  Some of these 500-count comment diaries crash my browser.

    "I believe in compulsory cannibalism. If people were forced to eat what they kill, there would be no more wars." - Abbie Hoffman

    by Jensequitur on Wed Jan 03, 2007 at 03:36:17 PM PST

  •  I don't (2+ / 0-)
    Recommended by:
    larryrant, nhcollegedem

    have a clue . Not going to vote , you can't make me .

  •  re (8+ / 0-)

    Have fun trying getting Dkos to scale with Ruby.


    Yes. I know its got all the Web 2.91002 buzzwordy-ness and yes you will see posts that say "it will scale" all I am saying is have fun if you go that route.

    And I really think  tossing out PHP "for security" is overblown by a mile.

    Look at yahoo and tell them PHP is a poor solution.

    In the end I would not use any of the choices you mention.

    But then again I am not going to program it or get up at midnight when things get messy.

    So its your choice.

    Just make a choice based on all the facts and not buzzwords or hyperbole.

    Best of luck! (really!) I want DK 4.0 to kick ASS!

    "Steve Holt (D - Arrested Development)" - Steve Holt

    by cookiesandmilk on Wed Jan 03, 2007 at 03:38:42 PM PST

  •  For a CMS, you might look at SNAP by Intermix (1+ / 0-)
    Recommended by:

    Where I work, here at Fox Interactive Media (I'm sorry! We hate Fox News too!), we're using SNAP (formerly known as 'Grab in a Box').

    This is partially 'cause Fox bought Intermix which made Grab in a Box - but, Fox bought the company 'cause it's good.

    "Think. It ain't illegal yet." - George Clinton

    by jbeach on Wed Jan 03, 2007 at 03:39:02 PM PST

  •  PHP (6+ / 0-)

    what other considerations besides security make you anti-php? and re: security, my understanding is that php is no more unsecure than any other language - it just has a lot of inexperienced coders who don't know what they are doing. I'm not an expert by any means though.

    Re: Speed - Google loves Python and manages to make it handle some amazing loads. Certainly dkos lacks a Google-sized budget, but Google provides a great example of what Python can do.

    I'm no coder, but I did start out with Perl and always loved how you could do the same thing a hundred different ways. The flexibility of the language may be useful for DK4. Not to mention a LOT of people know Perl and there is a lot of pre-existing open source code that may come in handy.

    "The power to dominate rests on the differential possession of knowledge" -Foucault

    by Jett on Wed Jan 03, 2007 at 03:39:47 PM PST

    •  PHP? Don't make me come over there. n/t (4+ / 0-)

      /* You are not expected to understand this. */

      by ct on Wed Jan 03, 2007 at 03:42:37 PM PST

      [ Parent ]

      • know where I live! (2+ / 0-)
        Recommended by:
        Canadian Reader, taylormattd

        Did you see that the big redwood in my backyard blew down? It fell away from your old place and missed our neighbors house by about 9 inches. They couldn't open their back door because the trunk was up against it. It ruined my birthday.

        "The power to dominate rests on the differential possession of knowledge" -Foucault

        by Jett on Wed Jan 03, 2007 at 03:45:53 PM PST

        [ Parent ]

        •  Holy shit, no (2+ / 0-)
          Recommended by:
          taylormattd, AnonymousArmy

          I've actually got to drive by there in a little bit to pick my daughter up from daycare -- I'll have to keep my eye open.

          At least it fell the other way. Having it hit the old place would have been a little bit too much.

          /* You are not expected to understand this. */

          by ct on Wed Jan 03, 2007 at 03:55:40 PM PST

          [ Parent ]

      •  re: security (6+ / 0-)


        Could you pass a link or two that best summarizes PHP's security holes as you see them?

        Personally, my argument for PHP is a bit like my argument for gun control:  PHP doesn't kill; crap programmers kill.  Handling globals and user input properly tightens up PHP nicely.

        he he he

      •  Seriously, what are you problems with it? (0+ / 0-)


        I was the lead developer for DFA for the past year and a half.

        Mostly because of legacy, I started out using PHP and stuck with it for my tenure there (I still do development work for them).

        Using ugly, spaghetti code PHP I wrote DFA-Link:

        More recently, our other two sites were built with CakePHP, a Ruby on Rails port.

        There's nothing wrong with PHP just as long as best practices are used, and CakePHP really enforces those.

        PHP is used to power Wikipedia, Yahoo, Drupal, and other huge, huge projects. I actually now prefer Ruby on Rails, but there's nothing wrong with PHP in the hands of a professional developer.

        •  It seems to me that the majority of the web is (0+ / 0-)

          done in Apache/MySQL/PHP, including most retail sites where security is necessary.  It's powerful, easy to use, and has a huge number of preassembled library files for download.  I'd choose PHP over Perl any day.    

          Good point about Google/Python, tho...

          War is not the continuation of politics by other means. On the contrary, it represents a catastrophic failure of political skill and imagination. - Kofi Annan

          by Arclite on Wed Jan 03, 2007 at 06:11:30 PM PST

          [ Parent ]

        •  It was mostly a joke (0+ / 0-)

          Jett lived behind my old house, so I know where he lives and could actually go over there.

          That said, I don't personally care for PHP; I can certainly work with it, but I've always liked perl more.

          /* You are not expected to understand this. */

          by ct on Wed Jan 03, 2007 at 06:26:06 PM PST

          [ Parent ]

    •  PHP an insecure language? (1+ / 0-)
      Recommended by:

      There was a huge thread on the Bugtraq mailing list about PHP being secure or insecure. I mostly just skimmed it, but archives are online at Securityfocus.

      re: Python. I think Youtube is mostly done in Python too.

      •  if it isn't, it will be soon (1+ / 0-)
        Recommended by:
        Canadian Reader

        This may no longer be true, but I read somewhere that Google converts any website they own over to Python.
        I know Blogger is using Python now - it handles a huge load.

        "The power to dominate rests on the differential possession of knowledge" -Foucault

        by Jett on Wed Jan 03, 2007 at 03:47:42 PM PST

        [ Parent ]

    •  the same thing in different ways thing about perl (1+ / 0-)
      Recommended by:
      Boreal Ecologist

      Try maintaining a code base in perl written by lousy programmers who don't work in the office anymore sometime :)

    •  re: PHP (2+ / 0-)
      Recommended by:
      Jett, Mr SeeMore

      I think most web developers will admit that PHP has it's place if a) handled properly and b) is the right tool for the particular technical requirements of a job.  PHP's low entry-point and easy learning curve gives it a bit of a stigma -- much like an email address...

      It could just be that Hunter doesn't know PHP and has been scared off by what he's read.  No problem; lots of other tools in the shed.  What's bugging me is that the poll doesn't even list PHP or Java as an option to the question "what's your favorite web development language?"

      And...(PS) think twice about hiring a developer who doesn't have at least 2 languages under their belt.

      •  PHP 5 is worth another look (1+ / 0-)
        Recommended by:

        I've never been big on language wars.  Certainly, it's possible to write lousy code in any language.  I've done web development in Python, perl, and PHP, and they all have their place.

        That said, there are several considerations I'd hope Hunter, Kos & Co. took into account:

        • Availability of high quality libraries.  I've seen too many developers "roll their own".  Stop it.  If you're not using well engineered libraries developed by really good people, your ego is getting in the way of your job.  BTW: this is where you're likely getting your notions about PHP security.  Bad projects like PHPBB have bad bugs.  Well engineered PHP projects have far fewer liabilities.  I don't care how good you are (or how good you may think you are);  very few developers are fabulous in more than a fairly narrow area of specialty.  Perl and PHP have a lot of really good code available.  Python has somewhat less.  I'd guess that Ruby has less than that.
        • Good support for building large systems.  Python is the gold standard.  I understand Ruby is nearly as good.  Perl's notion of OO is a bad joke, and writing good large scale apps takes abonormally large amounts of discipline.  PHP 4 was mediocre, but PHP 5 (esp. 5.1 and later) has most of the nice support for large scale development that Python has (hell, I'm guessing the developers looked at Python and stole much of its good OO ideas).
        • Good tools for making large deployments scale.  PHP is probably better here than you're giving it credit, esp. PHP 5.  In particular, some of the stuff I've seen people to get Ruby to scale scares the hell out of me;  if you've every played seriously with lhttpd, you know exactly what I'm talking about, too.
        • Comfort level for the development team.  If you like developing in perl, you like developing in perl.  But like Larry Wall likes to say, TIMTOWTDI.

        I agree with some of what I've heard about Java, mostly because it encourages you to overengineer things.  I've only started to do Java web engineering recently, but it has a lot of C++'s liabilities, including complex syntax, and some insanely complex technologies like EJBs, etc.

        I'm curious what y'all will end up doing.  But I hope you'll resist the urge to enforce NIH, and concentrate on what you do best.

        •  What happens when php6 comes out? (0+ / 0-)

          My problem with php thus far has been that the developors don't give enough of a damn about backwards compatibility.  Php5 is already well into its release cycle and there is every indication that the transition from php5 to php6 will break existing sites once again.  

          By the look of things right now I'll be gradually migrating my sites over to moinmoin, django, and zope/plone.  

          Oh. Look at Ellington CMS:

          It's major pricy though.

          •  PHP 5.1 is a serious improvement over 5.0 (0+ / 0-)

            A lot of the backwards compatibility problems with 5.0 get resolved with 5.1, which is turning out to be a very nice release.

            I think the best thing about the new 5.x object model is the use of "magic methods", more or less on the Python model.  It makes it much easier to write "what you mean", especially for things like web services.

            Right now I'd say the problem is less about having 4.x code break, and more with the inability to move frameworks to use 5.x features, since 4.x is still in wide use.

            I've heard that 6.x is on its way, but I'll believe it when I see it.  Like, when's perl 6 coming out....

            Also, all kinds of fun things have happened with Python in the 2.x releases.  This is one of the reasons I stopped doing Zope development and switched mostly to doing Drupal work in PHP.

  •  If you're gonna reinvent the wheel... (11+ / 0-)

    you should design an interface that other blogs would be willing to switch to -- all this work deserves some extra income. It is stunning that blogs like C&L, AmericaBlog, and others still handle comments like a late 80s BBS.

    •  Reinvented or not... (1+ / 0-)
      Recommended by:

      I would love to see a common interface for all the blogs I read--and if it looks like this one I would be quite happy.

      Are there any thoughts about creating some sort of package from DK 4 that would be available for other blogs?

      (That's a completely naive question, in that I also don't know anything about the currently available options...except that I do know that if I can't respond to a comment right where it sits, any thread gets completely useless almost immediately.)

  •  I feel like (2+ / 0-)
    Recommended by:
    Wee Mama, splinterbrain

    Marcus Brody in the bazaar.

  •  Slow start (0+ / 0-)

    So far just a lot of posts from non-techies about how confusing this all is. I'm not much help on the web application side. I do more with hardware, networking and OS. If you need any ideas about hardware or picking a Linux distro, feel free to ask.

  •  Wish I could help (2+ / 0-)
    Recommended by:
    Shapeshifter, joanneleon

    but I gotta get back to wrestling with ASP

    -6.25, -5.18

    "If you treat people right they will treat you right - ninety percent of the time" - FDR

    by assyrian64 on Wed Jan 03, 2007 at 03:43:21 PM PST

  •  Imprecise Perl -- a blessing and a curse (10+ / 0-)

    PHP "security issues" really isn't a fair brush off.  PHP may have some inherent vulnerabilites that are unsettling, but any one of those languages will allow you to shoot yourself in the foot just as easily (and far more likely).  I understand that people are inherently more suspicious of things that are outside of their control.

    I'm biased towards mod_perl because its easier to "just do it" in a crunch time with some horribly obfuscated run-on sentence of code instead of having to abstract your way up six levels and back down five in a different level just to fix some parsing error or over-reaching SQL select statement.  I'm equally biased against it because nobody ever refactors that Perl code post crisis to anything resembling sanity.

  •  Regarding CMS's... (0+ / 0-)

    There is one company out there with products that I find interesting: Vignette.  Basic out of the box, but HIGHLY customizeable.  For the front end, you use DHTML combined with a proprietary scripting language (it's not bad).  On the back end, you use Python (ok, actually it's Jython) to access their API (which I believe is total Java).  I've done a fair amount of work with this CMS recently.  It really is not very practical for building a bigger, more robust application on top of (which we are), but for straight customization and nice features, it ain't bad.

    "I am not a member of any organized political party — I am a Democrat."
    ~Will Rogers

    -5.25, -4.87

    by cotasm on Wed Jan 03, 2007 at 03:44:49 PM PST

    •  I hope it's gotten better (0+ / 0-)

      I used it about five years ago (I know, a lifetime in this business) and it was a disaster.  At that time, the frontend stuff was not really ready for primetime (required using TCL, company was promising and still developing other options) and configuring was a royal pain.  But the worst issues were the performance problems we encountered when the data base started to get bigger.  

      We also had a lot of trouble finding people who had any experience with it.  I'm sure the product has matured and gotten better by now, but back then it was a real problem.  I got off that project just in time because I was about to own it and the group who developed and maintained it.

      "War against a foreign country only happens when the moneyed classes think they are going to profit from it." -- George Orwell

      by joanneleon on Wed Jan 03, 2007 at 06:01:05 PM PST

      [ Parent ]

      •  I agreen with Joanne (0+ / 0-)

        My best friend worked for Cheap Tickets when they tried to develop their ticketing sales system using Vingette (four years ago).  It was a disaster.  The stuff was so unoptimized that they need four times the number of servers to get the same performance they were from ASP.  

        War is not the continuation of politics by other means. On the contrary, it represents a catastrophic failure of political skill and imagination. - Kofi Annan

        by Arclite on Wed Jan 03, 2007 at 06:22:35 PM PST

        [ Parent ]

  •  Don't buy your premise... (2+ / 0-)
    Recommended by:
    cjohnson, montpellier

    About rejecting Java.  Of course, you're going to be programming it, not me, so it's your call.

    I know a little Python, but not Ruby.  I know a bit of Perl, but HATE the fact that anything can be done in "a hundred different ways" (to quote a comment upthread).  That kind of "flexibility" just makes maintenance hard.

  •  I do this for a living (7+ / 0-)

    I do, among other programming tasks, web programming for a living.  I have used PERL, ASP.NET, Servlets, Ruby on Rails but not python(complete honesty here: python's enforced whitespace drives me up the effin wall).  In my experience, Ruby doesn't scale as well as I would like.  I think that for the combination of speed and flexibility that you seem to need, I would recommend mod_perl.  mod_perl is fast, it scales well, and there a ton of modules and platforms that can provide you with ready-made or easy-to-modify-to-what-you-need solutions.

    •  Can you elaborate on Rails scalability? (1+ / 0-)
      Recommended by:
      Josh or Con or Both

      Reviews on that point are mixed, at best, and I've never seen anyone go into meaningful detail about, if it did indeed have a problem, why it might be a problem.

      Mod_perl has its own problems, of course. You can make things quite fast, but for many applications it's a huge friggin' memory hog.  Just huge. Sometimes that counteracts whatever speed boosts you can get from it, in my experience.

      •  There are ways of shrinking its memory footprint (3+ / 0-)

        If you organize things carefully and load dynamically where appropriate, you can keep mod_perl small.  There are lots of tricks to employ.  I've been working with it for many, many years, and I'd be happy to contribute that knowledge toward the effort.

        The biggest memory hog in my experience has been the proprietary CMSs.  I have thoughts about how to improve upon that; I'm confident there are ways of customizing to do so. (Or building your own, as an alternative.) With the sites I've worked on it never really got to that point that I had to do this, but having anticipated it, I think I could be made to work and work well for dKos.  Again, I have suggestions any time you want to pick my brain.

      •  I don't have solid numbers in front of me (4+ / 0-)

        But the one that sticks out in my mind as when we did a proof of concept with a fairly data intensive Ruby on rails app and it did well until we got into the tens of thousands of simultaneous connections.  Performance didn't fall off a cliff, but it did drop off noticeable, relative to the number of connections being added.

        I have done smaller Ruby on Rails projects and they performed fine, but have had some complaints about responsiveness as the apps get heavier usage.  I have even been asked to improve the performance of a Ruby web app.  I turned that done after an inspection because I didn't know how I would do that, which may be the root of the problem.  I am not a Ruby expert so I may be doing things that are contributing to the scaling problems and not doing things that would help.

        You are right about mod_perl and memory.  But if you load dynamically and are careful, I have found that the trade off is worth it for the extra performance and flexibility.

        One thing that perl lets you do, that I don't know that Ruby does, is run C code through PERL. That can be a help when you are really, really desperate for speed.  Similarly, if you are already writing PERL apache modules, going to C apache modules for speed is easier, at least in temrs of mindset.

        But then, if you hate Java, I can only imagine what your opinion of C is ;)

      •  you can make a lot shared (0+ / 0-)

        Just preloading everything makes things way way better, but I assume you're already doing that.  In fact just looking at something like top can be misleading if you're not taking the shared memory into account.  

        Also Scoop instantiates a hellacious lot of short-life variables for each request, so child processes require a lot of memory compared to tighter mod_perl code.  

        BTW do you guys use memcached or don't you?  If you aren't you really ought to check it out.  

        •  so what? (2+ / 0-)
          Recommended by:
          klondike, jfm

          Language performance isn't even in the Top 3 list of performance bottlenecks for a web application. Mostly it has to do with RAM size and bandwith, DB latency, and how good your caching is. Most web server boxes spend most of their time idle.

          "Instead of asking what you could do, you ought to have been asking what needs to be done."

          by khaavren on Thu Jan 04, 2007 at 10:49:59 PM PST

          [ Parent ]

      •  Interfaces to web servers not real mature (1+ / 0-)
        Recommended by:

        I think that ruby is just a bit too new.  Even with apache, hooking a ruby app up with it lacks a certain... I don't know what :-)  

        That people spend a lot of time hooking it up with FCGI should scare you at least a little.  That people would consider running real web sites on top of lighttpd should scare the hell out of you.

      •  I don't think "scalable" is the right word (1+ / 0-)
        Recommended by:

        Rails is slow, primarily because Ruby is slow. And as everyone in the Ruby world waits for YARV or JRuby or Rubinius, this speed gap is mainly dealt with by:

        • optimizing code
        • caching
        • throwing hardware at the problem

        None of which I see as very problematic. I think the development speed and code quality that are encouraged by Rails more than makes up for a bit more resources at the end. Rails tends to be used by very skilled web developers who are often working on innovative or relatively large projects, and as such there are some really good tools and guides out there for setting up load-balancing, database cluster, using memcache, etc.

        Speed-wise Rails starts at a handicap, but given the benefits of the platform and surrounding community, it's not much of a concern. I think much of the criticism of Rails comes from people who used it in its infancy - before 1.1, before mongrel, before the large community of people designing solutions for "serious" Rails apps

    •  I would just like to echo kcr (2+ / 0-)
      Recommended by:
      zericm, Josh or Con or Both

      I would like to add another vote for mod_perl. In my limited programming expereince I have found mod_perl to be great with its flexibility and the mods have been a life saver.

      What would prevent Captain America from being a hero "Death, Maybe"

      by Doughnutman on Wed Jan 03, 2007 at 04:23:29 PM PST

      [ Parent ]

    •  Python (2+ / 0-)
      Recommended by:
      kurt, jfm

      (complete honesty here: python's enforced whitespace drives me up the effin wall)

      I've seen a number of people say this, and I just don't understand it. You properly indent your bracketed blocks in C-style languages already (except for one-liners, which Python supports), right? So what's the problem?

  •  Requirements? (28+ / 0-)

    I didn't get the link to your requirements and really that is what SHOULD drive your technology choices.  Also, what is the current site using and what challenges are you faced with in doing DK4 in that same technology.

    I'm a java architect and like they said above, lots of sites run lean on java but I don't know anything about your requirements other than speed and in-memory changes (concurrency issues)...and AJAX support.  I have to tell you that I'm not a fan of RoR - mostly for lack of DB support and just some basic security stuff that should be in the platform.  My guess is that you will find most of what you are looking for with python.

    Also ask the slashdot folks what they use...

    Feel free to talk to me offline about this if you want...I wouldn't mind doing some free work for DKos.  My way of giving back.

  •  I like Python (9+ / 0-)

    Because the name sounds more kickass than Perl and Ruby.  And that's the extent of my ability to help on this subject.

    Most of the things worth doing in the world had been declared impossible before they were done. Louis D. Brandeis

    by pontificator on Wed Jan 03, 2007 at 03:50:10 PM PST

  •  Stop! I'll Tell! (1+ / 0-)
    Recommended by:

    This is what we say when we want to torture someone and get them to reveal their secrets! It's better than waterboarding!

    You think your Thanksgivings with GOP relatives get bad, try wandering into a room full of programmers and saying "Java sucks!" or "I just got a 2.18% speed increase in my data archiving cronjobs by converting them from Ruby to Python."

    This President is in a league of his own. The BUSH league!

    by Tuba Les on Wed Jan 03, 2007 at 03:50:59 PM PST

  •  1. Java, 2. Perl, 3. Python (10+ / 0-)

    Not putting Java on that poll is like not putting Al Gore on a 2008 prez straw poll. Why are you shutting out the heavyweight that should obviously win?

    IMFO Java is about the best mix of run-time performance and ease of development for lots of tasks including web-app development. There are some huge sites out there running it, bigger even than

    I wrote in Java, and like pacified says, all the SoapBlox sites like My Left Wing, Swing State Project and many others are running that Java based software.

    1. Election Reform 2. Kucinich 3. Richardson 4. Edwards

    by bolson on Wed Jan 03, 2007 at 03:57:06 PM PST

  •  Creating translation intensive applications... (8+ / 0-)

    takes a lot more than just choosing a language. Scalability is much more than throwing more machines at the problem. Excluding java with all of it's available frameworks is just plain laziness.

    I learned the lessons of picking the wrong platform and the price you pay. An application I helped designed and build started out using VB. When we ended up having 4 of the top 10 largest global companies as clients we found out the hard way that VB didn't scale for enterprise solutions. We had to switch to J2EE and EJB as our language and framework. This took us almost two years to do.

    We were paying a host to place servers on their rack space. When we converted to J2EE from VB we got back 25 servers. That was a little over 2 1/2 'racks' at our host provider. We saved a ton of money.

    But the biggest bang for you efforts come from using well established design patterns. I currently use the MVP (model-view-presenter) over MVC (model-view-control). I like the pattern because it allows me to write unit test that cover more of the logic plus the added bonus of removing all business logic from the UI. With a presenter all I have to do on the UI side is wired up the events and set up the properties. It makes changing the UI around very quick and easy.

    I'm not that knowledgeable about using Ruby, Perl or Python. My concern if I moved to these languages would be scalability and maintain the code line.

    Good luck on your endeavor. On a translation intensive environment like DailyKos I sure won't 'bet the farm' on using either of those three languages.

    The bottom line is you need to do your research (this obviously is a start). You need to do your prototyping. You need to do your architecture. You need to use design patterns. I will be interested in someday hearing about how DKos4 development went. I'm always open to learning new things.

    •  MVP v. MVC? (0+ / 0-)

      Does Struts do MVP?  Or is there another framework?  I haven't read about MVP.

      Don't start a blog, build a community with SoapBlox - the NEW blog framework.

      by pacified on Wed Jan 03, 2007 at 04:01:11 PM PST

      [ Parent ]

      •  Struts is MVC... (2+ / 0-)
        Recommended by:
        shock, colorless green ideas

        the controller is defined by the servlet and the events are configured by the .xml file.

        The cool part of MVP is that the events are controlled by the backend code. You don't have to mess around with tweaking some .xml file. The idea is pretty simple.

        On your page you have a 'view' interface and you instantiate a 'presenter'. The 'presenter' handles the events and the 'view' handles any 'property' changes that are caused by the events. If you 'Ajax' the events you can eliminate the page 'blink'.

        MVP has been around for quite a while. Martin Fowler calls the presenter the 'Supervising Controller'. Just google 'model view presenter' and you will get links to investigate this.

    •  I meant to write transaction intensive... (1+ / 0-)
      Recommended by:

      not translation intensive. I'm currently working on translation software so my mind is stuck on that word. Sorry for any confusion.

    •  Not a language issue... (1+ / 0-)
      Recommended by:

      "My concern if I moved to these languages would be scalability and maintain the code line."

      Ever heard of slashdot? It runs on perl and has the power to disable other sites. They even made a word for it: "slashdotted".

      Yes, I know that the power to slashdot is a function of the community size and not perl itself, but it has brought down at least one site implemented in every programming language currently under discussion, and itself scales to serve enough links to send a crippling wave of users to another site. Scaling is a function of the programmer and her use of patterns and libraries, not the core language. Code maintenance again has more to do with the programmer than the environment, although many environments do their best to interfere.

      •  And that is why they are important... (0+ / 0-)

        Scalability and code maintenance are truly a function of architectural design and the programmer's ability to implement it. This holds true for most of the website that are out there. However, I'm sure DailyKos is not running on a couple of machines. There is no doubt that on November 7th a few more machines had to be used to respond in a timely manner to the requests that were being made. Sites like Google dedicate whole buildings of machines to provide not only quick request/response exchanges but also reduncancy and monitoring to proactively respond to IT type problems.

        If your application doesn't scale well then you end up paying more money then you should in both equipment and rack space. You only have to look at what happen to Lieberman's website to understand this. If your application is hard to maintain you end up paying expense developer wages to add new functionality and integrate it with the current application.

        So I'm not against using perl, ruby or python. I don't know enough nor I have had enough experience using these languages to 'bet the farm'. I am very familiar with slashdot and their site seems solid enough. But I doubt if slashdot sees the type of request/response hits as DailyKos did in November. However, this is just my opinion and not based on any facts about either slashdot or DailyKos traffic that I have at my disposal.

  •  Hibernate baby! (0+ / 0-)

    Ok, I've never made some successfully in Hibernate more than a "hello world"  Damn lazyUpdateExceptions (or something).

    Though most people I've talked to say Hibernate isn't that great, adding a layer of abstraction that gets in the way sometimes.

    Don't start a blog, build a community with SoapBlox - the NEW blog framework.

    by pacified on Wed Jan 03, 2007 at 03:58:37 PM PST

    •  pppt (0+ / 0-)

      For the level of performance this site needs, I would avoid Hibernate, as query optimization is much harder when the database is abstracted

      "I have sworn upon the alter of God eternal hostility against every form of tyranny over the mind of man." - Jefferson

      by SWicklund on Wed Jan 03, 2007 at 04:12:55 PM PST

      [ Parent ]

  •  Technology (5+ / 0-)
    I'm a Python bigot, so I'm going to restrain my comments to a comparison between Zope and Django.

    Zope is a lot like J2EE in that it has a big, big learning curve, and does a lot of powerful stuff once you figure out how to use it. It you want something that will potentially give you the world, strongly consider Zope.

    If you just want a Slash++ site, Django might be better. I'm not sure how Django is for the persistence layer, though.

    If you want to be really, really ambitious, consider coding the whole thing in D. They have all the modules you need for data and basic connectivity, and the resulting code runs on bare metal instead of a vm. It's a snap to learn if you know C or Java and perl. The MAJOR downside is that it doesn't have all the well-developed bells and whistles of the other languages, e.g., AJAX support is probably beta at best.

  •  These questions are backwards (18+ / 0-)
    1. Define your business requirements
    1. Then look at "off the shelf" software that could fulfill your requirements. Or spec a system that you build yourself.
    1. That will almost certainly dictate the framework you'll need to use, or will at any rate narrow down the choices.
    1. That in turn will almost certainly dictate the programming language you'll need to use.

    Hunter, you're asking these questions exactly the wrong way around, starting with the last one and working your way up backwards. And the most important question -- what are your business requirements? -- doesn't appear to come up in your text at all. Not that I can recognize, anyway.

    Absolute power corrupts absolutely.

    by Buckeye Hamburger on Wed Jan 03, 2007 at 03:59:12 PM PST

    •  Based on the requirements we already (1+ / 0-)
      Recommended by:

      have been mulling, including our desire to do a great many things using AJAX, we already 90% know there's no off-the-shelf higher-level (CMS) tool that we can easily use. Sorry to be vague on that, but... I have to be vague on that.

      So at this point it's just a question of choosing which one can most easily be drastically modified -- if any.

      I know it looks completely backwards -- but we've already got many of the most critical business requirements, we're just interested in hearing from folks whether or not they've got some flashy tool that they think can jump the moon, in case they mention low or middleware frameworks we haven't heard of that we should be considering -- or problems with some of the referenced frameworks that we might not have considered.

      And as it turns out (to commit a bit of tech blasphemy) many of the lower level frameworks are largely interchangable -- they all solve the same core problems, it's just the philosophy of how they do it that's marginally different. CMS philosophies differ wildly, of course, but for lower-level core frameworks like Rails, mod_whatever, PHP, Java, it's all pretty standard stuff -- they call things different names, but the capabilities of most of them are more similar than not. But that doesn't mean people don't have very strong opinions about every single one of them.

      •  Sorta Hard to Answer. . . (1+ / 0-)
        Recommended by:

        Without knowing the requirements. I understand why you are not sharing, but in all honesty I can't make a language recommendation without knowing what you are trying to accomplish. Indeed, any person who recommends a language based on the limited information that you have given is just making a knee jerk response based on their own biases.

        Don't get me wrong: this has been an interesting conversation to follow. But as an architect, I would not follow your path to selecting a solution.

        My recommendation: select the language that your development team is most comfortable with, then find the best framework for that language. Depending on your choice, your CMS may also be driven by your language choice.


      •  Not really true Re: AJAX (0+ / 0-)

        That's not really true Re: Ajax, but seeing as how you've already ruled out PHP it doesn't really matter.

        Drupal 5.0 has implemented jquery, which in my opinion is superior to a lot of the libraries out there that do similar things because it is simple and compact.

        As a CMS it would also make for an excellent framework if the API was properly utilized.

        Pre-order unConventional, the official photo documentary of YearlyKos 2006!

        by Raven Brooks on Wed Jan 03, 2007 at 07:13:01 PM PST

        [ Parent ]

      •  Rails is the bomb for AJAX. oh yeah, TDD. (0+ / 0-)

        I've been doing a lot of AJAX stuff in Rails lately, and it's super for that. The RJS system is very nice, especially the integration with the testing framework.

        Speaking of testing, Rails has the best setup for doing integrated testing I've ever used. I've been doing TDD (Test Driven Development) in Rails and it's like heaven.

        But here's the kicker. I can concede your point that the basic technical framework stuff is roughly equivalent. But the communities for the different frameworks are quite distinct. And the Rails and Ruby communities are totally great. People are helpful, friendly, and put a lot of energy into contributing to the community. Very much like DailyKos. If you went the Rails way for DK4, I can predict you'd have a lot of volunteer help, myself included, and when it comes to Rails I very seriously know what I'm talking about.

        "Instead of asking what you could do, you ought to have been asking what needs to be done."

        by khaavren on Thu Jan 04, 2007 at 10:58:25 PM PST

        [ Parent ]

  •  PERL, it's gotta be PERL (1+ / 0-)
    Recommended by:
    Bill Evans at Mariposa

    cuz, uh, i've never heard of the other two choices?

    (hiding my head in techno-ignorant shame)

  •  A post for me! (5+ / 0-)
    Recommended by:
    Hunter, rwsab, hhex65, 4jkb4ia, joanneleon

    I have never used, nor looked at Ruby, so I cannot comment on this. I have however, used Perl and Python. Currently, I mainly use Perl for all of my programming automation tasks ... it beats living hell out of that Microsoft command-line interpreter!
    Both Perl, and Python, in my opinion are pretty good scripting languages. Both scale pretty good. I have used both for very small quickie type stuff, to large projects, (large software build automation, test and hardware verification). In my opinion, I say that Perl is probably a bit easier to maintain. Alot of my IT friends say pretty much the same thing. Python has alot of nifty O-O type stuff that makes it a bit easier to scale to very large systems. (You probably already know this ... Perl has a bunch of cool O-O extensions too) So, it is your choice ... a bit easier to maintain, or a bit easier to scale? Perl? or Python? I like ... a bit easier to maintain, so I usually choose Perl. Most of the stuff I have been doing lately is not so large as to warrant a bunch of large Python modules.
    As to CMS, you will most likely get some really expensive out-of-the-box solution, and completely rip the thing apart. Almost everyone these days does it this ways anyway.... (shhhhhh, don't let the managers that pay for this stuff hear this!). My belief, is you will probably windup doing some kind of deeply embedded XML prestidigitation along with a bunch of heavy duty Perl or Python scripting, lather, rinse, repeat ... until you get something that resembles what you want.
    You will never get exactly what you want, but hey close enough, especially with a wonderful website like yours is pretty good eh?
    Who am I? Why, I am not a web designer, nor claim to be one. I am just an Sr Embedded Systems Consultant. Most of the stuff I do is hardware/software stuff. Everything from broadband fiber optic video networks, digital signal processing, medical equipment, to that fancy FASTRAK thingy used on the bridges of the SF Bay.

    No more gooper LITE!

    by krwada on Wed Jan 03, 2007 at 03:59:49 PM PST

  •  On Zope and Other Stuffs (2+ / 0-)
    Recommended by:
    Hunter, 4jkb4ia

    I've used Zope, and for what I was doing, it really really sucked. There was the problem of no single area for development; some of the tasks were completed in the Zope "front-end" web app, some were done by hand editing .py files. All of it usually took about 15 minutes of orientation before all affected objects/files could be assembled in my head, and at no point were all of these visible all at once. I even considered using emacs to get the unity I craved, but then I realized I was on the edge of insanity. (What's a tech thread without some obvious hijacking flame-bait? ;] ) Short version: I really hope, for your sake, that you don't select Zope.

    Don't get me wrong, Python as a language is good for many things and I personally like its rapid prototyping aspects. It does however get pretty slow and isn't the greatest at constantly updating dynamic processing. I'm very language neutral and all 3 that you listed are languages I've had reason to develop in at some point or another. The framework is a much better evaluation point in my opinion.

    I'd recommend against picking a canned CMS; I've been involved with CMS systems (creation and use, including some of the stuff from a certain Washington state developer we all know and love to hate) since the initialized acronynm emerged from the buzz-o-sphere, and they all have seemed to fit like a burlap sack. Sure, they cover your ass and keep the wind out (mostly), but you will never make them fit right without alteration. Metaphors in tech discussions are cheap and easy to break, but I'm serious, the only CMS solutions I've seen work for a customer were customizations, and the ones that work best were built ground up on more basic frameworks.

    Knowing how to implement design patterns to suit your specific needs will get you much further than buying someone's idea of the Universal Design Pattern Implementation. If you are going to spend most of your time shoe-horning or trimming down an out-of-box solution to fit your needs, it may be better to just start your own framework tailor built to fit exactly what you do now, with just a bit of care to extend to what you think you are most likely to do later. You will never hit 100% on either, but you will be much closer than with the oob CMS, and you will also have a known code base that you control and can modify on your own schedule instead of waiting for a release.

    I love that I'm involved in this in any way. Even if it's only to be a crucial griper at the first design meeting, I feel I've done something to evolve the Daily Kos community. For that I think I have enough "good" karma to now retire and reincarnate in absolute peace and bliss.

    Speaking of karma, have you ever looked at slashcode and its many variants? That would satisfy the perl language cadre, and you'd be in the same group as Carl Steadman.

    Best of luck on the new version.

  •  Hijack alert - Q for techies. (0+ / 0-)

    I posted this in open a number of times and haven't got a good answer.

    1.  What is the best, progressive, ISP to use?  We have several websites we like to host.  We have tried GoDaddy and Network Solutions and they are lacking.

    Key features are those of any small media biz: large bandwidth for video, fast Webmail and design help would also be important.

    Thanks in advance for any suggestions!

  •  Language: Perl, Ruby, or Python? (2+ / 0-)
    Recommended by:
    4jkb4ia, montpellier

        I'm Presuming we're talking Server side code. I'm also presuming that the diaries,Comments and metrics are stored in a SQL database.

        Now It would appear that from the server side a building a diary and discussion  blog site is the manipulation and presentation of this SQL data(the client side with AJAX is another matter). This based on work I  was involved in on a product called Participate a Early text based BBS over 25 years ago.

    So the question remains which of these languages is best at manipulating the widest range of Databases with the most efficient use of Server computing power and that it would appear to be Perl

    With Ruby
    Following behind in my professional opinion.

    The Async Javascript and XML code on the client(browser) side i'll leave for some future discussion.

    Be carefull what you shoot at, most things in here don't react well to bullets-Sean Connery .... Captain Marko Ramius -Hunt For Red October

    by JML9999 on Wed Jan 03, 2007 at 04:02:41 PM PST

  •  Just outta curiosity (2+ / 0-)
    Recommended by:
    zericm, JML9999

    Can anyone explain why .NET was completely thrown to the curb before the gates were even opened?

  •  python (10+ / 0-)

    i haven't heard of any serious webapps starting with perl from the ground up... assuming you're looking for a next-generation site that is both scalable and easy-to-maintain, it's really between python and ruby.  

    i agree that the focus should be on the preferred web app framework.  i personally prefer django (the auto-admin interface is phenomenal), but rails is popular for good reason...

    in terms of language, however, i would recommend python for a few reasons:

    1. as compared to ruby, the python community is much larger and more mature.  you can find modules for just about anything you can think of.
    1. python's focus on code readability and developer productivity suggest that it will remain popular for a while.  can the same be said for ruby/rails?  not as sure...
    1. python has large institutional support from organizations like google and nasa.  ubuntu linux (which is making large strides here and in the 3rd world) uses python and pygtk as its language and gui-toolkit of choice.
    1. with the psyco JIT compiler, you can now get python code to run at C-like speeds, taking away what was formerly python's biggest weakness (performance).  memory/CPU-bound applications like dKos should benefit tremendously for such optimization.
  •  In favor of Perl (0+ / 0-)

    Myself, I like Perl and voted for it, but I get the impression that Python is new and innovative enough to distinguish itself yet old enough to enjoy a large development pool.  

  •  Take a look at Django (16+ / 0-)

    You ought to give Django a serious look. It was originally developed in a newspaper environment by programmers with journalism backgrounds, and accordingly it has a bias toward rapidly evolving, news-based sites. It is a general-purpose web framework to be sure, but, for instance, you get a lot of date- and time-based functionality "for free".

    As far as scalability goes, Django is behind a number of high-traffic and high-volume sites, like's Congressional Votes Database, for example. It comes with a caching API built-in, which makes it easy to plug into memcached, so you're not hitting a DB or building pages on every request.

    A lot of Django-partisans tout the "free" admin you get. This is definitely a nice plus: it's slick, and it lets you get up and running quickly and populating data. It may well be enough for many of your admin tasks.

    That said, I've developed a number of sites using Ruby on Rails, too -- don't discount it, it's a very powerful framework in its own right, and it has a very passionate and loyal developer community.

    No matter which you pick, ultimately, you will be happy with either Django or Rails. Python and Ruby are a joy to program in -- both are beautifully designed languages with a lot of developer friendliness. The same goes for the two frameworks built on them. Also, Rails and Django are strikingly similar in many ways -- they address and solve many of the same web development problems -- so don't sweat having to choose; either way, you'll find much more happiness than the alternatives.

    One last point: don't let anyone give you any flack about how "dynamic" languages like Python and Ruby are too slow and don't scale well on the web. That's nonsense: just look at the success is having with Rails, for example. These days, with smart caching (think memcached) and good load balancing, the productivity and pleasure wins you get from developing in these frameworks/languages outweigh hand-wringing concerns about production performance.

  •  Apache + mod_perl + custom CMS (2+ / 0-)
    Recommended by:
    kingubu, montpellier

    That's my personal and professional opinion.  It's very, very fast and gets right into the core of the Apache web server; it can be optimized and scales very well. I perfer it greatly to Python.  Admittedly I haven't worked extensively with Ruby, but I share your concerns about it.

    Also, if you're looking for help, I have many years of development experience in this area (backend, database, search) and would be glad to offer my assistance, as a long-time community member.  Seriously, shoot me an email.  I'd love to get involved.

    •  I agree on the Perl part... (0+ / 0-)

      I agree Perl is the best of the languages under consideration, but I'm unconvinced that Apache is really the best choice under these circumstances.  I have a couple servers running lighttpd, and find it to be infinitely preferable in terms of performance to Apache when you don't need any special features Apache provides.  I'm not sure how well it handles Perl, though, especially at the levels of traffic this site sees.  It's output compression is less resource-intensive (and plays better with others) than mod_gzip, though.

      Failing that, I'd think Apache 2 would be better than the 1.3.34 DKos is now running.

      •  And I agree on moving beyone 1.3 (0+ / 0-)

        ... at least, I never work with it anymore if I can help it and make the effort to migrate sites to apache2.  I haven't worked with lighttpd, and you're right that there may be alternatives, but I don't know of any that offer comparable speed to the Apache + mod_perl combination.

        ps - it's easy to debloat Apache if you know what you're doing.  I guess it helps when you're trained as both a system (and database) administrator and a developer. :)

        •  Sure... (0+ / 0-)

          Sure, it's easy to debloat Apache (mod_frontpage? WTF still used FrontPage?) to a pretty good degree; it's just not a terribly efficient ensamble of software (it is, after all, "a patchy" webserver), and I think that for something like this site, some of the alternatives might well be worth looking at.  From what I recall, a lot of the images are already served by thttpd, but that project really hasn't gone anywhere in the last year or two (not that that's necessarily bad; I like qmail, which hasn't been updated in years...) and it might be worthwhile looking at successors.

        •  DKos uses apache2 on the backend. (0+ / 0-)

          The front-end proxy apache still uses 1.3, mainly because I didn't feel like mucking about updating the proxy configs and figuring out the differences between mod_gzip and mod_deflate.

          /* You are not expected to understand this. */

          by ct on Wed Jan 03, 2007 at 06:52:11 PM PST

          [ Parent ]

  •  Drupal for CMS? (2+ / 0-)
    Recommended by:
    Mordecai, genocideisnews

    I have never used this, but have heard a lot of good things about it.  The development is very active.

    Don't know if this would work for you -- but check it out.

    •  Drupal for CMS! (10+ / 0-)

      I'm a Drupal Developer so I'm biased, however that being said, here's my redundant top 10 reasons to use drupal.

      Now you can take this with a grain of salt, and I am utterly and completely biased, being a Drupal developer, but I highly recommend Drupal for the following reasons:

      1. It is wonderfully developed tool with a quick turnaround for new versions.
      1.  Every client we get who has a community ends up requesting that it behave exactly like "Daily Kos."
      1. It's modular, so all the hacktackularness that you add to it, could also be contributed back to the Drupal community, while still allowing you to  upgrade the standard drupal features when new versions come out without breaking the rest of the site. You could build a new kos comment module. We've converted many scoop sites to drupal (including tpmcafe, air america and securing america.) and our clients are always asking for "Daily Kos" features. For Example, we did our best to mimic mojo with the karma module, and role delay for troll management. If you guys unleashed a suite of new modules of Drupal, you'd make a gargantuan community of people very happy. Including a lot of progressive orgs who could then use the tools you built.
      1. We host drupal sites with the same service provider you have. TpmCafe, which we built early in our careers, and garners oodles of traffic, although not completely bug-less, managed to survive the election without a scratch. Drupal scales incredibly well in the right hands.
      1. Google loves drupal. Drupal sites get significantly more traffic, thanks to all the integration google has done with it, and continues to do every summer with college students.
      1. Drupal's cache system in version 5.0 is rad.
      1. PHP is a much easier code to handle than perl or python.
      1. Drupal has 1000's of developers, who are improving the system every chance they get. Its an extremely active community that shows no signs of letting up.
      1. I've used scoop, tried to develop scoop, and yes it is a nightmare. Drupal is much easier and user-friendly to expand and hack.
      1. Drupal once wrestled a bear to the ground with its bare hands. Drupal isn't afraid of being hacked.

      Any questions? We'd be glad to help. Seriously, if you could take the kos comment system and build it in Drupal, you would be very loved.

      •  If it can scale to DKOS Traffic (1+ / 0-)
        Recommended by:

        I love the ideas of helping an existing Open Source project with new modules etc. It would help a thousand flowers to bloom.

      •  I've got a question about Drupal (0+ / 0-)

        Does the update to 4.7.4 finally stop the spambots from registering users without email validation?

        I'm running 4.7.0 and it's epidemic. =^(

        You're not kidding about google LOVING drupal. Every time the bot shows up with re-crawls the whole freaking site.

        I inherited the site, and it's pretty good. Better than Mambo or Xoops, but I wouldn't choose it given my experiences with 4.7.0. There's a good reason that Wordpress is so popular. =^I

      •  I'm a Systems Engineer (1+ / 0-)
        Recommended by:

        who runs production servers and I do not understand why PHP is purportedly insecure - or any more insecure than either Python, Perl, Java or Ruby.  There are plenty of ways of handling security in the coding.  Languages are not inherently secure, unless you're claiming the runtime engine is more prone to vulnerabilities.  In which case, I'd say Perl and Python are at least as bad.  I don't have enough experience for Ruby to say.

        mod_perl is at least as unstable as mod_php, but both should scale just fine.

        Python is for masochists and I do not see any inherent advantage.  PHP and Perl both have scads of source (libraries) lying around which can be modified and adapted as needed.

        I'm agnostic on CMSes, though the two small sites I maintain as a volunteer run PHPnuke, mostly because that was the thing when I set them up, and I'm lazy.  

      •  Seconded. (0+ / 0-)

        I'm active on the project too. Always wanted to see this happen.

    •  I don't like Drupal (0+ / 0-)

      but mostly because I don't like PHP. It's great if you want to put together a site out of the existing pieces. I used the CivicSpace distribution to whip up a campaign web site in a week last year. But my great distaste for PHP as a language and as an implemented runtime (both grossly inefficient and unfriendly) made me feel like that site was a dead end. I didn't want to develop for it. Hunter rules out Java for his uses, I rule out PHP for mine. Eh.

      (and I do like Java)

      1. Election Reform 2. Kucinich 3. Richardson 4. Edwards

      by bolson on Wed Jan 03, 2007 at 05:49:00 PM PST

      [ Parent ]

      •  It has evolved significantly since last year... (1+ / 0-)
        Recommended by:

        just as a side bar, and so has civicspace.

        Drupal allows you to put together a site easily and quickly if you only look at the top layer.  The real power is that it is a very powerful framework and if you know the API the sky's the limit.

        Pre-order unConventional, the official photo documentary of YearlyKos 2006!

        by Raven Brooks on Wed Jan 03, 2007 at 07:18:23 PM PST

        [ Parent ]

  •  OT: Question about Wiki's (0+ / 0-)

    My father wants me to help him set up a couple sites using Wiki software - one for a local democratic party site.  I figure it'll save me and him time if I use a hosting service that will take care of the set up for you.

    Does anyone have any recommendations for good hosting services for Wiki sites?

    I'm thinking about SiteGround:

    Any thoughts?

    A slip of the foot you may soon recover, but a slip of the tongue you may never get over. - Benjamin Franklin

    by meowmissy on Wed Jan 03, 2007 at 04:08:50 PM PST

    •  Wiki software (0+ / 0-)

      I played with PmWiki for a little while just to see how it worked. It’s amazingly quick and simple to set up and get running.

      I have no real need for Wiki software so I went no further. It runs on any web server with php (which means most).

  •  You're Doing It Backwards (6+ / 0-)

    You should start with the requirements (which I haven't seen posted, correct me if I'm wrong) and then work on the implementation.

    Start with the "what" and then progress to the "how".  You're posing the "How" question before we know what the site is supposed to do.  When you have a good idea of what you want to do, this should give you some informed insight in evaluating the implementation choices.

    This is a very common error made by techies, who love to argue about implementation details, and who can fall into religious wars over languages, frameworks, etc.

  •  C#, Visual Studio, Visual Web Developer, ASP.Net (7+ / 0-)

    ,SQL Server

    If you havent' tried out the latest (2005) versions of these then I don't want to here about it...

    And "they" have (just this year) come out with free versions of all these products that are extremely powerful if a person is interested.

    Resistance is Futile...

    "If they can get you asking the wrong questions, they don't have to worry about the answers." - Pynchon

    by HairOnFire on Wed Jan 03, 2007 at 04:16:29 PM PST

    •  I agree (0+ / 0-)

      I would have to hedge my bet on the .NET framework, C#, SQL Server 2005 for the love of all that is holy ... hire a graphic designer.

      There are both leaders & followers in this world ... which will you be? --- My Mom.

      by DBW21 on Wed Jan 03, 2007 at 04:28:29 PM PST

      [ Parent ]

    •  How much are you paid to shill for MicroSoft? (2+ / 0-)
      Recommended by:
      nasarius, jfm

      Whatever it is, your soul really is worth more.

      this message is intended to inform. any annoyance, abuse, threat, or harassment is solely in the perception of the reader, not the intention of the poster.

      by horsewithnoname on Wed Jan 03, 2007 at 04:33:20 PM PST

      [ Parent ]

      •  How 1990s of you. (4+ / 0-)


        Ah the 1990's when we thought Bill Gates was evil, and then 2000 and Bush gets elected. Reality smack. That's the true meaning of evil and bashing MicroSoft which had been so "cool" started to seem so silly.

        For me, it became about doing the best job for my customers and using the tools necessary to meet their requirements.


        "If they can get you asking the wrong questions, they don't have to worry about the answers." - Pynchon

        by HairOnFire on Wed Jan 03, 2007 at 04:44:28 PM PST

        [ Parent ]

        •  Oh! There can't be TWO greedy people? (0+ / 0-)

          I guess if Bush is Hitler, then Gates must be Mother Teresa.

          Do I have that right?

          •  Gates Foundation (2+ / 0-)
            Recommended by:
            unfounded, Arclite

            However evil their practices may be, bill gave a shitton of money to a foundation and he wants to eradicate several diseases.

            Most MS products aren't crap (just Windows and IE). C#.NET is a very, very nice framework.

            •  Nice logic there. (0+ / 0-)

              Bill Gates generously donated a large chunk of his personal fortune toward the cause of eliminating some horrible diseases.

              Therefore, .NET is our best choice for a web application framework, SQL Server 2005 is the most compelling option for a stable, well supported database infrastructure, and Microsoft will always be an ethical business partner.

              I'm sold. I mean how much more evidence do you people need?

    •  Thank you (0+ / 0-)

      I can use Visual Studio at school but a free C# is appreciated.

      -4.00, -5.33 "I expected to hear from the usual well-read laymen, and overeducated computer scientists."--DovBear

      by 4jkb4ia on Wed Jan 03, 2007 at 04:47:48 PM PST

      [ Parent ]

    •  I second. (1+ / 0-)
      Recommended by:
      el zilcho

      I use C#.NET, Visual Basic.NET, ASP.NET at my job and can have fast development. Plus the Microsoft.NET platform is being designed for Delphi, and 15+ other languages. Pretty soon, you'll have more flexibility in designing in your favorite language because all will have a common backbone.

    •  So, when do you MS guys start talking Price? (4+ / 0-)

      "Free, lightweight, easy-to-use, and easy-to-learn tools for the hobbyist, novice, and student developer."

      With the definitions of "hobbyist", "novice" and "student" being under the control of MS. And, of course, you will also need several WinXP Enterprise servers running MSSQL 2005 when you get ready to deploy, which they don't have free versions of last I checked. I am always ready to be proven wrong...

    •  I'm not a MS guy, but we use it at work (0+ / 0-)

      We get great performance for minimal hardware.  We're a F500 company and our apps get 100,000s of hits per day.  It's relatively easy to code and maintain, and it's fast since it's precompiled.  

      As much as I hate MS, I've been thoroughly impressed with the platform.  

      Enterprise level stuff is what it's designed for.  I don't know what the licensing fees are, tho.

      War is not the continuation of politics by other means. On the contrary, it represents a catastrophic failure of political skill and imagination. - Kofi Annan

      by Arclite on Wed Jan 03, 2007 at 07:04:17 PM PST

      [ Parent ]

    •  While I'm partial to the solution... (0+ / 0-)

      As a .NET developer, I agree the platform can't be beat for ease of development and speed of execution.

      It's a losing battle to try to recommend it to people who are in love with PHP.

  •  I'd like to add another language (0+ / 0-)

    MySpace, which has similar dynamic requirements, has had really good results with the BlueDragon recode of of CFML (Cold Fusion) into .NET. Of course that makes some assumptions about OS that may not fit. If yu are looking for speed and can deal with .NET you should put it on the table. You can compile custom .DLLs using IronPython or whatever you want to do the heavy lifting, and wrap it in CFML.

    I've played with Django. I love Python above most other languages. But Django seems to me great for OO RAD but not great on speed.

    I love Zope, but the CMF and Plone have too many levers. Plus, I don't know if it is still the case, but you used to have a hell of a problem if your data.fs file got corrupted. Don't know if that's still the case, been a while

    I'd really take a look at the Blue Dragon CFML products though... I know it's embarrassing to tell people it's in CFML, but true geeks have no shame. Plus you can always code the couple DLLs you write in C, just in case you get in a pissing contest. And you can mention the most highly trafficked site in the world uses the same framework...

    Nobody puts Blue Hampshire in the corner. Nobody.

    by keener on Wed Jan 03, 2007 at 04:20:58 PM PST

  •  Use Stored Procedure Language in the Database (17+ / 0-)

    Most web development languages are best at handling the back and forth and the display.

    You can reduce the amount of code you need to write by using the 'Stored Procedure language - SPL' that is available on SQL databases.

    Then calls like:

    save_comment(story_uid, parent_comm_uid,subj,txt);



    display_banner_ad(client_uid, ad_uid);

    display_column_ad(col_uid, client_uid, ad_uid);

    Can do most of the heavy lifting on the database engine spread across multiple storage servers. Several SPL statements will handle lots of the constant juggling act with a lot less CPU since the pages are created and stored in the database and reused instead of having every connection redoing the same things constantly.

    Also, I really like the AJAX-ified comments, etc, but I am experiencing 'pregnant pauses' on long strings of comments generated by popular stories/diaries. Firefox keeps asking me to continue the script or cancel it. So I'd take a look at how to render the comments say once every X minutes and insert new ones for display at that time. I'd bet that currently the system is working itself to death re-generating the pages a gazillion times right now.

    My 2 cents.


    "There is a time for compromise, and it is called 'Later'!"

    by LeftyLimblog on Wed Jan 03, 2007 at 04:26:07 PM PST

    •  first useful comment of the thread :) (0+ / 0-)
    •  right on (0+ / 0-)

      SP's 4 teh win!

      There are both leaders & followers in this world ... which will you be? --- My Mom.

      by DBW21 on Wed Jan 03, 2007 at 04:30:36 PM PST

      [ Parent ]

    •  Which SQL? (0+ / 0-)

      Last time I checked (and it's been ~9 months), MySQL doesn't have any stored procs. Are you talking about MSSQL, or some other variant? Is it a free variant (PostGreSQL perhaps?), which is probably a requirement on this site?

      If you are talking about MySQL, which version includes stroed procs, so I can tell my ISP to get off their ass and update! :)

      •  Stored procedures... (3+ / 0-)
        Recommended by:
        Hunter, 4jkb4ia, Josh or Con or Both

        Any commercial database has them, and if you want to stay away from Microsoft them something like Oracle would be the way to go.

        MySQL 5.0 has support for both stored procedures and triggers.  But this is the first version where that was added and I understand the implementation is not as full featured as it could be.

        Pre-order unConventional, the official photo documentary of YearlyKos 2006!

        by Raven Brooks on Wed Jan 03, 2007 at 04:37:59 PM PST

        [ Parent ]

        •  Thx (1+ / 0-)
          Recommended by:

          I'll look at MySQL 5.0, thanks for the tip.

          My comment was more that the word "commercial" is probably yet another non-starter for this. I'm pretty sure "free and open" is the watch phrase for tech on this project. This excludes both MS and Oracle. That's why I pointed out that the major free SQL (MySQL) doesn't (quite) have them yet. I mentioned PostGreSQL because I think they do have them, and they are free, I just haven't done any PostGreSQL work.

          Yes, I know about sprocs and use them daily; they aren't the solution to everything, and the world had software that worked before they existed. I have done plenty of MSSQL and MySQL work, and so I know how to do DB stuff both with and without sprocs. Sprocs do have their own problems. One of which is revision control, because revisions of the sproc generally can only be maintained outside of the DB and the natural dev environment. Not a serious counter-argument against sprocs, just an annoyance and caveat for those attaching holy significance on the "CREATE PROCEDURE" keywords existing in a given implementation's lexicon.

          •  I've worked in postgres (0+ / 0-)
            It's a solid database, with a lot of extended features before MySQL had them... but IMHO they left out some of the meat and potatoes basic things that are more critical to me in order to do those.

            So yes, I could do subqueries years ago, and that was nice. But formatting a date nicely was always hellish, and the documentation was very difficult to search. As it turned out, rather to my surprise, I was far more annoyed about always having to reformat my dates using perl for every result on a site instead of getting them the way I wanted from the postgres DB than I was about those times when I was in MySQL and wanted to do a subquery. (The client could not live with dates with leading zeroes, sigh.)

            Postgres is a very solid platform though, and there are good reasons to choose it. But it was a(nother) lesson to me that it's the little things that you just assume that Will Just Work Because They're Easy and Obvious that always bite you.

            Fry, don't be a hero! It's not covered by our health plan!

            by elfling on Wed Jan 03, 2007 at 06:25:43 PM PST

            [ Parent ]

    •  Talk to about their TableTalk (0+ / 0-)

      hassles. This is all so deja vu to me. I know zilch from code, but I sure recognize this discussion.

      When TableTalk started they were the first customer of any size for webXcrossing and the hassles were incredible. The search implementation crashed every other day, threads over 2500 posts (I had one) never finished loading.

      Every once in a while Management would post a plaintive note about how Salon and WebXing were living in each others' laps, trying to grow together without driving their customers away.

      Might be worth talking to the people who made it work, finally...

      And anyone else who actually lived through some massive expansion...

      Misery loves company.

      Listen Before You Talk.

      by ormondotvos on Wed Jan 03, 2007 at 05:06:09 PM PST

      [ Parent ]

    •  Stored procedures (0+ / 0-)

      Isn't that a given?  I'm not being sarcastic, but really, are they not used as a matter of course in open source world?

      Also, does the fact that Markos won't consider a MS platform completely rule out SQLServer as a database solution?  I guess so, but figured I'd ask anyway.

      "War against a foreign country only happens when the moneyed classes think they are going to profit from it." -- George Orwell

      by joanneleon on Wed Jan 03, 2007 at 06:51:20 PM PST

      [ Parent ]

      •  MySQL (0+ / 0-)

        vast number of PhP scripted Db backends were written around MySQL, which has not supported SP's until v5 & even the v5 implementation is somewhat limited.

        There are both leaders & followers in this world ... which will you be? --- My Mom.

        by DBW21 on Thu Jan 04, 2007 at 04:44:54 AM PST

        [ Parent ]

    •  Back End Persistence Layer (1+ / 0-)
      Recommended by:

      Stored procs are a no-brainer - a litmus test if you will for a hack vs. a well engineered solution.  The real trick is a implementing a cache layer sitting above the database which refreshes the busy blog page hits every 10 seconds or so instead of re-executing queries hundreds of times a second.

      I assume the subject of this blog was with regards to the presentation layer technology.  Sorry, I've got to agree with the Java people.  Parsed script (PERL) was out years ago for high traffic use.  

      ASP.NET is indeed platform independent (providing the platform is Windows)...

  •  I'm not being flippant here, (1+ / 0-)
    Recommended by:

    but could somebody tell me why this change even needs to take place?  I think the site is pretty fine as it is.  Well, okay, the searching could be better.

  •  IANA programmer, but my husband _loves_ Perl (0+ / 0-)

    He made and maintains a medically relevant database that serves hundreds of thousands of pages from a Linux based PC. This summer he put together a database that matched abstracts/reviewers/reviews, with automailing capabilities, in what seemed to me like just a few days. This is second hand, but I can testify that the bystanders may be happier if Perl is used.

    Be very kind, for everyone you know is fighting a great battle.

    by Wee Mama on Wed Jan 03, 2007 at 04:38:14 PM PST

  •  YOU FORGOT MATLAB!!!!11!!!! (1+ / 0-)
    Recommended by:


    Anything's possible with Commander Cuckoo Bananas in charge. -Homer J. Simpson

    by Cheez Whiz on Wed Jan 03, 2007 at 04:41:04 PM PST

  •  I'm staying outta this one (1+ / 0-)
    Recommended by:
    Sharon in MD

    I'm an infrastructure guy. I make the servers work and I don't care what software you run on it.

    So many impeachable offenses, so little time...

    by Cali Techie on Wed Jan 03, 2007 at 04:42:28 PM PST

    •  Same here... (2+ / 0-)
      Recommended by:
      montpellier, Cali Techie

      I know enough Perl to parse logs and do other administrivia, I've fixed someone's broken Python form-based mailing script (though I've never actually coded anything in Python from scratch), and I've done a bit of PHP coding for the web.  I guess I'd choose Perl by default.

      But I'm surprised to see how reasonable this discussion is, considering it's top quality flamebait, like a vi vs. emacs troll, or Linux vs. BSD ("Netcraft confirms...").


      "In spite of everything, I still believe that people are really good at heart." - Anne Frank

      by ktakki on Wed Jan 03, 2007 at 05:52:01 PM PST

      [ Parent ]

      •  It's civil because (1+ / 0-)
        Recommended by:

        Everyone genuinely wants to help. I'm definitely in the "you gotta start with requirements" crowd. The problem is that gathering requirements is tedious and not fun. It's more fun to decide what technology to use then make the requirements fit the technology. It's a mistake I see over and over and over and over again usually with less than optimum results.

        The only advice I have to the geniuses behind this site is to keep it as simple as possible. Too much complexity will doom any project.

        So many impeachable offenses, so little time...

        by Cali Techie on Wed Jan 03, 2007 at 06:54:44 PM PST

        [ Parent ]

        •  This is applicable quite generally. ... (1+ / 0-)
          Recommended by:

          Too much complexity will doom any project.

          So many impeachable offenses, so little time...

        •  If it ain't broke... (1+ / 0-)
          Recommended by:
          Cali Techie

          Personally, I'm surprised that the Scoop engine scaled this well.  Kuro5hin, the ur-Scoop site, has a tenth of the number of active users.

          On the other hand, Slashdot has ten times the traffic of DKos, with a million registered users (as opposed to 100K DKos users).

          Scalability and the graceful handling of hundreds of thousands of users would be my first requirement, and towards that end I would look at the codebases used by high traffic sites like Slashdot (coded in Perl, and mostly hack-free despite being a high-profile target).


          "In spite of everything, I still believe that people are really good at heart." - Anne Frank

          by ktakki on Wed Jan 03, 2007 at 09:23:54 PM PST

          [ Parent ]

  •  I'll chime in on the requirements bandwagon (4+ / 0-)

    a few people have said it already but I'll chime in on the requirements bandwagon as well.

    Requirements very well may be defined already to some extent, but to me debating which technology should be used is rather pointless before you've nailed down business requirements and technology requirements (things like security, performance, etc).

    If this exercise is undertaken first then the architecture decisions and the decision to build vs. use a product become a lot easier.

    Right now I'm not even sure how someone could provide an answer here other than stating personal preference and past experiences.  But their project might have been completely different so if someone says language X works really well for me and scales, etc then it may be completely irrelevant for this project.

    Without requirements the only sensible thing to do would be to look at what other equally large sites use, but even that doesn't really tell you anything because their code may be an absolute mess, but they may just be throwing hardware at the problem.  But it'd be worth thinking about what sites like Google, Yahoo, Digg, MySpace, Facebook, etc use and why they use it if no requirements are on the table.

    Pre-order unConventional, the official photo documentary of YearlyKos 2006!

    by Raven Brooks on Wed Jan 03, 2007 at 04:43:02 PM PST

    •  Exactly. (1+ / 0-)
      Recommended by:

      And in the spirit of the cagematch let me also second that the framework language choice should be driven by the design, performance and biz needs, as well as how much reuse you'll want to get out of the existing perl DKos/Scoop backend. If you want to leverage some of the existing code obviously perl may be the tool to grab out of the toolbox.
      That said, I find large perl applications to be problematic in terms of testing and maintenance, YMMV.

      Ruby/RoR can be very useful, PROVIDED you accept the nothing-shared/threaded architecture, grok its unicode caveats and are prepared to use something like memcached for cache/db buffer/etc.

      And as a java user since the beta days let me just say that J2EE/Struts/EJB make me very angry indeed, and I avoid them whenever possible. Plain J2SE, on the other hand (for now) is usually my tool of choice for "big stuff" (e.g., blogging engines, streaming media servers, biz backends, etc).

      Disclosure: I'm currently using RoR frontends with legacy java backends (and mySQL) for a couple of projects.

    •  strongly agree (0+ / 0-)

      Though also rx a deliberate effort to use "agile" strategies.  It's true that cooking up trivial barebones prototypes and having people use them and give you feedback leads to more genuinely useful software than a "big design up front"...  

  •  I recommend Python language. For example (1+ / 0-)
    Recommended by:

    "What do you do when a man attacks you with a raspberry?"

    "You've got two 'alves of cocoanuts and you're banging 'em together."

    "NOOO-body expects the Spanish Inquisition!"

    "I never wanted to do this job in the first place.  I wanted to be . . . a LUMBERJACK!"

    MAUDE: What do you do for fun? DUDE: Oh, you know, the usual. Bowl. Drive around. The occasional acid flashback.

    by Cartoon Peril on Wed Jan 03, 2007 at 04:45:06 PM PST

  •  ACS! (0+ / 0-)

    It's all about the appearance of security, stupid!

    by zeke7237 on Wed Jan 03, 2007 at 04:51:45 PM PST

  •  C# & .NET - makes it so easy (3+ / 0-)
    Recommended by:
    guy123, el zilcho, The Other Steve
    •  I don't care if Jesus and the Dalai Lama (2+ / 0-)
      Recommended by:
      mimi9, jfm

      both endorse .NET.  Don't get in bed with the evil empire.  You always pay for it in the end.  

      Good technologies in service of a bad company is a bad choice.  Just look at the DRM knots MS has tied in Vista to see where you MIGHT get taken if you marry Microsoft.

    •  Look at the top sites (4+ / 0-)

      there is not a hint of microsoft penetration into the top sites as per  There's a reason for that.  $$

      •  $$? (3+ / 0-)
        Recommended by:
        nasarius, OldPerlGeek, jfm

        There's a reason for that.  $$

        And security.
        And uptime.

        linux is the way to go.


      •  indeed... (3+ / 0-)
        Recommended by:
        mimi9, OldPerlGeek, jfm

        Google - Linux
        Amazon - Linux
        YouTube - Linux
        CNN - Linux

        And so on. There's no need to fork over tons of money for your servers when there are perfectly good free alternatives (Linux, OpenBSD, etc). If you absolutely must run .NET, there's always Mono. However, I've noticed that none of the proponents here have offered specific advantages over Python and Ruby.

      •  I am assuming (0+ / 0-)

        That this runs on just a couple of servers so the O/S cost would not be much. If it is a large server farm then that is a different story.

        But my basic point is that C# is very easy and very powerful. And the programmer time saved would be much greater than the cost of a couple of Win 2003 licenses.

        •  I concede the point on C#, it's nice (0+ / 0-)

          but not so nice that it's worth getting roped into the MS money machine.  Microsoft is in business to make money, and if you join their system, they will get it from you.

          I know nothing of dkos internals, but I'd bet all my mojo that the assumption of a couple of servers is way off.  

          In addition, thinking that way (it's only a couple of servers ...) puts a ceiling on your ambitions, and I'm pretty sure that there is no ceiling on kos ambitions.  MS really fucks you over as you go up the scale (look at SQL Server licensing if you don't believe me), so your perfectly reasonable C# architecture only looks completely reasonable if your project is a complete failure and doesn't need to scale.  If it's a success, immediately start rewriting in FOSS, or start forking over the dough to MS.

          Hey, if I have to learn something and develop some custom code to make MySQL scale, when I'm done I own something valuable.

          Proprietary servers penalize success.  FOSS rewards it.

  •  Think carefully before rewriting form scratch (1+ / 0-)
    Recommended by:
    Sharon in MD

    Of course, you may decide to, but read Joel's warning first.

  •  Use Python (7+ / 0-)
    Recommended by:
    jotter, ppluto, nasarius, mkc, kurt, sabishi, jfm

    There is a reason why Python is used by Google. And most of it is for programmer productivity versus speed of compiled code.

    Now possibly I'm preaching to the choir here, but to quote a Sir Tony Hoare "premature optimization is the root of all evil" (popularized by D. Knuth) No matter which server-side language you choose once you get functionality operational, then is the time to  profile, find the slow areas, and then optimize the algorithms or leverage C to boost execution speed. (That would obviously mean having a good interface to C available) Python looks to have one, I can't speak for the others with certainty but I'm pretty sure they do.

    While not throughly scientific The Computer Language Shootout can provide some idea of language performance as compared to others.

    Peter Norvig, Google's Director of Research, has some nice articles related to Python, most are in relation to Lisp, but they still give useful information.

    •  To be fair (1+ / 0-)
      Recommended by:

      Google uses Python, yes, and they also use Django (the framework originally developed at the company where I work). But they also use a lot of other stuff, including Ruby, PHP and even Java :)

      Also, comparing to Google isn't always useful; they do use a lot of different technologies, but they also bend those technologies to do the sorts of things that only Google needs. Personally, I'd say that better examples could be found for comparison, and that Python, Ruby, PHP, Perl, Java, .NET and lots of other things are all worthwhile candidates to evaluate.

      I do wonder at the utility of the poll on this post, though -- we geeks aren't known for our objective, unbiased opinions about our tools ;)

  •  Ideas from a non-techie (1+ / 0-)
    Recommended by:

    A couple of things I thought would be neat to add to Kos are:

    • Daily action items.  Imagine how many people a day would respond to posted action items!
    • e-mail functionality.  There are so many good posts, it would be great if there was an e-mail link that could be used to send them to our friends.
    • A "bat" for fundraising.  Prior to the last election, when Kos was asking for one more round of contributions, I thought it would be cool if we could track our progress in a more real-time manner than following occasional posts that updated us on how much had been raised.

    Anyway, I love DKos and think it is already an excellent website.  But it can be made even better :-)

  •  $.02 (2+ / 0-)
    Recommended by:
    montpellier, klondike

    Java is the only one of these languages I can use for anything complicated, and for some things I prefer writing in plain ordinary C.
    After the meta explosion has died down, perhaps there should be a thread about forms of community building.  Although search is also important, this is as strong a requirement as having a mass-accessible site.

    -4.00, -5.33 "I expected to hear from the usual well-read laymen, and overeducated computer scientists."--DovBear

    by 4jkb4ia on Wed Jan 03, 2007 at 05:02:20 PM PST

    •  if you're talking about community-building (2+ / 0-)
      Recommended by:
      Abou Ben Adhem, 4jkb4ia

      you're in web 2.0 land - semantic web and web services.  I've beat the drum for web services further down and I'll do it again.  The first time you see a mashup doing something you'd never thought of, all the aggravation of web services will seem worth it.

  •  Google likes Python (3+ / 0-)
    Recommended by:
    theran, nasarius, klondike


    Its not definitive, but I would have to think that Google's seal of approval for Python in production applications should make it at least a safe choice.

    -2.38 -4.87: Maturity - Doing what you know is right even though you were told to do it.

    by grapes on Wed Jan 03, 2007 at 05:05:28 PM PST

  •  One actual suggestion (7+ / 0-)

    take a page out of the Amazon/Google playbook and provide a web service layer to enable the 'community' of geeks to percolate and mashup dkos with the rest of the semantic web.  I am personally working on a mobile framework that attempts to project the power of the wide,wide net through mobile devices by enabling easy and secure integration of web services into mobile applications.

    So for my sake, give us a web service API.

    •  Yeah (1+ / 0-)
      Recommended by:

      I don't know what that might look like in practice, but I like the theory.

      I do know we plan on pulling in other services via their APIs. For example, I want a Googlemap which tracks the locations of site users.

      •  Theoretically it'd be a layer that (1+ / 0-)
        Recommended by:

        your ajaxy front end would access so you get a two-fer.  Your ajaxy ui uses it as the data access layer, and the community gets access to it to do their cool mashups - a google map mashup to show the prevalence of right-thinking progressives in the People's Republic of Taxachusetts for example.

        But to do that you really want to architect it in from the start.  Layering web services on after the fact just to 'serve the community' is a losing proposition.

      •  kos, see the thread further down with cjohnson (0+ / 0-)

        this comment in particular:

        and this statement within it referring to his use of xdoclet and EJBs:

        It's also how we got SOAP interfaces to our EJBs by just changing the build script and annotating our existing business interfaces

        The way Hunter is thinking about this right now, imho, precludes the possibility of getting a Web Services API into dk4 as a simple side effect of the user-facing site development process.  You could get that SOA with a more middleware-oriented approach.  Whether it's worth it is another question entirely.

    •  something RESTful, please n/t (0+ / 0-)
  •  PHP (6+ / 0-)

    Disregarding PHP because "it's not secure" makes me wonder what kind of conversations you guys are having. It's a great language with lots of support, and can be used to write secure web applications. The PHP of today is not the PHP of 10 years ago.

    I personally think the language to choose is the language that you and your developers are most comfortable using, no matter what the language is. Or, choose a different language as an excuse to learn it. Web languages are at the point where you can do anything in any of them, and it comes down to a matter of taste or (more often) religion.

  •  perl is a write-only language (7+ / 0-)

    I know that a lot of seriously talented people love Perl, and it's powerful and poetic, but it's extremely expensive and confusing to sight-read. And what's hard to sight-read is hard to maintain. Perl is probably the most "abusable" language that has ever achieved wide usage. (Worse than ksh.)

    Even if you think you will never ever write bad or confusing code: to the extent that the language permits it, you probably will, sooner if you are in a hurry, later if you are deliberate and careful. Perl not only permits it, it encourages and tempts and provokes it.

    Python is at the other extreme. It takes effort to write confusing Python. The language is the closest thing to pseudocode that I've ever written, that can actually execute. Lovely, fun, and fast.

    Ruby is like a hybrid of the two languages, and oddly charming and a thrill to write. Also quite powerful, but Hunter, you are probably right about how you won't need some of its more spectacular features.

    In sum: avoid Perl. Go with either of the other two, if you find a suitable framework.

    Also, consider XSLT for a lot of the presentation. It's kind of hard to wrap your head around, but because it's declarative rather than imperative, your bug rate (once things are actually coded) will be quite low.

    •  Re: perl is a write-only language (1+ / 0-)
      Recommended by:

      Perl wouldn't be expensive in this environment.

      I do agree that it can "abused" and isn't the best choice for sloppy programmers.  The solution of course is to hire good programmers. I wouldn't trust a bad programmer's code in Python either. :)

      Python would likely be okay too; Ruby is the one I might warn against.

      More seriously, my feeling is that Hunter and ct already have their preferences.  With good organization (i.e. project management), good programmers, and a good CMS framework, I think Perl would work well here.

    •  agree, except for the 'avoiding perl' part (0+ / 0-)

      The ease of abuse and crappy legibility are the things I like least about Perl, but it has (IMO) benefits that make up for that.  Anyway it's just as easy to write good, legible Perl as obfuscated, illegible Perl.  People frequently don't, because they're in a hurry.  But that's also true of C/C++ and Java and yes, Python.  

      I used to work with a guy who was a really fine programmer but who, even in high level languages, named all his variables with no more than 3 or 4 chars and called them things like "ref."  I still run into that code sometimes.  Grr...

    •  It's not that bad (0+ / 0-)

      I've done a little hacking on the giant pile of perl that is Scoop (from whence this site is derived). It can be built with decent structure, it's also very easy to not do that.

      1. Election Reform 2. Kucinich 3. Richardson 4. Edwards

      by bolson on Wed Jan 03, 2007 at 05:44:59 PM PST

      [ Parent ]

  •  Here is the biggest problem (7+ / 0-)

    You have HUNTER, for chrissake, of all people, asking questions like this:

    ... how should the brand-spanking-new backend be built? What language? What frameworks? What tools?

    Who gives a flyin' fuck? HUNTER is only the best freaking writer on the site. He should be creating the content, not managing it. His questions should be about the investigations, about what's going on now, and aired for all to hear!

    Markos lives in Berkeley. There have to be about a zillion highly successful tech enterpreneurs, committed Progressives one and all, within a vigorous frisbee throw, who can point you in the right direction on tech issues and might even support you. Not all useful networks run on TCP/IP, you know what I'm sayin'?

    Language: Perl, Ruby, or Python? That's up to the implementation team. You specify the functionality, you pick the team, they implement it.

  •  What's best to use???? (0+ / 0-)

    I'm a main frame kinda girl. So, I'm kind of biased toward c and c++. However, Perl is what most of the people I know, outside the main frame world, use for their programming.

    If I were doing the programming, I would try all 3, and figure out which one is easier for me to work with.

    Good luck.

    "Government will be in trouble if the people think." Hitler

    by KatGirl on Wed Jan 03, 2007 at 05:22:36 PM PST

  •  Python (0+ / 0-)

    is the way to go.

  •  Lisp (6+ / 0-)

    I vote for Lisp.

    Sure, it's incomprehensible and hard to maintain, but on the upside, once your LOC count gets to a certain point DailyKos will become self-aware.  Think AI.  Think Skynet.  Think "control of USAF strategic missile forces".

    Now, that's what I call crashing the gates.


    "In spite of everything, I still believe that people are really good at heart." - Anne Frank

    by ktakki on Wed Jan 03, 2007 at 05:27:11 PM PST

  •  mod_perl (2+ / 0-)
    Recommended by:
    EightBall, montpellier

    If you need reliable speed, rails (or any Ruby at all) is prolly not a good idea.  That leaves python and perl.  I consider python to have better (though not more) framework choices, be slightly easier to decipher (or at least harder to obfuscate), arguably easier to refactor, and you don't have to be so careful about leaks (or do you?).

    But perl is faster, terser and easier to write, has IMO better libraries and idioms, and is more flexible when you really do need to do something sneaky (which you often do).  You just have to make sure you document, and use 'strict' and some common sense.  The request cycle in apache essentially gives you a framework anyway.  Override the request phases you need and ignore the rest and you're halfway there.  And there are good, fast templating systems (and if you don't like any of 'em it's easy enough to write your own.  everybody else certainly has ;-)

    Python seem about like perl as far as optimizing and shifting things off into C, but I'm really a perl guy, so I've never actually tried it in python.

    If there's a out-of-the-box-ish CMS good enough for DK4 I'd sure as hell like to know about it too.  

    •  I'm more of a hacker and not a coder but... (3+ / 0-)
      Recommended by:
      jotter, montpellier, kurt

      I was going to post something similar. Perl seems faster on many of the bench mark sites I have seen (although I think the python code could be optimized so it does not look SO much worse). However, I can never look back on my perl code and figure out what I was doing. My python code is so easy to read, it is actually pretty.

      Since code always has bugs and always needs modifications, I vote for python due to its readability.

  •  Steve Martin said it best... (2+ / 0-)
    Recommended by:
    justme, kurt

    This lawn supervisor was out on a sprinkler maintenance job and he started working on a Findlay sprinkler head with a Langstrom 7" gangly wrench. Just then, this little apprentice leaned over and said, "You can't work on a Findlay sprinkler head with a Langstrom 7" wrench." Well this infuriated the supervisor, so he went and got Volume 14 of the Kinsley manual, and he reads to him and says, "The Langstrom 7" wrench can be used with the Findlay sprocket." Just then, the little apprentice leaned over and said, "It says sprocket not socket!"


    "...sometimes they ignore you, then ridicule you, then fight you and crush you like an overripe eggplant." - Billmon

    by Astoria Chris on Wed Jan 03, 2007 at 05:36:55 PM PST

  •  Ruby on Rails is the way forward (0+ / 0-)

    Perl is soooo Web 1.0. We had good success at Democracy for America with CakePHP, which is a Ruby on Rails port.

    Ruby on Rails (and the general MVC framework) is the way of the future. Just ask the Forward Together PAC tech team -- it's a shame we won't everything they were cooking up.

  •  my vote: ruby. my hope: openid (0+ / 0-)

    i'd go with ruby just because i think the it is the most elegant and sophisticated of the options, and the 2nd easiest to look at (*ahem* perl).  will you really be working around the rails data abstraction framework for everything? in that case, why bother with a framework at all!?! :)  but really, it is the nicest such framework, and you'll likely still have use for such a thing in parts of dk4.

    scalability? ok, so ruby won't be as fast as perl, BUT... you will save tons on hacker productivity, and you could spend that difference on dozens of extra servers, and load balancers. afaik, basecamp is entirely on rails.

    openid: please, please, please, please, PLEASE!!


    •  Forward Together PAC (2+ / 0-)
      Recommended by:
      colorless green ideas, jfm

      Echoing my comment above, the Warner tech team did a lot of work with OpenID (in Ruby on Rails) in preparation for the campaign and they'll release something under the domain soon.

      •  thanks for the plug (1+ / 0-)
        Recommended by:

        Obviously as one who was designing the technical architecture of a national campaign's internet platform with Ruby on Rails and OpenID, both would generally get my vote for building web apps. I think RoR is a killer app for the fire-drill environment of political campaigns. You can build highly functional web apps quickly and if you've designed a strong architecture from the beginning you can plug them into a scalable structure. Baking in open standards like OpenID is just trying to move the ball down the field on my belief in user-centric online identity. Hopefully we'll have something to show everyone on that soon!

        Others have had success with Ruby on Rails including ActBlue, the online fundraising engine of the netroots, and they seem to be chugging along just fine.

        I come from the internet consulting firm world where being a one-language/database/framework-to-rule-them-all approach is just bad business. We based our recommendations on the requirements and did everything from Java to .NET to Cold Fusion. That doesn't mean you don't have preferences or proficiencies. Currently, for me, those are Ruby on Rails and PHP, but I'd love to check out Python/Django. Along those lines, I personally believe v4 could be built with a LAMP or RoR Shared Nothing app layer and a kickass database layer (PostgreSQL for mature stored procedures?).

        But, let's figure out those requirements in detail first eh?

    •  Yeah (1+ / 0-)
      Recommended by:

      I'm very intrigued with openid.

  •  Python, methinks (3+ / 0-)
    Recommended by:
    mark, jotter, vivacia

    It's a rough choice. I like Java/C# because they remind me of object-oriented programming, which I <3.</p>

    Ruby on Rails is awfully esoteric, and their data abstraction seems like somehting that's "magic". I don't trust magic.

    Python seems to be solid. Google likes it.

    Perl should die.

  •  And the answer is: vote on it (0+ / 0-)

    But not all us peons. The 5-10 people who are actually going to get to code on it should vote on it.

    1. Election Reform 2. Kucinich 3. Richardson 4. Edwards

    by bolson on Wed Jan 03, 2007 at 05:52:01 PM PST

  •  On the database end... (1+ / 0-)
    Recommended by:

    .. if you do not go for content management and get tied in to  database, please I urge you to look at Intersystem’s Cache. Not many people know about it, but that’s not to say it’s a fringe product. It's mainly used in Healthcare, Banking and Shipping. Some of the biggest databases in the world are using it, the V.A (Vista), Keiser and many health groups all keep their critical hospitals systems running on  Cache. It’s just far more stable than MS SQL.

    The benefits are simple, low cost and high speed. It's better than MS SQL for speed. It's on par with Oracle and far, far cheaper to implement. It can be used as a regular object oriented (SQL) DB or you can use some of its more advanced features to improve speed. So your DBA's will pick it up quickly, and even if they are not familiar with the software will find it easy going. You can get a similar speed as the other two, with a far smaller cost of hardware, also less server to go wrong.

    Anyway it's the hidden gem, and its far more cost effective than most commercial options. Don’t let it’s apparent lack of main stream appeal (lack of familiarity or even the "Post relational database" marketing sound bite) scare you off.

    You can download a trial version at..

  •  Pick the first one that has the features you want (0+ / 0-)

    and then have good programmers.

    everybody love somebody...

    by toys on Wed Jan 03, 2007 at 06:06:13 PM PST

  •  Perl (1+ / 0-)
    Recommended by:

    I think Perl sucks least in that it's been around so long at least you know where it sucks least.

    by BobOak on Wed Jan 03, 2007 at 06:14:45 PM PST

  •  Quality Assurance help offer. (2+ / 0-)
    Recommended by:
    EightBall, kurt

    I'm a software quality assurance systems analyst and engineer for a fortune 500 company.  Let me know if I can support your efforts.  I usually am involved in all stages: design, development, testing, and maintenance phases.

    Let me know if I can help when you need it.

    War is not the continuation of politics by other means. On the contrary, it represents a catastrophic failure of political skill and imagination. - Kofi Annan

    by Arclite on Wed Jan 03, 2007 at 06:18:49 PM PST

  •  Wrong, wrong, WRONG questions. (6+ / 0-)

    It strikes me that the server side scripting part of dKos is the least problematic.  Who cares what language you write it in?  Perl, python, Ruby, they're all cool in their own ways.

    Where you have a problem is clearly with your database layer -- that's where you keep having to throw hardware and it's the apparent reason behind a lot of your crashes, at least according to your off-line messages.

    As a professional SQL database programmer, I think you might want to look at something other than a typical SQL database.  The vast majority of your lookups are for diary-by-id, comment-by-id, or user-by-id.  You might want to look at a journaling database like Berkley DB to speed this stuff up and make it more reliable.

    Your biggest area for advancing the "coolness" of the site is in the AJAX stuff, and for that you might want to see which frameworks support which AJAX packages or functionality built in.  In fact, choice of the AJAX package may well be more important (I don't know how far you're thinking of taking AJAX in the future) than the backend scripting language and may be important enough to drive toolkit selection.

    Don't blame me -- I voted for Weicker.

    by LarryInNYC on Wed Jan 03, 2007 at 06:25:33 PM PST

    •  I neglected to mention this in my previous post (0+ / 0-)

      But the database layer has been bitched about more than once and I've not seen any statements either about which database they seem to be using.  Not that I recall exactly what problems they had with the DB, at any rate. ;) But there is a particular karma to be had choosing a language that best fits a particular database.  

      I wouldn't recommend BDB, myself, I've been much more happy with sqlite where a small database is concerned these days.

      •  This is the opposite (0+ / 0-)

        of a small database, I think.  The issue is very large numbers of additions and very few, if any, deletions.  BDB is not for everything (I've never had ocassion to use it myself and for my needs it's the opposite of what I want) but I know a fellow who was a performance engineer for them for a while.  If it's speed you need for a simple, write-only (or write-mostly) database it's where I'd look.

        Don't blame me -- I voted for Weicker.

        by LarryInNYC on Wed Jan 03, 2007 at 06:40:10 PM PST

        [ Parent ]

    •  ding ding ding - the language debate is stupid (2+ / 0-)
      Recommended by:
      deepintheheartoftx, Luetta

      it's the architecture that counts.  2 recs for that comment.

      I don't have any experience with alleged XML indexing engines but ... once upon a time we built a system that was actually a lot like dkos in the output side.  We had to store a lot of content, and search it quickly.  A very sharp db consultant suggested leaving the content as it already existed (as small XML files on the file system) and layering an XML indexing engine on top for the "databasy" functionality.  He mentioned a specific package (OSS?) and claimed that performance for what we needed would be better than a traditional RDB.  We didn't go with it, sadly, but keeping dkos content as a simple collection of XML files on a filesystem has a LOT of attractive facets.

      Choose whatever framework suits the front end types, but just make sure that its middleware doesn't require you to violate Klondikes second rule of architecture - to be fault tolerant a system must allow you to bash any single machine to smithereens without falling over.  Klondikes third rule of architecture is that if you find yourself swimming in a sea of inscrutable and interdependent configuration files in order not to violate rule #2, you bought the wrong middleware.

  •  A little more information would have been good... (1+ / 0-)
    Recommended by:

    ...for half the arguments I've read pro and con.

    I'm not a hardcore Kos user (I do read nearly everyday) but if there was ever a discussion about load, balancing, server hardware and configuration, things of that nature... I've completely missed it.  And given that Hunter has not stated such parameters in this post, discussions on scalability are moot.  There's simply no basis for saying "X doesn't perform well."  Each language can be made to perform well given an appropriate environment.

    So its left to the merits of the language itself.  And frankly, as a long-time Perl programmer, I'm appalled that Perl has gotten so many votes.  Python is very good, and Ruby is as well.  If you had to pin me down on the basis of framework, I'd have to hand it to Ruby on Rails.  Python does very well, but on a framework to framework matchup, to something like TurboGears or Twisted, I have to give it to Rails.  You can do far more with far less with Rails, and migrations are the greatest thing since sliced bread.

    I could take my digs out on a few comments I've read, but I've bitten my tongue this long, and I will continue to do so. ;)

  •  Architecture and Language... (1+ / 0-)
    Recommended by:

    Let me start by saying I love Java.  The control you get is what you pay for by the more precise requirements of the language vis a vis scripting.  Struts is not the only framework out there.  JSF is a powerful tool, justifiably called a framework in its own right, and there are open-source frameworks for integrating it with AJAX (see AJAX4jsf for example).  Security will be an increasingly important factor in the coming years, and Java beats the scripting languages hands down for that.

    Architecture-wise, I would hesitate to put too much application logic on the data tier unless you plan to use some very robust hardware. I've never worked with MySQL but I gather it lacks some high-end availability/manageability features that a commercial DBMS would have.  I would never use MS SQL Server for anything because it means using Windows, which is a bloated, unstable OS riddled with security holes.  On the other hand, I am the architect on a large Oracle project, and I have to say that we have run into some serious bugs in 10g lately that are of the "change your underwear" variety.  The good thing is that they crank out patches quick, the bad thing is that they frequently break stuff with those patches...

    Full-disclosure: I work for a certain azure giant and my opinions are entirely my own.

    Peace in a world free of Religion, Peace in a world where everyone gets Heaven... -- Toni Halliday

    by Wintermute on Wed Jan 03, 2007 at 06:27:32 PM PST

    •  okay azure man (0+ / 0-)

      Oracle is expensive, buggy and customer unfriendly, SQL Server is evil empire, MySQL immature.  Which way do you turn for the DB?

      What do you say to the argument that says 'lose the middleware entirely' (no ejbs) and have server farms running simple web servers serving some interpreted page language (JSP or whatever) hitting the data tier directly through something like recordsets and only allowing session primary key (which lives in the db and on the client) to cross server boundaries.  This way, you take a hammer to any individual web server and the system hums right along.

      The trick there is to bulletproof the data tier, which imho is more known technology (mirroring etc) than bulletproofing middleware.

      •  Middleware (and J2EE) is much more than EJBs (3+ / 0-)
        Recommended by:
        Hunter, montpellier, klondike

        First of all, with respect to EJBs, there are session beans, which are intended for business logic, and entity beans, which are an O/R abstraction layer.  I presume you are referring to the latter.

        For a site like this, where most of the database work is inserts and reads, EJBs are almost certainly inadvisable.  That doesn't invalidate the concept of middleware, or J2EE.  One of the scalability problems associated with scripting is that the interpreters typically run out of process on the web server, which has other duties to perform, ie managing http connections and formulating and transmitting HTML.  Even with AJAX this is still a heavy duty function.  Also, decoupling app code from HTTP servers makes it easier to manage the whole architecture if you scale the HTTP layer out.

        There is much more to bulletproofing the data tier than mirroring, which is just a storage management technique.  You get in to clustering the database, for which Oracle has the only real solution (RAC) IMHO.  Most commercial app server middleware (Oracle, BEA, WebSphere) can handle either load balancing or true clustering with sessions serialized to the database.  This still allows you to isolate functions to specific architecture segments to eliminate points of failure and improve scalability by optimizing HW to function.

        Despite Oracle's bugs, I would still pick it over anything else for industrial uses.  All software has bugs.  I just wanted to alert folks that the price you pay for Oracle's power and performance is higher management overhead.

        Hope that clarifies what I was trying to say.  :-)

        Peace in a world free of Religion, Peace in a world where everyone gets Heaven... -- Toni Halliday

        by Wintermute on Thu Jan 04, 2007 at 02:30:28 AM PST

        [ Parent ]

        •  ok, well said (2+ / 0-)
          Recommended by:
          Hunter, jfm

          and yes, I was referring to entity beans because a lot of my ill-feeling toward app servers in general stems from pre-EJB3 work with entity beans, and the feeling that I was paying way too much (either $$ or dev time) for object relational mapping.

          A couple of random thoughts while my thinking is evolving on this - middleware/app servers are a great way to achieve OO elegance in your application, and at a certain level of business logic complexity, that elegance can mean the difference between a project that is doable, and a project that is doomed.  

          But, my experience of dkos is that the business process is not nearly as complex as even the smallest app that I've ever delivered to one of our financial services clients.

          Is there a point on the left of the business process complexity curve where you can almost entirely eliminate the app server in favor of, for example, a simple servlet container?

          •  They all moved towards heavier... (1+ / 0-)
            Recommended by:

            JBoss, JRun, they all support EJBs now.  But again, if you can't serialize sessions and decouple it from the presentation tier, you lose scalability and manageability.  

            To quote Joe Pesci in Lethal Weapon IV, who utter a phrase that has become for me a kind of Zen koan: "You always get f*cked at the drive-thru!"  :-)

            Peace in a world free of Religion, Peace in a world where everyone gets Heaven... -- Toni Halliday

            by Wintermute on Thu Jan 04, 2007 at 06:29:29 PM PST

            [ Parent ]

      •  I use MySQL everywhere (0+ / 0-)

        But I do have performance issues - right now I'm trying to optimize by migrating the backend off of the old ISAM tables and onto the Sleepycat DBM engine - but haven't done it yet on the big databases.  I have found MySQL to be very, very reliable.

        Your points about architecture are dead on.  

        •  Thanks! (0+ / 0-)

          My only regret is the last time I touched any code in the last two years is when I helped one of my DBAs debug a shell script.  I LOVE writing Java code! :-)

          Peace in a world free of Religion, Peace in a world where everyone gets Heaven... -- Toni Halliday

          by Wintermute on Thu Jan 04, 2007 at 06:32:37 PM PST

          [ Parent ]

  •  A sysadmin's two cents - go with mod_perl (0+ / 0-)

    As a sysadmin who's maintained some fair sized Internet and intranet web infrastructures (which have been predominantly Java-based), my preference is mod_perl. I agree with those who have commented that Perl is often the most efficient language to "just get it done". I also think that efficiency carries over to resource use and administration (not to mention licensing costs). I'd also be interested to know which, if any, of the frameworks on CPAN people would recommend assuming mod_perl was the choice (Mason, etc).

  •  Ruby. (3+ / 0-)
    Recommended by:
    MackInTheBox, Siberian, klondike

    I strongly suggest Ruby. Python is almost as good. I advise you to stay far away from Perl. If you do not know any of the three above already then i strongly advise you to stay away from Perl.

    (I also advise you not to let a majority vote determine your language--that's not just because Ruby is losing, it's actually the first thing i thought of saying. I'll actually come back to this below.)

    Perl is all well and good... or, rather, it was well and good some time ago... but now, both Ruby and Python are really quite a step up. I am just some jerk on a website, so i will let some jerk on a website who is now employed by Google do the talking:

    Insanity is Not Good Business Practice

    Is that too harsh? I don't think so.

    He also likes Ruby--and Haskell, Lisp, and Scheme even though those are not being considered at this point (and i think sensibly so). For instance, on his conversion between Ruby and Perl:

    Within about 3 days, I was more comfortable with Ruby than seven years had made me with Perl. I was still accidentally sticking "my" in front of variable declarations, typing "sub" instead of "def", little things like that. But I was already starting to think in Ruby, and suddenly I had all this extra space in my brain. You have to learn Perl by memorizing, but you learn Ruby by understanding.

    Within a month or two, I'd totally given up on Perl. My Ruby programs were shorter, clearer, and more fun to write. Everything I'd actually liked about Perl was there in Ruby, and I realized there was a lot of crufty old legacy baggage in Perl that it was never going to lose.

    Nowadays I can't look at a Perl program without snickering; there's so much suffocating boilerplate junk in there. And I feel for folks who still have to write in it. Going from Perl to Ruby is as big a step in expressive power as going from C++ to Java.

    (He also, in that link above, goes into specific reasons why you would actually want to use Ruby.)

    Since i mentioned Lisp, here is another essay on why you should pick the language you think right, rather than just do what everyone else is doing. In this case, why choosing Lisp when everyone else was choosing some other language allowed the author's company to defeat all its competitors. Paul Graham, the author of that essay, is also highly regarded within the geek community.

    As far as Rails goes: you are right that it is not a very well proven system, but i think it is pretty solid anyway. For instance, Penny Arcade (an ENORMOUS web comic thing--check out the comparison to uses Ruby and Rails, last i checked, and works pretty darn well.

    So, okay: now that i have a little Ruby evangelism out of the way, why would you not want to use Ruby?

    First of all, Ruby is slow. Very slow. Slower than everything else. I won't bother with charts or what have you, here, because it basically loses to everything.

    For now.

    The Ruby team is working on moving Ruby to a VM which will improve things quite a lot. It is scheduled to show up in the next major version of Ruby, but some preliminary benchmarks show YARV (the new VM) kicking ass and taking names. Bonus: easily reproducible benchmarks. Note, however, not everything is rosy.

    Another problem is that Ruby's debugging is an atrocity. Some of the errors are completely opaque, and this really does matter occasionally. This is not as bad as it sounds at first, because Ruby code is so clean a lot of things are totally obvious once you know something is wrong. But when they aren't, it might hurt bad.

    The worst part is that, as far as i can tell, this is not an inherent feature of the language. The debugging support could be better, but it appears nobody in Chez Ruby cares for debugging all that much. I do not know how this will affect a very large project.

    Threading support in Ruby is maybe not all there. Certainly not Java-quality threading, last i checked. I do not know if this will affect your design or not.

    Finally, i am suspicious of Rails. I know it is great and all, and it appears to be genuinely great. But the Rails team makes me nervous. This is my own paranoia and not entirely justified, but it is also my intuition.

    Now keep in mind that Python and Perl both have their own quirks and problems, as well. (I am not as familiar with them as i am with Ruby's, however.) These are just Ruby's particular burdens. It is also not comprehensive or necessarily correct, just sort of what came to mind while thinking about the choice. For me, i have already made it and it is Ruby.

    The Shapeshifter's Blog -- Politics, Philosophy, and Madness!

    by Shapeshifter on Wed Jan 03, 2007 at 06:42:44 PM PST

    •  I was going to write something but ... (1+ / 0-)
      Recommended by:

      You covered it far better than me.

      I've worked with PHP, Java, Perl and Python.

      But my last 3 projects have been all in RoR and I've picked that up faster and enjoyed working with it more then any of the others.

      I don't have any performance problems with it, but none of my sites are even close to Kos's traffic.

      "Ever get the feeling you're in the wrong alternate universe?"

      by Siberian on Wed Jan 03, 2007 at 06:55:36 PM PST

      [ Parent ]

    •  One other thing... (0+ / 0-)

      I sort-of didn't go over this because i thought it would be obvious, but after reading all the other comments i have re-thought that.

      Ruby's primary form of scalability is one you do not get from other languages. It is not a machine sort of scalability, but a programmer sort of scalability. Execution may not scale super-great, but development does.

      And if you need more performance? But a handful of cheap, generic Linux boxes. Isn't that part of what made Google so great?

      The Shapeshifter's Blog -- Politics, Philosophy, and Madness!

      by Shapeshifter on Wed Jan 03, 2007 at 07:28:40 PM PST

      [ Parent ]

    •  great summary, and my advice (0+ / 0-)

      from a project management perspective would be to hack up the dk4 prototype in RoR and if you fall in love with it, as many do, then go with it for the front and middle tiers in production and put your limited funds into a commercial, bulletproof, highly redundant, non-stop back end (see LarryInNYC's comments).

  •  Got questions? I may have answers (3+ / 0-)
    Recommended by:
    Abou Ben Adhem, kurt, jfm

    Hunter, I used to do contract work on a few Scoop and, now, former Scoop sites (did some work on TPMCafe, a few others). These days I work in Python and I'm knee-deep in the world of Web 2.0 frameworks and tools (been working about a year now at the company which originally developed Django).

    If you've got any questions about platforms, scalability, anything, on any platform (I like Django, but I don't believe in one-size-its-all tools and I do my best to keep up with and know people from most of the big projects) feel free to drop me an email and I'll do my best to answer them or point you to knowledgeable people who can.

  •  Catalyst (0+ / 0-)

    I'm a Perl guy and don't have much to say about Ruby or Python, but I would think the most important factor would be the experience of your current team, unless you're going to hire all new people.

    If you end up going with Perl I would highly recommend taking a look at the Catalyst MVC framework.  I've been using it for a couple of months now and I really like it.  It's probably the best framework for Perl right now.

    PS.  I don't know why you don't just keep ripping Scoop apart, that seems like the easiest way forward.

  •  Come to my site to see what I did with Perl. (0+ / 0-)

    The entire site is generated using Perl CGI.  CGI has limitations no matter how it is done, and I hope to incorporate some Javascript to get around some of those, but the neato thing about Perl is that it's wicked easy.

    Oh, the site URL (the one index.html points to):

    •  perl is not easy (1+ / 0-)
      Recommended by:

      bad perl is easier than bad code in other languages.

      good perl is harder than good code in other languages.

      good perl is worth more than pretty much anything else.

      strange things happen with a free alphabet

      by dirge on Thu Jan 04, 2007 at 12:54:17 AM PST

      [ Parent ]

  •  Blosxom! The best CMS ever! (0+ / 0-)

    flat files rock. =^)

  •  Wait: Does this mean I lose my LUID? (1+ / 0-)
    Recommended by:
    Bill Evans at Mariposa
  •  Java and the Choice of Language (0+ / 0-)

    You know, we ridicule "W" all the time for making choices based on stiffnecked prejudice and here we have a classic example of that very process.  The choice of backend for this site will be based, not on what is the best product and process, but on the personal prejudice of the chooser.

    There are only two major web backends -- Sun (Java) and Microsoft (dotNET/ASP).  Every enterprise-level web presence is coded in one of those two backends.  Every one.  

    Now, you can play with the big boys.  Or, you can play "hobbyist" and cobble something up with perl or python.  One guarantee:  if you "roll your own," you'll be doing site maintenance and code upgrades constantly.  Hey, as long as you're getting paid for it ... why not?

    I like Java.  The crack about "too much typing" is just a meaningless one-off.  Any major modern IDE (I use NetBeans) can structure your application and build out the method stubs automatically.  You have builtin tools like FindBugs and profilers that can audit your code.  You have packaged unit-testing modules with JUnit -- unit test stubs built automatically if you write the code first then test.  For web development, you have automated tools that connect your page elements to the backend database with drag-n-drop.  You have UML design modules that will stub out your classes and methods based on the elements of your model.  And, they have bidirectional support, so when you change the model the code is updated and vice versa.  You can encapsulate all your private/protected variables with two clicks of your mouse.

    But hey, you go ahead and do all that work by hand.   Isn't that the more manly way?  Maybe after that, you can hack down some brush on your ranch, too.

    Any decent Java web developer could have the site up and working with a lot less "typing."


    mp, a reformed perl hacker

    Michael Powe Naugatuck CT
    -7.0, -6.15
    For if ye love them which love you, what reward have ye? do not even the publicans the same?--matthew 5

    by pdxlooie on Wed Jan 03, 2007 at 07:05:22 PM PST

    •  Every single one? (0+ / 0-)

      I'm not sure what the hell 'enterprise-level' means, but I guess none of the huge sites that use Perl are, like Amazon, Yahoo, Salon, IMDB, Wired, Slashdot, ValueClick, or any of the other millions of perl powered websites.

      •  Amazon et al (0+ / 0-)

        Hey, slick, check this out:

        Software Development Engineer: General Overview

        Your Background

           * Strong object-oriented design and coding skills (C/C++ and/or Java, preferably on a UNIX or Linux platform)

        [ ... ]

        Gosh, the number-one requirement for software engineers at Amazon is ... C/C++ and/or Java.

        At Yahoo, the requirement is * Strong Experience in C or C++, and UNIX development environment.

        ValueClick:  Our technology stack is: Struts, "plain old Java objects" with Spring, Hibernate and Oracle 10.

        I'll be durned.  Which orifice were you talking out of?

        Beyond all that, pulling a half-dozen middling sites, or even "millions of perl powered sites" out of your hat doesn't make my original assertion false.  Slashdot, for example, of which I have been a reader for nearly 10 years, has struggled with scalability issues almost from the beginning.   The plain fact is, web technology has grown up.  

        Additionally, that Amazon uses perl actually doesn't tell us much.  How do they like it?  What percentage of their operating costs are associated with maintaining it?  I know for a fact that priceline (a Java-based site) determined that a complete site redesign would cost them $5 million and for that reason, postponed the project.  And their redesign was based on the desire to update the feature set, not deploy a new technology.  

        So, big legacy sites like Amazon are not, of themselves, poster children for a technology.  They may simply conclude that they can't afford the investment in new infrastructure, the way you keep driving that clunker because the incremental maintenance costs are easier to manage than the large additional costs of a new car purchase.

        I would wager real money that if Amazon or Yahoo were to undertake a complete site redesign this year, they would not be deploying perl, python or ruby for the site infrastructure.  And, I'd win that bet, too.



        Michael Powe Naugatuck CT
        -7.0, -6.15
        For if ye love them which love you, what reward have ye? do not even the publicans the same?--matthew 5

        by pdxlooie on Thu Jan 04, 2007 at 06:59:50 AM PST

        [ Parent ]

        •  I made the statement about the LAMP (0+ / 0-)

          technology stack at the biggest consumer sites (Google, Amazon, Yahoo), and I stick by it because I heard it firsthand from the people who know.  I believe it was an episode of The Java Posse in fact (which is kinda wierd but at least they're honest) and the discussion included top level architects at Google and other big sites.  They don't use the LAMP stack because they're legacy.  They use it because it fits their requirements.

        •  Damn it, (0+ / 0-)

          you made me waste time searching job postings.

          First, in that Amazon link you have, the next bullet point says * Well-versed in Perl and SQL

          The only web developer posting I could find at Yahoo is for a PHP developer for a new project and it says - Perl, XML/XSLT, JSON, REST, SOAP, AJAX, SQL, MySQL

          You do seem to be right about Valueclick, but my conclusion is that you resemble that orifice you mention.

      •  enterprise-level (0+ / 0-)

        the way I generally use and understand the term implicitly excludes b2c applications, especially the very-large-scale b2c apps like Google.

        For instance, when I talk about 'selling to the enterprise' the unspoken implications below that are usually that:

        1. the app is hosted internally in the enterprise's data center and must be able to fit into that enterprise's best practices and general architecture.

        1a. This implies (without requiring) that the app cooperates with the rest of the enterprise's applications rather than dictating to them.  So, for instance, a proprietary app written entirely from scratch, adhering to no public standards and without interapplication communication abilities would NOT fit the enterprise profile.  An app like that could, however, be perfectly fine for a Google or Yahoo.

        1. the audience is employees, not the general public

        which implies that the app UI must scale, but the upper range of that scale is in the 10's of thousands, not the millions (like Google).

    •  you'll be doing maintenance and upgrades anyway (1+ / 0-)
      Recommended by:

      Every enterprise-level web presence is coded in one of those two backends

      Amazon and others use mod_perl/Mason.

      More importantly:

      There is no substitute for good application design.

      There is no substitute for good project management.

      There is no substitute for talented developers.

      If you're smart, you choose a language after you've designed your application.

      If an IDE is saving you more than a little time, your application design is wrong.

      When the application that your java IDE vomited forth needs serious work, I'm the guy who gets hired to do "maintenance and upgrades".  It's going make the end user very angry because it will be expensive.  It's going to make me angry because it should've been done right the first time.  When was the last time you lifted up the toilet seat to see what you're actually producing?

      (I spend most days (re)writing java.  I find it adequate for moderately complex tasks.  I have used most of the major java IDEs.  I find them fucking useless.  If you like working in that environment, they must have you doing insultingly dull shit.)

      strange things happen with a free alphabet

      by dirge on Wed Jan 03, 2007 at 09:29:38 PM PST

      [ Parent ]

      •  Problem Resolution (0+ / 0-)

        If you're smart, you choose a language after you've designed your application.

        Well, that would make Hunter stupid, since he's planning to base the site design on his choice of language.

        And, of course, that was my major point, that the decision was not being made on the basis of what is the best tool for the job, but on the basis of what tool The Decider wanted to use.

        None of your points obviate mine.  Nor does pulling a rabbit out of the hat (Amazon) demonstrate that hand-coding millions of lines of code is a serious substitute for the kind of coordinated and controlled code production and maintenance you can get using modern IDEs in a Java-based environment.  Finally, Amazon has dozens, probably a hundred, developers to maintain that codebase.  What is kos going to have? Three?  Six, maybe?  They'll all be bald inside a year.  

        I spend most days (re)writing java.  I find it adequate for moderately complex tasks.  I have used most of the major java IDEs.  I find them fucking useless.  If you like working in that environment, they must have you doing insultingly dull shit.

        Sorry, this "I'm a manly coder" chest-thumping nonsense bores me.  Save it for comp.lang.perl, where you are surrounded by similar thoracic drumming.  If you've found somebody to pay you four hours' wages to do two hours of work, good for you.  That's not my approach to my job.  

        I like perl and I like Larry Wall.  I've written thousands of lines of perl in emacs with cperl-mode.  It's good for what it is.  And, despite the occasional crowing from kos about the size of this site -- dk is not in the league of or, or even American Eagle, Java-driven sites all.  Frankly, it's unlikely that it ever will be.  

        But that doesn't mean that hundreds of hours of hand-coding what amounts to the code equivalent of boilerplate will result in a site more scalable or more easily maintainable than a site developed in a modern Java environment using state-of-the-art design and coding tools.  It just means that the developers prefer to spend their time "typing" rather than bugfixing, profiling or testing.

        A lot of people don't like Java, obviously.  I figured I'd get flamed.  Don't waste my time dressing up your prejudice in pseudo-rational arguments that, in fact, have nothing to do either with the actual deployment of Java or the design and implementation of Java applications.  When designers want security (Financial Services industry) or scalability (enterprise web sites), they aren't choosing perl, python or ruby.  They're choosing Java or dotNET.



        Michael Powe Naugatuck CT
        -7.0, -6.15
        For if ye love them which love you, what reward have ye? do not even the publicans the same?--matthew 5

        by pdxlooie on Thu Jan 04, 2007 at 06:04:56 AM PST

        [ Parent ]

        •  I don't really have a problem with java (0+ / 0-)

          You didn't get flamed by me for suggesting java, or even an IDE.  You pissed me off by acting like you've discovered the Alchemist's Stone.  You haven't.

          Probably you didn't intend to come off like that, so I apologise for the tone of my earlier post.  I've got a lot of other people's messes to clean up, so I guess I was a bit cranky and maybe you pushed my buttons.

          Anyway, I find java slightly frustrating for a variety of technical reasons, and deal with a huge amount of extraordinarily bad java, but very little of the bad code is the fault of the language.  I do have a problem with the way most people use most IDEs.  Like the IDE users I work with on a modern java web app.   They're really frustrating.

          Any tool that helps produce more boilerplate code faster strikes me as a mixed blessing at best.  If you're producing boilerplate code at all, there's clearly something wrong with your design.  If an IDE helps you live with that, well, that's a compromise I'd make if I didn't have better options.

          I mean, I use code generators and such when I have to, but I always take it as an indication that there's a deeper problem I ought to fix.  And when I do fix those, I'm saving a lot more man-hours than I'm putting in.  When I'm doing my job properly, the code base gets smaller.

          Incidentally, I find that sort of productive coding easier in perl.  I find that IDEs and the like do nothing but get in the way on the big wins, because those often involve unconventional techniques.  Your mileage may vary.

          And maybe you didn't follow that link.  It's not like Amazon is the only significant outfit on that list.  And I'm not actually a big fan of IBM's site.  But that's really not the point;  commercial success is a weakly correlated proxy measurement for technical utility.

          And finally:  emacs?  Can we have an emacs vs. vi flame war too?  csh vs. ksh?  Oh, I know:  x86 vs. 68k!  That's a really good one.

          So yeah...  Peace.

          strange things happen with a free alphabet

          by dirge on Thu Jan 04, 2007 at 10:54:56 PM PST

          [ Parent ]

  •  PHP is out, huh? (1+ / 0-)
    Recommended by:

    I guess those fools at Yahoo must really be out of it, eh? What with their billions of dollars in market capitalization and all, basing it on an insecure house of cards.

    You should be engaging the community around a more substantive discussion than "what language do we use?" if you want this to be of any value. This thread is an exercise in fanboy-ism, nothing more.

    If this is how you want to run yr development project, be sure to double your budget now (and again when you actually make some decisions).

    But good luck guys. Hopefully you will embrace open standards if not open source.

  •  Language observation (3+ / 0-)
    Recommended by:
    montpellier, klondike, jfm

    The languages are gathering votes in order of age. Ruby, the newest, gets the fewest votes. Perl, the oldest, gets the most.

    Two comments. (1) If you took Perl out of the equation, it might be interesting to see how the vote changes. (2) Ruby is much closer to Perl in terms of ease of coding for regexps and such. Perl programmers wouldn't complain as much as they would with Python.

    Also, what LarryInNYC said. You need to fix the database layer first. If you get that right, it makes the front end much easier.

    •  Perl > Python > Ruby (0+ / 0-)

      I suspect the poll results would be almost exactly the same if the question were "Which is your least  favorite language?"

      Just like with political polls -- favorables and unfavorables are often both just a factor of name recognition.

      <div style="color: gray; font-size: 80%">(-7.88, -8.97)</div>

      by Abou Ben Adhem on Wed Jan 03, 2007 at 08:52:23 PM PST

      [ Parent ]

  •  Not Perl (0+ / 0-)

    I would not write a new application in Perl. Perl is the web's version of COBOL. I know with ModPerl it would be faster and multithreaded and all that, but that comes with overhead too.

    I am managing a legacy application in Perl and I can't wait to reengineer it into something scalable. I am not sure if even with ModPerl or FastCGI you can write a scalable Perl application.

    And Perl is not object oriented. Why bother to use it? I don't know much about Ruby on Rails, but of the three Python would be my choice. After all it can be compiled.

    Many an insightful opinion and observation can be found on my blog Occam's Razor.

    by Guy Noir on Wed Jan 03, 2007 at 07:15:54 PM PST

    •  I am writing a new application in perl (2+ / 0-)
      Recommended by:
      mykej, montpellier

      First, you can (and should) write object-oriented perl code.  I've got an entire object relational mapping interface that I use routinely:  on application bootstrap it examines the database schema and rewrites its own API to suit.  I'm planning to overhaul it in the near future, because I've gotten to like Hibernate, but think java is too crippled to do what I want.  In a lot of ways, OO perl is much more flexible than OO java and others.  If you know what you're doing, you can invoke some very useful black magic.

      I suspect the problem you have with perl is that there's nothing about the language to stop people from writing bad code.  In fact, there are a lot of things about it that make it easy to write worse code than is possible in most other languages.  It's deceptively easy.  There are cool tricks that get you into trouble.  I also deal with legacy perl apps, and I've seen plenty of that.  The good ones are great, but the bad ones are terrifying.  None of that stops anyone from writing good perl code if they can.

      And the performance issue is entirely solvable.  If your web app is CPU bound these days, you're doing something wrong regardless of language (or maybe something way more exotic than kos).  And in practice, beyond a certain level of complexity, I expect a perl app to dramatically outperform java, since you'll never have to fall back to reflection, delegators, visitors, wrappers and other cheezy workarounds for your crippled environment.

      And mod_perl/Mason.  Whatever environment I find myself working in, I wind up reimplementing Mason's core facilities.  Mason has the incredibly cool effect of encouraging good code organization.

      In the end, I'd say if you have a small team of smart people you trust, then perl is an excellent choice.  Because while writing very bad perl is very easy, writing good perl is hard but entirely worth it.

      If you have a large team who may or may not be morons, you're pretty well fucked anyway, but something like java might help limit the damage.  Because you have to be either a certifiable genius or a head injury victim to write anything other than mediocre java.

      I hear Ruby and Python are pretty interesting though.  I'd comment on them if I'd written more than a few thousand lines.

      strange things happen with a free alphabet

      by dirge on Wed Jan 03, 2007 at 08:36:28 PM PST

      [ Parent ]

    •  Not so. (3+ / 0-)
      Recommended by:
      mykej, montpellier, dirge

      I am managing a legacy application in Perl and I can't wait to reengineer it into something scalable. I am not sure if even with ModPerl or FastCGI you can write a scalable Perl application.

      False. Critical Path's webmail system was written in mod_perl in 1998, and was hardware scalable to millions of users. ValuClick also mod_perl/MySQL, and they serve half a billion requests a day. I've worked on several n-tier systems built in mod_perl, it's fast and scalable. It is also adaptable to small CGI hack scripts, which are slow. The difference between perl CGI scripts and mod_perl applications is that a perl CGI script incurs the interpreter overhead every time the script is called, whereas mod_perl embeds the interpreter in the Apache process. With some skillfull engineering and configuration, most of the shared parts of the mod_perl application can be loaded and compiled in the root process at startup. Perhaps your legacy app was badly engineered?

      And Perl is not object oriented.

      Also false. It is not as OO as python, java, or Ruby, but its OO facilities are more than capable of providing the benefits of OO architecture for large codebases.

      Additionally, mod_perl will give you full access to the Apache API, and lets you build custom handlers for each stage of the request cycle.

  •  If I was paying for it... (1+ / 0-)
    Recommended by:

    If I was paying for it I would try to improve the existing site.  I realize no one wants to read other people's code but I think it is cheaper and more efficient.  Find out where the inefficiencies are and try to speed them up.  You will have to re-design and it is confusing working with old and new code, but that is what I would do.

  •  Uh, ooops (0+ / 0-)


    Wrong room.

  •  and another actual architecture suggestion (0+ / 0-)

    entirely self interested, but don't develop tunnel vision on the desktop form factor - 1024X768 resolution, kbd and mouse, 2 handed operation, scrolling, integrated advertising, negligible latency and unlimited bandwidth.

    From an "ebb and flow of technology" perspective, a lot of the frameworks now strike me (entirely subjectively) as so cool and over-the-top (but just for the desktop) that the technology undercurrent just has to be moving in a direction that will make them irrelevant.

  •  aoeu (1+ / 0-)
    Recommended by:

    Haskell for the language, I'm not sure what sort of geek wouldn't even consider it!

    dulce bellum inexpertis

    by TealVeal on Wed Jan 03, 2007 at 08:04:35 PM PST

  •  four podcasts that speak directly to these issues (0+ / 0-)

    Don't take all this advice on my say-so.  Listen to some real experts.  Tell them klondike sent you.

    Software Engineering Radio.  A no-bullshit hardcore systems architecture show.  These guys kill me.
    Web Dev Radio A real web developer.
    Web 2.0 Show The 37 Signals crew - round buttons and laid-back, west coast, web 2.0 hype.
    The Java Posse All things Java.

    I can strongly recommend ALL the back issues of all of these shows.

  •  My free advice, worth what you've paid for it (2+ / 0-)
    Recommended by:
    Abou Ben Adhem, klondike

    I'm just another Perl hacker. It's the language I use when I have a choice. It's what I write stuff in when I'm writing for fun. It's a messy language, which suits my messy mind.

    But honestly, Python and its enforcement of design principles would make for much more maintainable code. (And just as honestly, much of the reason people don't like one language or the other has nothing to do with speed or maintainability of code. Some people think Perl looks like an explosion in a punctuation factory and hate its lack of variable typing. Others are turned off by Python's use of white space as a delimeter. Me, like I said I use Perl when I'm writing for fun or quick hacks, but I find it easier to build user interfaces in Python with Tkinter than I do with Perl::Tk. It's all a matter of perference, and I don't get into messy flame wars either way.)

    You can't go wrong with either one. I don't know enough about Ruby to comment intelligently on it.

  •  Frameworks: Be Very Afraid (6+ / 0-)
    Recommended by:
    EightBall, Hunter, permanentE, njn, zappini, dirge

    I've had some experience working on reasonably large scale web apps, and my personal suggestion for a site like dKos would be to run the hell away from frameworks as fast as possible, regardless of language. The reason is scalability.  

    Typically frameworks are designed around the core assumption that programmer time is a scarcer resource than system resources.  Their goal is to help the programmer who's stamping out lots of sites, typically with small user bases.  So they aim to speed up development, and do so by abstracting out a lot of stuff into framework-specific libraries.

    For a site like dKos, that assumption is wrong wrong wrong.  dKos' traffic is mongo and getting mongo-er every day -- and is subject to sudden spikes, if, say, a Congressman gets caught diddling pages in his spare time.  And you guys have only one site to worry about -- unlike consultants who can turn a profit if 80% of their clients are happy, you have one client, Markos, and one chance to get it right.

    With a soup-to-nuts framework, you run into problems in this scenario.  Say traffic spikes, and suddenly the database servers are crashing, seemingly randomly.  How do you diagnose and fix the problem?  You can't, because the database interaction is a black box -- one of those libraries that made development so much easier.  Suddenly you're searching Usenet in a panic, hoping that somebody else has (a) had the same problem with this framework, (b) figured out how to fix it, and (c) bothered to write the solution up.  

    (Sure, you can inspect the source code of the library and try to figure out why it's flaking.  Do you really want to be doing that when the site is crashing and Markos is screaming at you to get it back up?)

    Whatever language you use -- and frankly, my advice would be that the language is irrelevant, use whatever your team knows best -- when you're working on this scale, you want to be sure you know every inch of it intimately, or at least every inch that potentially could present a performance bottleneck. Yeah, that's gonna slow down development, but them's the breaks when you're the #1 political blog on teh Internets, right?

    The frameworks advertise that they make your life easier by doing magic. Anything that involves the word "magic" in a high availability project should give you the willies.

    (Caveat: Hunter mentions mod_perl in the same breath with RoR and Django when talking about "frameworks".  IMHO mod_perl is a different beast.  My comments are more about the currently-popular everything-and-the-kitchen-sink frameworks ala RoR, Django, CakePHP et al.)

    •  One more thing (2+ / 0-)
      Recommended by:
      EightBall, zappini

      Replying to my own comment because I was too dumb to mention this in the original: to see what I mean about the value of "owning" the code that handles the bottlenecks, just look at Google.  They wrote their own filesystem, BigTable, for pete's sake.  I wouldn't recommend going that far but it's telling that BigTable is universally cited as a key reason for Google's scalability and performance.

      •  productivity, performance, scalability (1+ / 0-)
        Recommended by:

        Productivity: I've written in Perl (cgi and mod_perl), Java (mostly servlets) and ROR. I've found ROR wonderfully productive. I'm working on an app now in ROR. I do Java for my day job (J2EE on the way). I do ROR for my own stuff. I'll never do Perl or Java for my own stuff ever again. Way too much unnecessary work.

        Performance: Any db intensive application is ultimately bounded by the db. Getting that right can be derived from usage analysis that drives indexing and caching decisions. DK - with years under the belt - should have plenty to work with there

        Scalability: See the following link. Basically: tiers and balancing can get you anything you want in scaling - up to the db.

        •  database bound apps (0+ / 0-)

          For an app like dkos, you can cheat the database bottleneck, because it's non-transactional read-mostly.

          If you're clever about it, you can drive reads off of cached and/or best-effort-replicated databases (Amazon and others do this).  The only penalty is that it's a pain to set up and maintain, and we'll all see a little latency between posting a comment and seeing it show up. (With a little gimmickery, you could see your own show up right away, but you'd still wait a few seconds for everybody else's.)

          So you can even scale the database, for reading at least.  You're still stuck with a single (presumably powerful) machine for posting though.

          In theory that could be fixed too, since it's pretty much all inserts rather than updates, so a lot of locking problems can be dodged -- that sort of thing can work peer-to-peer, though I don't imagine kos funding a bunch of from-scratch distributed  database dev (but if he did, I'd love to pitch in).

          strange things happen with a free alphabet

          by dirge on Wed Jan 03, 2007 at 09:52:02 PM PST

          [ Parent ]

          •  One of my favorite caching tricks (4+ / 0-)
            Recommended by:
            Hunter, damn furriner, dirge, klondike

            Comes out of the Django framework: a cache setting which caches everything, but only for users who aren't logged in (the reasoning being that anonymous visitors can't post new content, so there's no need to worry about them not seeing their posts show up immediately). Since in most cases the majority of visitors aren't logged in and don't even have accounts, that can be a gigantic performance gain at little cost of interactivity. Of course, Django also has a variety of other cache settings -- high- and low-level, fine-grained and non -- but that's by far my favorite :)

    •  programmer time vs. CPU time (2+ / 0-)
      Recommended by:
      Hunter, klondike

      Generally, I don't think there's much competition here:  I'd rather blow the CPU time -- even buy a couple extra servers.  Because good programmers are hard to come by.

      The huge problem I have with the "frameworks" I've dealt with is the programmer time vs. programmer time tradeoffs I've encountered:  most of these things wind up making easy things even more trivially easy, but hard things almost impossible (where, as a nod to your point, hard things include making the occasional pathological page perform well).  Unless everything kos plans to do is easy, stay away.

      And mod_perl isn't a different beast merely in your opinion.  It's not a framework at all;  just an embedded interpreter and a collection of C<->perl bindings.  mod_perl/Mason is a framework, and a good one because it doesn't attempt to do too much and doesn't attempt to anticipate what you intend to do with it.

      Sure, you can inspect the source code of the library and try to figure out why it's flaking.  Do you really want to be doing that when the site is crashing and Markos is screaming at you to get it back up?

      What?  You mean other people have days that aren't like that?

      strange things happen with a free alphabet

      by dirge on Wed Jan 03, 2007 at 10:07:36 PM PST

      [ Parent ]

    •  Be afraid... of misinformation (1+ / 0-)
      Recommended by:

      "Run away from frameworks because they focus on programmer time" is just as misinformed and ignorant a statement as most of the framework hype ;)

      Frameworks scale just like anything else: through a combination of caching and hardware. The difference is that you get to the point where you can start working on scaling much more quickly.

      Most of the rest of your criticisms can be leveled at pretty much any piece of software ever developed; frameworks are not any more or less prone to random bugs or lack of documentation than any other software products.

      If you'd like to bring up specific criticisms of specific frameworks, I'd be happy to discuss them, but generalized misinformed bashing isn't a constructive way to help dKos make a decision ;)

      •  there are trade-offs (0+ / 0-)

        Analogously to what I said earlier (frameworks will tend to make easy things trivially easy and hard things nearly impossible), frameworks will tend to make common cases perform extremely well and unusual cases perform pathologically badly.  Of course, really good ones provide a way to get around that.

        A good framework is probably fine for kos actually, because I think the requirements conform to pretty straightforward patterns, but it's still something to be wary of.

        In any case, you have to be wary of the shear number of moving parts, and another layer of middleware is another moving part.  If you can avoid it, you should.

        strange things happen with a free alphabet

        by dirge on Thu Jan 04, 2007 at 12:39:45 AM PST

        [ Parent ]

  •  Don't rewrite from scratch (1+ / 0-)
    Recommended by:

    Joel Spolsky has good comments about this here and here.

    Do you really need to rewrite from scratch?  That's what it sounds like you're planning.  I would consider instead keeping the current system -- which after all, works well and is time-tested, even if it doesn't do everything you want -- and just rewrite chunks of it as you go.  That way DK3 can evolve into DK4 gradually (and gracefully).  A bit like Apple is doing with Mac OS X (compare 10.1 with 10.4) as opposed to Microsoft's Big Bang approach with Vista.  

    Some of those chunks may be large (eg. 30% of the system) but I doubt you really have to rewrite the entire thing in one hit.  It would no longer be Scoop, but who cares?  It sounds like Scoop has already been modified enough that it's no longer Scoop anyway.

    •  Every once in a while... (0+ / 0-)

      You need to bite the bullet, like Apple did going from OS9 to OSX.  Generally the need isn't going to be obvious to anybody but the developers in the basement or some self important genius/prick like Steve Jobs (pretty sure Vista's problem is that they'd been telling those poor sods to piss off since around '96, and now it's too late).

      The thing is that if that 30% chunk you're talking about that needs fixing happens to be scattered at random through 90% of the code base...  Well...

      Anyway, if kos thinks it needs fixing, well, he's got to live with the miserable beast more than we do...

      strange things happen with a free alphabet

      by dirge on Thu Jan 04, 2007 at 01:25:15 AM PST

      [ Parent ]

      •  well... (0+ / 0-)

        Apple didn't write OS X from scratch.  They took BSD, an already-working OS, and extended it.  Big difference.

        •  actually descended from NeXT (1+ / 0-)
          Recommended by:

          Jobs left Apple to start NeXT, and a lot of that technology was brought back to Apple when he returned.  Even when they started on OSX it wasn't really BSD any more.

          You're right that the kernel for both is BSD based, but that's only one piece amongst many.  The OSX package manager is taken from a Linux distro, they kept but reworked their file system, but the display-postscript driven UI is all new, as are all the system configuration interfaces and high level APIs, and that's just the first couple things to come to mind.  The amount of work that went into getting all of it working was immense.

          Anyway, my point wasn't that they didn't reuse any technology, but that they were willing to scrap a most of their own (some of it pretty cool) when it was prudent to do so.

          strange things happen with a free alphabet

          by dirge on Thu Jan 04, 2007 at 09:18:52 PM PST

          [ Parent ]

    •  I agree, do not write from scratch (0+ / 0-)

      I agree.  I posted something similar but but less detailed yesterday.  I have worked on stuff, through no fault of mine, did not work or was very much over budget.  People frequently shoot the messenger so I have mixed views on reporting problems to management.  I have worked with those who wanted to work on interesting things,  and justified doing technical things that absolutely made no sense, and screwed it up.  For some reason management, not only wants to manage, but be considered brilliant technicians and frequently overrule technical decisions.

      Programs grow over time and after years are much bigger and more complicated than management originally estimated.  One of my managers at a large automotive company grossly underestimated a conversion from one platform to another and lost his job.  

      Testing is much easier piece by piece.  I am not an expert at any software because I very seldom do the same thing twice.

  •  age weighted scores: ruby dominates (0+ / 0-)

    While Perl and Python appear to dominate Ruby in raw scores. Ruby dominates the age weighted scores.

    Perl and Python have both been used in creating web applications for over 10 years. ROR went 1.0 in December 2005.

    If we divide votes by years we get the age weighted scores below - using raw scores around the time of this post.

    Ruby: 239/1 = 239
    Python: 403/10 = 40.3
    Perl: 418/10 = 41.8

    Hmmm. Seems like that's worth thinking about.

  •  I would vote Perl... (0+ / 0-)

    From my understanding, Perl is the oldest and best supported. But with any pure language you are "re-inventing the wheel" by writing a web app from scratch.

    I would also check out ModX. Yes, it's based on PHP, but, it has AJAX as well. Security concerns over PHP are mostly overblown.

    Consider; is it better to plot a strategy and wait, or set a course, and run? - BMM

    by keechi on Wed Jan 03, 2007 at 10:01:08 PM PST

  •  Democratic choice (0+ / 0-)

    Ruby on rails. Full stack of open source goodness.

    Scaling. That myth has been debunked. Some very large sites use it.

    It is all about the hardware architecture.

    The RoR guys are the best out there.

    Trust me on this one ;=)

    •  No it hasn't (0+ / 0-)

      "Scaling" in Ruby is still in question, and that doesn't even include Rails in the discussion.

      There are no large web apps being run on RoR that I've ever seen. It's new. It needs to mature.

      Not to mention, every other option he listed is also open-source.

  •  new server arche. (3+ / 0-)
    Recommended by:
    Terri, corncam, montpellier

    Hunter, I wouldnt be asking which tool to use. You should be deciding which tools to use. It seems to me that at the base what this site is a massive db query engine with xml (albeit in html format) output. I would think it would be best to reduce down the core processes, and by core i mean, maintain a database connection (preferably to SQL Server, because it will do native XML) roll up your queries into some hand tuned stored procs. and use some perl to match the XML output with some XSLT templates to generate your html. you should be able to match the speed of static html production. as far as the rest of the site, the no comment section/ thread pages, i would look at a stripped down drupal, its cache algo. is great. and for search i would highly reccomend a google mini box, its $2,000 for 1 year and it does a good job...

    anyway, i dont know if that made sense, a thread like this is not the best place for a discussion of high performance web apps. you can email at hubris.sonic at gmail if you like.

    i have used something similar to the above to run a quad lingual airline website for all of asia pacific on very little hardware because the performance is great.

    oh, and i would stick with AJAX for the commenting, etc...

  •  Any plan to let other blogs pay for your code? (0+ / 0-)

    You all have done a hell of a lot of work to the platform -- any plan to make it available to other progressive community sites?

    Since there's already been so much put into this, it seems like you could expand the tech team a bit, let other sites pay you a fair price for the work, and enhance progressive infrastructure everywhere while taking advantage of economies of scale/enhancing the capacity of your development team.

    Just a thought.

    I tell you truly, whatever you did for the least of these brothers of mine, you did for me. -- Matthew 25:40

    by mSnook on Thu Jan 04, 2007 at 12:52:43 AM PST

  •  My ten cents (2+ / 0-)
    Recommended by:
    Hunter, Hubris Sonic

    Before I start, I'm a web development project manager. I've run (and currently run) projects on Ruby on Rails and Django, and I've worked with Perl in the past, although not with the frameworks mentioned.

    Some general comments

    Regardless of technology choice go for agile development. The opportunity DKos gives to get feedback on ongoing development work will be priceless.

    Regardless of technology choice worry about speed. It's not uncommon to be able to make parts of an application 500% quicker through refactoring and good design. If those are the parts that are slowing you down - that's crucial and far more important than language choice.

    I've had good experiences with the Google Mini / Google Search Appliance for introducing off the shelf search. Also has the advantage of moving search / spidering to a separate server, big disadvantage is that it only updates daily. Also not sure how it would hold up to your levels of traffic (I deployed it in a corporate intranet).

    Some specific Comments

    Ruby on Rails
    My biggest issue with Ruby is that is doesn't support character UTF-8 encoding that well. But since you're not developing for the international user group I am that's less of an issue for you.

    Ruby is very nice to work with, code is very readable and developers tend to like it. Rails is extremely good for churning stuff out quickly, the Ajax libraries are great and it's probably the nicest development platform I've worked with.

    However - it does have issues with scaling. Not the huge scary issues some folks talk about, but it doesn't scale as well as (say) Java. but regardless of the technology I picked I'd want to do some load testing before I began development - there's a nice presentation at

    that covers how to do what you need to.

    Someone who can answer more questions about scaling with Ruby than I can is Obie Fernandez ( ) who has worked on a lot of enterprise scale Ruby stuff.

    Ruby programmers are easy to find and in my experience tend to be of a very high quality.

    Django seems to be nice for use as a publishing tool, and less nice for use as a platform for application development. My biggest issue with Django has been the availability of documentation (it's well behind Ruby on Rails) and a less active community.

    Python developers are plentiful, Django developers are rare and if you want to get a the most out of the framework you'll need someone who really knows their way around it.

    Probably scales better than Rails for publishing given it's background and purpose in life. Not so sure it will do more interactive stuff on a very large scale.

    Don't know enough about the current state of play to add much.

  •  I've used em all... and Ruby wins hands down (1+ / 0-)
    Recommended by:

    I've used all these languages before and nothing has been more of a joy to work with than Ruby.  The Ruby/Ruby on Rails community is an awesome, incredible group of developers that are so much fun to work with.  The language is simply beautiful.  And the amount of work I can get done in a short amount of time is incredible.

    • Ruby
    • Rails
    • Memcache
    • Ferret (the Ruby port of Lucene)

    Awesome combo perfectly capable of handling the heavy loads that DK gets.  Ruby gets a bad rap for being slower, but I have yet to see any difference in my live applications.  Besides with Memcache it doesn't matter anyways.  Memcache will take that heavy load off the database store the frequently accessed data in memory.

    And even if it did make a difference, the new VM (YARV) was just merged into the trunk of Ruby 1.9 and its 2-10 times faster than 1.8.5, depending on the operation, making it as fast if not faster than Python and Perl.

    Ferret is an awesome port of Java's Lucene in Ruby.  Its relevance and accuracy is superb and gives all that fuzzy searching goodness that we need.  And with Dave's continuing work on this, its getting faster and close to competitive with Lucene performance.

    It would be a joy to add DK to the Ruby family.

  •  Whichever you choose... (0+ / 0-)

    Is there any way interested developers can help?

    US Citizen Overseas? REGISTER TO VOTE ALREADY!

    by kevin lyda on Thu Jan 04, 2007 at 03:03:00 AM PST

  •  I vote "None of the above" (0+ / 0-)

    Clearly Hunter has a bias against Windows, since he hasn't considered C# and .NET. All three of those languages suck. Use a REAL programming language, why don't you?

    •  Not unhead of (1+ / 0-)
      Recommended by:

      ... anti-Microsoft decision making, on principle alone.  It is fair enough, especially if there are budget contraints on top the philisophical reasons.

      There are both leaders & followers in this world ... which will you be? --- My Mom.

      by DBW21 on Thu Jan 04, 2007 at 04:54:40 AM PST

      [ Parent ]

    • Perl's not real? (0+ / 0-)

      That assertion provides all the information one needs.

      Actually, they ruled out Java, so maybe they're ruling out the whole IRL re-incarnation of the MS Java Engine.

      •  No. (0+ / 0-)

        It's a scripting language. Not a programming language.

        •  hahaha (0+ / 0-)

          As somebody who works in Computer Science research and has graduate work in PL, I think that's entirely ignorant.

          Do you really understand how .NET framework operates?  What exactly is the IRL?  

          Compile time for various stages hardly differentiates "real" from "fake" in programming languages.  

          A script is a program.  

          •  IRL? I'll give you a min to take your foot out... (0+ / 0-)

            First, don't make me laugh. Please. You bring up "computer science research" and then show absolutely no mastery of what you're talking about: There's no such thing as "IRL." You basically combined MSIL and CRL into one. Normally I'd let that slide, but if you're going to use "graduate work in PL" as some sort of credential, you ought to get the acronyms right.

            Second, why would you jump to some conclusion about "compile time?"

            Perl is not compiled. Period. It's code is interpreted at runtime.

            CLS languages are, in fact, compiled to MSIL. And compiling is compiling. Whether it's being compiled into MSIL or assembly is irrelevant.

            I appreciate how dumb you made yourself look here. If you really are doing "graduate work in PL" I suggest you ask your undergrad institution for a refund. You're not exactly displaying a mastery of your field.

            And seriously, don't make yourself to look like such a douche bag. While you're doing "research" I'm spending 60 hours a week actually developing commercial software.

            I was going to wrap this comment up, But this sentence just makes me double-over in laughter:

            "Do you really understand how .NET framework operates?  What exactly is the IRL?  "

            I mean, you couldn't have done this better if you planned for it. You try to be all superior and ask "do you really understand.." and then in the very next sentence you illustrate to the entire world that it's YOU that doesn't understand.


            •  not true: (1+ / 0-)
              Recommended by:

              mod_perl embeds the interpreter in the Apache server process, and any modules loaded in BEGIN blocks or from a are loaded and compiled in the root process. All the overhead happens at server startup, from then on it's compiled.

              See here for details.

              •  Still.. (0+ / 0-)

                Yes, I'm familiar with that, but it is still interpreted with every new httpd instance.

                •  also not necessarily true (1+ / 0-)
                  Recommended by:

                  Anything loaded in the parent process is compiled once at startup. For modules that require loading by child processes, those are compiled once, and kept in memory. For the most part, if intelligently configured, those modules should be minimal.

                  As an aside, I've rarely seen the mod_perl part of the application as the performance bottleneck. Most of the  time performance can be more easily be wrung from query caching and templating. In the rare cases where very large object instantiation was a bottleneck, object caching took care of the problem. (I do love me some memcached.)

            •  compile time (2+ / 0-)
              Recommended by:
              montpellier, jfm

              MSIL is translated to machine code at run time just like Perl.  Whether you call this run-time translator a JIT compiler or an interpreter is a pretty stupid way to differentiate "real" vs "fake" languages.

            •  My Apologies (1+ / 0-)
              Recommended by:

              When MS Research sent someone out to peddle .NET to us, their rep. labeled MSIL IRL (intermediate runtime language).  Pick on them not me.

              The final poster in this thread made the point I was trying to make - your precious C# compiles at runtime just like Perl and other "scripting" languages.  

              No, compiling to machine language is handled at the final stage in both cases.  MSIL ain't machine language.  

              Any distinctions drawn between a "script" language and a "real" language based on this are entirely arbitrary.  Pre-processed (and semi-optimized) object code can speed things up a bit.  

              •  Riiight... (0+ / 0-)

                Microsoft "got it wrong." Sure They Did.

                Seriously, learn to accept that you were wrong, and that arrogance just makes you look dumber. This can be a learning experience for you.

                More importanly, you compile C# to MSIL.

                You fail to realize this step.

                "Compiling" has nothing to do with machine code. C# is compiled. Perl is interpreted. Just that simple.

                The funniest thing is how you're taking this personal. "your precious C#" you say. I'm not even a C# developer! 95% of my day is spent developing C++.

                And anyone that knows a real programming language knows that Perl, like PHP, VBScript, etc, is a scripting language

                Sorry, you're not a winner today: play again soon!

                •  No bright line between compiled and interpreted (1+ / 0-)
                  Recommended by:

                  "Compiling" has nothing to do with machine code. C# is compiled. Perl is interpreted. Just that simple.

                  C# is compiled to bytecode, which then gets interpreted by a VM, same as Java (I suppose both can also be compiled to native machine code, but usually aren't).  Perl is compiled to bytecode, but the compilation step gets run every time the script is run (the compiled bytecodes are not written out to disk).  Python is compiled to bytecode the first time the script is run, and every time it's run after being modified, but the compiled bytecode is saved to disk and used when available.  Indeed, Python can be compiled to JVM (Jython) or CLR (IronPython) bytecodes.  The difference is not as significant as you think it is.

                  And anyone that knows a real programming language knows that Perl, like PHP, VBScript, etc, is a scripting language

                  Repeating something doesn't make it so.  Serious Versus Scripting is a pretty subjective argument.  I don't care for Perl, but that's because  I don't like some of its design features, not because it's "a scripting language."

  •  Cross-post on Slashdot? (1+ / 0-)
    Recommended by:

    Although the dkos community has a wealth of collective intelligence, the group expertise is politics, not technololgy.

    The types of questions you are posing are daily subjects of discussions on Slashdot, the technology blog, where uber-geeks hang out. Might want to think about asking over there.

  •  sum of the parts (1+ / 0-)
    Recommended by:

    Last bit of advise from someone who has worked on many large-scale projects.

    After reading though all the comments ... take the time to disassemble the big picture back down to its parts.

    This will ultimately be your bottle-neck & be the first to require scaling.  Choose wisely. As a SP proponent, I would discount any DB engine not supporting SPs -- you will tweak out the best performance via SPs & can implement the more secure solution vs passing SQL commands across the data transportation layer.

    Don't skimp, despite what some would lead you to believe, a spectacular Python, Ruby, Delphi, Java, C++ (insert your favorite language here) programmer is not always (if ever) the best candidate for indexing, optimizing queries & fine-tuning DB performance.  Trust me, as you get into the hundreds-of-millions of records, this will be the bottleneck!  Watch your hardware here too, there is a big difference between an i/o bottleneck & a cpu bottleneck & having someone at your disposal who can identify the difference is very, very valuable.

    Data Transportation
    Another don't skimp recommendation.  The scripting language you choose to ultimately collect, assemble & present the data to\from the end-user may not be the best choice to actually move the data to\from the backend.  I have seen too many data transport layers designed around the middle-ware.  PhP, Cold Fusion, ASP.NET, Perl (insert your favorite middle-ware here) used as the defacto source for data transport is just poor design.

    So far, the heart of this discuss from what i have read. While very important, it should not be the end all, as mentioned above. You business logic is typicaly here most all mentioned script, JIT or pre-compiled languages above could do the job of collecting the data, applying business logic & assembling it for storage or presentation.  Some are more scaleable than others, but I would bet the ultimate botttleneck would be in the backend & data transport, not here, due to the amazing performance of hardware these days.

    Front End
    Keep design to the designers & scripting to the programmers. I get a kick out of engineers who whip up front ends & think they look awesome or designers who drop in script blocks from thier favorite design tool and think they are programmers.
    Conceptualize the user experience ... "this is what I would like the user to experience", get feedback from existing users (good & bad) ... Then develop a design con presentation (i.e. code) layer. This is where the AJAX, CSS, etc. discussions should take place.

    TEST, TEST & did I mention LOAD TEST
    Major don't skimp here ... test every aspect, even if you go with tried & true open source, or commerical, solutions.  Dump in 10,000,000 mock up threads, search, post, retrieve, simulate 100 users, 1000 users ... there are plenty of script-driven solutions to simulate load testing, find someone who can taylor one to your needs & beat the crap out of the system before you deploy it.  The biggest failures in development I have seen are those that do not take this step seriously by not affording the time & effort required.  Ah, those chumpware development teams\houses that spend months of design & development and a couple weeks of testing with a handful of users ... it is almost comical ... if it wasn't so very sad.

    Your host will be ultimately tied to your budget contraints ... if you need a 100 gigabytes of bandwidth per month & can only afford 10, well that is what it is.  Whether you host internally, co-locate or lease dedicated it all comes down to budget here.

    Post Deployment Support
    Just a small word of caution from many years of experience ... whomever develops initially may not around when needed in the future.  Be sure all of the above choices are not so esoteric that only a sole source can mantain them.  If you go the open source route ... be sure that you have peeps who stay up-to-date with new releases & have a staging area to apply updates & test again before deploying into production.

    Well for what it is worth, that is my 2.5 cents.

    There are both leaders & followers in this world ... which will you be? --- My Mom.

    by DBW21 on Thu Jan 04, 2007 at 06:21:03 AM PST

  •  Python's probably the best choice. (2+ / 0-)
    Recommended by:
    TalkieToaster, jfm

    I've used all three languages, though python by far the most. Perl, while certainly a gem in the right environment, would be a nightmare to maintain for an app like DK4.

    Both Ruby and Python are fast to write, fairly easy to maintain, and execute fairly quickly. I have a Python bias, but I would suggest Python simply due to maturity. Ruby seems to do most things just as well as Python, but Python has a much longer history of usable web frameworks. But honestly, developer preference should really be the deciding factor. In either language, proficiency will make a huge difference in effectiveness.

    As far as web frameworks go, that's a little tougher. I love Zope, and I think Zope 3 is just about the bee's knees, but I find it very hard to recommend for new projects. It's a complex system, that has a fairly significant learning curve before you can feel comfortable developing in it. That said, developing in Zope 3 goes a long way towards encouraging proper system design, so if you are up for the challenge, it can be incredibly rewarding. But they don't really have an effective AJAX implementation yet (you can put one together, but there's no standard for it), and I think that is a must.

    Among the others, I've used a little bit of all of them, but not enough to give strong recommendations. However, I've read numerous comparisons, and my understanding is that TurboGears would probably be a better fit thant Django. Again, this is all academic, but my understanding is that TurboGears is more appropriate for sites that require a lot of customization.

  •  Biased poll to begin with (1+ / 0-)
    Recommended by:

    Kos and team:

    This whole discussion seems somewhat backwards - a statement like 'letting people work with what they are comfortable' is the reason we have such bad IT projects in the first place. This survey seems to resemble voting in a popularity contest rather than finding out what the best solution is.

    As many have said in this thread - the stated objectives (speed) and the stated options (php, python, perl) are not the ideal match. Hence, as an avid Kos reader I would request this "project" team to sit and figure out what the prioritized list of objectives really are. Please have this discussion on what framework to use after you have had this session. I come into this discussion with my biases as any other poster here but thats precisely why you need to consider all options.

    BTW - check out



  •  can only really speak for mod_perl, but... (3+ / 0-)
    Recommended by:
    EightBall, mykej, montpellier

    I can't speak for Ruby or Python, never worked with them. I haven't worked much with Java either, so most of my impressions of it are from the outside.

    I worked for a very large company that's in the top 5 busiest sites for e-commerce, with a development staff of around 80 PM's, developers, and QA engineers. We used a combination of mod_perl on Apache2, and mod_perl on Apache1.3 for certain application servers where it had to be.

    More important than the language used, though, was the architecture. Our requirements (which aren't unique) were along the lines of:

    • needed to be able to separate services into server classes so that apps handling heavyweight transactions didn't get in the way of those doing very fast requests, and so that particular part of the site could be hardware scaled without having to scale parts of the site that were handling the load just fine.
    • needed to be able to cache arbitrary data (whole pages, heavyweight objects, etc) and access them across several tiers.
    • needed to be able to deploy the same codebase to all servers and have configs automatically deployed for each.
    • needed to be maintainable by a large team of developers.
    • needed to provide metrics and diagnostics so that problems could be rapidly diagnosed and fixed.

    ...etc, etc. For the most part, the difficult and important decisions had to do with the architecture's ability to fulfill the requirements, development and testing environments, code management, release strategies, etc.

    That said, we built our own MVC architecture, implemented in mod_perl. It gave us access to the Apache API, request handlers are very fast, there's lots modules available that are very mature and actively maintained by the community, and we could find decent Perl developers experienced in building mod_perl handlers and fine-tuning DBI.

    The DB parts of the architecture weren't so much a matter of "Oracle vs. MySQL" throughout the platform so much as "Is there a part of this application that would benefit from using MySQL instead of Oracle?" We ended up using Oracle for transactional applications and MySQL for read-only data. That's just how it worked out.

    The difficult part of using Perl, for us, was building a unit testing framework. Not that there's not lots of tools available for unit testing (there are, and some great profiling tools for mod_perl apps specifically), but it had to be built from the ground up for the kind of testing we were doing. My experience with Perl is that you end up having to build many tools yourself that other languages provide for you. I'm personally okay with that.

    The other important thing for us was maintaining discipline. Coding guidelines had to be followed, and documentation was important. Perl will give you more than enough rope to hang yourself if worst practices show their many faces. Profiling was critical, since it doesn't do much good spending a week tracking down some minor performance gains in the request handler if you're spending 95% of your time in the template doing complex conditional tests.

    As to templating systems (and again, I can only comment on Perl's) here's a broad generalization (feel free to object): templating systems in Perl tend to fall on a spectrum between power and speed.

    Typically, Mason templates provide more power to the templater than is needed, and therefore needlessly sacrifices speed. I like Mason for quick jobs where I don't mind violating MVC-separation, but for a system where you'd have different groups working on backend code and templates, I'd think TT or HTML::Template would be a better choice.

    TT will mostly suffer if you try to mix a lot of data types (hashes and arrays and objects) in the template, since it has to figure out what the dot-operator is doing in each instance. Also, TT has a rich templating language, which is either a pro or a con depending on who has to code the templates. But TT's XS stash makes it competitive with the more bare-bones templating systems, while providing plenty of templating features.

    If you're mostly going to be passing hashes and arrays into the templates, then you get the most MVC-separation and speed with HTML::Template. It's kinda primitive compared to the features of the other two, but it's real fast. Real fast.

    So there's a buncha random info, don't know if it helps much, but it's what I got right off the top of my head. Lemme know if there's anything else I can do to further muddle the issue.

  •  +1 for Drupal (0+ / 0-)

    Drupal has an unbelievable active development community, the 5.0 release has made strides in performance, and is very well architected for extensibility.

    The 5.0 release is in release candidate mode now and the 6.x feature list is under discussion. Daily Kos could have a huge input into that.

  •  Starting with requirements (0+ / 0-)

    To build out a back end, you first need to answer two questions:  1) What do I want the back end to do that my currrent back end doesn't do, and 2) What fundamental requirements are there that any back end has to serve.

    Let's start with the second requirement first, because it will rule out all the things that, however whizzy, simply won't get the job done.  Let me make some guesses about DK4.0 requirements:

    1. Traffic - DailyKos is widely visited site (, but not in the Yahoo / Google category.  Conclusion: DK4.0 can trade off some raw performance for ease of development or flexibility of delivery.
    1. Traffic Spikes (e.g. Election Day) - DK 4.0 will have to be configurable to be able to server monumental increases in short-term demand.   Any architecture has to support multi-tiering, and be able to add components in the stack (load balancers, web servers, app servers, DB servers) easily and at low cost.
    1. Staff and technical capabilities.  I don't think DK4.0 staff are that numerous, but by all accounts they are good (unlike some (cf. Lieberman)).   Any framework has to be able to be manageable by a small number of people who are probably very broadly versed and capable with a range of technologies.
    1. Content.  Mostly text, but probably a growing list of other media types going forward.   Open source databases like MySQL are terrific at flinging web pages, but proprietary databases may have an advantage with other media types.
    1. Bling.   DailyKos is the one of / the premier site(s) of the progressive movement.  This is not LGF, with one-fingered mouth-breathing morons in the reader base.  The move to Ajax has been terrific, and DK4.0 should be able to quickly test and adopt technologies that best serve a smart and demanding user base.
    1. Money.  DailyKos presumably has the funding base to be a premier site, but don't have the funding to support human waves of support personnel or bad/uneconomic design decisions.

    So:  Here are the conclusions I'd draw, assuming the basic requirements are correct:

    1. .Net is OUT.    Microsoft imposes a million taxes (certification effort to cut dev costs, expensive dev environments, costs for every piece of the stack, licensing unflexibility, upgrade management (.Net 1.1 -> 2.0 -> 3.0, with compatability issues at each step, etc.) that they fail 6) above.  Microsoft support for ORM (object-relational mapping) is weak, so most big sites have to roll-their-own ORM layer - not an easy thing to do, and a killer to maintainability if done badly.  I'd turn .Net down even if it were offered free.
    1. J2EE is overkill.   The J2EE framework, as others have noted above, required a mountain of technical knowledge (violating 3)) before the first line of code even runs.   With Spring and Hibernate, the J2EE camp has emerged from the EJB hellhole that sank most development efforts in the early 2000's.  J2EE stacks can be cheap (linux on cheap hardware), and open-sourcing of Java will help with flexibility, and a well-designed stack can be scaled to support any traffic load.
    1. Perl is the right track, but fails on supportability.  Dynamic languages are the way to go - cycles are cheap enough and enterprise support is good enough that you can put up a great, sexy, scalable, flexible site without a ton of investment or load on your staff.  Still, Perl is not quite the answer -- hard to read code the offers you everything IF you're willing to dig down deep at code level.   Ditto for php - a fine language that does power some very large sites, but security concerns are rising and it's not what I'd choose today.
    1. Django / Python - closer still.  Django is a good (if immature) framework, and Google has shown some flexible stuff that supports high load.   Everything I've seen / read says D/P will be a player, but I can't say this combo is mature enough to support a big new effort.
    1. [The Winner] - Ruby on Rails.   DailyKos is comparable in size to 37Signals in regular traffic, so there should be no great concern about steady-state traffic.   Scaling RoR is straightforward (, and can be accomplished though tiered additions of cheap hardware.  Ruby and Rails can be picked up quickly - Ruby is elegant and easy to read, and Rails is "opinionated software", that makes it easy and straightforward to manage large systems and code bases.  Support for ORM (activerecord) and REST are good, and getting better.  Ajax support is very good, and "baked into the framework" - so support for "bling" is good, and easy to adopt.   It's all open source, so a server farm with Linux, database of choice, and RoR is inexpensive to procure.

    There are only two concerns with RoR, as other posters have noted.  1) Ruby is not fast.  The Ruby runtime environment is undergoing a much-needed rewrite, and the new runtime might be terrific but isn't here now.  2) The RoR framework isn't very mature either - it's only been a year from 1.0 -> the current 1.1, with 1.2 expected shortly.  Still - the code is open and the RoR community is very active, and some large applications have been deployed.

    Bottom Line: Look HARD at RoR - in a world of expensive programmers and cheap hardware, it's a great solution for meeting the requirements guesstimated above, and for creating and managing rich sites.

    •  I'm not sure we'll come to the same conclusion (1+ / 0-)
      Recommended by:

      as you did, but your analysis is exemplary, and your conclusions are very close to what ours are.

      I'll also note that you were able to come up with the most crucial core requirements we're working with, in your points 1-6, while others simply railed that we didn't have any.  ;-)

      There's of course a large number of behind-the-scenes design requirements too, in terms of things we're going to AJAXify, new features we know we want, etc.  But we're satisfied that those are doable in any framework.

  •  Better Web App Development (0+ / 0-)

    I started off as a Perl aficionado but have since migrated to using Python. Programming language wars can be as acrimonious as, well, political debate. Python seems to hit the sweetspot for me. It's object oriented and it's also a scripting language.

    I watched a nice video called Better Web App Development in which Plone comes out pretty good. Zope and Plone do have a steep learning curve for developers which is ironic because Python doesn't.

    Plone is powerful and cool, but Django and TurboGears show much promise.

    Pycon 2007 will be in Dallas, TX Feb 23-25, 2007. I went last year and it was a valuabe learning experience. I plan to attend again.

    When in danger or in doubt, run in circles scream and shout.

    by londubh on Thu Jan 04, 2007 at 09:20:41 AM PST

  •  Pycon 2007 Dallas, TX Feb 23-25 (0+ / 0-)

    Python fans and those interested in Python check it out. I attended my first Pycon last year and it was great. Also, for a conference of it's type it's inexpensive, and it was very well run.

    I don't know what the lead time for DK4 is but those interested can check out a lot of Python based technologies. Go to for more information.

    When in danger or in doubt, run in circles scream and shout.

    by londubh on Thu Jan 04, 2007 at 09:29:44 AM PST

  •  Technical Myopia (1+ / 0-)
    Recommended by:

    No technical decision has ever been made based on technical merits. For example, Hunter wrote:

    Java is right out, because I hate it and that much friggin' typing screws with my carpal tunnel issues. PHP is right out, because of security and other considerations. The only three languages seriously being considered are Perl, Ruby, and Python.

    I stopped reading right there. Thirty years of software engineering and yet every geek makes like they're the first. Good grief.

    If anyone cares...

    The first correct answer is to first define the problem (and constraints). Then and only then choose the tools that best solve that problem.

    Secondly, it sounds like you're considering a rewrite. Huh? That's very ill-advised. Never start over. If you have a working system that must remain working, evolve it. Please read Fred Brooks' "Mythical Man Month."

    Good luck. Have fun. And don't screw up.

    •  Things You Should Never Do, Part I (0+ / 0-)

      I only quote famed software guru Joel Splosky when he's right:

      Things You Should Never Do, Part I

      The money quote:

      It's a bit smarmy of me to criticize [Netscape] for waiting so long between releases. They didn't do it on purpose, now, did they?

      Well, yes. They did. They did it by making the single worst strategic mistake that any software company can make:

      They decided to rewrite the code from scratch.

      Wise devs know these things. Mostly by embracing learning opportunities called "failure".

      •  Different context. (2+ / 0-)
        Recommended by:
        chiggins, jfm

        I can't speak definatively on the issue, but my personal feeling is that a code rewrite can be an extremely healthy process, and that Joel's context is different from that of DK.
        There are huge issues with rewriting when it comes to released software, because you lose all the bug fixes, accumulated from years of user experience (unless, of course, you have an excellent document management system, and each of those fixes becomes a feature requirement. But no one does, that I know of). But that's not what DK is. DK runs in 1 place. Contextual bugs that have been fixed in earlier versions will mostly be irrelevent to the context DK4 will run in.

        Furthermore, Joel is incorrect that code doesn't "rust". While it's certainly accurate that the code will still do the same thing it always did, it's neighbours will do things better, and more uniformly (possibly taking advantage of newly developed standard libraries). So the experience of "rusting" will occur. There is a lot of value in a code base that is uniform, because altering the design of the system (which will always occur) will be much easier. Ideally, the code should have been written originally to evolve gracefully, but that is even more rare than the document management I suggested above.

        For myself, I am merciless with rewrites. If I realize I've done something entirely wrong, I will do everything I can to get time to gut it and replace it. I'm very confident in my unit tests, so usually the process isn't too painful.

        Of course, there are significant arguments against rewriting the front-end, because that does run in many environments, and likely contains bug fixes for each of them. But as that has already been done recently, I would hope it is not high on the list of priorities.

        As an aside, Joel may have been right in the short term, but if Netscape had not decided to do that rewrite, we wouldn't have Firefox, which has actually succeeded in taking the fight to MS.

        •  Oops (0+ / 0-)

          Please reread Joel's essay. And then read the followups.

        •  call it reverse-rust (1+ / 0-)
          Recommended by:

          and two points for identifying the phenomenon.  Code may not get less shiny over time, but everything around it keeps getting shinier.

          •  Dormant code rots. (0+ / 0-)

            From Wikipedia's entry on bit rot:

            The term "bit rot" is often used to refer to dormant code rot, i.e. the fact that dormant (unused or little-used) code gradually decays in correctness as a result of interface changes in active code that is called from the dormant code.

            If you keep your production systems up-to-date, which you must do for security reasons, old code may well require maintenance in order to maintain compatibility with new versions of libraries it uses (etc).  Eventually, the cost of that maintenance may become unbearable relative to the cost of starting over from scratch.

            •  perfectly put (1+ / 0-)
              Recommended by:

              I've been in a couple of failing startup situations where the money players have confided their secret plan to put the intellectual property 'on ice', while they fight with the rest of the vultures over the corpse.

              The last thing I say before I go out the door is, "if you keep that code on ice for six months, you'll be lucky if you can get it to recompile let alone link and run".

              •  to clarify (1+ / 0-)
                Recommended by:

                both phenomena are present - reverse rust and bit rot.  Reverse rust is always the easier one to sell to non technical people who have the brilliant idea to let code hibernate for some amount of time.

      •  I'm with Passive on this one (1+ / 0-)
        Recommended by:

        Joel gives good advice, but it shouldn't be taken as dogma. Starting from scratch if never the right thing to do... until it's the right thing to do.

        From what I gather, this isn't just a refactor of the current platform, it's the next platform. Did Netscape make a mistake? According to Joel they did. But I'd also point out that there came a point when starting the Gecko Engine from scratch was the right thing to do (further, releasing it to the open-source community, which at the time was the subject of much negative conjecture, turned out to be a very good idea).

        I worked on a system that had served admirably for a good long life. But it was getting more and more difficult to maintain, and the feature requests that were coming from product were impractically difficult. In one meeting, the person in charge of looking into integrating one of the features happily reported that it could be done... in around 8000 person-hours. That was the point at which it was decided to start over with a platform that was by-itself fairly featureless, but provided exactly the foundation to build the other applications as plugin-modules.

        It wasn't that one platform was necessarily better than the other, but the first had a long life and the requirements had changed enough over the years that it couldn't possibly do the job we were asking of it. By the same token, if the first incarnation of the platform had been built the way that the second one had, it would've been way over-engineered for the requirements it originally had, would've taken much longer to develop (with what originally was a small development staff), and would've taken so long to get to market that they never would've gotten through the first round of funding.

        So that's my anecdote on starting over. It kinda seems like the requirements-folks have made the best points here today (which you've ably contributed to). Once those have been more fully fleshed out, we can start having religious conflicts on languages and architecture that have some real teeth :)

  •  Smalltalk! (0+ / 0-)

    The seaside framework is coming on strong. Look at

    Just kidding kind of but really seaside has the potential to overtake everybody.

    Of course all the l33t know that lisp and smalltalk are the languages of the gods.

  •  my 2 cents (1+ / 0-)
    Recommended by:

    I'm an OS guy, not a web guy. However, I have about a decade of experience in large system design (with the systems getting larger over time, of course) and have written and maintained projects in a variety of languages.

    Languages: I'm syntax-agnostic. It doesn't matter what you choose -- Python, Perl, Ruby, Java, all fine choices. Mostly you need to ask yourself who the developers are and what their experience is, and what they're going to be comfortable with. You can write great code in all of them and you can write awful code in all of them. I think Python and Ruby are better at rapid prototyping, but once you build it up into a real system that needs maintenance they're all about the same. A few percentage points here or there don't matter. Seriously.

    Frameworks: Perl and Java probably have the largest sets of existing pieces that you can use. Ruby and Python are getting there but simply don't have the volume of stuff that comes from age. They benefit from fresh designs, but they also have bugs and frustratingly unfinished little things that come from simply being young. (e.g. it amazes me that Python made it to 2.5 before it got support for files over 2GiB ... but then again I work on filesystems for a living.) There's nothing more frustrating than having a piece of existing code that almost does what you need, but needs a change to its fundamental assumptions before it can do what you really need.

    Really you need to ask your team how much you think you're going to use existing pieces vs rolling your own. Again, it comes down to who the programmers are, what languages they know, and what their style is, as well as the problem domain. I'll let the web guys give you advice on the specifics. :-)

    Finally, two points that have nothing whatsoever to do with language choice:

    Modularity: Think very hard about this in whatever design you choose. More modularity is always better in the long run. Define module boundaries and use them religiously. It makes things much less painful when you find you need to suddenly change some major detail of the implementation.

    Scalability: Always design your project to scale to 100x - 1000x what you think you need. You might make some wrong guesses here and there, but remember that things in technology tend to grow exponentially. A growth of 3x per year seems like it would be manageable, until you realize that five years means growth of around 250x. (!!!) If you always have those kind of numbers on your mind while you design, you'll be much better off in the long run.

    In a past job I wrote a filesystem generator for CDs (700MB) which became a generator for DVDs (9GB) which is now being used for Blu-Ray and HD-DVD (30-50GB). These days it's pushing around 70x as much data as when I wrote it but, because it was designed with 100x-1000x scalability in mind, it's only required minimal maintenance so far.

    The world won't get no better if we just let it be.

    by drewthaler on Thu Jan 04, 2007 at 02:59:02 PM PST

  •  as this thread winds down (2+ / 0-)
    Recommended by:
    Hunter, jfm

    and this is certainly not MY discussion, but I personally wanted to thank everyone who stuck their necks out and put their opinions and experience on the line here.  I really enjoyed the entire discussion, even the comments I disagreed with.  Special thanks to cjohnson and ubernostrum who both gave me a lot to think about.

    Hopefully Hunter has gotten some of what he needs out of this.  There's a lot of good stuff here, and you've identified some great resources.

    •  Actually, this was very valuable. (2+ / 0-)
      Recommended by:
      EightBall, klondike

      Obviously, we have some behind-the-scenes requirements that we're not fully able to tell people right now, which handicaps the conversation. But as many people guessed, some of them are the reasons that both .NET and Java are out of the running, and all of them point to a LAMP architecture of some sort, a small, light framework, and little to no middleware, but we have no showstopper design requirements that would make one framework clearly dominant over the others.

      But this was also a good (if sneaky) social test to see who in the community might be most helpful with these efforts -- whether there were people who intrinsicly understood our unstated requirements enough to base thoughtful answers on them, for example.

      So the people who gave good, solid advice on this thread, no matter what platform they were advocating, just got major brownie points from us. We may need to hire some of them to help us implement this stuff, after all, so it's good to see the various approaches that people in the different framework communities take, and the arguments they make for those approaches.

  •  Ruby on Rails (0+ / 0-)

    I have nothing to add on this debate that hasn't been said already.

    But, if you choose Ruby on Rails, I'd be happy to volunteer my time helping to build stuff, or consulting, or whatever.  Let me know if you need help.  I LOVE working on Rails projects!


  •  that teaches me (1+ / 0-)
    Recommended by:

    Damn, Hunter, here I go and take two days off to be with the kids and without a laptop and you have to go and post the "which language sucks worse" diary.

    Everything already seems to have been said, but I'll add my piece anyway. As an old perl jock (over 10 years, a former perl porter for the VMS os), nothing would please me more than a dKos4 written in perl. This is for a couple of reasons. First, I assume you already have db access taken care of. And second, most of the tag processing stuff I'm writing is in perl, and would therefore require little modification to plonk it in if that is desired.


    Rollout is scheduled to be about a year off; I recall a couple of months ago kos said he was looking at 14 months out. So you have to try to imagine the language/framework/whatever landscape in early 2008. From what I've seen Rails seems to have the momentum, and so figure that Rails's speed and scalability issues should be much improved at that time.

    (As far as the tag stuff goes, well, I'd like to try and get some of it in dKos3 during the coming year, and to that end perl is fine.)

    Forget the myths the media's created about the White House. The truth is, these are not very bright guys, and things got out of hand. -- Deep Throat

    by The Centerfielder on Fri Jan 05, 2007 at 06:43:59 PM PST

  •  Timeframe (1+ / 0-)
    Recommended by:

    The timeframe Centerfielder notes is a key point.  CJohnson's comments on Java are well-written and valid; but I question whether a small, non-Java-veteran team can come up to speed on: Hibernate / Hibernate Annotations, Data Access Factories, Data Transfer Objects, Tiles, JSF, Spring / Spring Attributes,  JBoss / WebLogic  / WebSphere , EJB 2.0 / 3.0 / Stateless Session Beans, XDoclet, Lucene, Google Web Toolkit, Tomcat,  JUnit4,  JAXB and Toplink BUT NOT JSP Tag Libraries to launch a DailyKos production system in that amount of time.

    In my experience, patterns and frameworks are developed to cover deficiencies in a language or platform, and the volume of toolkits that CJohnson notes might be taken as a sign of Java's weakness, rather than it's strength.

    My brain is limited, and one of the joys of the work I've done in Ruby on Rails is that it has enabled me to create a multi-tier, scalable, ajaxified enterprise web application by myself with only one or two books-worth of education, rather than dozens.  I've also found RoR wonderful and straightforward at integrating code written by others.

    Rails and Django were developed well after Enterprise Java and generally improved on Java's mistakes.   I believe you're on the right track looking at the new frameworks, and will be well pleased with what you choose.  Good luck in your evaluation.

    •  Re: Timeframe (1+ / 0-)
      Recommended by:

      Well put frostieb!

      Java makes perfect sense for large teams bolting on significantly sized additions to existing enterprise systems (any ERP solution, etc...).  With standard APIs and frameworks for doing transactions, messaging, and just about anything else it's quite easy to keep several developers on task implementing new functionality in a consistent manner.

      Conversely, learning all of those frameworks, APIs, and configuration files takes a significant time investment.  Also using them day in and day out incurs its own overhead.

      Ruby, and the three P's make perfect sense for Greenfield projects.  In my humble opinion its much easier to realize the vision of a project in each of these languages because they aren't so quick to get in your way with terse syntax and API requirements.  With a little thought and a dash of design you can have an easy to maintain, extensible, and modular system written in either Ruby, Python, Perl or PHP.  Java is not the sole owner of that market!

      -Chad Johnson

Subscribe or Donate to support Daily Kos.