While I was in drug-induced la-la land after my nose surgery, I watched all 300 (or so) hours of extra material that comes with The Fellowship of the Ring Extended Edition. There's two whole DVDs worth of material for you to watch. Also, recently we watched the HBO series Project Greenlight.
What, you might ask, do The Fellowship of the Ring and Project Greenlight have in common? Very little, actually, but one difference really struck me.
The Lord of the Rings (LOTR) movies were in pre-production for two or three years before they started shooting film. Additionally, the basis for the movie has been in print for over 50 years and the principles involved with bringing the production to film were very familiar with the source material prior to working on the project.
Contrast that to the Project Greenlight movie, The Battle of Shaker Heights, which is a new script written by a very talented writer, then directed by two directors. This project had something like six weeks of pre-production. The Directors knew the script, but I don't think they had more than a superficial understanding of the meaning and thought behind the script.
The LOTR movies are winning Oscars. The Project Greenlight movie is a trainwreck.
Too often in Software development, we don't spend the time getting to know the problems we're trying to solve, the reason that the problems exist, the real needs of the customer, or even the goals of the project before diving right into design sessions. Designing software without a complete understanding of the problem seems to be the way things are done these days. Of course, by the time the project is finished (assuming it gets finished), the design is so badly out of date and wrong that it's not even relevent.
In the LOTR movies, the directors and writers made script changes on a daily basis, and communicated those changes to the cast and crew each morning. On Project Greenlight, the directors and writer made script changes on a daily basis also, but the communication of those changes was frequently verbal or non-existent, with conflicting directions being given to cast and crew. Additionally, since so little planning went into the Greenlight movie, changes to the script resulted in a dramatic shift in direction for the movie.
Sound like any Software projects you've been on? Sure does to me. It's not a bad thing to better understand the project as work progresses, but not communicating that understanding and those changes to the rest of the team constitutes a gross failure of responsibility.
The analogy of LOTR vs. Greenlight, of course, breaks down in a number of key areas, the biggest being budget. I'm sure the Greenlight folks would've loved to have spent more time in pre-production, but there just wasn't a budget available to do that.
On a general level though, I think that lack of planning remains one of the biggest problems in software development.