Here are a few questions from a famous developer, asking about tradeoffs in XP:
Worst things first vs User choice of stories We have Worst Things First, which tells us what to worry about. On the other hand, our division of responsibilities tells us that the customer decides which stories to work on. Does WTF only apply within the particular story we have in hand now, or do we somehow exercise programmer forethought in deciding what we need to worry about?
Remember that when we do the Commitment Schedule, at that time we assign two priorities to stories: business value, and risk. Then we move stories toward the front which are high on either one, favoring risk reduction as much as possible.
Because one of the Extreme Values is Communication, it’s quite important that the programmers educate the customers on the risks that they foresee. Much of this education happens during the commitment schedule, but effective teams do it all the time.
Perhaps a couple of developers sit down with customers and talk about risky areas: “You know, some of the approaches we’re taking could slow down worse than one-for-one. When we get to 100 users, parts of the system could be 1,000 or 10,000 times slower, not just 100 times. You might want to be sure to write a story about performance and put it in the hopper.”
There are two key things to remember: first, XP programmers don't care at all if a story they've never heard of shows up in the next iteration. So if a story they suggested shows up, like "System must support 100 users with response time under one second", it's no problem at all.
Second, remember that the XP response to risk is to reduce it, not to solve the entire problem. So favor doing an experiment, or a spike solution, to find out how to deal with the performance issue. Once you know how you'll solve it, you can wait to put it into the system until the customers give it business value.
XP vs The real world At some level, XP principles don't apply in the non-programming world. We can't build a bridge and have a user story when we're 75% done which says that we'd really like it to be about 12 feet to the left over here. XP seems to be built on the malleability of software, and on practices that help maintain that malleability. Do you have any feeling for where the boundary lines in these fall. One obvious grey area is the non-software parts of software projects. e.g. we need to train N-thousand users on this system, providing inertia once we start training them.
You're on the right track. There are no magic answers in this area. Certainly you try to defer decisions that will lock you in. The rule Model First, which says to defer working on the GUI as long as possible, is an example. As soon as you start working on the GUI, you're in an area where you can't change things without confusing your users.
The C3 team put off GUIs for a long time. But now even they find that when they make changes to the GUI that wouldn’t bother a technical person, the users often get confused. We have to inform them, make changes really obvious, and so on. Even so, mostly the users come and ask.
C3 has the advantage that all the users are right there. With a delivered product, changing things that humans know about is much harder. There are lots of tricks out there: compatibility modes, configurable human interfaces, and so on. None of them are very satisfying. All I can offer is to try not to lock in, make the system as self-documenting as possible, and keep the locked parts of the system isolated from parts that may need to change.
If we get lucky, maybe the other XPer you asked will have a better answer. If he does, I'll publish it here right away!
Incrementalism vs Rewrite XP seems built on incremental modification of software, fixing things that exist by gradual transformation, even if those things are really bad. Is there a point at which you say forget it, start again from scratch. Presumably there is, since I think that's what happened with the original C3 system. How does one tell?
XP is about ways to keep a system from needing a rewrite. By refactoring mercilessly, you can keep the code clean and malleable. When you walk into an existing system that wasn't built that way, or a system that somehow got out of hand, the temptation to start over is strong.
You’re right that C3 started over. We have even reported that we wish we had trashed the few things from the old system that we did keep! But it’s a rare manager who will let this happen, since tons of money have probably already been invested in the old system.
If you do get the chance to start over, using XP is very important. Remember the famous "second system syndrome", where the developers throw in all the stuff they didn't get to use in the first system, resulting in a Fat Albert program that never gets done, and would consume all the computers in the universe if it ever did run. Now, more than ever, you need to do the simplest thing that could possibly work.
Much as I hate to say it, as a guy who hates working with horrible code, often your best chance is to clean it up as you go. It's possible to address one area at a time, write tests for it, refactor it, and move on. But the pain is sometimes just too much to bear.
Remember the rule, Doctor, it hurts when I do this. If the most mature and strongest developers on the team think that the program is doomed, it very likely is. Even if management is pushing to maintain the existing program back to health, measure your progress and feed it back to them. If it’s good enough, fine, keep going. If it’s not, help them to see that fact.
And don’t forget spike solution! Sometimes you just have to take a week off and rewrite the system, or build a prototype. I’ve used code written in my free time to sell new ideas, and I’m sure you have too.
Final Comments You've asked some hard questions in these last two. XP is a process allowing development to proceed smoothly, if it's applied throughout the project. If you're playing catchup, it'll be harder to do. And never underestimate the value of your past experience. A team of experienced developers doing XP will blow away an inexperienced team. Let your educated intuition work for you. If you think the program should be destroyed, quite possibly it should. If you think an area is risky, it almost certainly is. And if you think some part of the system will be hard to evolve, you're probably right.
Follow your intuition and apply the XP rules as you develop. No one can do any better!
Thanks for asking!