It's all in the mindset

I don’t know if it has something to do with the Shu-Ha-Ri theory that Alistair Cockburn uses to describe the three level of experience in the agile practices (In the Shu phase you practice and learn, in the Ha phase you start applying variations to the lessons learnt, in the Ri phase you do the right thing by instinct, because you know that’s he right thing to do), but there’s something in common between the most successful approaches to software development I’ve seen. It doesn’t necessarily have to do with agility, many of these teams or individuals were indeed applying agile practices, but I’ve realized that they were doing something deeper, and agile practices was only part of it. On the other hands I’ve seen so many people completely screwing up software development, regardless of the process in place, blaming waterfall first, and then RUP, and Scrum later on.

Technical excellence does make a lot of difference, but still I’ve seen really skilled people doing awful things, despite their qualities. What does still make a lot of difference is in he approach to software development, is something that is a combination of a quest for continuous improvement (something that we could maybe call lean, but that’s to close to be another label) and of desire to experiment and challenge what we already know.

Learning is a non-linear process. It’s not piling up bricks of knowledge. It’s challenging what we already know against new experiences. It’s experimenting different ways to do things, and analyze the outcome. It’s something different from reading books or following advices, pretending to be right just because somebody else said that. It’s dare to be wrong. It’s trying to do something different from the last time we did the same thing, because doing the same thing twice in the same way won’t teach us anything.

Put this approach in a person, or in a team, and you probably won’t need so many books. Books can provide you inspiration, for “proven” solutions, but proving a solution, is ultimately the key of our job.

Does agile add costs?

Another interesting question that popped out during my “Agile: the B-Plan” presentation at IAD2008 regarded the different cost schema of an agile process, compared with a traditional one. Somebody asked “How can I sell an agile process, given that I need some kind of presence on the customer site, and this will trigger extra costs for travel and hotel expenses?”. This sounded like “if you want an agile process, it will cost more”. Strong objections arose: “This is not truly agile”, which is true, just adding a communication channel towards the customer doesn’t make you agile. But I think the problem is a little deeper than that. I’ll try to elaborate.

Selling the process
Does it make sense to sell your development process? I think it doesn’t. As software companies, we are expected to sell results, not the way we use to achieve them. Agile is better to achieve results? Great, this is our business.

Unfortunately the last statement is not entirely true: agile does not work without customer collaboration in place. So the customer must be aware somehow that starting an agile project involves some commitment (dedicated time from key resources) from the organization. Not all the organizations are equally ready for this type of engagement, so you might find yourself desperately looking for some type of feedback, and this feedback might be arriving only close to the deadline. Ouch, we’re back to waterfall.

The myth of the on-site customer
XP approaches the customer collaboration problem introducing the on-site customer role. Well, in practice this is not likely to happen, for dozens of reasons. So a negotiation phase is necessary, to establish a communication channel with the customer, as well as a tuning phase to ensure results are good. This is an area were simply “playing by the book” will lead you nowhere: you’ll need to understand why you need this type of communication, how can you achieve that, and which are the shortcomings and risks of the agreed trade off. And both sides must be aware of that. Scrum places much of this responsibility on the Product Owner. The efficiency of the communication channel between the team and the PO is critical to project success.

Frequent feedback loops are necessary to both sides. The team needs feedback to know they’re developing the right thing. And the customer needs to see how the things are going, to reduce project risks. Risk management is probably one of the most powerful key words if you still want to sell an agile process. But I think talking of risk management sounds better to the management’s ears.

Project specific cost schemas
However, there are no dogmas, even in this area. A face-to-face communication with the customer, is a lot more efficient. But it does have a price. Sending 2 people to the customer offices for a day or a week has a cost, which is a lot depending on the project specific constraints. In Italy, hotel prices went up in the last years, while IT salaries went down. Traveling to other countries might be really expensive. In some circumstances, the cost of knowing that I am doing the right thing might be higher than the cost of “learning-by-mistakes” (but remember: there are some vicious hidden costs on this side, to consider). Unfortunately, almost always it takes the right person in the right place to understand exactly what we are missing. So the only way to know if we can safely reduce some costs by having a less efficient communication channel towards the customer, is to have an efficient communication channel towards the customer. Otherwise we won’t know what we’re actually missing.

Getting back to the title question: I think agile processes have a different cost schema (you save a lot in printed paper and pay more in post-it... ), but are generally more efficient overall even in fixed-scope, fixed-costs projects.

The alibi of creativity

One of the most intriguing parts of the discussion, that was part of my speech in IAD2008, was a question (interestingly asked by one of the few non-developers) regarding possible negative effects of time-boxing (especially in the form of “pomodoro”) on a creative activity such as developing software.

I quickly dropped my opinion, but I was more interested in making others opinion on the topic emerge. Then I had some tome to think better about it: here are my thoughts.

Coding is not a creative activity
I mean ...there’s creativity involved, but most of the time we are solving problems, in ways which are probably been explored by many others before us. So, basically, coding is a problem solving activity. Creativity slips in when the problem to solve is new, non conventional or pretty hard, or does not strictly involve coding (in this respect, the process might be far more creative than coding). Sometimes we have the luck to deal with applications which are pioneering, but many more times, this is not the case.

This doesn’t mean that we have to be mere follower’s of somebody’s else ideas. But “being creative” or pretending to be, too often has the undesirable side effect of “reinventing the wheel” which is definitely not what we want. Put in another way, “being creative” is often just an excuse for being “too lazy to study”.

Time-boxing does not constrain creativity
Ok, then there are times where the strict time constraints imposed by time-boxed activity, a pairing session, a pomodoro, a day or a sprint, is not enough to allow for the “inspired” solution. As Simone Genini pointed out, “as a manager, you don’t want to wait for developer’s inspiration”. So what you need is a repeatable attitude for this problem solving activity.

Back home I realized that this happens in creative fields too: comics are written on a monthly basis (even if not every author can fit in the schedule), ad campaigns are creative, but organized as time-boxed projects, and so on.

Even in software development, the time-box constraint is less stringent than it might be: if one of your best developers came out with what he consider a sub-optimal solution, something he doesn’t really like. You can be sure that he’ll keep thinking about it, and probably will attack the problem again, once a better solution appeared to him in his dreams or under the shower. Just allowing more coding time to be explicitly allocated to that task won’t probably get the solution any closer.

Pair programming’s side effects

While attending JAOO 2007 I had the chance to talk for a while with Pramod Sadalage, which gave me some interesting hints about pair programming. I was basically concerned about the amount of time spent by development teams with Instant Messaging tools, such as Skype, ICQ, GoogleTalk. There's basically nothing wrong with those tools; normally they're something that acts as a development process accelerator, especially in offshore or de-localized development situations. Unfortunately, this is not always true: instant messaging can be easily abused, and cause frequent interruptions in people's work. The number of interruptions is related to the team size, but has something to do with the way tasks are assigned to people, with the underlying architecture, which increase the number of people you are depending upon to complete your task, assuming that all the incoming messages are related to the job … which is basically not true. Given that every interruption destroy concentration in the receiver and that it takes about 15 minutes to become productive away, preserving the so-called state of flow is a crucial task.

Pramod's suggestion was to adopt pair programming, on a dedicate workstation. When you're pairing you are less inclined to chat with others, because your peer is just sitting there. The dedicate workstation also helps, because you can configure it not to have IM software installed on it. Pairing sessions anyway aren't for the whole day. So after a couple of hours – and with a bunch of well done lines of code – developers can check out the e-mail, and so on. But productive time is preserved.

Tags: JAOO2007, Agile, XP

So …what is lock in anyway?

When designing an application architecture, one of the things to consider is how to deal with lock-in on third party components. The problem is generally multi-faced (a component might be freely available as open source, or being tied to a license fee) and is generally managed on a risk analysis basis. It can also be tied to large scale aspects such has hardware choices, Operative systems etc. I'll focus here on choosing 3^rd party libraries and components for a generic enterprise application (thinking of JEE, but not necessarily only of it). On the practical side this is managed by taking care that the component substitution cost is kept as low as possible. Typical approaches were to intercept and wrap dependencies, with a proxy or an adapter class: no direct access to the 3^rd party component is allowed, access to the provided service is available only via the adapter class. In a J2EE context looked like everybody was concerned about Log4J, so the logger wrapper became the architect-level counterpart of the String-manipulation class that every developer felt obliged to write at a certain point of his career.

Technically speaking, you want to reduce coupling between the caller and the provider of a given service, so you extract a more or less generic interface from the service, and provide a (usually brainless) adapter to the 3^rd party component. This way you achieve compile time decoupling from the component, at the price of a small increase in complexity (a bit in the code, a little more in documentation, guidelines and control). This price is generally proportional to the complexity of the used portion of the component API (which is one of the reasons why everybody wrapped Log4J…).

The point is that this effort is not really worth a lot of times. For basically two reasons:

1. You are trading a certain cost now for a possible one in the future (ok, you've done risk analysis on the subject, but… you've got the point.)
   
2. You are probably overestimating substitution cost. Ok, I am telling you but I am thinking me, 'cause I know I did… Common refactoring techniques, enhanced IDE support, or even a trivial search and replace capability make this type of code-level change really faster. Developers are generally worried about the number of occurrences of a given required change more than the diversity of the single ones.
   

What's left out, is more subtle effects of lock in… which results in way of organizing the surrounding components, or in the way to structure the application. In those cases, you can still enhance the capability of your adapter, providing an higher level of service abstraction, but as before you end up paying more at the early stages for a possible risk that could be better mitigated some other way.

Tags: Agile, XP, Software Architecture, OOD

Communication in the Agile Team

Agile methodologies, particularly XP, strive to achieve better process efficiency by reducing the amount of the required documentation to a “no more than sufficient” level (among other things, of course). Having a relatively small team of people who get along quite well, and putting the team members spatially close together should provide the desired background for an informal but highly efficient communication, to spread between team members.

What “spontaneously” happens is something slightly different: communication happens, but developers aren’t saying the right things. They’re asking each other

• how do you configure this?
• how do you setup the environment?
• This page looks odd, how did you solve this in yours?

Which are good questions, but could be better answered by an agile HowTo, or a script. Or are problem determination questions, which are good for sharing knowledge but still with a backward perspective. What is less likely to happen is an efficient communication between mates, about what really matters (to me, at least) which is putting the pieces together to produce software that works. Developers tend to focus on their task only, asking somebody else only if they are in need. Otherwise development proceeds silently, or with headphones, wasting all the competitive advantage of space proximity.

Constructive communication
Adding cooperative documentation didn’t prove itself to be an efficient glue. What did prove so was re-assigning tasks in a way that forced team mates to cooperate: instead of trying to have developers work simultaneously on two separate parallel tasks (which is what you normally do to avoid deadlocks), I asked them to work together on the same, short task. The result was a lot less messy than expected, in fact we had working code faster than expectations; there was a deadline pressure, to be honest, but didn’t result in overtime. Instead it produced a spontaneous design discussion on the classes to be shared, something I haven’t seen for a while.

Some might say that I just discovered the XP hot water: pair programming, and continuous integration. Which is partially true, but we still aren’t doing real pair programming (for a lot of reasons), while we normally have continuous integration practices in place. Still we are talking here about integration at communication level. Asking is only one form of communication (and if DeMarco’s theory about the state of flow is true, asking is a sort of stealing) and a pretty primitive one. Forcing tasks overlapping might look like a suicide choice, but generates instead positive effects on development speed, and on the team mechanics.

Tags: Project Management, Agile, XP

Sticky Standards (Coding, the IKEA way - part 2)

Right after publishing this post, I’ve got the feeling of something missing, that still could be extracted from the IKEA metaphor. Mounting drawers handles was indeed only the second part of the job, after building the whole closet (well, in fact you could have chosen another way too: mounting handles, or simply drilling before building the whole stuff).

So, how do you do that? You just unpack the pieces, and start following the instructions. Don’t believe those folks blaming on the quality of IKEA papers, they’re pretty good: only drawings, no text (to get rid of i18n issues) but if you follow the plan you can’t miss. Drawings provide details about which side goes up, which part to start from reducing degrees of freedom that you might have in doing even the simplest stuff without having a plan. This way they can avoid maintaining a huge F.A.Q. section answering things like “how to attach legs to a closet after you filled it up” and so on.

What’s the difference from coding? It’s in the fact that many developers tend to favour copying some colleague’s code instead of following a detailed HowTo. A good reason for that is that you can ask clarifications to the author of the code if needed (which is efficient on a small scale, but it’s not on a large one). A not so good reason it’s just in developer’s mind: following a plan might be easy, and leads to predictable results. Put in another way it’s boring. Fun is solving a problem, and if there’s no problem there’s no fun.

Some might already have spotted the underlying danger in this practice, but to achieve a little more thrill, I’ll start telling a completely different story.

The unpredictable standard
The keyboard in front of you has an interesting story: the so called QWERTY standard, was originally developed for mechanical type writers, replacing the dominant hunt-and-peek system (requiring two actions for every key) which was dominant at that time. In absence of standards there was plenty of freedom in choosing how to place the chars on the keyboard, and the main reason which led to the odd QWERTY disposition was to protect the underlying mechanics. Put in another way, it was designed to slow down typing to avoid collisions between mechanical parts. Well it was still a lot faster than the previous standard, but the decision was to sacrifice part of the potential for short term needs. If you are interested in a full review of the QWERTY story, please read this article.

History made the rest. QWERTY standard survived far beyond expectations, and once mechanics were not a bottleneck anymore the standard itself become the next. But every attempt to move a step forward failed, due to the critical mass achieved by QWERTY users.

Drawing conclusions
Many things, and code is one of them, persist more than they were intended. In a single project lifecycle, the thing you don’t want is an anti-pattern calling for refactoring. The most efficient way to shield you from this is to provide the team with a bullet-proof prototype, that developers can sack in the usual way. This won’t ensure you that the team got it right, but will reduce a lot the diffusion of “alternative solutions” to already solved problems.


Tags: Project Management, Agile, XP

Estimating Change

Everybody knows that the average developer tends to (largely) underestimate development time, or to run out of the estimated time. A less investigated phenomenon is that, despite all the evolution in IDEs, developers instinctively also tend to overestimate refactoring time. When asked to change some features of the existing code – except their own – estimations always jump up to the worst-possible-scenario-on-earth.

There are several reasons for this phenomenon, the most obvious one is a lack of trust, meaning that you don’t trust somebody’s else’s code. Having automated tests in place might of course help, but, even if it increases operation’s safety, doesn’t have a direct effect on the estimation numbers: a 20 minutes stuff still is perceived as a “half a day” stuff, while repeating a stupid 30 seconds workaround hundreds of times it’s not perceived as a waste.

Still I don’t think this has a lot to do with safety and trust, but a lot more with the taste developers had when they were learning java. I remember the “good old days” when even renaming a single class was a mess (remember JBuilder 1.0?) due to the classname=filename.java constraint, plus naming convention issues and OS issues as well. Well, let me say one thing: THOSE DAYS ARE GONE. It just takes a minute to use a refactoring command to rename a class or move it to another package, and this is true on all the most used IDEs, even the free of charge ones.

Not all refactoring tasks are that simple: changing an attribute from one type to another might require some extra work were IDEs stop being that helpful. But also in those cases, if you have a bunch of red markers for compile errors in your code and solve them one by one, you are not going to loose that much time. You are going to lose a lot if your starting design was poor (ok, we have a code smell here), but if you have an average level of encapsulation in place we are still talking ‘bout minutes, not hours.

The domino effect
What’s worst in this situation, is that this poses the basement for making development speed get stuck in the hell of spaghetti code. The domino effect goes like this: 1) somebody writes some bad piece of code, maybe marking it as a todo; 2) somebody developing another use case, looks for guidance in that code, and copies the anti-pattern; 3) now the anti-pattern is spread like a virus, and shows heavier effects; 4) still correcting it is delayed, cause there is not so much time to correct it (for some obvious reasons, like time was lost before).

Hmmm, sounds like an XP evangelist paper, and honestly it is. Good news are that the domino effect really works in the other direction also, but you have to try before you believe it.

• Good designed code makes refactoring tasks faster (which is a little different from the common belief that the first XP iteration should produce crap, just to be refactored).


• A good starting point helps a lot, cause younger developers are anyway going to copy (if not reuse) a solution, so it’s better to get inspiration from a good one.


• test helps building the safety net for refactorings, speeding up change time.

Only after you apply changes quite a few times you’ll have estimations get smaller. If you’re going for refactoring with a “shy” team, my suggestion is go for it regardless of the opposition, and possibly make a bet to make it a remarkable lesson. Or use the heaven sent extra time to have a party.

Tags: Project Management, Agile, XP