Doing Agile in Italy

During the latest JAOO conference, Alistair Cockburn deliberately provoked the audience stating that "we can't do agile in Europe" and this was due to EU regulations for contracts, which have to be filled with the requirements of the software to be developed by a contractor. This was a serious issue also, because in this case the law (unconsciously, I suppose, unless there is a powerful waterfall lobby in Brussels) poses a serious constraint on the development process. A contractor has to face the fact that requirement discovery in an agile fashion might drive the development team out of the officially signed project goal, some might go for a more ambiguous contract (but the customer has to agree on that, and this is not a piece of cake, especially with new customers), some others might require more paper to change the project scope (but this doesn't agile), the latest option is to take the risk and be "paperless" for a while.

Setting up an Italian development team

In Italy we have also some peculiarities related to the job market, which have some interesting effects on the way software is developed. I remember an old yet interesting article by Steve McConnell, dwelling on the notion of "problem programmers" meaning programmers with a negative impact on productivity. In other words programmers which are normally unable to finish assigned tasks, who need external help to finish the tasks, whose tasks needs extra cleanup work afterwards or which are sucking time an productivity to other members of the team. Put in this way this sounds somewhat scary, but it's not infrequent to meet such a beast. McConnell has only one solution to the problem: get rid of the problem programmer as soon as you can, which goes to a small set of options:

1. Resolve the causes for him/her being problematic
   
2. Move him/her to another team (hoping that this is a variation on option 1)
   
3. Fire him/her before the damage is too big
   

In Italy we can't fire. Job market regulations make firing an employee a very long nasty and complicated process, and 99% of the cases a non-viable option. I mean, in some places, you can't fire somebody even if you found him stealing (which puts the problem programmer in a much better light). The average Italian solution to the problem (there is always an Italian solution) is to go for temporary recruitment, hiring programmers from body rental companies, the third millennium version of mercenaries. Unfortunately this option has some drawbacks.

1. In many cases you have less control on who you are hiring. Even if I've seen pretty bad recruitment strategies in many places, being the one to choose who you are going to work with it's still a recommendable option.
   
2. You'll have less control on turnover issues: people might leave the company which hired them (and your project) for reasons which have nothing to do with you.
   
3. You have less control on motivation, you can't do much to improve it (at least on the salary side, you can do a lot, and probably more effectively on other aspects), you can do a lot to destroy it (with bureaucracy, holidays management, etc.).
   
4. Being hired by a third party is often a treat to motivation per se. One might ask "why doesn't this company want to actually hire me?" and feel they are just a replaceable part of the process, which helps them feel less guilty the day they'll decide to leave.
5. Last, but not least, the control chain sucks money. So the sub-employee is also usually sub-paid compared to the official counterparts.
   

So you generally end up with external teams with potentially lower motivation, or internal teams with people which tend to "seat" on a safe job, without being challenged for a while, and that need "something to do". I don't really want to end up arguing on the ongoing political debate which is going on in Italy these days on the topic, and please don't derive conclusions on my political orientation from this article. If you have to achieve high productivity from a software development team the best way is hire the best people, and get rid of the problematic ones. In Italy is hard to do both, so the net result is that you have to squeeze the best out of the existing team, which is sometimes …challenging.

Tags: agile, software development, Italy

IDE refactoring capabilities

I had to spend some time recatoring some classes, working with Eclipse, a couple of days ago. I selected some code inside a long method, and went for an "extract method" refactoring. Good surprise! The IDE (I was using Eclipse, but I bet that IntelliJ IDEA has the leading edge here) also checked out the entire file looking for occurrences of the code portion that was cutted and pasted all over the place, and replaced all of the occurrences with the new method (whic was the part I was planning to do afterwards...) the net result was that a 20 minutes operation became a 10 seconds operation.

On al larger scale this has some interesting consequences: if you know what to do (if you have read and understood the first 70 pages of Martin Fowler's Refactoring), refactoring a large amount of crappy code is a lot easier that you might think, so its worth doing it in many more situations. To be effective you must anyway be confident on your team OOP capabilities, and on their IDE power user skills, being the former much more important - and harder to achieve - than the latter.

Tags: OOP, Refactoring

Things they wouldn't teach you of in college - Part 2

In my previous post, I focused on some of the scope differences between software project assignment in the university, and real world projects, Pointing out that there is no maintenance phase. Well, it ca be if you fail the exam and have to evolve your personal assignment, ....but don't take it as an advice.

You are not alone
Well, time is not the greatest scope limitation you have to cross: being in a team can be far more challenging. One of the first thing you'll have to learn is sharing your code with your team, a practice which involves knowledge of SCM tools mechanics (first encounter with a CVS, Subversion or VSS is often a sort of initiation ceremony) and of the group mechanics as well. Something like "who breaks the build takes the blame" or "the last one to commit gets the conflict", but it's just the fact that the team as a whole carrying on the burden of the whole development. So you need coordination, and you need to agree with your teammates on the borders of your own activity.

Sharing the code also shows the symptoms of the "it works here!" syndrome, making it clear that you can't rely on your IDE (or on your memory) anymore, but you need more specific tools such as Ant or Maven to carry on the build process. The notion of repeatability, platform independence, environments and so on (have you ever heard the term "production environment" while at the university? I didn't).

A more subtle question is related to the notion of good code. Writing code just for yourself is close being an onanistic practice: good code is something somebody else finds useful. Which means that:

1. it works
2. easily
3. understandably

Code must work. This might sound trivial, but the point is that code should do what is expected to do. Expectations, unfortunately, could be quite non deterministic if left alone. So you gotta get a grip on what people expects your software to do, and to the average developer's expectation, which leads basically to one word (and one adjective): good documentation.

Documentation can make a lot on ease of use, but doesn't address point 2 completely. Software must be easy to use: doesn't matter if steps from 1 to 17 are thoroughly documented in MyFabulousPieceOfSoftware.howto, if you can achieve the same result in 3 steps, or one. The two things come together: as Vincent Massol points out in an old post about documentation driven development, documentation could be a good test for code and an inspiration for refactoring. If documenting takes too much it's probably because interfaces are far too complicated, or are not placed at the right level of abstraction. It's too easy to mess with interfaces and method signatures when we design both the client and the server class, for example 'cause we are stuck in the mud doing it, but we forget that we don't want classes that call our methods to be stuck in the same complexity that we are.

Understandability of the code refers to the internal mechanics of our systems. And to the beauty of the design. This is a tough lesson: a clean model is potentially useless if nobody else can understand it. It will lead to a cathedral in the desert, or - worse - to the code equivalent of a ghost town. Even if you are the best OO designer around, you can't write code that nobody else can maintain. Elegance in software can have dreadful effects, if nobody but you can appreciate it. Don't get me wrong, clean elegant software it's most of the time better than spaghetti code, but in a team, design elegance must stabilize on a level that can be reachable by a large enough number of teammates. To achieve this you can lower down your expectations, or raise team modeling skills. I tend to favor the second option as much as I can.

Tags: OOP, Software development, Coaching

Things they wouldn't teach you of in college

I had an interesting conversation with one of my colleagues, a couple of days ago, about the best way to coach the newly hired developers who just graduated in university. He was doubtful about what to teach, since some of the technologies used in the project were new for him too.

Well, it's not about the technology, at all. A young, smart developer, can learn a new technology faster than I can do. He's probably already learned a couple of fashionable technologies for the thesis, still he desperately needs to learn all of the things that university doesn't teach (sometimes, doesn't even mention) but that are vital for becoming a good developer, or a good IT consultant.

So I started thinking about some posts, hoping that they could be meaningful or useful for the case...

The test of time
A student can write code. Can also solve difficult coding puzzles. But writing good code is another stuff. The main reason for this, is that coding for an exam is a pretty simplified job compared to coding for production (which is the typical developer's stuff).
Think about the context:

• you are given a task (sometimes you can even choose it)
• you are given some constraints
• you can work alone, sometimes in a team
  

But the most important question is: "What happens after I deliver the code?" and the answer is ... nothing. Well, nothing, unless you fail the exam. But the point is that in the university there is no such thing as the maintenance phase. Your code lasts the time of an exam (it's like a butterfly, in a way...) and then it's gone. Will you ever need that code again? I bet not.

So you're missing a big part of what the code is for, which is users and colleagues. The former will ask you to change how the code works, the latter the way the code is written. More about this in the next posts.

Tags: OOP, Software development, Coaching

Gotta get going...

"Excuse me, sir. I think you got the wrong shoes on!"
"What do you mean?"
"Looks like you put the right foot into the left shoe and vice versa."
"Yeah, looks like you're right. That's why they were hurting so badly."
"Aren't you switching them?"
"No, I'm late, I gotta get going now."

One of the strongest advantages of iterative development is that the concept of iteration leads also to the concept of checkpoint. The moment you release an intermediate milestone is also a moment to think and verify if you're going in the right direction. That's also a moment where you take a breath and think instead of just doing things.

Put in another way, iteration doesn't mean repetition. Splitting the project time line in iterations means that there should be fixed moment reserved to approaching the project from another angle. SCRUM makes this "what's the biggest slowing factor of the project?" question at the center of the project lead on a daily basis. Long lasting waterfall projects normally place this question far too late.

If you are in an agile or iterative project, and there's no difference, or perceived change from iteration n to iteration n+1, this is generally a bad smell. Normally it means that this is not an agile project or, more precisely, that it's not an adaptive one. Checkpoints are probably used only to track the elapsed effort and delays, and to re-adapt estimates. A more pervasive, SCRUM like, survey might lead instead to a different way of doing certain things, or to a suggestion about how to improve them.

It's just a larger scale application of the tuning methodology: test (iterate), measure (finish the iteration), diagnose (the iteration meeting), and correct (plan for the next iteration). If the iterations borders are blurry, you have a lot less test information, if you're not having a meeting, you're probably not getting all the information you need for a good diagnose, if you never stop and plan for a change, you'll never improve. If you keep on postponing needed changes, just because you are too busy, ... you'll always be.

Tags: project management, agile, software development

My personal workplace survey

Being a consultant, I frequently have the chance to work in different organizations, and see different approaches to software development. Being a reader of Alistair Coburn's Agile Software Development, and Tom DeMarco and Timothy Lister's Peopleware, I have a strong personal feeling on the way workplaces affects software development.

Some basic principles first...
For those who are not familiar with the topic, I'll squeeze the information a fer lines. Mr. Coburn puts a great emphasis on the way communication takes place in a software development team. Communication is meant as sharing of common useful information, and it's probably one of the biggest efficiency booster in a software development team. If you start analyzing what developers do during the day, you'll note that coding is actually only a small part, communicating can be often as big as coding. Differently from coding, though, communication time is not really tracked in productivity reviews, so most of is effect is below the observability source. He also introduces the notion of information radiators, those artifacts, such as writeboards, diagrams, calendars and so on that help indirect information diffusion in the workplace.

DeMarco and Lister, focus on the effects of the workplace on individual productivity. A developer, and, more in general, a knowledge worker, does a very special job which is a mixture of knowledge and creativity. This type of job is influenced a lot by environmental factors (researches show a factor 13 between the best and wort measured individual productivity). The underlying model is that a knowledge worker is getting in the productive state of flow only after about 15 minutes after an activity is started. Every time his/her concentration is interrupted by something, 15 minutes are lost.

... and then the survey
For places I've seen, the situation is pretty close to a Waterloo. One thing that must be considered anyway is that, being a consultant I was not a full time employee (so I didn't have the same rights of my employed peers). Also, we normally help organizations in trouble; healthy organizations normally don't need us, so we probably see places a little bit below the average. I tried to define workplaces according to the main distinction of software or non software company, meaning that in some cases developing software was the core asset of the company, while in some others it was just something that needed to be done. Anyway here's the survey...

• Small financial company IT department, small room (few people in it). We were free to attach papers on the wall (but looked suspiciously), the place was basically silent. High overall efficiency.
  
• Big financial company IT department, very large scale project, huge open space. No walls, so nothing could be attached on it; quite a few meeting room not to disturb the colleagues. Messy shared information system based on notes. Structured documentation, but very few readers, improved over time. Lots of windows, but they couldn't be opened for security reasons. Smoking forbidden only in the open space. Headache basically every evening. Very low overall efficiency, boosts during somebody's holidays.
  
• Average Financial company IT department. Small room, silent, about ten people in it. Meeting room available, but seldom used (maybe too formal). People often talking less than necessary.
  
• Software company. Average size open space. Lots of people talking loud. Whiteboards available in separate rooms. Low efficiency.
• Software company. Average size open space, high background noise. Lots of whiteboards available. High overtime ratio, to maximize silence's effects.
• Part of big financial company IT department. Small rooms, some people worked for years in a servers room (no windows), high background noise. Headaches. Only junk food available in the surroundings.
  
• Software company. Small rooms. Boss smoking in the aisles, and changing requirements. Extremely high overtime, enough to define efficiency as non measurable.
• Software company. Small room, many people. Whiteboards forbidden, meeting rooms sometimes available, very few people taking notes on paper. Later, the whole department moved to a single room, formerly a shop. No whiteboards, no meeting room, increased quit ratio.
• Software company. Small room, smaller desks, very high background noise. Only paper whiteboards (non-erasable). High overtime ratio, low productivity.
  

As you might notice, the picture is somewhat desolating. I managed to see a better place, where developers had better rooms, whiteboards, space and silence. But that was not in Italy, so it can't be included in my survey. But the facts that only a few of the company manage to reach a sufficient score (and none of them is even close to excellency) really makes me wonder...

Tags: agile, software development

The dark art of cheating software estimates - Part two.

Just a quick addendum to my previous post about software estimates (and how to fool yourself cheating with them).

The most effective trick to miss a deadline (and the following one) is to put unrealistic estimations over developers activities. To achieve this result you have basically two ways.

Define the estimates at the wrong level
This normally happens when the team leader defines the activities and assigns them over the head of the developers. Experienced developers know better what should be done and which activities take time, so you should rely on their point of view, after all they're the experts in their domain. Imposing an estimation on the developer's head has also the undesired effect to cause an emotional drift in case of activities taking longer than expected. A young developer could assume that he is the problem (and maybe the real problem is just something forgotten at a higher level) and feel responsible for the delay. If you just say "You have 3 days to finish this" you might end up having the only crappy piece of software the developer was able to write in those three days.
And, of course, skills and environmental factors are different, so the same activity could take 2 days if assigned to one developer and 4 to another one. If the estimates are coming from above, it's easier to forget about that, when quickly reassigning activities.

Ok, this is just something that might happen. I am not saying that one should completely rely only on developers' numbers. A good team leader has its own estimates in place, that can be used manage risk related to optimistic developers, and so on. But this is an activity that should be performed behind the scenes (my first team leader always added 30% to my developer estimations). Comparing developers estimates with yours could also help to spot potential problems, like activities that shouldn't take that much.
By comparing different developers estimations on similar tasks, you could also spot if somebody found a smarter way to do something, and have him teach to the others (or discover that somebody is not finishing the activity and leaving some dirt under the carpet as an undesired gift for the following iteration). Put in another way you should rely on the developers on the estimations and take yourself the burden of find out how to speed up activities.

Asking estimations in the wrong way
One thing you should never do is ask "Will it be finished for Tuesday?" and - even worse - the follow-up "You told me this was going to be finished for Tuesday". Of course, in the middle, everything can happen. If you want to be hated you can ask the question one day, then interrupt with a higher priority task, then on Tuesday ask the follow-up, with a blaming pitch.
The point is that, as a leader, you shouldn't forget the effect of your role. A milestone should be a leader's problem, not a developer's problem (and shouldn't be managed in terms of blame anyway). Asking a question in this way is no different from a woman coming out from the hairdresser with a new "transgressive" haircut asking "How do I look?". It's just a compelling invitation to lie.
At the end, the numbers are exactly the ones the team leader wanted, but in this case the blame is all on the developer. Playing this trick, is unfair, doing it repeatedly is just a way to increase pressure and to threaten team alchemy.

The right way to put question is "How much time you need to do this?" and then do everything possible to ensure that all of this time will be spent on that activity. A good leader manages pressure from above and should shield team from it. If the collected numbers are too high, there is probably a problem, which needs to be investigated and possibly solved, soon.

Don't forget that providing a detailed estimation of the activity, is an activity itself. It shouldn't take a day, but be careful when you get an immediate reply; it's a sign that somebody isn't probably thinking enough. So leave your developers the time to think about what should be do and how much time will be needed. As Joel Spolsky states, this is design activity, after all.

Tags: Software development, Software estimation, Project management