The final words about software estimation - Pecha Kucha talk

Here are the slides of my Pecha Kucha lightining talk I gave at #ALE14 in Krakow. Kind of a quick reminder (or kick in the butt, in case one forgot...).

Introducing Event Storming

Disclaimer: This is the seminal article about EventStorming. This is where everything started, so I think it's relevant to keep the original. At the same time, EventStorming has evolved a lot since I wrote this article, and the format described in this page is no longer my favorite one. In fact, there's not a single EventStorming format, the tool proved itself so versatile and powerful that I perfectioned a few different recipes. I also had to start writing a book about it, which is slowly progressing to completion. As of today, the better entry point to understanding EventStorming potential is probably my presentation 50.000 orange stickies later (which is also available in video) or reading the first chapter of the book, which is available for free. We are also working on a new version of the EventStorming.com website that should organize pointers to the interesting content about EventStorming that many practitioners published.

Agile Testing UK meetup

While in England for DDD Exchange, I caught the opportunity to participate in a meetup of the UK Agile Testing community. Gojko Adzic led a very interesting workshop about how to write acceptance tests.

Neil McLaughlin, who happened to be in my team in the hands-on simulation wrote a nice report on the evening, and then Gojko himself posted about what he showed us. I'll try add something, from the participant perspective.

The developer mindset

Given we had quite a few people in the room, around 60, it was really interesting to note all different behaviors, but there were also some striking similarities. Most of the discussion we had were focused on

• Formal correctness of the test
• Splitting tests to be testing one concern at that time
• avoiding duplication in test fixture setup

When discussing the outcome, Gojko pointed out that the main concern for acceptance tests was readability, instead. Readability is a subjective evaluation criteria, and might lead to a not-so-precise definition about how to make it right. But that's the only way to write a useful acceptance test. But the developer's mindset really had a strong influence on us, it was really hard to give up.

Constrained by the tools

Despite Skills Matter (that was hosting the evening) doing a fantastic job in providing an impressive amount of whiteboards for all the team, some of the discussions were constrained by the amount of space available on the whiteboard, so we ended up pruning some concerns, thinking they were of little value to the discussion. It turned out we were wrong.

But, as I said, we had an amount of writable space that was a lot larger than what I normally found in a typical workplace... and still this was constraining us. Buying one more whiteboard is cheap, making wrong assumptions because of writable space constraints is a lot more costly.

We're problem solvers, after all

During the exercise, Gojko was wandering around from one team to another one, playing the role of the Domain Expert. However, most of the teams seemed more interested in his Teacher role and asked some questions about how to solve the exercise, but barely none about ambiguities in the domain. We preferred to use assumptions instead.

Even if I knew the trick, it worked on me as well. But it's scary to note that even when writing acceptance tests, which are a lot more translation than design, our built-in problem solving brain leads us so easily on the road of guessing instead of asking.

About Entities, Aggregates and Data Duplication.

There's been an interesting discussion about Aggregates on the Italian DDD mailing lists. When things become complex, a simple example might just turn too simple. So I came up with this medium-sized one. Hope it won't be too long. Ok, so let's start from our first User Story

User Story #1: Placing an order
As a Customer
I Want to place an order
In order to purchase some goods

The simplest implementation of the story is essentially stateless: every time a customer wants to order something, needs to re-enter the data. In the DDD perspective, the resulting model is based on a single aggregate (we're deliberately ignoring the catalog for now) whose root is the Order class.

The stateless nature of the service makes it really easy to implement: a Customer is just a Value Object, created and eventually dropped at needs. Also we took some shortcut: we've chosen to implement Address as a String. Item is sort of natural value object, while LineItem is somewhat in the middle: we can change quantities, while the order is in open state, but we can implement this also using droppable Value Objects, easing the integrity burden for the aggregate root.

Some businesses (like buying train tickets) might just work like this, but our marketing is more inclined to manage customers in a more long-term way, so here are two more user stories.

User Story #2: Returning customer
As a Customer
I Want to retrieve my profile
In order to place more orders

Story #2 breaks our assumption about the aggregate boundaries. If we stick to the aggregate rule-of-thumb, if we want to delete an Order, we probably don't want to delete also the corresponding Customer. So we need a separate aggregate for that. What would happen if we decide to delete a Customer? Should we delete all orders? We don't have enough information to answer that, yet, we'll mark it as an outstanding question for our next meeting with the domain expert. Let's try the model with two aggregates and see how does it perform.

We now have a relationship crossing the aggregate boundary. We have promoted Customer to become the root of the newly created aggregate, so there is no potential integrity violation. Still we have to watch it closely, because this is where problems related to lazy/eager loading will arise. We also added username and password to Customer.

User Story #3: Different shipping address
As a Customer
I Want to specify a valid shipping address
In order to ship to a different destination

Multiple addresses are a call for a separate type to manage Address. We don't have so many responsibilities so far, for this class, except validation (which as Udi Dahan would say, doesn't necessarily belong to the domain layer) but the smell of duplication is probably enough to go for a separate class. We try to keep the model as simple as possible, so we treat Address like a value object.

User Story #4: Editable customer profile
As a Customer
I Want to edit my profile
In order to update it if needed

Story #4 makes explicit what we've been suspecting: Customer needs to be an entity, because it has a nontrivial lifecycle. No revolutions at Domain Model level, but this triggers a question: "What happens if I have an outstanding order and the customer changes its data before the order is dispatched?" This is the type of questions you don't want to answer as software developer. So we walk up the stairs to have a talk with the Domain Expert. We come back with two fresh user stories:

User Story #5: Specify Billing and Shipping address
As a Customer
I Want to specify independent billing and shipping addresses
In order to deliver goods to different locations

This one is relatively easy: just reinforcing our design. We'll now have two references from Order to Address, which are managed by the Customer. We just store a default Address in the Customer aggregate (we could do better) but's ok for now.

Story #6 is a little trickier, it comes from the legal department and tell us what we probably expected.

User Story #6: Track past orders
As a Legal Department
I Want to track orders
In order to in order to manage litigations

The domain expert state it clear: once an order is placed, it can't be changed in any of its parts, be it the content or the Customer. In case of litigation, it must behave exactly like printed paper. But Customer does not, its lifecycle is different from our needs, we'd need a separate class for that. We're lacking fantasy and call it CustomerData.

Everything is looking a lot different from the beginning. The two aggregates are now largely decoupled: we can change or delete an order without affecting the customer or deleting or unregistering a customer without losing tracks of its past orders. On the other hand we have explicit duplication here. Customer and CustomerData look so similar we're feeling guilty. Did we violate DRY principle? At first look the data is the same, but if we think about behavior, or class lifecycle, Customer and CustomerData are clearly two different beasts. But more often than not, when the starting point is the data model, instead of the domain model we end up thinking that's the same data, hence the same class. I bet experienced data modelers do not fall into these pitfalls as well, but I've seen these problems recurring quite often.

Once we accept that little bit of data duplication we have a system which is a lot easier to evolve and maintain, with aggregate roots as integrity enforcers within their boundaries, and agnostic about the rest.

To add some salt, don't forget aggregates are also building blocks of distributed systems: suppose we'll need to send orders to a remote system. Sending just the single aggregate and the referenced value objects is probably the cleanest way.

We can do better than this... Reloaded

In the last Italian Agile Day I delivered an open presentation called "Possiamo fare di meglio" (this post's title is just the translation), where I raised some questions about the way we develop software. I then tried to summarize some of the things that emerged in the discussion here, but there are still many things that are bouncing in my head...

One thing that really disappoints me is the low quality of many software applications I have to deal with in my everyday life. Some days I really feel like I am surrounded by crap. Just to make it clear what I am talking about: here is an excerpt of what a home banking service is asking me every single operation I do.

Choose the desired account:

My Company Account (all the accounts)

And every time I think: "It's one account, it's just one account, it's always been just one account, so why on Earth you keep asking me this question every time! Couldn't you just take me straight to the account...".

Some days I just feel like surrender, some other I see how some application are doing a fantastic job, on platforms like iPhone or on the web, and I feel like there's still some hope for us.

There is still a lot to do

Despite all of our efforts, to introduce TDD and continuous integration, velocity based estimations and the like. There's a lot to do in other fields as well. Let me say it in another way: many companies are looking towards agile methodologies as a way to improve their development processes, in order to deliver software on time and on budget.

Doesn't sound that bad isn't it?

It does. It doesn't mention the quality of the product. ...Oh, yes, we forgot, we need also tests to reduce our defect ratio. Tests are good, software quality is good, but still doesn't address the whole point. We should deliver better products. Emphasis on the process itself would satisfy the managers' ancient need for schedules and predictable outcome, allowing teams to produce crap more efficiently, but would not produce better products unless some key points are specifically addressed.

The most neglected areas at the moment - at least from my point of observation - are:

1. user's involvment,
2. understanding the domain complexity,
3. lack of an overall perspective,
4. inefficient learning during the product development lifecycle.

The items are not completely separated, in fact they're slightly overlapping, but I think these are the areas where as software developers we can (and must) improve a lot.

In the next posts I'll dig deeper in these topics.

Shared the Agile Day Presentation

The Slideshare version of my "Possiamo fare di meglio" (we can do better than this) presentation, is now available (in Italian) here.

I did my best to add comments to the many pictures, sooner or later also the video version will be available.

A crowdsourcing experiment

Like one year ago, I decided to give my talk at Italian Agile Day as an open session with roughly the same schema: 15-20 minutes of impact talk, and 30 minutes of open discussion. I like this type of sessions because it’s not a “I know the truth” approach (I don't), but rather a way to make a small portion of a possible truth emerge from anyone in the room. So, even if I had some answers in mind, I preferred to focus on the questions, leaving the answers eventually open. I’ve seen some open sessions, but it’s not the kind of things that you practice every week (unless you’re leading a talk show), so please forgive me if it anything went wrong. Personally, I think it went a lot better than the year before: the discussion flew naturally and touched many of the topics I wanted to touch.

See it in another way
Ok, now think of me like Dr.Evil, with a “masterplan”: I’ve managed to hire about 200 among the best IT professionals I know as consultants for one hour. Basically for free. That’s my crowdsourcing experiment. And I had the feeling of so many little things emerged. Maybe not so clearly, maybe not so well formed or well expressed, but enough to make me think a lot in the following days.

On the way home, and later in the weekend some thoughts were a little clearer. I guess my personal follow up is worth sharing: at least ...I owe it to the participants. :-)

We’re making products, not software
Software is what we know, what we like to write, what we learned to produce. Some of us are really good in writing software, or in making their team write good software. That’s only half of the story: from the user perspective we see a product, not code. In the discussion, many mentioned user interface or the user experience. Which is right. But it’s still trees, not the forest. If we start thinking about the overall product we might realize that we need to see the whole a lot better than we usually do. Scrum has a good shoot on this by emphasizing the role of the Product Owner, meaning that we have a single responsible person (which is a gift from the gods in many organizations) for the whole product. There’s a problem up there, because so many times the whole idea of product is flawed, and we’re just writing software. Guess what... with a high probability of being completely useless.

I’ve seen so many projects missing their goals or their potential, just because the responsible organization didn’t have a clear idea of the whole product not to see a pattern here.

Non technical integration is not so easy
Ok, so we should try to have an all-round view to our product. Including also non developers roles within our project scope. Interesting but tricky. Especially if the team boundaries do not match with the company boundaries, which might happen quite often in a cross-functional team. But software-only teams can control continuous integration to a very high degree of efficiency using TDD, Continuous Integration and SCM tools. But this is only a portion of the product. Other professional might follow a different approach, that has nothing to do with TDD (or maybe it has... only on a different time scale), and the perfection we strived to achieve becomes frustratingly fragile. I guess there’s no easy recipe here, we’ve got to fall back on mantras like “inspect and adapt”: solutions might be a lot different from team to team. After all we’re humans. But it’s funny to realize that we sometimes we want to “integrate early and often” but “only software artifacts, please”. Postponing integration with non-software components of your project is even riskier.

This is definitely a hot topic, especially if you approach it from an enterprise management perspective: enterprise 2.0 folks are talking about more flexible approaches to collaboration, and this might be at odds with the benefit of a jelled team.

Back from the Italian Agile Day 2009

I’ve seen skilled and curious colleagues. I’ve seen nice ideas emerge from different perspectives (more on this soon). I’ve met old friends and nice persons which I didn’t know and that I am now honored to know.

All this doesn’t happen everyday and everywhere.

Thanks to Marco Abis that every year make this magic happen. Thanks to everybody in the staff and to all friends from the Bologna XP Users Group, once more fantastic. Thanks to all the speakers, able to tell real stories without hype or fluff. Thanks to all the participants to my “talk show” sharing interesting ideas and thoughts. Thanks to the heroes that risked asphyxia to follow my Domain Driven Design mini-workshop.

Thanks also to Roberto Bettazzoni that reminded me what it means to be “Romagnolo”.

It has been a great day.

Thanks to everybody made it possible.

Italian Agile Day 2009

Registrations are open for the Italian Agile Day 2009, still in Bologna on November 20th.

I'll be giving a talk "Possiamo fare di meglio" (We can do better than this) and maybe a short workshop. Everybody interested in agile software development is warmly welcome. Hurry up!

Back from the London DDD-Exchange

I just came back from London where Gojko Adzic and I led a 4-day full immersion Domain Driven Design class in Skills Matter HQ. Wasn’t it enough, on friday we had the first edition of DDD-Exchange, in the Crypt with quite a few interested participants.

First of all, I’ve got to thank everybody in Skills Matter for the fantastic job. Organization was incredibly efficient with wi-fi, food and beverages, real-time twitter coverage and quick publishing of the talks on the company website (mine is here). Despite a long line of sleepless nights working on the presentation and my terrible accent, looks like everything went fine. We had quite a few interesting talks, and I also participated to my first park bench panel, which turned out being quite interesting and open ended. Also, beers with some of the folks that came to attend the DDD-Exchange were even more open ended.

So, despite being incredibly tired, at the end I felt like I did something good, many people actually liked my presentation and we had many interesting discussions during the breaks.

By the way I uploaded my presentation on Slideshare, after making some small corrections and adding some notes to prevent possible misunderstandings.

Tags: DDD, Software Development, Agile, Project Management

Sharing information within organizations

Naresh Jain tweeted code reuse being still an unsolved problem within organizations some days ago. This made me think that this is a symptom of a more general problem within organizations, who do not account for information and knowledge transfer costs.

I’ve seen a few attempt to promote reuse within organizations, and they all failed because they assumed a non realistic communication model within the organization. In some cases the approach was completely naive: just because some code had been written to address a specific problem, everybody was supposed to know about this code. There's only one organization capable to achieve such level of coordination: it's called "The Borg". They have a collective intelligent and are continuously in sync, one and another.
This type of information sharing simply does not happen in nature.

A more sophisticated approach involved use of repositories and/or knowledge management system to promote information sharing in a common tool. I've seen most of these approach fail as well. Having a common repository to share the information is generally a good idea, but expectations on those systems were really high, and probably unreasonable.

The starting point to promote code reuse within an organization is to know that there is some code that already solves your problem. From a communication perspective we have Developer A, a potential writer of an information at time T0 who knows something about what he's currently doing. Some times later, or maybe simply somewhere else, we'll have Developer B with a similar problem, at time T1. To have a successful communication:

• A should have written something meaningful about the problem he solved
• B should have searched the repository for an existing solution
• B should have found what A wrote
• B should have recognized that what A said was relevant to his problem

I think most of these steps are inherently non-trivial, basically because there is a huge amount of arbitrary decisions in every step (including whether to do the step or not). Words used by A might be different from words used by B to search. Or the information might be filed under an arbitrary label. Basically, this is a decoupled asynchronous communication. There's no feedback possible, and every arbitrary decision is an obstacle between A and B.

I personally find structured systems for storing information particularly bad: they force users to choose where to file their data, and the way people label their knowledge varies a lot form person to person and from time to time. Non-structured information repositories, such as Wikis with a Google-like search tool are definitely better, but they don't solve the core of the problem.

Sometimes we can connect our specific problem to a more general case, for example a "shortest path problem" or a "traveling salesman". This is probably how we'd call the problem after we've found the solution. The way we describe the problem before might be a lot different. Knowing how to call the problem is already a step towards the solution. This means also that who describes the problem and who describes the solution might use different terms. making our asynchronous-remote-decoupled communication at high risk of being completely ineffective.

Effective communication about partially unknown problems needs a higher amount of feedback. That's what we do when reading the menu in an ethnic restaurant: or (closer to our example) when we get into a shop to buy things we've seen but we don't know the name. A clerk might have a gotcha and tell us "Ah! you meant this!" showing us what we wanted and telling us how it's called. The most effective way to communicate about unknown things is the one with more feedback: face to face conversation.

Unsurprisingly, organizations trying to push communication through structured systems are often the ones that put a lot of efforts in preventing effective face to face communications to happen. Well ... in fact preventing good things to happen is one thing organizations are good at. And effective communication is one of these fragile things which might happen in a good environment, but there's no way to force them to happen.

The Big Agile Practices Survey

Jurgen Appelo is is currently doing an interesting all-round survey about agile practices. The more people will participate, the more accurate the results. If anyone is interested, just follow the link.

Blogged with the Flock Browser

Kanban vs Scrum

Thanks to Jurgen Appelo, I've found this incredibly clear and useful article about Scrum and Kanban on Henrik Kniberg's blog.
The type of reading that makes things perfectly clear. :-)

Blogged with the Flock Browser

The mechanics of continuous integration

I was reading this nice post about Continuous Integration, from Distilled Brilliance, by John Feminella (thanks to Marco Abis for pointing me to that, via twitter), and I have to say I wholeheartedly agree: setting up a CI dedicated machine is absolutely pointless if mechanics and disciplines are note attacked.

• Unless the teams move to a smaller increment policy, some of the advantages of CI won't be appreciated.
• To make consistent progress with small increments, TDD is your better friend.
• If the team is using a CI server, but team members don't quickly fix the build if it happens to be broken, then you're probably lost (I know this is rather obvious... but... )
• If the team is using a CI server but nobody checks the status before updating the local version of the code you're probably lost again.
• The team should be working on the trunk, and not on separate branches.

As John pointed out, it's much more a matter of discipline rather than of practice. And discipline takes time and effort. In this case it takes trust and coordination. The same kind you might find in a crowded bar at rush hour when many bartenders are communicating quickly and finding things exactly where they should be, by adhering to team standards (such as "she small dish calls for an espresso" while "the large one calls for a cappuccino").

It takes time to get there, but once you're there, the benefits are great.

It's all in the mindset

I don’t know if it has something to do with the Shu-Ha-Ri theory that Alistair Cockburn uses to describe the three level of experience in the agile practices (In the Shu phase you practice and learn, in the Ha phase you start applying variations to the lessons learnt, in the Ri phase you do the right thing by instinct, because you know that’s he right thing to do), but there’s something in common between the most successful approaches to software development I’ve seen. It doesn’t necessarily have to do with agility, many of these teams or individuals were indeed applying agile practices, but I’ve realized that they were doing something deeper, and agile practices was only part of it. On the other hands I’ve seen so many people completely screwing up software development, regardless of the process in place, blaming waterfall first, and then RUP, and Scrum later on.

Technical excellence does make a lot of difference, but still I’ve seen really skilled people doing awful things, despite their qualities. What does still make a lot of difference is in he approach to software development, is something that is a combination of a quest for continuous improvement (something that we could maybe call lean, but that’s to close to be another label) and of desire to experiment and challenge what we already know.

Learning is a non-linear process. It’s not piling up bricks of knowledge. It’s challenging what we already know against new experiences. It’s experimenting different ways to do things, and analyze the outcome. It’s something different from reading books or following advices, pretending to be right just because somebody else said that. It’s dare to be wrong. It’s trying to do something different from the last time we did the same thing, because doing the same thing twice in the same way won’t teach us anything.

Put this approach in a person, or in a team, and you probably won’t need so many books. Books can provide you inspiration, for “proven” solutions, but proving a solution, is ultimately the key of our job.

Subcontracting is a recipe for disaster

A friend of mine recently told me about a tricky situation he was trying to solve. A software company was supposed to develop an application, but they subcontracted development to another company. After a while, the project started going out of control: features were late, quality was poor, bugs were not fixed and release date slipped indefinitely. Things were so bad that lawyers started to warm-up.

As many of you, I had a sense of deja-vu, when hearing this story. The thing that struck me was that this type of project is doomed from the start! Why do people repeat the same mistakes over and over and over?

Money for nothing
Let’s dig into the scenario: customer asks Company A to produce some piece of software.
The price can be defined in many ways, but it’s normally related to a rough estimation. Anyway, Company A gets the contract. Even if they are not going to write the code, they want to be paid for the marketing, and everything that comes before the project starts, somebody will be paid to “manage” the project and some money will be allocated to cover project risk.

Budget splitting from Company A’s perpective.

Ideally, such project should provide a good ecosystem for communication between the customer and the developers, allowing developers to explore and to build consistently upon frequent customer feedback.

The supposed development ecosystem: project manager deals with high level issues, while detailed communication happens informally between the team and the customer

The subcontracting scenario
In a common subcontracting scenario, Company A decides to outsource development operations to Company B. However, since the original deal was between Customer and Company A, a new deal is necessary between Company A and Company B. Sometimes this deal is completely visible to the customer, but sometimes Company A is just pretending to develop the application while Company B is developing it behind the scenes.

Anyway, the new deal between Company A and Company B is based on a different project budget, since some costs (marketing, risk and project management) are accounted on Company A.

Now the project may be worth about 1/3 less than its original budget (the exact figure may vary according to greediness and other human factors) but Company B still needs to allocate budget for its own project management and risk coverage: nobody wants to be sued for somebody else’s mistake.

The project budget, now seen from Software Company B’s perspective.

Adding two separate management boxes around the project doesn’t provide any real value in return, the financial effect is that now the project is a low-budget project, so company B will probably start looking for the cheapest software development resources available.

One would expect that the extra money put in Project Management and Risk Coverage would turn into a perfectly managed, risk free project. Well... not exactly so.

By the way, this is often the schema which is applied in offshore software development, where the common belief is that developers are so cheap that you can compensate the extra costs related to offshoring with huge savings in product building. I am not going to dig into offshoring in this post, but it’s definitely not that simple.

Messin’up with things
Unfortunately, this type of two-levels management doesn’t only duplicate costs, it does more: it makes the whole process inefficient, and it’s actually effectively working against project success.

The subcontracting development ecosystem: formally the referring person is still belonging to Company A, but development is performed by Company B. The key communication channel is now obstructed.

• Communication flows from Customer to Company B through Company A. If Company A is the one to be paid, then all of the key discussion must pass through A, transforming the supposed Project Manager into a bottleneck.
• Indirect communication means delay, important information may rot in a mailbox, before being forwarded to the person associated with the solution.
• Indirect communication means also information loss. Some information must be understood, not only listened or read. Sometimes forwarding it’s not enough to ensure some key detail is not missing.
• Since the information is sensitive for two different contracts, all parties become more inclined to write key issues instead of using face-to-face communication. Some issues are then carefully evaluated and involve more negotiation on an official level. The entire process becomes then waterfallish: having three players, with higher risk and no effective direct communication, paper (specifications, change requests, and so on) becomes more important than face-to-face communication eventually slowing down the process.
• The customer role becomes awkward. The customer is the real customer, but for Company B, Company A is ‘the customer’. Unfortunately, the quality of information you can get from a customer who is not the user of the software it’s a lot lower. So the risk of doing the wrong thing it’s extremely higher.
• If Company A is still pretending to be the developing company, direct communication between Company B and the Customer might even be forbidden, making things even more awkward.
  

The overall effect of all these factors combined is often pure havoc. A project with a decent budget becomes staffed with inadequate people, in an ecosystem that obstructs good communication between the customer and the developers, and where management roles are duplicated adding costs and subtracting value.
Is there any hope?
In theory, there is one scenario where it could still work: it’s where Company B is so good at building software that its estimations are significantly smaller than Company A ones. But to put this scenario in the real world one should answer the question: “Why a company that is so good at building software needs marketing from another (and poorer) Software Company?”

As developers, we’re probably associating this situation with the decorator pattern, which is simply adding features (a more effective marketing) to some other (the development) without changing it. Well ...this is simply not the case with subcontracting. The act of subcontracting has a negative impact on development and should be avoided or limited. As a customer, it does make a lot of difference if the software company is actually developing the software or simply playing the role of a broker. Having seen many of these situations going straight to non-recoverable state, I really think that special care about these issues should be taken in the pre-contract negotiation phase to prevent unavoidable surprises.

Does agile add costs?

Another interesting question that popped out during my “Agile: the B-Plan” presentation at IAD2008 regarded the different cost schema of an agile process, compared with a traditional one. Somebody asked “How can I sell an agile process, given that I need some kind of presence on the customer site, and this will trigger extra costs for travel and hotel expenses?”. This sounded like “if you want an agile process, it will cost more”. Strong objections arose: “This is not truly agile”, which is true, just adding a communication channel towards the customer doesn’t make you agile. But I think the problem is a little deeper than that. I’ll try to elaborate.

Selling the process
Does it make sense to sell your development process? I think it doesn’t. As software companies, we are expected to sell results, not the way we use to achieve them. Agile is better to achieve results? Great, this is our business.

Unfortunately the last statement is not entirely true: agile does not work without customer collaboration in place. So the customer must be aware somehow that starting an agile project involves some commitment (dedicated time from key resources) from the organization. Not all the organizations are equally ready for this type of engagement, so you might find yourself desperately looking for some type of feedback, and this feedback might be arriving only close to the deadline. Ouch, we’re back to waterfall.

The myth of the on-site customer
XP approaches the customer collaboration problem introducing the on-site customer role. Well, in practice this is not likely to happen, for dozens of reasons. So a negotiation phase is necessary, to establish a communication channel with the customer, as well as a tuning phase to ensure results are good. This is an area were simply “playing by the book” will lead you nowhere: you’ll need to understand why you need this type of communication, how can you achieve that, and which are the shortcomings and risks of the agreed trade off. And both sides must be aware of that. Scrum places much of this responsibility on the Product Owner. The efficiency of the communication channel between the team and the PO is critical to project success.

Frequent feedback loops are necessary to both sides. The team needs feedback to know they’re developing the right thing. And the customer needs to see how the things are going, to reduce project risks. Risk management is probably one of the most powerful key words if you still want to sell an agile process. But I think talking of risk management sounds better to the management’s ears.

Project specific cost schemas
However, there are no dogmas, even in this area. A face-to-face communication with the customer, is a lot more efficient. But it does have a price. Sending 2 people to the customer offices for a day or a week has a cost, which is a lot depending on the project specific constraints. In Italy, hotel prices went up in the last years, while IT salaries went down. Traveling to other countries might be really expensive. In some circumstances, the cost of knowing that I am doing the right thing might be higher than the cost of “learning-by-mistakes” (but remember: there are some vicious hidden costs on this side, to consider). Unfortunately, almost always it takes the right person in the right place to understand exactly what we are missing. So the only way to know if we can safely reduce some costs by having a less efficient communication channel towards the customer, is to have an efficient communication channel towards the customer. Otherwise we won’t know what we’re actually missing.

Getting back to the title question: I think agile processes have a different cost schema (you save a lot in printed paper and pay more in post-it... ), but are generally more efficient overall even in fixed-scope, fixed-costs projects.

Agile Job Trends

The graph at indeed.com tells a lot on the reasons why there is so much debate within the agile community...

The dark side of self-organizing teams

Agile methodologies focus on the notion of self-organizing teams, as a key to software development success. This works for a lot of reasons: talent is not constrained to follow a pre-defined process, the process is adaptive and tailored on the individuals and so on.

But it looks so beautiful and simple in books, and so hard to achieve in reality. One reason is that there are two main patterns for self-organizing team creation:

• being already on one
• spontaneously creating one

Penguins are self organizing. Males stay in the middle of the antarctic continent for months with an egg on their feet keeping each other warm, and discussing about politics and soccer. At springtime female penguins arrive from their shopping and fishing just in time to start feeding the new born baby penguins. Young penguins follow the group habits without questioning and the ritual goes on and on like this year after year.

Creating a new group involves discovering common passion, is just like starting playing baseball in a small town. Maybe 2 kids are just playing with a ball and a bat. Another one joins the two, same day same hour, then later on, they discover they’re enough to form a real team. There can be some key moments in the growing phase, but many times, the original vision is shared by the majority of the team.

What is really hard, is trying to transform a team who’s always been directed from above, into a self-organizing team. There are so many things that can get in the way, that is often better to start with a newly formed group instead of turning an existing team into a self organizing one.

...not necessarily good
But the notion of self organizing team is not necessarily good in itself. Self organizing teams can be pretty nasty, (Mafia, Al-Quaeda, Ku-Klux-Klan are all examples of self organizing teams, as this interesting post says). But the question is not only tied to team ethics, it’s also related to what the team can do to achieve its goals, which involves powers and responsibilities.

If you follow soccer a bit you’ll probably know the story of Antonio Cassano. One of the most talented Italian players, he played in AS Roma for a while. After some initial success, its role (not his skills) started being questioned. AS Roma sold Cassano to Real Madrid and right after that started an impressive record of 11 victories in a row. After the embarrassing parenthesis in Spain, now Cassano is doing fine again with Sampdoria. This is a clear example of how a team can improve as a team by getting rid of some talent.

Talent is hard to manage (and this could be the subject of a dedicated post), but the key point here is that the team might decide to drop team members that don’t share the same values with the rest of the team. You need to grant this right to the team, otherwise shared vision and behavior will never emerge, but you’ll also need to be prepared to the consequences, as a manager or as a team member. And consequences can be pretty nasty: like telling a colleague that he/she is not welcome (maybe with some “unless...”).
One key point is to be sure that the most influential team members have a positive influence on the team. If you can’t guarantee that, you’re probably doomed, right from the start.

So ...where’s the fun part?

I have the feeling of having killed somebody’s dream in my last post. And also of walking on the edge of being misinterpreted. I think I’ve got to blog about it again from a different angle.

I like developing software. I think it’s fun. Even if I like solving complicated puzzles, I think the best part of software development comes from working within a team. I have some friends working for the Toro Rosso F1 Team (being from the same town makes it easy), and I know how everybody felt after winning in Monza. This is the kind of feeling a team should work for. Not every team could be so lucky, but ...you got the point.

Are we just executors?
So, are we just implementing a specification? Nope. If that was the job, pick some MDA tool and have it do the job for you. I just asserted that coding is a lot less creative than many developers think. Finding a proven solution (hopefully a good one) is often a preferable alternative to an original one. And “original solutions” often degenerate in in-house frameworks lock-in.

But being mere executors just doesn’t work either. Following specifications is a waterfall, but it’s also a waste of talent. And such a scenario will eventually lead to low motivation, and talent loss within your company.

Creativity might slip in when the puzzle is more complex than usual, or when tempting something new. In this cases, you have to fight with the creativity tools: get away from the workstation, have a (time-boxed) brainstorming session, involving somebody else, be provocative, avoid censorship. The key is to recognize which type of problem you’re facing and to use the right tool for the job.

Learn and challenge
Another area of software development where you definitely need some creativity is interaction with users and stakeholders. Understanding the problem domain is an interesting activity, which will lead you in unexplored areas of knowledge. Understanding the way users are interacting with your application, and the reasons for that, might lead you to propose some features that might come in handy, or make the difference!

Software development teams are often a concentration of pretty smart minds. Pretending that the only focus is code, is wasting talent and, often, a mess.