The mechanics of continuous integration

I was reading this nice post about Continuous Integration, from Distilled Brilliance, by John Feminella (thanks to Marco Abis for pointing me to that, via twitter), and I have to say I wholeheartedly agree: setting up a CI dedicated machine is absolutely pointless if mechanics and disciplines are note attacked.

• Unless the teams move to a smaller increment policy, some of the advantages of CI won't be appreciated.
• To make consistent progress with small increments, TDD is your better friend.
• If the team is using a CI server, but team members don't quickly fix the build if it happens to be broken, then you're probably lost (I know this is rather obvious... but... )
• If the team is using a CI server but nobody checks the status before updating the local version of the code you're probably lost again.
• The team should be working on the trunk, and not on separate branches.

As John pointed out, it's much more a matter of discipline rather than of practice. And discipline takes time and effort. In this case it takes trust and coordination. The same kind you might find in a crowded bar at rush hour when many bartenders are communicating quickly and finding things exactly where they should be, by adhering to team standards (such as "she small dish calls for an espresso" while "the large one calls for a cappuccino").

It takes time to get there, but once you're there, the benefits are great.

Toxic Code

During these days, the word toxic is often associated to investments hidden in some bank caveau, their risk have been underestimated and the result is a backlash with devastating effect.

I think many institutions have the same toxicity problem in their software, but still fail to admit it: key portions of their software act like small timebombs and are quietly undermining their businesses.

There are infinite examples of what I consider to be toxic code, here is a small checklist some might be familiar with:

• Useless code: it’s there but nobody uses it
• Untested code: it’s there, probably works but nobody wants to touch it
• Valueless code: Software which is not delivering any business value
• Annoying code: every change to the system takes more time than it’s reasonably necessary.
• Annoying software: the code is “working”, but in a way that makes things harder instead of easier. Users waste time every time they perform an operation.
• Leaking software: key portions of the application are not tested enough, some exceptions are unhanded resulting, in random untraced errors.
• Countdown software: software with a time-related flaw that will have a havoc effect at a given point in time.

• ...(add yours here)

Well... it’s basically the same idea of technical debt (and Ward Cunningham posted an excellent short video on that - thanks to Piergiuliano Bossi for pointing it) which should be familiar to any agile developer, and was created as an analogy to the financial world. But my worries are more related to the business side, of software rather than to its production process. Ideally, we should have 100% trust on our IT systems (don't laugh please... I said "ideally"), because every time some of them it’s not working there is a waste somewhere (be it time, money, or a customer abandoning the company).

Sometimes companies are aware of the real quality of their running software, and decisions to fix or improve specific portions of the running application are just the results of conscious strategic budget allocations. More often, this is not the case: like toxic assets, software with risky behavior is treated like normal production code, underestimating the associated risks and drawbacks.

Anyway, even when companies are taking into account the costs of a less-than-optimal solution, they’re generally considering only the direct effects of this choice. Something like “this feature might be more user friendly but it will cost us $xxx to rewrite it and the possible revenue for that will be only $yyy, so it’s not worth developing” which is reasonable, but doesn’t consider the cost of not developing it which is put on the user who is having a less-than-optimal user experience (well ...sometimes a miserable one) wasting time and/or money any time he uses the system. It's like polluting: if you don't count it, it's cheaper to do dirty things.

I am still intimately convinced that computers should make life simpler and not more complicated. And I am also convinced that they can.
For some reasons they don’t.

SOA desperately needs DDD

Talking with some colleagues, involved in SOA projects, I often have the feeling of a big hole in the perception of what SOA is really meant for, and what is needed for SOA projects to be successful.

I do believe that SOA is a great idea, it just sounds like large scale common sense. But I also do believe that “large scale common sense” is an oxymoron. Many dysfunctions in SOA projects are closely related to what normally happens in large scale projects, and in many cases those dysfunctions are transforming the whole stuff a whole big loss of money.

A common mistake, is to focus primarily on the architectural aspects of a SOA implementation. Which sounds like a rather obvious thing to do, considering what the A of SOA stands for. Unfortunately, this approach often turns out being narrowed to large scale OOP, only with a different terminology. Services (and their underlying model) are not reusable the same way objects are (and you remember a bit of the times where “reuse” was the buzzword and OO was hype, you probably know that also object reuse turned out being a lot different from the premises).

Behind its interface, a service is implemented according to a specific model. SOA makes no assumptions about the model, and allows for different implementation strategies: so far so good. Still one of the primary drivers for SOA is the need to rationalize the enterprise landscape, by removing unnecessary duplications and enforcing reuse of enterprise level services (we’re still in the “large scale common sense” here). Too often, the attempt to rationalize the landscape goes too far, trying to involve also the model in the process. This is often linked to the way SOA is conceived and implemented within your organization, a policy like “every significant entity will have to be wrapped by a service” will possibly lead to an enterprise scale CRUD moloch.

Enter Strategic Domain Driven Design
The point is that a service might be a significant reusable entity throughout the enterprise, while a model might not. A model should be the optimal solution to one specific problem, valid within a specific context. The context must have boundaries to allow for an optimal solution. An enterprise-level model will rapidly become blurred and some key abstraction will start to serve too many different purposes.

What is a Customer within an enterprise? Can you really find a one-size-fits-all model to represent a customer within the many context that must be supported by your enterprise scale SOA? My take on that is ...nope. Well, you’ll always be able to find some trivial entity that can be the same throughout the whole enterprise, but the odds will be against you if you start looking to some non-trivial entities existing in multiple domains.

In strategic Domain Driven Design there are some key principles to address this situation.

• “a model serves a specific use”: a model is a tool, and to be perfectly shaped for a specific use, the use must be well defined. Like a tool, a model can’t be really effective if the same object is used for many different purposes. Also, an effective model must be kept small enough to be coherent and manageable by a single skilled development team.
• “a model lives within a well defined context”: context and their boundaries are really important to define a coherent model. There are entities that can be used in different context, but sharing the same entity is not necessarily the best way to address the problem. Often the drawbacks are heavies than the advantages.
• “there will always be multiple models”: despite this sounding rather obvious on a large scale SOA, many times a lot of effort is dedicated to try to fight this situation, with minimal chances to win.

A typical problem with SOA is that implicit communication costs are rarely accounted. Sharing the vision of a model within a development team already has a cost which can be kept small enough if the team size is reasonable. Having the same vision within a 5 persons team is feasible. Sharing the vision among 40 people (or more) from different consulting or body rental companies (which is a common scenario in large-scale SOA development) is pure utopia.

Greg Young interview about DDD State Transition

In this interview on InfoQ, Greg gives a very interesting perspective on how (and also why) to use Domain Driven Design to build a complex system.

Playing with Domain Specific Languages in Groovy and Grails

I’m just coming back from the Italian Java Day, where I talked about Domain Specific Languages on the Groovy - Grails platform. I’ve uploaded the reviewed version of the presentation on slideshare, with some extra notes, if you are interested.

The whole idea is related to the possibility of adopting multiple DSL simultaneously. I took some interesting hints from Guillaume Laforge, Graeme Rocher and Peter Backlund, for the DSL part, as well as some from Ninja Coding Monkey, when it came to integrate my DSL specific value Objects with Grails.

Groovy meta-programming offers tools for designing DSLs that were not possible in a static language such as Java. Operators overloading and coercition make it smooth to deal with numeric types including non-primitive types such as BigDecimal or BigInteger. Still, the most powerful tool is the possibility to re-define the reference operator ‘.’
allowing for more compact and explicit notation such as

100.EUR == 50.EUR * 2

for Money types.

However, this involves adding new behavior on the Number metaClass, which is not entirely correct in terms of object oriented responsibility assignment: it’s basically a new Money constructor that we’re attaching on the Number class instead. It works, it’s cool, but it’s definitely not the cleanest code on Earth.
Different DSLs have many points in common and their readability relies heavily on the same features, so a centralized coordination role turns out to be necessary. At the moment I am working on a prioritized DSL engine that allows pluggable fine-grained DSL to be activated separately in the same application, for example I can write something like:

1000000.EUR == 1000.k.EUR
1000.k.EUR == 1.M.EUR

but that’s only a small a part of the fun. More details in the next posts

It's all in the mindset

I don’t know if it has something to do with the Shu-Ha-Ri theory that Alistair Cockburn uses to describe the three level of experience in the agile practices (In the Shu phase you practice and learn, in the Ha phase you start applying variations to the lessons learnt, in the Ri phase you do the right thing by instinct, because you know that’s he right thing to do), but there’s something in common between the most successful approaches to software development I’ve seen. It doesn’t necessarily have to do with agility, many of these teams or individuals were indeed applying agile practices, but I’ve realized that they were doing something deeper, and agile practices was only part of it. On the other hands I’ve seen so many people completely screwing up software development, regardless of the process in place, blaming waterfall first, and then RUP, and Scrum later on.

Technical excellence does make a lot of difference, but still I’ve seen really skilled people doing awful things, despite their qualities. What does still make a lot of difference is in he approach to software development, is something that is a combination of a quest for continuous improvement (something that we could maybe call lean, but that’s to close to be another label) and of desire to experiment and challenge what we already know.

Learning is a non-linear process. It’s not piling up bricks of knowledge. It’s challenging what we already know against new experiences. It’s experimenting different ways to do things, and analyze the outcome. It’s something different from reading books or following advices, pretending to be right just because somebody else said that. It’s dare to be wrong. It’s trying to do something different from the last time we did the same thing, because doing the same thing twice in the same way won’t teach us anything.

Put this approach in a person, or in a team, and you probably won’t need so many books. Books can provide you inspiration, for “proven” solutions, but proving a solution, is ultimately the key of our job.

Subcontracting is a recipe for disaster

A friend of mine recently told me about a tricky situation he was trying to solve. A software company was supposed to develop an application, but they subcontracted development to another company. After a while, the project started going out of control: features were late, quality was poor, bugs were not fixed and release date slipped indefinitely. Things were so bad that lawyers started to warm-up.

As many of you, I had a sense of deja-vu, when hearing this story. The thing that struck me was that this type of project is doomed from the start! Why do people repeat the same mistakes over and over and over?

Money for nothing
Let’s dig into the scenario: customer asks Company A to produce some piece of software.
The price can be defined in many ways, but it’s normally related to a rough estimation. Anyway, Company A gets the contract. Even if they are not going to write the code, they want to be paid for the marketing, and everything that comes before the project starts, somebody will be paid to “manage” the project and some money will be allocated to cover project risk.

Budget splitting from Company A’s perpective.

Ideally, such project should provide a good ecosystem for communication between the customer and the developers, allowing developers to explore and to build consistently upon frequent customer feedback.

The supposed development ecosystem: project manager deals with high level issues, while detailed communication happens informally between the team and the customer

The subcontracting scenario
In a common subcontracting scenario, Company A decides to outsource development operations to Company B. However, since the original deal was between Customer and Company A, a new deal is necessary between Company A and Company B. Sometimes this deal is completely visible to the customer, but sometimes Company A is just pretending to develop the application while Company B is developing it behind the scenes.

Anyway, the new deal between Company A and Company B is based on a different project budget, since some costs (marketing, risk and project management) are accounted on Company A.

Now the project may be worth about 1/3 less than its original budget (the exact figure may vary according to greediness and other human factors) but Company B still needs to allocate budget for its own project management and risk coverage: nobody wants to be sued for somebody else’s mistake.

The project budget, now seen from Software Company B’s perspective.

Adding two separate management boxes around the project doesn’t provide any real value in return, the financial effect is that now the project is a low-budget project, so company B will probably start looking for the cheapest software development resources available.

One would expect that the extra money put in Project Management and Risk Coverage would turn into a perfectly managed, risk free project. Well... not exactly so.

By the way, this is often the schema which is applied in offshore software development, where the common belief is that developers are so cheap that you can compensate the extra costs related to offshoring with huge savings in product building. I am not going to dig into offshoring in this post, but it’s definitely not that simple.

Messin’up with things
Unfortunately, this type of two-levels management doesn’t only duplicate costs, it does more: it makes the whole process inefficient, and it’s actually effectively working against project success.

The subcontracting development ecosystem: formally the referring person is still belonging to Company A, but development is performed by Company B. The key communication channel is now obstructed.

• Communication flows from Customer to Company B through Company A. If Company A is the one to be paid, then all of the key discussion must pass through A, transforming the supposed Project Manager into a bottleneck.
• Indirect communication means delay, important information may rot in a mailbox, before being forwarded to the person associated with the solution.
• Indirect communication means also information loss. Some information must be understood, not only listened or read. Sometimes forwarding it’s not enough to ensure some key detail is not missing.
• Since the information is sensitive for two different contracts, all parties become more inclined to write key issues instead of using face-to-face communication. Some issues are then carefully evaluated and involve more negotiation on an official level. The entire process becomes then waterfallish: having three players, with higher risk and no effective direct communication, paper (specifications, change requests, and so on) becomes more important than face-to-face communication eventually slowing down the process.
• The customer role becomes awkward. The customer is the real customer, but for Company B, Company A is ‘the customer’. Unfortunately, the quality of information you can get from a customer who is not the user of the software it’s a lot lower. So the risk of doing the wrong thing it’s extremely higher.
• If Company A is still pretending to be the developing company, direct communication between Company B and the Customer might even be forbidden, making things even more awkward.
  

The overall effect of all these factors combined is often pure havoc. A project with a decent budget becomes staffed with inadequate people, in an ecosystem that obstructs good communication between the customer and the developers, and where management roles are duplicated adding costs and subtracting value.
Is there any hope?
In theory, there is one scenario where it could still work: it’s where Company B is so good at building software that its estimations are significantly smaller than Company A ones. But to put this scenario in the real world one should answer the question: “Why a company that is so good at building software needs marketing from another (and poorer) Software Company?”

As developers, we’re probably associating this situation with the decorator pattern, which is simply adding features (a more effective marketing) to some other (the development) without changing it. Well ...this is simply not the case with subcontracting. The act of subcontracting has a negative impact on development and should be avoided or limited. As a customer, it does make a lot of difference if the software company is actually developing the software or simply playing the role of a broker. Having seen many of these situations going straight to non-recoverable state, I really think that special care about these issues should be taken in the pre-contract negotiation phase to prevent unavoidable surprises.