SOA desperately needs DDD

Talking with some colleagues, involved in SOA projects, I often have the feeling of a big hole in the perception of what SOA is really meant for, and what is needed for SOA projects to be successful.

I do believe that SOA is a great idea, it just sounds like large scale common sense. But I also do believe that “large scale common sense” is an oxymoron. Many dysfunctions in SOA projects are closely related to what normally happens in large scale projects, and in many cases those dysfunctions are transforming the whole stuff a whole big loss of money.

A common mistake, is to focus primarily on the architectural aspects of a SOA implementation. Which sounds like a rather obvious thing to do, considering what the A of SOA stands for. Unfortunately, this approach often turns out being narrowed to large scale OOP, only with a different terminology. Services (and their underlying model) are not reusable the same way objects are (and you remember a bit of the times where “reuse” was the buzzword and OO was hype, you probably know that also object reuse turned out being a lot different from the premises).

Behind its interface, a service is implemented according to a specific model. SOA makes no assumptions about the model, and allows for different implementation strategies: so far so good. Still one of the primary drivers for SOA is the need to rationalize the enterprise landscape, by removing unnecessary duplications and enforcing reuse of enterprise level services (we’re still in the “large scale common sense” here). Too often, the attempt to rationalize the landscape goes too far, trying to involve also the model in the process. This is often linked to the way SOA is conceived and implemented within your organization, a policy like “every significant entity will have to be wrapped by a service” will possibly lead to an enterprise scale CRUD moloch.

Enter Strategic Domain Driven Design
The point is that a service might be a significant reusable entity throughout the enterprise, while a model might not. A model should be the optimal solution to one specific problem, valid within a specific context. The context must have boundaries to allow for an optimal solution. An enterprise-level model will rapidly become blurred and some key abstraction will start to serve too many different purposes.

What is a Customer within an enterprise? Can you really find a one-size-fits-all model to represent a customer within the many context that must be supported by your enterprise scale SOA? My take on that is ...nope. Well, you’ll always be able to find some trivial entity that can be the same throughout the whole enterprise, but the odds will be against you if you start looking to some non-trivial entities existing in multiple domains.

In strategic Domain Driven Design there are some key principles to address this situation.

• “a model serves a specific use”: a model is a tool, and to be perfectly shaped for a specific use, the use must be well defined. Like a tool, a model can’t be really effective if the same object is used for many different purposes. Also, an effective model must be kept small enough to be coherent and manageable by a single skilled development team.
• “a model lives within a well defined context”: context and their boundaries are really important to define a coherent model. There are entities that can be used in different context, but sharing the same entity is not necessarily the best way to address the problem. Often the drawbacks are heavies than the advantages.
• “there will always be multiple models”: despite this sounding rather obvious on a large scale SOA, many times a lot of effort is dedicated to try to fight this situation, with minimal chances to win.

A typical problem with SOA is that implicit communication costs are rarely accounted. Sharing the vision of a model within a development team already has a cost which can be kept small enough if the team size is reasonable. Having the same vision within a 5 persons team is feasible. Sharing the vision among 40 people (or more) from different consulting or body rental companies (which is a common scenario in large-scale SOA development) is pure utopia.

The IT Communication Gap

As always, James McGovern got it right... check this

Tags: enterprise architecture, humour

Designing to lower the TCO

I was reading this post from Steve Jones, and had mixed feelings: as I commented on his blog, I am trying hard not to be that type of architect. But forcing an organization to consider the Total Cost of Ownership for the whole project lifecycle is often a tough job.
Sometimes is the organization itself to be badly shaped, maybe with separated budget between development and production, so managers aren't enforced to save somebody's else's budget. Sometimes the cost of ownership is perceived as a "normal mess" and becomes alarming only when it's totally out of control, which can be ten times bigger than acceptable, or more, due to the "buffer effect" of the dedicated people.

Sometimes is the overall project planning that plants the seeds, for the "touch and go" architect. Time is allocated mainly before the whole development starts, so the architect can't see the developers in action. I mean, architecture is challenged by development, and by time too: developers could not behave as predicted and find different ways to solve coding problems, and evolving tools and framework could make the balance of forces that drove some design choices not any more valid (that's a good reason for architect to document the reasons of their choices). It's often not a matter of being right or wrong, but instead a matter of seeing the whole picture , which is - obviously - much easier at the end. Following a project till the late stages clearly makes a better architect, but seeing everything possible from the early stages is what software architects are paid for.

There's some sort of analogy with the traditional drawbacks of the waterfall approach with the analyst role. Agile processes have put of a lot of efforts in introducing iterative release cycles, which are just a way to anticipate real feedback as much as possible. Iterating architecture, for some reasons, seems to have a longer path, but I'd say it's probably the same problem, only with different category of users.

Tags: SOA, project management, enterprise architecture

The Flawed Interface Principle - Part 2

After re-reading my last post, I felt quite disappointed: the main point could have been squeezed up in

1. the more you add stuff to a system, the more you’re likely to bring mess


2. if you do things by yourself you’re probably getting a better result (but in more time)


3. interfaces are dangerous.

Ok, point number one, is right, but really sounds too obvious to deserve a post (I'll never win a Nobel with that...). We’ll get on it later.

Point number two is simply dangerous: you simply don’t start a project developing everything from scratch, but start choosing the available components on the market (let’s say… JBoss, Hibernate, Log4J etc.) No way I am telling that you should re-invent everything from scratch! If time is a constraint (and in software development always is) then the better trade off is somewhere else, remember that we moved from a two-minutes micro waved junk food to an 6/8 hours preparation delicacy. Moreover, there is normally no reuse purpose in cooking. While reusable components are the keys to successful projects.

Point three. The real meaning is that interfaces are tricky. In OOP theory, an interface is just a substitution point. As a client class, I declare the need of an object instance able to perform certain operations, namely the public methods of the interface. I can’t make no assumptions about the implementing class, I just expect that somebody (a factory method or the classloader) will provide the needed runtime implementation. What I don’t know is just what’s behind the interface. Which is both powerful and dangerous.

What lies behind
In complex systems, the value of interfaces is their ability to simplify the overall picture. A complex subsystem can be masked by an elegant Façade allowing designers to ignore implementation details. I’ll make another example with a pretty simple interface: the electricity plug on your wall. It’s a pretty simple interface: 3 holes, 220V, 50Hz. Every electronic device can work everywhere (at least in one single country). The plug mask the whole complexity that lies behind so you can happily ignore about tri-phase cables, power plants, and so on. Unless they start building a nuclear power plant near your home, or stocking plutonium in your garden…
We got closer to the point. Complexity is still there. Interfaces make it more manageable, by allowing us to ignore details, not to delete them.

Crossing competence boundaries
While developing complex J2EE applications, a common problem to face is a version clash between (indirectly) required libraries. Xerces is a common suspect in such circumstances, cause it’s often referenced by the application server, the application itself and one or more used libraries often use it as well. Normally the solution is nontrivial and has to do with fine tuning of the application server classloading mechanism. Still you need an expert J2EE architect to drill and solve the problem. Sometimes is not so easy to find a good one, but it’s a well defined professional profile.

Things are not that easy when you start crossing competence boundaries. One of the first borders is the Java-SQL. How many applications have you seen that could be told to be optimized both on DB and Java performances?

If you start adding more layers, like running on a virtual environment instead of a real one, such as VMWare. How many details are you missing in the meanwhile?

Problem determination in Babylon
Here we go. We built our beautiful tower, by putting freely available components together. The tower is high and majestic. Then we have a problem. If the problem is in the glue we used to put the things together, it’s an easy task. After all gluing is all we did. If the problem is in one brick, we have to determine where the problem is (which can start to be a nontrivial task), and update, refactor or substitute the faulty component.
If the problem is in the interaction between two or more components, or in what happens behind the scenes (or the public interfaces) then you’re in deeper trouble: cause you need cross concerns competence and you probably won’t have it. And it’s not easy to be found as well.

Tags: Enterprise Architecture, OOD, J2EE

Food processing chain and The Flawed Interface Principle

While I was discussing with a friend, preparing a speech about complex system management, I realized that there is a close analogy between food processing industry (and its worst effects) and what usually happens on large complex systems.

Grandma’s good old taste
Suppose you want to make some pasta, tagliatelle Bolognese for example, that you used to like in your childhood (ok, it’s getting pretty clear that this post it’s gonna get targeted to Italian geeks that like to cook… a pretty narrow audience niche…).
You can go to the supermarket and buy the ingredients you need, but then you have to make a choice

1. buy the “all-in-one” package that you just have to warm up (microwave or frying pan)


2. buy the pasta and buy the sauce, cook the pasta and then mix it by yourself

…

• buy the eggs, the floor, the meat and the vegetables and actually make the pasta and the sauce

If you are a good enough cooker, you’ll notice that there’s a close connection between how much time you spend in cooking, and the final result.

Still, even if good, the result still doesn’t taste as good as your grandma’s. Next step is forgetting about the supermarket and buy the staff straight from the farmer. Suddenly eggs start to taste differently and so do the vegetables.

Ok, in the end you have to surrender: you can’t beat your grandma in cooking. It’s just an axiom. But sometimes you can get close. The interesting thing is that once you managed to get that close to the taste of your childhood you can’t even think about eating option number one at the top of page. And even if the label (and the pictures) tries to tell you that it’s the same thing, you just know it’s a completely different stuff from what you really want.

Question is: “How could it get that far?”
The answer is pretty simple: by systematically doing a substitution of the original ingredient with its industrial equivalent. And here is where we get back to software.

The Flawed Interface Principle
The keyword is “substitution”, how could you possibly do that (remember the oop substitution principle…)? Because every ingredient’s decoy looks or smells (almost) like the original, and if you change only one ingredient the difference could be small enough to go unnoticed.
Put in another way, food decoys implement just part of the original ingredients’ interface, generally the look (maybe with the help of some good photographer), but not others, less documented or testable, such as smell, for example.


If you start reading labels of pre-prepared foods, you realize also that they have more than twice the ingredients needed of the original recipe. Stabilizers, preservatives, enhances, colourings (whose role is just to satisfy the visual interface), and so on.

The point is that nontrivial interfaces are seldom complete. Even if they can be thoroughly documented, there is probably nobody checking if the whole system can fit together (at least I’ve never seen it happening in the software development industry). Every component you add in a system has a burden of internal complexity (libraries, for example, but not only) that can’t be completely masked by encapsulation, but only 99%.
The net result in complex software systems is that if you consider that every implemented interface between components has a certain probability to be flawed, you come to the conclusion that every time you add (or more fashionably plug-in) some component in a complex system, you are getting one step closer to the whole system collapse.


Tags: Enterprise Architecture, OOD