Can SOA governance technology be distracting?

In a recent post, David Linthicum asks "Can SOA governance technology be distracting?". His answer is yes, and he offers the following sound advice:

First, only purchase SOA governance technology, if it's indeed needed, after you have a complete semantic-, service-, and process-level understanding of the problem domain. Never before.

Amen to that. In my opinion, for all but the most mature and involved environments, the procurement of an SOA governance platform should be well down the list of priorities. I'd add to David's list of things that need to be 'worked out' before you get that cheque book out:

• What is your vision for governance itself? Do you want to adopt a 'iron fist' or 'hand in glove' approach? Is your registry going to be a mechanism for governing or a side effect of it?
• Who's going to populate it? Have you got your analysis, design and development processes sufficiently honed that your repository isn't going to turn into a dumping ground of candidate services?
• Have you actually got any services live yet? Governance is a whole lifecycle thing. Until you've worked out how you're going to deploy and manage services in the production environment and demonstrated that this works, how do you know what capabilities your governance platform needs to offer?
• Most importantly: What are the use cases for your governance platform? Can you demonstrate that these use cases can't be addressed using your existing tooling (even if that's Microsoft Excel)? Be honest with yourself about when you're likely to implement these use cases. If the answer is further than one year away, then for the time, you might be wise to forget them. There is little point in spending good money on runtime governance or automated deployment technology when in a year's time you'll be able to get more for less.

A lot of projects using SOA governance tools at the moment treat them as glorified databases. If that's where you're at, consider using something less specialised that allows you to evolve your ideas, understanding and schema before you commit to something that will make this innovation harder and more time consuming. When you've spent six to twelve months getting your ducks in a row, so to speak, you'll be in a much better place to make decisions.

I'd really welcome stories from people about how they've implemented governance platforms in the past, whether they're informal (e.g. Wikis, bugtrackers, spreadsheets) or formal (e.g. IBM Websphere Registry and Repository, CentraSite from Software AG): What did you implement? What worked? What didn't? What would you do differently next time?

Choosing an Open Source 'ESB' technology

I'm currently working to find the best Enterprise Service Bus for a project. Nothing unusual there. Something I've done a few times before. Except this time the requirements are a little more unusual than which kinds of transformation the tool supports.

Functional requirements:

1. The ESB must fit with Smart421's SOA patterns.
2. The ESB need not be a one-box solution. We're happy to mix and match tools around the outside. BPEL in particular is not necessarily a mandatory part of the core product.

On the face of it, not too tricky. In practice, maybe a little harder: In our view, the ESB isn't a piece of software in the first place, it's a collection of (continuously changing) standards and policies that govern the interactions that take place across the essentially empty void between two services, so we're looking for something that's compatible with that view, rather than something that wants to sit at the centre of the SOA universe (more on this later, perhaps). Let's look at the non-functionals:

1. The ESB must be Open Source, be based on an Open Source product, or there must be an Open Source version available.
2. The ESB must have an established presence in the market - the latest and greatest features aren't enough. We're looking for something that has some industry buy-in.
3. It must be possible to buy in support for the product from a third party, should we need it.
4. The ESB must support light weight development. We must be convinced that easy things are easy achieve, and hard things are proportionally (and not disproportionately) harder.
5. The ESB must offer a non-functional envelope that allows it to support a large scale enterprise application, preferably without restricting us to vertical scaling.

I'm intending to follow a fairly standard product procurement process to help select the technology, so the next step is to set some more formal selection criteria and identify some candidate products to form the ESB core.

At the moment, the obvious candidates for me are:

• Mule
• OpenESB
• Apache ServiceMix + IONA Fuse
• JBoss ESB
• Apache Synapse + WSO2

I'll keep you posted as I progress... Comments/suggestions/vitriol welcome!

So what is service reuse anyway?

Just read an post over at InfoWorld by David Linthicum. The central point of David's post is that reuse shouldn't be considered or advertised as the 'core value' of SOA. I totally agree with that, particularly as often, organisations still insist on setting up SOA projects, rather than changing the mindset and method of working to encourage the adoption of SOA across projects.

To me, half the problem with the reuse argument is that people don't have a clear understanding of what is meant by the term. Often, people consider that reuse is only achieved when a service has more than one consumer within the SOA at a point in time, but (to me) that's missing the point. An SOA is not a static environment; in a well managed organisation, it'll change almost continuously. To me, you're achieving some level of reuse whenever you fundamentally change a business process without having to re-hash all of its services, or you don't have to bother re-testing every combination of parameters to a complex business process because you know already that the services that underly it, which make all the hard decisions about someone's credit worthiness have been tested thoroughly already. You even get re-use whenever you say "Ah, well, we'll apply our standard grade message integrity policy to that service, so we know how to achieve that".

If you take this broader view (let's call it long-term reuse), then long-term reuse and agility become somewhat interchangeable concepts - you get agility through being able to reuse a service across two versions of the same process, by knowing that you've already tested a given service to death, and by not having to make complex technical design decisions each time you create a 'new' service.

So. I agree that the reuse should not be used as an advertising 'headline' for SOA, because in an advert (be it literal or spoken), you don't really have the time or bandwidth to get some of the subtleties of the above across. But in reality, over the medium term, I think we will find that the long-term reuse achieved (assuming for a moment you buy into my definition) is significantly higher than people currently expect.

The last point I'd make is this: If you design your services for immediate reuse, you will likely fail to achieve that re-use. If on the other hand, you design your services for change, and aggressively factor new functionality into them as new consumers require it, I suspect you'll find even 'point in time' reuse increases dramatically over the medium term.

IBM WAS: Fast? Yes. Central to SOA? Only in marketing.

IBM recently issued a press release stating that "WebSphere Application Server, a key building block for services oriented architecture (SOA), shattered a popular industry benchmark for scalability and performance by more than 33 percent using technology that costs half the price of the competition".

First of all, Kudos to IBM for continuing to improve the performance of WAS. Anything that drives the performance of key bits of infrastructure has to be a good thing. However, that's not the thing that caught my eye about this article. Instead, my eye was drawn to the statement that WAS is a "key building block for services oriented architecture". Let's consider where IBM was coming from when they wrote this:

• IBM WebSphere Application Server forms the foundation of IBM WebSphere Process Server, IBM's middle of the road (as opposed to specialised) 'SOA Platform' offering.
• IBM WebSphere Application Server is the leading (in terms of sales, at least) J2EE platform.

Maybe this is where IBM were coming from. It's true that WPS has significant take up in the market, usually within SOA projects, so I guess you could argue that WAS is the foundation to something which acts as part of the integration infrastructure underlying SOA projects. Alternatively, perhaps they're considering WAS in its own right. It's certainly an excellent platform for building and deploying the business logic which drives the behaviour of service implementations.

But (and in my opinion, it's a big but) the key to SOA is hidden cunningly in the name. It's Service Oriented Architecture not service oriented application-server, service oriented platform or service oriented infrastructure. SOA is about structuring your solutions according to service oriented principles, governing these solutions to ensure that they all contribute to an enterprise-wide ecosystem of services. To me, a key SOA principle is to be globally independent of any one third part product, and understand before you start how you can migrate away from the platform in the future. Therefore, to state that a product is a central component of SOA as a whole is misinformation at best, and could if taken literally be very dangerous for the ongoing health of your company.

I know that sounds melodramatic, but there's method in my madness: If SOA really is The One True Way, then it'll outlive any product (and possibly several of the vendors) currently in the market. If you couple your SOA vision to an individual platform, then you risk creating yourself a huge migration issue, comparable to nothing you've seen previously when the market inevitably moves on to new products in the future.

Web Service Threats

I've been doing some work recently that involves enumerating the potential threats faced by SOA implementations. To help Google, here are a few of the resources I've found useful in doing so:

• ZapThink - XML Threat Management: As you'd expect of ZapThink, a good high-level description of XML threats in general, but no specific terminology for individual threats, and no categorisation of threats.
• Forum Systems - XML Threats: Forum Systems sell XML/web service security products, so as you'd expect they're hot on illustrating the risks associated with using Web Services to customers. Nothing like a rainstorm to sell umbrellas. This page provides a very high-level summary of the types of attack vectors used by malicious service consumers; a companion white paper, "Anatomy of a Web Service Attack" provides a lot of additional detail explaining these types of attack.
• ZDNet - Five things you need to know about Web services threats: A useful article by Scott Morrison, Director, Architecture and Security at web service security firm Layer 7. Gets away from talking about specific threats, and talks more about the goals and high-level approaches used by attackers. Well worth a 10 minute read.

Interestingly, there doesn't seem to be a consensus over a taxonomy (categorisation system) for these threats. Possibly the closest we get is in Scott's article, where he suggests that all web service threats can be put in three categories: API, Infrastructure and Transaction attacks. Sadly, Scott doesn't go so far as categorising the threats identified by the other vendors. Hopefully when this picture emerges, it'll represent industry wide consensus. This should allow vendors such as Layer 7, Forum Systems and IBM (with their DataPower range) to communicate with customers in a consistent manner, and allow them to sell their wares based on customer need and product capability, which after all is the key to long term relationships and repeat business.

Runtime SOA Governance

I've recently been asked to define Runtime SOA Governance in the glossary of a project, and having looked high and low, have come to the conclusion that there is no widely accepted definition of the term, which is somewhat surprising given the number of product vendors who mention the term. Despite the risks to my personal safety of doing so, I have come up with my own, which I am only vaguely happy with: Runtime SOA Governance provides a mechanism allowing information about the operation of a running SOA environment to be fed back into an organisation’s SOA governance function. The information gathered may include:

• How often each service is invoked.
• Which services each service invokes.
• Whether services are meeting their service level agreements in terms of availability, reliability and performance.

This information helps to support investment and change decisions by giving the governance function a better understanding of the runtime behaviour of the services. If you have a better definition, do let me know...

Paul Freemantle: A new kind of (SOA) Registry

Forgive the misleading title; Paul isn't actually a new kind of SOA registry. But he does know a bit about them, it would seem. Especially the bad bits. Paul is proposing a REST-based registry implementation which completely sidesteps UDDI and friends. Paul's main accusation of the SOA Registry and Repository market is that it's very biassed towards commercial software, which is absolutely true. While I'm not convinced that this is UDDI's fault per-se, I do think there are plenty of things wrong with UDDI anyway, so am more than happy to see some alternatives on the scheme. What I love about the approach Paul's describing is its simplicity. Every other registry product I've looked at before (which isn't many, to be fair) has been almost defined by its complexity. Many even feel like they use it as a selling point. What Paul's describing does away with all that, relying instead on simple foundations (REST and ATOM), and then allowing the complexity to be built outside of the registry, just like in the blogging world. One to watch, I think. I'm off to download the code.

After Service_1.0: The Service Interface Version Adapter Pattern

Stellan recently wrote an excellent article about dealing with the versioning of service interfaces. I thought I'd chip in with a slick way of dealing with the versioning of the service implementations themselves. To summarise his approach, Stellan talked about including versioning information in the namespace of your service interfaces, which as he says:

• Means that clients already using the interface won't 'break' the service interface.
• Allows service implementations (and clients, potentially) to dynamically determine the service version based on a message.

One further advantage that Stellan doesn't directly allude to is the co-existence of multiple versions of the same interface in the same runtime environment, which can be exceedingly useful, as I'm about to explain. Stellan explains in his article that the namespaced interfaces can be used on the server to route the message to either the latest and greatest or original versions of the service. Unfortunately, the side effect of that is that you have to keep maintaining two versions of the service, which is made even worse when you have some kind of underlying data store which has to be shared by both versions, because any structural change to that database will require you to modify the data access layer of every version of the service you currently have deployed. A little while ago, I came up with what I think is a nifty solution to this issue, which I called the Service Interface Version Adapter pattern. The pattern allows you to maintain total backward compatibility with clients, while only keeping one version of the service live. I'm sure I'm not the only one to arrive at the same answer, but I've never seen it documented anywhere. Basically, the pattern makes the assumption that version N+1 of your service is backward compatible from a functional point of view with version N. The interface can have changed beyond all recognition, providing it still possible to achieve the same result. Instead of keeping the old version of the service around, we introduce a transformation between the old version of the service and the new. If the transformation is simple, it could be implemented as an XSLT stylesheet, for more complex cases, bespoke Java/.NET/Python might be required, but it's complexity will be far simpler than the original service in the vast majority of cases. What's more, since there's now only one implementation of the service, clients calling the old version of the service will automatically benefit from bug fixes with no extra effort. Okay, so we now have one service implementation, but two working interfaces. What happens when we need to release version N+2? Easy: We add another interface version adapter which converts between version N+1 and version N+2. We could have achieved the same result by modifying the original transform to target version N+2, and introducing a new one between N+1, but if we had done that, we'd have to maintain an ever increasing list of transformations - back to square one. By just adding one new transform, the effort for each new version is the same: one new transformation. Clients of version N and version N+1 can continue to benefit from bug fixes, no matter how long they take to upgrade. Obviously, it'd be right to consider the performance implications of this. In reality however, most of the transformations involved will be so simple (add a default value here, move an attribute there) that even an XSLT based implementation will be fast (as long as you don't use DOM based transforms), so unless you're supporting 100s of versions, I don't believe it'll be a problem, and consider the alternative: 100s of fully fledged service implementations to maintain. If you're really worried about performance, and your budget can stretch to it, you could even consider implementing the whole pattern using an XML appliance like the 1U IBM DataPower which will do most of the heavy lifting in hardware. I have it on good authority that they are faster than an a flock of exceptionally fast things.

SOA Rules of Engagement

Many organisations struggle to get their SOA off the ground. The reasons vary, but mostly stem from getting the very simple things wrong at the start. The following are my 8 simple rules for a successful SOA implementation:

• Don't forget the constructive tension. For SOA to be a success, there needs to be a tension between the desire to do things now, and the desire to do things right. This means setting up a governance function to complement (note: complement, not obstruct) your delivery function.
• Don't forget to deliver something useful early. SOA business cases are notoriously hard, because many (not all) of the benefits arrive in the longer term, and will only be realised if other projects behave as expected. By far the best way to get on in this world is to deliver some business benefit early. Your sponsors will thank you, and it's not as hard as you think - SOAs are all about managing change afterall, so you can do a lot just by delivering a sequence of relatively small incremental projects.
• Don't boil the ocean, the fish won't like it. Many many SOA projects try and 'SOA-enable' huge swaithes of their technical architecture in one foul swoop. It's a well established near-fact that it's difficult or impossible to achieve. I've only worked on one project where this worked, and that was because it was a spin-off from a large company, and so had only a small amount of 'heritage' (nee legacy) systems to contend with - most of the services were built from the ground up.
• Don't worry about the technology. Technology is usually far from the biggest risk to a large SOA programme. Weak benefits cases, poor governance, poor expectation management or a changing political/economic environment are far more likely to be your downfall.
• Do worry about the technology. Having said that technology is not the biggest risk, it is a significant risk. Keep it simple. Implement just the functionality and infrastructure you need in the first instance. Don't believe people or companies that tell you that SOA is a product, or is simply a technical problem that they can solve for you.
• Do get strong project management, but keep them in their box. As I've alluded to above, there are tons of risks waiting to consign your SOA aspirations to the great ESB Gateway in the Sky, so you'll need an excellent project manager to manage them for you. By excellent, I don't mean they're a dictator, I mean they're an enabler for your team, and a black belt in motivation, reporting, and risk management. Whatever you do: If they're running the delivery of a project, do not allow them to become responsible for your organisation's SOA governance. There be dragons.
• Do get an all star cast (at least initially). The nature of SOA is such that there is a lot of opportunity for working smarter rather than harder, which will get you to market faster, make your SOA easier to maintain, and cost you less. Get yourself a team of technical people who really know their bananas; quality is vastly more important than quantity. Seriously consider going outside your organisation to secure at least part of your core team - the people in the next room might be brilliant, but experience is everything in the SOA world. Architects, business/system analysts, developers and testers are all important, and you'll need them to work in perfect harmony for your SOA to really take off.
• Do make sure you're involved. It might be tempting at times to bury your head in the sand, and allow someone else to 'do SOA to you', but don't let this happen. While it's important to get that all star cast, it's also critical that your organisation learns from any 'outsiders', and will be capable of taking the reigns in the long term. That said, do consider keeping outsiders in the mix for the long haul - the world is changing rapidly, and keeping people who've seen other organisations make mistakes close to you will enable you to dodge the pitfalls that consume your competitors.

Comments and your rules, as ever, welcome.

IBM WebSphere Process Server and Apache Tuscany: Not in love. Yet.

I've just spotted a post on Bobby Wolfe's blog discussing the relationship between Apache Tuscany and WebSphere Process Server. He has confirmed that WPS will not be based on the Tuscany runtime.

I've been using Process Server since before it went GA way back in late 2005, and at the time, Tuscany was more embryo than fully fledged citizen of the SOA world. It drew my attention even then, because WPS - despite being a brand new product - was flying the SCA flag without actually being capable of implementing the 1.0 standard. Unfortunately for IBM, the last point release of the standard had made some fairly significant changes to the way code was structured into modules, which might have made it hard for IBM to implement the 1.0 standard on their existing architecture.

The combination of these inherent limitations, and IBM's bundling of Tuscany within the (snappily titled) IBM WebSphere Application Server Version 6.1 Feature Pack for SOA meant that we always assumed IBM would ultimately migrate Process Server across to the Tuscany, with the value add coming from IBM's established process, mapping and rules engines. Since the feature pack was out for 6.1, we had hoped (perhaps beyond hope) that this would happen as part of the WPS 6.1 release, which it would seem now is not the case.

This is something of a disappointment for me, as our architecture was being made significantly more complex than it needed to be because of the lengths we had to go to to re-use 'service capabilities' (more on these another day) across services.

I'm still 99% sure IBM will make this transition at some stage - they'd be crazy not to in my opinion, since there is plenty of value add IBM can provide in the components that sit on top of the SCA runtime, and every dollar they save working on the runtime they can use to add new glossy features to the remainder of the product.

My money is on WPS 7.0. How about you?

100% True (honest) SOA Facts

A colleague helpfully provided me with the following link yesterday, thinking I could do with some enlightenment.

http://soafacts.com/ 

Despite his assurance that they're 100% verified and accurate my Spidey Senses are still tingling...