Last week I outlined an idea, that of the service anti-pattern, as part of a presentation I gave to the Resource Discovery Taskforce (organised by JISC in partnership with RLUK). The idea seemed to really catch the interest of and resonate with several of those members of the taskforce who were present at the meeting. My presentation was in a style which does not translate well to being viewed in a standalone context (e.g. on Slideshare) so I have decided to write it up here. I would very much welcome comments on this. (The presentation will be published on the Resource Discovery Taskforce pages and I will ask for this post to be linked to from there when it does appear).
The following diagram is meant to represent a design ‘pattern’ which I have seen often proposed, and sometimes implemented, in the JISC Information Environment (IE) as well as in the wider higher education (HE) sector in general:
It is my belief that readers who have been involved with the IE for some time will recognise this, at least in a general sense, if not in specific cases. In this arrangement, an aggregation of data is presented to the end user, through the development of a user-facing application or service. The user-facing service will in almost all cases be a web-interface, somewhat similar to the ‘portal’ concept of old but in a centralised, single, global deployment. Because it is generally accepted to be desirable to make such data available to other services (in keeping with the larger goal of interoperability through open standards), one or more machine interfaces or so-called APIs, giving access to the ‘backend’ of the system, will be offered. What this design pattern aspires to is a service implemented to be both user-facing service and machine-facing infrastructure component.
However, I contend that this is, in fact, what software engineers might call an anti-pattern. An anti-pattern is a design approach which seems plausible and attractive but which has been shown, with practice to be non-optimal or even counter-productive. It’s a pattern because it keeps coming up, which means it’s worth recording and documenting as such. It’s anti, because, in practice, it’s best avoided….
There is much which is implicit in this pattern, so I will attempt to surface what I believe are some hidden assumptions in a new version of this diagram: this is what this design pattern, once implemented, reveals:
In this second diagram, the orange colouring indicates the parts which actually get built and are supported; the yellow indicates the parts which might get built, but which won’t really be supported as a service – in a sense, this is stuff which is believed to work but actually doesn’t; in the case of the users, the yellow colouring indicates that their demand for this service is believed to exist; those components in the diagram which are neither orange, nor yellow, are the product of little more than speculation. In the end, the investment in creating a user-facing application based on an expectation of future demand which doesn’t materialise is wasted while, at the same time, the investment in providing unused machine interfaces is also wasted.
I believe that this design pattern rests on several assumptions which are actually fallacies, and is, therefore, an anti-pattern.
Fallacy 1: “Build it and they will come”:
While infrastructure services can, indeed should, be developed with future opportunity in mind, it is helpful to have an existing and real demand to satisfy, which the new development addresses. If the service is demonstrably useful to users, and is developed effectively with future opportunity in mind, then there is more chance of the service actually working, and of it being attractive to developers working on future opportunities.
Fallacy 2: Interoperability through additional machine interfaces:
Machine interfaces need as much specification, development, testing an maintenance as user-interfaces. Simply making a machine interface available through the adoption of a platform which has a built-in facility offering some standard interface is not enough. A system which proposes to offer three or four APIs is quite likely not going to support any of them adequately. I have argued before that ‘interoperability is not enough‘: in fact, this arrangement does not often lead to interoperability, let alone actual exploitation of the capability to interoperate.
Fallacy 3: People/organisations who can make good infrastructure are also going to be good at building end-user-facing services (and vice versa):
Effective infrastructure supports services which in turn support end-users. The skills and knowledge required to support service-providers are generally quite different from those needed to deliver good user-facing services.
I call this the infrastructure service anti-pattern because the result comes from conflated requirements to deliver both infrastructure (machine-to-machine interfaces) and compelling user-facing services and applications. The result can be something which satisfies neither requirement. The users, requirements and priorities are often completely different between these two problem spaces. I suggest that the following are some possible reasons for this anti-pattern appearing:
- funding (naturally) tends to follow services, happy users and, importantly, new features.
- funders like to see their investment showcased
- infrastructure is mostly invisible making it hard to ascertain impact from users
Proposals for alternative design patterns
Here is a suggested alternative design-pattern:
In this design pattern, the API is developed before any user-facing application, or at least in parallel. An application is developed to exploit this API based on real users requirements. No service is developed until such requirements can be identified. This means that an API will be developed, and it will be being used in at least one case. Opportunities for third party integration for usage of the service are, ideally, identified beforehand. The API is properly supported from the start, or else the service fails completely. The value proposition being offered for further, opportunistic third-party developments, whether real or imagined, is now real and, crucially, supported.
An interesting alternative to this is the approach of combining the user-facing web pages and the machine-actionable API into one interface, through embedded RDFa for example:
It remains to be seen how this approach is going to work out over time, but we have seen hints of simpler approaches to combining user and machine interfaces in the past, such as RSS being styled to give a decent human-readable interface, or earlier attempts to do interesting things with XHTML.
I wonder if readers agree that the first diagrams represent an anti-pattern which they recognise. And would the proposed alternatives fare any better?
Agree totally with this post – though assumed it wouldn’t make a lick of sense to me after reading the title!
I think in 1st and 3rd Fallacies have been around as long as there has been a web and we don’t seem to be any closer to getting beyond them. Identifying a genuine demand for applications and services is vital – there will always be an element of crystal ball gazing but there should always be a core demand that makes people want to use the tool/app/service. Then there is the issue of not assuming that creating the user end of something is just something that can be bolted on and left to people whose expertise is behind the scenes.
I like the 1st example of an alternative design pattern – building a genuine example of a use of the API from day one seems like a sensible way forward – the 2nd alternative worries me a little as sometimes I think we try to run before we can walk..
Matt – glad you read beyond the title then! I think your use of the word ‘genuine’ is key – often, a ‘showcase’ takes the part of a real usage example.
The second alternative is already working in places – I’m including such tricks as styling RSS feeds to be easily browsed by users (ePrints does this).
And then there is a growing appetite for REST and content negotiation as well – some frameworks support this as a default approach (e.g. Ruby on Rails)
Just picking up on the e-prints example, see this discussion on the OU Research Online blog http://www.open.ac.uk/blogs/ORO/?p=55
I think the crux of the anti-pattern is that people construct APIs that they don’t use themselves in any way, shape or form to do their job. This is likely why the multitudinous APIs aren’t really used by a 3rd party, because they are not fit to purpose, bug-strewn and not particularly usable.
There unfortunately are very many open-source academic-related projects that fall foul of this.
But I’d make this note – if the API is used in the wild *already* for a similar purpose (eg RSS, AtomPub, JSON, etc) then the implementation can not only be tested – ‘does a known client for this common API understand the service out of the box?’ – but easily used once this test is passed.
Ben – yes, the ‘drinking your own kool-aid’ aphorism applies here strongly!
Regarding your second point – I guess this becomes ‘drinking the kool-aid which everyone else is drinking’. That way you can drink your own kool-aid in good company!
I’ve heard this argument presented before as a typical case of ‘pre-engineering’ functionality before you actually know what you are going to be using it for … and then inevitably getting it wrong when someone does actually sit down to try and use it.
There are a few circumstances where API can’t realistically be deferred and some API needs to hold a place in the sand. I saw this frequently when I was working for an OS company.
However, in my work now building graphical, embedded frameworks I try and use a development model more like Eclipse’s where you expose plugin entry points as they are needed to roll out internal API’s.
It is working well so far for our Storyboard application.
Thomas
http://www.cranksoftware.com
BTW: Great diagrams, really gets the point across well.
Thomas,
I’m glad (and not surprised) that this issue is familiar to people in industry and not just in my sector!
Thanks for the comment on the diagrams also – I put some thought into this approach and comments have all been very positive so far.
Great post. I especially like the idea of certainty, belief and speculation in the layers.
I’m not sure that “this design pattern rests on several assumptions which are actually fallacies” is the whole story – although I agree with the analysis of common assumptions I’m not at all sure that these assumptions are really the things that drive the anti-pattern.
I guess first and foremost ‘customers’ like ‘product’. This applies whether you are building a publicly funded service or selling a commercial product – what most end users want to see is a good interface. I realise you aren’t suggesting we deliver project without UIs of course – but it is this focus on the UI that drives projects/vendors to focus on these at the expense of underlying APIs.
Based on my own (very limited) experience in writing software I’d also say that building a specific use interface is quicker and easier than building an API. This means that the cheapest route to fulfilling the demands from the customer/funder becomes a UI with no underlying API.
Finally I’d suggest that there may be a skills issue as well. My own impression (again anecdotal of course) is that many developers do not have the background in developing software in the way that you suggest we should be aiming for – especially not in the combined API/UI configuration. I have a feeling that ‘programming for the web’ – understanding HTTP, content negotiation and RESTful approaches – is something that hasn’t (yet?) fully permeated the developer community. Along side this is that in my experience the ‘technical but not developer’ staff in libraries also don’t understand these things – and so aren’t as comfortable with them. Compare this with an ‘add on’ interface such as Z39.50 – which most library staff have heard and are comfortable with.
Thanks Owen – some useful comments – appreciated!
I think the skills/complexity issue is an important point – something we (JISC, UKOLN) should, perhaps, be addressing more directly.
I’ve had good feedback on the certainty/belief/speculation device. In truth, my post was partly intended to try this approach out so I’m really pleased with these comments. I have plans to apply this device in a project (more on this very soon).
This is a great post and I completely agree with nearly all of it. If I was feeling cynical (who, me?) I might tend to disagree with your greater belief in the embedded API and UI combo and wonder if that will also fall foul of the same kind of anti-pattern as the other forms of API. The risk is that the API part of the combo is again not actually used to drive the local UI and therefore isn’t subject to the same kind of ‘quality’ and ‘sustainability’ pressures as the rest of the UI.
In all such discussions, I’m minded to ask, “why has RSS succeeded as a good pattern (one exactly in line with your first diagram above) whereas other APIs (OAI-PMH, SRW, …) have failed?”. My answer, and this has been alluded to in some of my own posts, has partly to do with mainstream adoption (IMHO). Part of the anti-pattern is our reluctance to let go of APIs that are now clearly not going to get mainstream adoption (probably, as you say, because they don’t meet a general enough use-case).
Thanks for the comments Andy. I take your point about the UI/API combination potentially introducing the same anti-pattern. I can see how this would be a real risk for, say, RDFa embedded into a web-page. And, to be honest, my diagram somewhat glosses over the reality of different contextual requirements between machine and user interfaces! However, I think we agree that this is a pattern worth exploring further, even if my optimism is premature
Ben O’Steen, in an earlier comment, makes a similar point to you in your advice about considering the main-stream standard first. In many ways, this might be the big lesson of recent years….
@andy In terms of why RSS has succeeded, it feels like saying ‘because it has mainstream adoption’ is avoiding the question. I’m definitely in favour of looking at standards that have mainstream adoption and I’d agree with the point that we have to do this more, and not cling to sector specific standards (and this of course reflected in your recent blog post and accompanying comments http://efoundations.typepad.com/efoundations/2009/12/on-being-niche.html)
However, we also need to answer the question – what was it about RSS that led to widespread adoption? In the HE/Library sector we are often trying to solve problems that have not yet been tackled by the wider sector (coming back to the comment on your post that we are in a niche business – at least sometimes). What was it about the standards we developed like OAI-PMH, SRW, OpenURL, etc. that meant they never made the jump to a wider community?
So, I guess that what I’m saying is not just use mainstream standards, but develop standards that have the potential mainstream – and the latter is a much more difficult problem.
Hi Paul,
Outstanding post, and a lot here I’d agree with.
Having been involved in/responsible for around 20 of these projects, I absolutely recognise the pattern. I think, to be fair, that it is only really in the last 3-4 years that this kind of aggregation/syndication model has gained favour with funders, so my charitable instinct is to suggest that we’re on a learning curve!
Your model is also exactly the approach we’ve taken with the Culture Grid, although in some ways the use cases came before the infrastructure, so the intended outcome was less speculative.
I have long suggested a couple of your basic tenets, and after a decade or so, I’d probably write it as ‘if you build it, they might come’. It’s worth noting that for many of these projects, usage is not the primary success factor. The main criteria are (a) simply delivering something and (b) the approval of a very small number of people who control budgets.
Finally, the ‘anti-pattern’ concept also betrays one of the flaws in understanding the world through the eyes of the engineer – the political, economic and psychological factors at play exert a much stronger influence over decision-making than either logic or experience!
All best,
Nick
Thanks for the comment Nick.
I tend to agree with your observation that, “…usage is not the primary success factor. The main criteria are (a) simply delivering something and (b) the approval of a very small number of people who control budgets.”
I think these sorts of issues are important factors in the appearance of the anti-pattern I’ve described
I’m not sure I really understand how the anti-pattern betrays a flaw in the world-view of engineers. It’s a model, no more, and good engineers understand the limitations of models. While I completely agree that “political, economic and psychological” factors exert a strong influence (I might be tempted to add “social” and even “fashion” to the list!), it is still the role of the engineer to point to the engineering consequences.
An important point I would make about the anti-pattern (which perhaps I should emphasise more) is that the anti-pattern does not explain why there is no demand for the service. It explains why there could not have been any real usage, even if there had been demand….
+1 for making the “certainty/belief/speculation” categories more explicit. i’ve seen speculative requirements posed next to real ones too often.
while i commited some “let’s add an API so we can say we are interoperable” sins, what brought me to appreciate good APIs in my own products was:
* testing: testing APIs is typically much easier than testing web UI.
* AJAX: as soon as you start to access your backend in a more fine-grained way, you have to think about APIs.
* linkeddata: since linkeddata does pretty much specify the access protocol for your data, creating an API comes down to thinking abput resources and vocabularies.
nowadays my apps do almost commit the sin of delegating a good web UI to third party apps “because there’s an API”
Hi Robert, thanks for the comment.
The point about testing was echoed in an earlier comment – this is an aspect worth bringing into the pattern debate I think.
The “certainty/belief/speculation” device seems to have really worked for some people – I’m thinking about how to apply this to some other problem spaces for some work we’re doing for the JISC. I’ll be announcing and linking to this soon from this blog, so if you’re interested in seeing more of this then watch this space!
Like the others, let me congratulate you on a good post
I note that your final alternative design approach is pretty much exactly the REST approach advocated by Roy Fielding in his thesis. Roy’s fully-REST idea uses hypertext documents not only to transfer state but also to express what actions clients can perform in a given state. Most REST-lite APIs just layer CRUD on top of HTTP without making the state transition graph machine-readably explicit.
Hi Tony – thanks for the comment.
Yes – I’m still very interested in REST as an approach in the general ‘resource discovery’ problem space I tend to be focussed on nowadays and I find that the Resource Oriented Architecture (ROA) informs my thinking more and more.
I deliberately didn’t invoke it in this post because I wanted to see if anyone else raised it. Actually, a couple of others have mentioned it in response to this post but on Twitter, rather than as comments here.
It may be that REST’s time still hasn’t quite come as a more mainstream solution, even though we’ve had the wherewithal to implement it for a long time. Then again, I’m beginning to think that the RESTful (i.e. CRUD/HTTP) developments I’ve seen, although of variable quality, probably represent one of those “disappointingly short of the ideal but kinda working nonetheless albeit in a limited way” solutions which tend to win the day….
Great post Paul.
The idea of implementing a predicted future is something that has been recognised by many as an anti-pattern for many years. For example, extreme programming has the never add functionality early.
I’d also like to remind you that open source software development, when managed properly, makes it possible for those needing to expose parts of a core system via an API to do so. In many cases such additions are better made as apart of the core project, something that is also possible in a well run open source project.
This is important because whilst it does not change your proposed model
it allows collaboration across partners to ensure the best, most efficient and most appropriate solution can be built. It means that there is a rapid feedback between upstream and downstream developers. It also means that there is no need for the “donor” project to provide the necessary resources to extend the API or core app with unplanned extensions.
In other words, whilst I agree your alternative pattern is superior (and indeed can be commonly found) we must also consider how we support the ongoing development of the additional API features that new requirements present.
We must either centrally fund this work in response to market demand (central funding) or we must allow the market to make the changes themselves (distributed funding).
VIAF and WC Identities are architected along the lines of your last picture.
For VIAF see:
http://outgoing.typepad.com/outgoing/2009/08/viaf-and-opensearch.html
http://outgoing.typepad.com/outgoing/2009/12/viaf-and-rdf.html
Http://www.viaf.org
with user interface, sru/opensearch, linked data.
Interesting examples Lorcan, thanks.
I’m fascinated to see that VIAF merges the human/machine interfaces in two ways:
a machine-centric, but with a human-readably styled version if accessed from a browser
and
a human-centric, get the machine view with a little addition to the URL approach here
the second of which is, in turn, also styled for human consumption if accessed from a browser.
Excellent post! I’ve been coming to the same conclusions so it’s reassuring to know I’m not alone.
Pingback: An Opportunities and Risks Framework For Standards « UK Web Focus
Pingback: Discovery Summit: Paul Walk keynote » Overdue Ideas