paul walk’s weblog

For some time now I’ve been thinking about what I think of as the ascendency of the opportunistic developer in web application development. The phrase has unfortunate connotations for those who remember the ‘personas’ meme from some years ago when it was revealed that Microsoft had characterised three type of developer for three of its software development products. [1] and [2]. This post is not directly related to these archetypes (the opportunistic developer was called ‘Mort’ in the meme, a name which has become derogatory). Rather, I’m talking abut the developer who, regardless of their ability or their occupation wants to make quick use of something when they discover it, typically on the web.

The opportunistic developer prefers to use someone else’s service/component in the majority of cases. They will create their own software when necessary, and will choose to do so under certain circumstances, but they will accommodate a certain amount of compromise if it means they can get away with using something off-the-shelf. The opportunistic developer is still a developer, as opposed to a power user: they will still write code, just as little as they can get away with.

The proliferation of freely available web-services with simple APIs has created a happy-hunting-ground for the opportunistic developer - a few years ago they were inhibited by a lack of choice of available services to use. In addition to the usual concerns - stability, provenance, price… ease of use is becoming a more important differentiator.

In the JISC Information Environment, the norm has been to develop SOAP interfaces to services, almost by default. There are, no doubt, reasons why this has made sense in the past. However, if there is one thing which became abundantly clear at last week’s IE Demonstrator/CRIG event, it is that institutional repository developers do not want to have to use SOAP interfaces. Aside from the hard-core which is interested in pushing REST as the approach to use in repository-service interactions, the consensus was that the use of SOAP for public service interfaces, rather than being an enabling mechanism, is actually a barrier to adoption.

Whether RESTful or not, services are going to have to start having very good reasons for not offering very simple APIs over HTTP, if they are to attract the opportunistic developer.

Tags: iedmonstrator, JISC IE, opportunistic-developer, ReST, SOAP

I haven’t minted a TLA for ages - I think I might be the the first to come up with PPP for Personal Profile Portability as a convenient handle to wrap around the current flavour of ‘data portability’ being touted by the major ‘walled-garden’ social network sites.

Both MySpace and Facebook have recently launched initiatives to open up a little….but not too much.

MySpace has announced its Data Availability project with some major partner applications. Essentially, this will encourage the user to manage ‘profile’ information on MySpace, with a view to surfacing this information in other, partner applications (initially Yahoo, eBay, Photobucket and Twitter. It will also allow users to share some data such as photos which they have added to the MySpace site. Facebook has a similar initiative called Facebook Connect, initially in partnership with Digg. In both cases, a set of usage policies will be imposed such that the user retains control over what is shared, with the power to revoke the sharing agreement. I’m really encouraged to note that in the case of MySpace’s Data Availability, the mechanism adopted to solve the inter-authentication/authorisation issues between these systems is an implementation of OAuth.

Amit Kapur (MySpace’s Chief Operating Officer) says that Data Availability is:

“…founded first and foremost on allowing users to have comprehensive control over their content and data.”

Dave Morin of Facebook believes that:

“…the next evolution of data portability is [...] about giving users the ability to take their identity and friends with them around the Web, while being able to trust that their information is always up to date and always protected by their privacy settings.”

The extent to which users ‘have control’ over their content and data even while it has been completely locked up within the MySpace and Facebook applications has been argued about extensively. The relationships between these sites, their users, and their users’ data have evolved over the last year or two, as users have become a little more savvy. Pressure from groups such as DataPortability appears to have had an effect, with MySpace also signing up to this recently.

So, it seems as though the walled gardens are opening up, getting ready to participate in the wider web. Or are they?

In a web of distributed social networks, the most likely way in which users might manage their participation would seem (right now) to be through a single entry point. Essentially, if the web of social networks is going to allow ’single-sign-on for the user, and allow a re-use of profile information, and even content across multiple applications, then one model is to give the user a ‘gateway’ service, where they sign-on and manage their ‘account’. Both Facebook and MySpace are going to battle hard to be that gateway service for the masses. Both have accepted that they can no longer remain as a completely walled garden - they must open up, just a little, to avoid being eventually marginalised. But now that they are not totally closed, they may find it difficult to retain control. They may find others are waiting to seize the initiative. Enter Google, and its Friend Connect service.

Friend Connect is different to the previous initiatives from Facebook and MySpace. Google’s new offering is designed to provide a ‘middleware’ services, sitting between the big social networks, and sundry web applications which might want to exploit the new openings in these services. It also utilises components which have been developed with the OpenSocial API. Friend Connect is, I think, a very significant development, because it shows how more distributed social networks might work. It is significant also in a particular detail - notice how Friend Connect can become a social network of sorts simply by integrating existing social networks. Suddenly, the huge headstart enjoyed by Facebook and MySpace doesn’t look so unassailable. This is, presumably, the real reason why Facebook have taken steps to block Friend Connect.

I suggest that because they have been walled gardens for so long, neither Facebook nor MySpace really know how to succeed as middleware. They have always been the destination - never really a component in someone’s workflow. By contrast, Google has always offered services which the user employs en route to a different destination. Google understands this kind of arrangement fundamentally. Expect to see increasingly desperate measures from MySpace and Facebook to retain control while Google quietly grows its Friend Connect service.

Tags: data-availability, Facebook, friend-connect, Google, myspace, oauth, personal-profile-portability, ppp

Here’s an interesting approach. Bernhard Haslhofer at Media Spaces has developed OAI2LOD Server, a system which harvests metadata with OAI-PMH, processes the records to create a triple store and exposes interfaces to this for linked-data clients, SPARQL clients and web-browsers.

According to the web-page:

The OAI2LOD Server exposes any OAI-PMH compliant metadata repository according to the Linked Data guidelines. This makes things and media objects accessible via HTTP URIs and query able via the SPARQL protocol.

I find myself wondering if there is an application for this software in the institutional repositories space. Leaving the SPARQL aspect aside for a moment, note that this system makes resources available via URLs, having harvested metadata via OAI-PMH. I know from experience that there are all kinds of issues with simply identifying a link to a ‘thing or media object’ in many metadata records harvested from institutional repositories, so how well this works in practice remains to be seen. However, this could provide another approach to getting digital objects buried in repositories exposed as resources in the web-architecture. And while I don’t suppose that OAI2LOD is particularly aimed at institutional repositories, the SPARQL & linked-data interfaces do perhaps offer a route for some suitable repositories to participate in the web of data.

I’m also currently working with large, heterogeneous aggregations of metadata from repositories, so I’m curious to see how this software might fit with that kind of dataset. My guess is that this system will work best with collections which already contain some semantic coherence in the sense that it might suit a subject-based repository rather better than an institutional repository, although the three examples demonstrated on the OAI2LOD site are for national libraries.

So, what’s the real value of this software?. There are some perfectly good alternative systems offering triple stores with similar interfaces. And there is plenty of OAI-PMH harvester software out there. I haven’t seen these two things joined together directly in this way before, which is what has piqued my interest initially. But I assume that the real value must lie in the processing of the metadata records (and other information gleaned as part of the OAI-PMH transaction) into the triple store.

Anyway - it’s an interesting idea coupled with some working code - always a valuable thing in my book!

Tags: linked-data, OAI-PMH, OAI2LOD, SPARQL

Back in February I was asked to give a talk to the JISC Digitisation Programme meeting. I blogged about this shortly beforehand asking for comments and suggestions. The response was fantastic - I received a bunch of great suggestions and incorporated many of them into the presentation. Everyone who commented got a public ‘thankyou’ at the event, and I included all names in the slides I used.

I have finally gotten around to making the slides available (someone who was at the meeting has asked for them so they made some sort of impression with someone!).

Thanks again.

Tags: Digitisation, harvesting, JISC

For some time now I have occasionally advised people involved in repository administration that they should consider registering the Base URL of their OAI-PMH interface (if they have one) with Google as a proxy for a Sitemap. Until recently, Google has supported the use of OAI-PMH Base URLs in its Webmaster Tools which site owners can use to create and register sitemaps in order to give hints about the structure of the website to Google’s web-crawler.

A while ago, I noticed that there was no longer any reference to this particular support in any of the documentation and began to suspect that this was being deprecated. Today, Google announced via their official blog that:

…we’ve found that the information we gain from our support of OAI-PMH is disproportional to the amount of resources required to support it. Fewer than 200 sites are using OAI-PMH for Google Sitemaps at the moment.

In order to move forward with even better coverage of your websites, we have decided to support only the standard XML Sitemap format by May 2008. We are in the process of notifying sites using OAI-PMH to alert them of the change.

Fewer than 200 sites…..

There are a few ways of looking at this. Perhaps ‘open access’ repositories are less concerned with Google rankings than the typical website owner. Perhaps the penetration of OAI-PMH in the world is still below any level that Google could find particularly interesting - certainly they never went to great lengths to advertise this support while it lasted. Clearly, Google have come to the end of a ‘trial period’ for their support for this protocol in their main indexing service.

Can we conclude anything from this? Probably not - surely OAI-PMH can thrive without Google Sitemap support? It certainly plays a fairly significant part in my professional life at present! Or should we view this as a symptom of decline….?

The official Google announcement is here.

Tags: Google, OAI-PMH, Sitemap

In a recent post, Facebook Or Twitter - Or Facebook And Twitter , Brian Kelly says:

…in some circle such use of Facebook is being derided with comments such as “It’s a closed garden“, “Its popularity is on the wane” or “Twitter is a better development environment” being made. I have to say that I foind that such comments tend to miss the point.“.

Brian tackles the “popularity on the wane” comment with some web statistics, but leaves the “closed garden” and “better development environment” arguments. I’m not at all sure what the argument is about development environments, but I am very interested in the walled garden aspect - I wrote about this in July last year, and I have seen nothing since to change my mind. I’m not sure I’m deriding Facebook, but I do maintain that it is a walled garden. I still keep an account in Facebook out of interest but I rarely access it.

I attended a session on digital libraries earlier this week at the JISC conference, at which Lorcan Dempsey spoke about how where once the user built their workflow around the library, now the library must build services which fit into the user’s workflow. Facebook, it seems to me, is a destination. I go there sometimes, almost always because someone has uploaded some photos of an event I have attended. I go there for occasional amusement. According to the figures, Facebook is very successful at being a destination. But is it embedded in anyone’s workflow I wonder? Twitter is very much part of my workflow - it is the single most used application on my iPhone.

Twitter is an eminently ‘composable‘ service by design, while Facebook is an attractive (for many) destination. Twitter participates in any number of mashups, and has, given rise to an extraordinary range of user-interfaces. It fits into people’s workflows because they can choose how to access it. I use a combination of the mobile web interface and SMS: others use these and a variety of rich desktop interfaces.

So I think my response is still: use Twitter and Facebook, or both, or neither. But I believe that Twitter is more interesting, really because it’s composable nature will allow it to fit all kinds of workflows.

Your mileage may vary

Tags: composable, Facebook, Twitter

The headline to Guy Dixon’s post on vnunet.com is “Facebook user numbers fall in the UK”. The sub-title is: “Social networking fatigue sets in at last”.

I don’t think the one follows the other. I think that what we are really seeing is simply Facebook Fatigue.

I felt the first effects of Facebook Fatigue months ago and stopped actively using it although I still respond to the alerts that it sends me about people communicating with me in some way. I wouldn’t want to be rude!

When Facebook announced their platform which would allow third parties to deploy applications within the Facebook environment, I was momentarily interested, until I had a closer look, and concluded:

I just can’t get all that excited about facebook as a platform. From my point of view, in an exciting era of mashups, facebook is only seriously mashable in one direction, and it’s the wrong direction. If facebook’s social networks were exposed to the web, ‘mine-able’ and mashable - now that would be exciting.

I have yet to see a compelling use of Facebook’s platform. Very quickly, during the period I actually used Facebook, I decided that the only value it offered me was the status and news updates provided by my ‘Facebook friends’. The RSS output facility for these allowed me follow them without having to log in to the application proper. Of course, this relegated me to the status of ‘lurker‘ but I was already getting bored with the thing anyway.

I was actually alerted to the possibility of more widespread Facebook Fatigue by someone ‘tweeting’ about it on Twitter. Twitter is, from my point of view, interesting in all the ways that Facebook just isn’t. From one point of view, Twitter provides the social network and ’status updates’ functionality of Facebook, and nothing else. Importantly, it does so in an open way - it has a very good (and simple) API which has allowed a number of applications which use Twitter to spring up already. Where everything developed for the Facebook platform is only usable within Facebook itself, Twitter-based applications can be deployed anywhere.

I’m a fan of Twitter. It took me a while to ‘get it’, but now it is becoming increasingly useful to me. It’s my virtual ‘water-cooler’, where I catch up on the gossip in my network. It’s my alerting system for breaking news. It’s agile - I can easily start/stop following people. Now I can do the same to ‘tags’ - if I get interested in something, I’ll follow it for a while, then stop. By embracing the constraint of the 140 character limit per post, we get a very different communication channel - one which seems to fit a need for an increasing number of people. Where my network on Facebook peaked to a plateau quite early on, on Twitter I’m gaining new contacts frequently.

Facebook has failed (so far) to get embedded, in systems, workflow or practice on a large scale. Given it’s massive user-base, this is interesting. Facebook seems to want to be the destination, and the only destination. Twitter is already both destination and component - I now habitually turn to Facebook to see what my network thinks about the latest news for example, and have already started roughing up application which could use Twitter to add to my ‘finely-tuned antennae‘.

If you haven’t already, give Twitter a try.

Social networking fatigue? We’re only just warming up!

Tags: Facebook, facebook-fatigue, Guy Dixon, Twitter

In Repositories thru the looking glass over on the eFoundations blog, Andy Powell gives a summary of a keynote he gave to the Vala Conference last week. It’s interesting stuff, and I will take the time to look at the presentation slides as well. I mostly agree (vehemently in some instances) with Andy’s points, though I do find myself questioning some parts of this, so I’ll quote some snippets and make a few comments here.

Firstly, that our current preoccupation with the building and filling of ‘repositories’ (particularly ‘institutional repositories’) rather than the act of surfacing scholarly material on the Web means that we are focusing on the means rather than the end (open access)

It’s hard to deny that there is a current preoccupation with establishing repository systems of one kind or another and populating them with content, and also that there is a focus on institutional deployments. However, I’m not convinced that open access is (or at least is going to remain) the sole driver behind the development of institutional repositories. From an institutional perspective, it absolutely makes sense to want to manage the outputs of research conducted within the auspices of that institution.

A common use for an institutional repository is to house eprints. Were it not for the open-access imperative, we might have expected software designed to manage eprints to fall somewhere between a document-management and a content-management system - both familiar to a large number of institutions. I think it is interesting that it might be considered to be open-access which has skewed the development of repository software in some respects - the community has largely started from scratch, building repository software, where it might have made more sense to simply adapt what was there.

So I half agree with Andy - we do seem to be focussed on the means, but I think I am sympathetic to those (institutions at least) who find themselves pre-occupied with this.

Secondly, that our focus on the ‘institution’ as the home of repository services is not aligned with the social networks used by scholars, meaning that we will find it very difficult to build tools that are compelling to those people we want to use them. As a result, we resort to mandates and other forms of coercion in recognition that we have not, so far, built services that people actually want to use. We have promoted the needs of institutions over the needs of individuals.
Instead, we need to focus on building and/or using global scholarly social networks based on global repository services.

There are four sentences here, and I completely agree with the first three and a half! I find myself wondering who ‘we’ are in this. Now that institutional repositories are becoming a reality, the ‘we’ is going to expand to include people who simply have institutional interests - who have no real interest in open-access for example beyond it being a requirement for them to support. The MIS Manager of your average institution, for example, will start to get involved once institutional repositories get embedded into the business which is a university. The half sentence I don’t quite buy is the “global repository services”. Why can’t we “focus on building and/or using global scholarly social networks” (which I support) based on institutional repository services? We don’t have a problem with institutional web sites do we? Or institutional library OPACs? We have certainly managed to network the latter on a global scale, and built interesting services around this….

Finally, that the ’service oriented’ approaches that we have tended to adopt in standards like the OAI-PMH, SRW/SRU and OpenURL sit uncomfortably with the ‘resource oriented’ approach of the Web architecture and the Semantic Web. We need to recognise the importance of REST as an architectural style and adopt a ‘resource oriented’ approach at the technical level when building services.

Absolutely - couldn’t agree more. Yesterday, at a JISC committee meeting, I argued that a resource-oriented-architecture and the service-oriented-approaches being encouraged by the e-Framework could complement each other if intelligently and judiciously applied. Incidentally, last Friday, I attended an excellent CRIG workshop devoted to exploring the relevance of ReST to repositories. Matt Zumwalt of MediaShelf showed a working ReST interface on Fedora, and Oxford University’s Ben O’Steen used this to develop a client app, in real time, in Python.

I think we agree that the individual’s interests may often be orthogonal to those of the institution. This may have always been the case but it is, perhaps, increasingly an issue as recent developments and trends on the Web empower the individual at an accelerating rate. I wonder if the user-centric/institutional/global debate around repositories is just symptomatic of a tension about to become apparent all over the (institutional) Web?

Having said all this, when visiting the outer limits of repository software development, I am occasionally reminded of the Knight:

‘I see you’re admiring my little box.’ the Knight said in a friendly tone. `It’s my own invention — to keep clothes and sandwiches in. You see I carry it upside-down, so that the rain can’t get in.’
‘But the things can get OUT,’ Alice gently remarked. Do you know the lid’s open?

(from Alice Through the Looking Glass, via Project Gutenberg)

Tags: Andy Powell, Ben O'Steen, Matt Zumwalt, repositories, ReST

Facebook, Google And Plaxo Join The DataPortability Workgroup. So, parts of the blogosphere are quite excited by the news that Facebook, previously criticised for being a closed system, has agreed to join the DataPortability Workgroup. According to Duncan Riley, the author of this TechCrunch post:

The DataPortability Workgroup is actively working to create the ‘DataPortability Reference Design’ to document the best practices for integrating existing open standards and protocols for maximum interoperability (and here’s the key area) to allow users to access their friends and media across all the applications, social networking sites and widgets that implement the design into their systems.

Of course we can only speculate on the real significance of this. While Facebook would not be impossible to copy and compete with a technical level, especially for someone with the resources of Google or Yahoo or the like, it is the established user-base, and consequently the brand, which would be very difficult to match.

It seems clear to me that Facebook has been looking for ways to open up access to its data - it has to in order to be able to exploit it beyond the simple ‘page view’ advertising model. But Facebook has already found this difficult - witness the strange mistakes they made with RSS and the Beacon debacle.

What the participation in the DataPortability initiative gains Facebook is three-fold:

1. Facebook becomes immune to the ‘walled-garden’ accusation for the short/medium term
2. Facebook is not alone in navigating the uncharted territory of social network portability
3. If DataPortability actually delivers, it puts competitors on the same playing field as Facebook - rather than Facebook being scrutinised and expected to lead the way

However this pans out, it is certainly a boost for the credibility of the DataPortability initiative - I’ll be keeping an eye on this now.

Tags: DataPortability, Facebook

I’ve been having a look at Ruby on Rails again, now that version 2 has been released. Generally, I like what I find in this new release. Rails is opinionated software - it isn’t afraid to commit to an approach, rather than trying to be ‘all things to all men’. In a way, this is a continuation of the convention over configuration philosophy which underpins much of Rails’ design. For example SOAP services support has been moved out of the main release and made available as a plugin, while Rails has been refactored to support ReST functionality by default via ActiveResource.

One major change in Rails 2 does strike me a somewhat odd though: the decision to remove the ‘dynamic scaffolding’ from Rails 2 removes one of the very things which make Rails attractive to me. Dynamic scaffolding is the functionality which, in a Model View Controller (MVC) framework (like Rails), allows the developer to prototype the models and have controllers and views which support basic create, update and delete (CRUD) operations generated at run time. I have used Rails successfully as a very rapid prototyping framework, knocking up a basic web-application in real time, with the ‘customer’ sat beside me while I did so. Rails was the first tool I found which allowed me to do this - and it worked brilliantly, in part due to dynamic scaffolding. The thinking behind the removal of this, to my mind, extraordinarily useful functionality is unclear to me. It is explored by Steven Harms in his post, Is the rails 2.0 scaffold system philosophically ( not technically? ) broken?.

This post generated quite a long comment stream - much of which seems to miss the point and go off on tangents about what should or should not be ‘allowed’ to change in a major upgrade. Other comments point out that the developer can’t learn Rails properly if they are using dynamic scaffolding because the code is invisible to them. Somehow it doesn’t seem to matter that part of the appeal of Rails is how much you don’t have to learn. Martin-234 asks, reasonably:

Many older tutorials (and books) suggested a method of working models and relations like: generate model, scaffold, migrate, check, migrate, check… (e.g. the original blog-video and the depot tutorial) [...]What is the new preferred way of working?

I’m with Martin. While I understand that experienced rails developers might eschew dynamic scaffolding altogether, it has been a major factor in attracting new developers to Rails to date - I’m puzzled that it would be left out altogether now.

This story has a happy ending however. Having a look around, I found reference to the Scaffolding Extensions Plugin for Rails. Originally designed to augment and extend the existing dynamic scaffolding in Rails 1.x, it occurred to me to try it as a replacement for this lost functionality. Having installed it I immediately hit the following error:

ActionController::InvalidAuthenticityToken in CrudController

I mailed the developer, Jeremy Evans, with this issue and as much detail about my environment as I thought could be useful, and I’m pleased to say that I got an exemplary response with an explanation within 6 hours, followed by another response a couple of hours later with a patch attached to the message. (I won’t post the patch here as Jeremy is working on a fix for the codebase right now). Havin followed Jeremy’s instructions and applied the patch he supplied, this plugin is now working for me - which means I have dynamic scaffolding back, and better than before!

First-class support for open-source software - I urge any readers missing a dynamic-scaffolding capability for Rails 2 to look no further than Scaffolding Extensions Plugin for Rails.

Update: the the Scaffold Extensions plugin has now been updated (rev 89) to fix the issue I reported above.

Tags: dynamic scaffolding, MVC, Rails2