paul walk’s weblog

In Repositories thru the looking glass over on the eFoundations blog, Andy Powell gives a summary of a keynote he gave to the Vala Conference last week. It’s interesting stuff, and I will take the time to look at the presentation slides as well. I mostly agree (vehemently in some instances) with Andy’s points, though I do find myself questioning some parts of this, so I’ll quote some snippets and make a few comments here.

Firstly, that our current preoccupation with the building and filling of ‘repositories’ (particularly ‘institutional repositories’) rather than the act of surfacing scholarly material on the Web means that we are focusing on the means rather than the end (open access)

It’s hard to deny that there is a current preoccupation with establishing repository systems of one kind or another and populating them with content, and also that there is a focus on institutional deployments. However, I’m not convinced that open access is (or at least is going to remain) the sole driver behind the development of institutional repositories. From an institutional perspective, it absolutely makes sense to want to manage the outputs of research conducted within the auspices of that institution.

A common use for an institutional repository is to house eprints. Were it not for the open-access imperative, we might have expected software designed to manage eprints to fall somewhere between a document-management and a content-management system - both familiar to a large number of institutions. I think it is interesting that it might be considered to be open-access which has skewed the development of repository software in some respects - the community has largely started from scratch, building repository software, where it might have made more sense to simply adapt what was there.

So I half agree with Andy - we do seem to be focussed on the means, but I think I am sympathetic to those (institutions at least) who find themselves pre-occupied with this.

Secondly, that our focus on the ‘institution’ as the home of repository services is not aligned with the social networks used by scholars, meaning that we will find it very difficult to build tools that are compelling to those people we want to use them. As a result, we resort to mandates and other forms of coercion in recognition that we have not, so far, built services that people actually want to use. We have promoted the needs of institutions over the needs of individuals.
Instead, we need to focus on building and/or using global scholarly social networks based on global repository services.

There are four sentences here, and I completely agree with the first three and a half! I find myself wondering who ‘we’ are in this. Now that institutional repositories are becoming a reality, the ‘we’ is going to expand to include people who simply have institutional interests - who have no real interest in open-access for example beyond it being a requirement for them to support. The MIS Manager of your average institution, for example, will start to get involved once institutional repositories get embedded into the business which is a university. The half sentence I don’t quite buy is the “global repository services”. Why can’t we “focus on building and/or using global scholarly social networks” (which I support) based on institutional repository services? We don’t have a problem with institutional web sites do we? Or institutional library OPACs? We have certainly managed to network the latter on a global scale, and built interesting services around this….

Finally, that the ’service oriented’ approaches that we have tended to adopt in standards like the OAI-PMH, SRW/SRU and OpenURL sit uncomfortably with the ‘resource oriented’ approach of the Web architecture and the Semantic Web. We need to recognise the importance of REST as an architectural style and adopt a ‘resource oriented’ approach at the technical level when building services.

Absolutely - couldn’t agree more. Yesterday, at a JISC committee meeting, I argued that a resource-oriented-architecture and the service-oriented-approaches being encouraged by the e-Framework could complement each other if intelligently and judiciously applied. Incidentally, last Friday, I attended an excellent CRIG workshop devoted to exploring the relevance of ReST to repositories. Matt Zumwalt of MediaShelf showed a working ReST interface on Fedora, and Oxford University’s Ben O’Steen used this to develop a client app, in real time, in Python.

I think we agree that the individual’s interests may often be orthogonal to those of the institution. This may have always been the case but it is, perhaps, increasingly an issue as recent developments and trends on the Web empower the individual at an accelerating rate. I wonder if the user-centric/institutional/global debate around repositories is just symptomatic of a tension about to become apparent all over the (institutional) Web?

Having said all this, when visiting the outer limits of repository software development, I am occasionally reminded of the Knight:

‘I see you’re admiring my little box.’ the Knight said in a friendly tone. `It’s my own invention — to keep clothes and sandwiches in. You see I carry it upside-down, so that the rain can’t get in.’
‘But the things can get OUT,’ Alice gently remarked. Do you know the lid’s open?

(from Alice Through the Looking Glass, via Project Gutenberg)

Tags: Andy Powell, Ben O'Steen, Matt Zumwalt, repositories, ReST

Facebook, Google And Plaxo Join The DataPortability Workgroup. So, parts of the blogosphere are quite excited by the news that Facebook, previously criticised for being a closed system, has agreed to join the DataPortability Workgroup. According to Duncan Riley, the author of this TechCrunch post:

The DataPortability Workgroup is actively working to create the ‘DataPortability Reference Design’ to document the best practices for integrating existing open standards and protocols for maximum interoperability (and here’s the key area) to allow users to access their friends and media across all the applications, social networking sites and widgets that implement the design into their systems.

Of course we can only speculate on the real significance of this. While Facebook would not be impossible to copy and compete with a technical level, especially for someone with the resources of Google or Yahoo or the like, it is the established user-base, and consequently the brand, which would be very difficult to match.

It seems clear to me that Facebook has been looking for ways to open up access to its data - it has to in order to be able to exploit it beyond the simple ‘page view’ advertising model. But Facebook has already found this difficult - witness the strange mistakes they made with RSS and the Beacon debacle.

What the participation in the DataPortability initiative gains Facebook is three-fold:

1. Facebook becomes immune to the ‘walled-garden’ accusation for the short/medium term
2. Facebook is not alone in navigating the uncharted territory of social network portability
3. If DataPortability actually delivers, it puts competitors on the same playing field as Facebook - rather than Facebook being scrutinised and expected to lead the way

However this pans out, it is certainly a boost for the credibility of the DataPortability initiative - I’ll be keeping an eye on this now.

Tags: DataPortability, Facebook

I’ve been having a look at Ruby on Rails again, now that version 2 has been released. Generally, I like what I find in this new release. Rails is opinionated software - it isn’t afraid to commit to an approach, rather than trying to be ‘all things to all men’. In a way, this is a continuation of the convention over configuration philosophy which underpins much of Rails’ design. For example SOAP services support has been moved out of the main release and made available as a plugin, while Rails has been refactored to support ReST functionality by default via ActiveResource.

One major change in Rails 2 does strike me a somewhat odd though: the decision to remove the ‘dynamic scaffolding’ from Rails 2 removes one of the very things which make Rails attractive to me. Dynamic scaffolding is the functionality which, in a Model View Controller (MVC) framework (like Rails), allows the developer to prototype the models and have controllers and views which support basic create, update and delete (CRUD) operations generated at run time. I have used Rails successfully as a very rapid prototyping framework, knocking up a basic web-application in real time, with the ‘customer’ sat beside me while I did so. Rails was the first tool I found which allowed me to do this - and it worked brilliantly, in part due to dynamic scaffolding. The thinking behind the removal of this, to my mind, extraordinarily useful functionality is unclear to me. It is explored by Steven Harms in his post, Is the rails 2.0 scaffold system philosophically ( not technically? ) broken?.

This post generated quite a long comment stream - much of which seems to miss the point and go off on tangents about what should or should not be ‘allowed’ to change in a major upgrade. Other comments point out that the developer can’t learn Rails properly if they are using dynamic scaffolding because the code is invisible to them. Somehow it doesn’t seem to matter that part of the appeal of Rails is how much you don’t have to learn. Martin-234 asks, reasonably:

Many older tutorials (and books) suggested a method of working models and relations like: generate model, scaffold, migrate, check, migrate, check… (e.g. the original blog-video and the depot tutorial) [...]What is the new preferred way of working?

I’m with Martin. While I understand that experienced rails developers might eschew dynamic scaffolding altogether, it has been a major factor in attracting new developers to Rails to date - I’m puzzled that it would be left out altogether now.

This story has a happy ending however. Having a look around, I found reference to the Scaffolding Extensions Plugin for Rails. Originally designed to augment and extend the existing dynamic scaffolding in Rails 1.x, it occurred to me to try it as a replacement for this lost functionality. Having installed it I immediately hit the following error:

ActionController::InvalidAuthenticityToken in CrudController

I mailed the developer, Jeremy Evans, with this issue and as much detail about my environment as I thought could be useful, and I’m pleased to say that I got an exemplary response with an explanation within 6 hours, followed by another response a couple of hours later with a patch attached to the message. (I won’t post the patch here as Jeremy is working on a fix for the codebase right now). Havin followed Jeremy’s instructions and applied the patch he supplied, this plugin is now working for me - which means I have dynamic scaffolding back, and better than before!

First-class support for open-source software - I urge any readers missing a dynamic-scaffolding capability for Rails 2 to look no further than Scaffolding Extensions Plugin for Rails.

Update: the the Scaffold Extensions plugin has now been updated (rev 89) to fix the issue I reported above.

Tags: dynamic scaffolding, MVC, Rails2

Correction: In a comment, Justin Crites points out, correctly, that Simple DB does not offer ‘relational’ database functionality, in the sense that it is not an RDBMS. While this is true, I think Simple DB clearly offers functionality which many people get, perhaps inappropriately, from a relational database system.

Amazon continue to expand their excellent infrastructure web services. Simple DB - relational databases in the cloud. I’m definitely going to have a look at this - clustering/replicating databases for resilience and performance was extraordinarily difficult when I was developing and supporting enterprise systems a couple of years ago. In fact, more often than not, we’d cluster and load-balance all the layers in the enterprise stack except for the database, and then just rely on a singe very expensive box running Oracle Enterprise 9i or whatever the equivalent is nowadays.

Like the other Amazon infrastructure services (S3, EC2 you pay for what you use. Linear costs. Simple. Brilliant.

Tags: Amazon, EC2, S3, SimpleDB

I’ve just listened to a podcast of David Heinemeier Hansson’s keynote at RailsConf 2007 (which actually took place back in May of this year). David describes the changes and new features being introduced into Rails 2.0. Firstly, he is at pains to point out that Rails 2.0 will not represent a radical change, or a complete re-write. Also, 95% of what will constitute Rails 2.0’s new features are already available in the bleeding edge EdgeRails and are being actively used. To use his phrase, Rails 2.0 is no ‘unicorn’ - it is not a mythical beast.
Having said this, there do appear to be some fairly important changes in Rails 2.0. Rails has for a while had an effective plugin system. Plugins are relatively easy to write for Rails and so good, new ideas can be disseminated and incorporated into working code very quickly. The popular, successful plugins can then be rolled into Rails ‘core’. What I like about Rails is that this can work the other way too: after time, things which no longer seem to belong in Rails core can be removed and made available as plugins. It is very, very easy to make a plugin available to your Rails installation so this type of deprecation strategy is kind to the developer, especially if they are using legacy approaches or code. And if nothing else, Rails is all about being kind to the developer. Significantly, the ActiveResource plugin which drops a full ReST framework into Rails is in, while the ActiveWebService functionality to support SOAP is out although, of course, still easily available as a plugin. This is really about making a statement, something neither Rails, nor it creators and developers are generally shy about doing. As David puts it in his keynote (my transcript from the audio):

Unlike a lot of frameworks which pride themselves on being all about flexibility, which pride themselves on being all about being neutral, neutral to choices, being Switzerland in other words….Rails is not. Rails is not Switzerland. Rails has an opinion.

It was through engaging with Ruby on Rails that I first encountered the design paradigm of Convention over Configuration. As a veteran Java programmer I immediately saw the appeal of this. Of course Java programmers have historically favored the “abstract enabling-framework” over the “direct problem-solving” approach, so it can be difficult to adjust from one to the other. Michael Coté describes something similar when he talks about Java’s Fear of Commitment.
This is what makes Rails interesting for me. It has an opinion. It is not ashamed to show a strong preference for a way of doing things. As a developer, you can still choose to do things differently (and sometimes this is just unavoidable) but you get so much for free by just following the conventions.

David also characterised Rails as having ‘friends’ and ‘allies’. Components or aspects which Rails strongly supports, such as ReST and AJAX are friends. Those which it is starting to support, but which still need to prove themselves are allies - among these David listed Atom and OpenID.

Interestingly, Rails will now have simple support for HTTP Basic Authentication. Previously, Rails developers have eschewed this non-user friendly approach to securing their applications, as it fails to offer the user features which they have come to expect, such as “remember me” and “mail me my password because I’ve forgotten it” options. However, it is now recognised that HTTP Basic Authentication is useful for non-user mediated authentication, such as client access to secure APIs. David cites, as an example, the RSS reader which typically can store and use a user’s credentials on their behalf t access a secured RSS feed. Eventually, perhaps, this will be made redundant by newer approaches such as OAuth, but I can immediately see how this could be useful in developing mashups in the meantime.

Technorati Tags: Convention-over-configuration, David-Heinemeier-Hansson, Java, OAuth, Rails, RailsConf, REST, Ruby-on-Rails

A couple of years ago, Dave Winer posted an article in which he outlined the notion of a River of News, describing the use of an RSS aggregator to simply scroll through the latest new items from many sources (or categories), all merged together in one ’stream’. I don’t tend to use my RSS reader (the excellent NetNewsWire) in this way, preferring to browse particular sources as the fancy takes me. But this is for news, serious comment, articles etc. What about more lightweight uses?

I’ve been using Facebook and Twitter for a few weeks now (I’ve actually been subscribed to Facebook for far longer, in the sense that I created an account and had a cursory look at it more than a year ago, but I’ve been actively using them recently). I’m quite persuaded of the usefulness of Twitter in terms of its functionality. What I’ve found is that I’m using Facebook in the same way. I now ignore the multitudinous applications which have been developed on Facebook’s much-vaunted Platform. But I’m still reading my friends’ status-updates, along with occasional invitation to an events.

Now it turns out, as I described recently, that I can get this information from Facebook in RSS format. Twitter also allows me to subscribe to an RSS feed of the stream of posts from the people that I’m ‘following’. The only reason I’m still bothering with Facebook is that it is used by quite a few people that I care about and who aren’t using Twitter.

So with both Facebook and Twitter, what I really want is to be able, from time to time, to read the gossip from my friends. I don’t want to have to really care whether they are using Facebook or Twitter (or whatever comes along next to join or usurp these).

And so to Yahoo’s Pipes. It seemed to me this morning that it would be useful to be able to merge these two tributaries from Facebook and Twitter into one combined river of gossip. Of course I can approximate this by creating a group of feeds in my RSS reader. But it occurred to me that Pipes offered a neater solution. And so, using Pipes, I created a merged feed of RSS feeds from Facebook and Twitter, and then subscribed to this feed in the normal way.

From realising the need, to figuring out a simple solution, to actually implementing the solution with Pipes, took less than ten minutes. If you haven’t yet had a play with Pipes, you really should give it a go. My solution is completely disposable - I have had to install nothing.

It’s clear that the availability of a machine API to a service can utterly change your human experience of that service. Mike Ellis, a long term user of TechCrunch, noted his surprise at how poor its user interface was - his experience of TechCrunch had been mostly through its RSS feed. I don’t think that Facebook’s user interface is bad - I think it’s actually, objectively speaking, quite good. But I’m happy, nonetheless, that most of the attention I give to Facebook can be through an interface I have available all the time, whether online or offline - my RSS reader. When I want to tell my friends what I’m up to, or send out a general plea for help etc. I can use Facebook or Twitter’s respective websites. But when I simply want to access the river of gossip, I’m more than happy to use a tool I use all the time, my RSS reader.

Technorati Tags: Dave Winer, Facebook, river of gossip, river of news, RSS, Twitter, Yahoo Pipes

Brian alerted me to a post by Dave Winer, called Facebook *is* opening up. It’s true! Some Facebook content is now easily available as RSS feeds: for example a feed of items which have been posted by my ‘friends’. Note how you don’t need to be logged in to Facebook (or have an account for that matter) to use this. Dave reports that, in fact, these feeds have been available for some time. Nonetheless, as he says:

According to convention wisdom, Facebook was, until today, considered a sandbox, a walled garden, a silo. Now that we know that the feeds are being implemented (many are still needed to make it really open) it’s possible for Facebook-generated data to percolate into other Internet applications. As Fred Wilson has wisely pointed out, there is no winner-take-all outcome possible, and closed sandboxes just encourage route-arounds, so what Facebook is doing is smart and necessary.

For me, Facebook just became considerably more interesting. As I said in a previous post:

If facebook’s social networks were exposed to the web, ‘mine-able’ and mashable - now that would be exciting.

I’m not excited just yet…. we need to see more ‘opening up’, but I’m watching with real interest now.

UPDATE: The default privacy setting for most of the content a user can add to Facebook appears to be “All my networks and all my friends“. This is the most ‘open’ of the settings possible - there is no ‘public’ or ‘absolutely everyone’ setting. So when when one of my friends changes their status message for example they might, if they care at all, be under the impression that this can only be viewed by their friends (including me) and people in their network(s). If I then go and publish the feed URL to the world, this information is now available without restriction.

Have I betrayed the trust of my ‘friends’ by making such an RSS feed available? Is this model broken?

Technorati Tags: Facebook, privacy, RSS, security

Quite a bit of excitement has been generated by Blaise Aguera y Arcas’s Jaw-dropping Photosynth demo. The demo starts with a quick run-through of the capabilities of Seadragon, now owned by Microsoft, and then follows this up with a demonstration of Photosynth. I think it’s rare to hear a round of applause in the middle of a technology demonstration….

Photosynth is the real jaw-dropper, and I’m inspired by the promise of being able to construct elaborate 3D models of spaces from the aggregation of many, tagged, 2D images in systems like Flickr. But it’s Seadragon, which takes progressive refinement to another level which has really piqued my interest, because of its potential for general applicability. The embedding of an advert into the Guardian newspaper page was a fascinating moment in the demo - I can imagine all kinds of ways in which this approach could start to challenge our notions of information and information linking. Taken to a (possibly ridiculous) extreme, within a single website this approach could do away with the need for linking at all from the user’s perspective - where we currently have a hyperlink for example, this kind of technology could allow the ‘linked’ information to be simply embedded at a ‘deeper’ level of resolution.

Anyway - it’s just a technology preview/demonstration at this point, albeit a very slick one. Good to see Microsoft innovating - or acquiring innovation at least….

Windows only. Makes me wish I’d bothered to install a Windows emulator on my MacBook, and I don’t find myself thinking that very often!

Technorati Tags: Blaise Aguera y Arcas, Microsoft, Photosynth, progressive refinement, Seadragon

I came across this quote:

The coolest thing to do with your data will be thought of by someone else

attributed to Rufus Pollack, in a in a post on the Talis blog, Nodalities.

Absolutely, couldn’t agree more. The point is not so much whether this statement might be true or not, so much as what it does to your thinking and planning if you decide to take it as an article of faith. This reminds me of conversations I had with colleagues last year when I was working on the XCRI project (a project to develop a schema and related tools for the creation and management of standard and interoperable course catalogue data). We started to evolve the idea that Universities should consider the approach of opening up the non-contentious data in their considerable internal management information systems, slapping public facing APIs on to these data sources and inviting the world in to use what they could. We surmised that:

• the barriers and costs to doing this kind of thing were rapidly diminishing
• the possible gains, in terms of new business opportunities, partnerships etc. might be worth this small investment.
• the more you did this, the more chance of discovering a new opportunity

During many a debate about addressing all kinds of thorny issues surrounding IPR, security, access control etc. we wondered aloud why we didn’t start with the peripheral stuff - the data which didn’t need to be secured or controlled in these ways. We recognised that this sort of development had long since taken place with other systems in a typical university - notably with library catalogues for example. So why not with course catalogues?

In business terms, this kind of activity might, perhaps, be described as a loss leader - giving something away for free in the hope that it might open up a channel to more profitable business. Amazon did this long ago with their E-Commerce Service, and it seems to be working well for them! Crucially, they go out of their way to show-case the solutions built by other, third-party developers on top of the Amazon services.

As me move into a world of institutional repositories, with Universities beginning to accept the benefits of providing open-access to scholarly output, will we see this trend extend to other types of less sector-specific information? Some quick examples might include accommodation details and bookings, expertise directories, course catalogues(XCRI!), calendars of public lectures etc.

Technorati Tags: AWS, management information, open data, Rufus Pollack, Talis, universities, XCRI

An interesting post from Philipp Keller on Tag history and gartners hype cycles from back in May of this year which I missed first time around. Now part of me thinks it must be possible to plot just about anything on the Gartner Hype Cycle, but it can be a useful tool for provoking reflection and discussion.

Note how Philipp indicates that we now find ourselves in the Trough of Disillusionment in 2007. Gartner define this stage in the cycle thus:

Technologies enter the “trough of disillusionment” because they fail to meet expectations and quickly become unfashionable. Consequently, the press usually abandons the topic and the technology.

I’m not sure that this is where we are.

Phillipp suggests that:

There are no blog posts any more. Tagging is not really unfashionable but the topic is “done” à la «if that’s all what’s tagging adds to the web experience, I’m not interested in this technology any more». There isn’t much thinking and innovation going on.

I half agree. I agree that tagging is not unfashionable, but that people aren’t blogging about it to the same degree. But I can’t help wondering if we haven’t already reached the Plateau of Productivity - the last stage in the Gartner curve and absent from Phillipp’s diagram. Surely del.icio.us and Flickr can be considered mainstream and accepted?

Did we go through a Trough of Disillusionment already? Or does the Gartner curve not really apply in this instance?

Technorati Tags: folksonomies, Gartner Hype Cycle, Philipp Keller, Tagging, Trough of Disillusionment