Comments on: Google gives up on supporting OAI-PMH for Sitemaps http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/ Sun, 12 Oct 2008 20:31:43 +0000 http://wordpress.org/?v=2.6.1 By: da blog (ulcc digital archives blog) » Blog Archive » OAI-PMH: decline or fall? http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-965 da blog (ulcc digital archives blog) » Blog Archive » OAI-PMH: decline or fall? Wed, 07 May 2008 14:29:08 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-965 [...] have gleefully rejoiced in the possible demise of the standard. However the discussion on Paul Walk’s blog is more balanced, lucid and informative, and covers well the many areas where OAI-PMH is more [...] [...] have gleefully rejoiced in the possible demise of the standard. However the discussion on Paul Walk’s blog is more balanced, lucid and informative, and covers well the many areas where OAI-PMH is more [...]

]]> By: Owen Stephens http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-943 Owen Stephens Fri, 02 May 2008 14:28:28 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-943 @Wobbler - it depends what 'interoperability' you are looking for. Unless we define how we want repositories to inter-operate, then we can't assess the best tools for the job. OAI-PMH is about 'metadata harvesting' - and in theory provides a lightweight way of an automated agent finding out what is in a repository. However, in reality this has only seen take up within the small world of repositories. Google, Yahoo etc. simply crawl the web following links - they don't see the need to implement a new protocol to deal with a tiny (in web terms) amount of content. Given this, I would argue we should concentrate on how to integrate the material that is stored in repositories into the 'web' environment. Now, this means making the content crawlable and linkable. As the Semantic Web is now getting broader takeup (with the key players such as Yahoo and Google now taking an interest), I would guess that embedding microformats and looking at other semantic web technologies could be a fruitful approach. @Wobbler - it depends what ‘interoperability’ you are looking for. Unless we define how we want repositories to inter-operate, then we can’t assess the best tools for the job.

OAI-PMH is about ‘metadata harvesting’ - and in theory provides a lightweight way of an automated agent finding out what is in a repository. However, in reality this has only seen take up within the small world of repositories. Google, Yahoo etc. simply crawl the web following links - they don’t see the need to implement a new protocol to deal with a tiny (in web terms) amount of content.

Given this, I would argue we should concentrate on how to integrate the material that is stored in repositories into the ‘web’ environment. Now, this means making the content crawlable and linkable. As the Semantic Web is now getting broader takeup (with the key players such as Yahoo and Google now taking an interest), I would guess that embedding microformats and looking at other semantic web technologies could be a fruitful approach.

]]> By: Google dice addio a Oai-Pmh per Sitemaps « The Geek Librarian http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-941 Google dice addio a Oai-Pmh per Sitemaps « The Geek Librarian Tue, 29 Apr 2008 19:58:35 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-941 [...] alla notizia - i pochi apparsi sul blog di Google Webmaster, e soprattutto quelli apparsi sul blog di Paul Walks. Che ne [...] [...] alla notizia - i pochi apparsi sul blog di Google Webmaster, e soprattutto quelli apparsi sul blog di Paul Walks. Che ne [...]

]]> By: Wobbler http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-940 Wobbler Tue, 29 Apr 2008 19:11:51 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-940 I have read that blog post earlier (thanks to your blog). I think it is an interesting post, but unless with "alternative", he means: ditch the concept of institutional repositories altogether and go central repositories (which I do not think is realistic), it does not actually give any alternative approaches to handle this. And assuming "the web", e.g. Google Scholar?, can do it just as good without protocols such as OAI-PMH, why have that protocol in the first place? I am going to assume that it was necessary to optimize interoperability between institutional repositories? I have read that blog post earlier (thanks to your blog). I think it is an interesting post, but unless with “alternative”, he means: ditch the concept of institutional repositories altogether and go central repositories (which I do not think is realistic), it does not actually give any alternative approaches to handle this.

And assuming “the web”, e.g. Google Scholar?, can do it just as good without protocols such as OAI-PMH, why have that protocol in the first place? I am going to assume that it was necessary to optimize interoperability between institutional repositories?

]]> By: paul http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-939 paul Tue, 29 Apr 2008 15:24:37 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-939 Wobbler, I guess the short answer to your question would be 'The Web'. The case has been made for this more strongly elsewhere - I suggest that you have a look at <a href="http://efoundations.typepad.com/efoundations/2008/02/repositories-th.html" rel="nofollow">http://efoundations.typepad.com/efoundations/2008/02/repositories-th.html</a> for one argument along these lines. Wobbler,
I guess the short answer to your question would be ‘The Web’. The case has been made for this more strongly elsewhere - I suggest that you have a look at http://efoundations.typepad.com/efoundations/2008/02/repositories-th.html for one argument along these lines.

]]> By: Wobbler http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-938 Wobbler Mon, 28 Apr 2008 23:43:54 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-938 First I would like to say that since I have found your blog, I have been finding a lot more interesting blogs on scholarly communication/ OA/ repositories. So thanks. Anyway, maybe I have been following the Stevan Harnad/ Alma Swan crowd too much, but I was under the impression that OAI-PMH, in the context of institutional repositories, was currently the best approach to achieve a high level of accessibility of eprints? And also that it is flexible enough to be extended with other metadata (such as comments)? If not, can someone tell me about (better) alternatives for accessibility/ interoperability between institutional repositories? What other protocols are there like OAI-PMH (but better and more flexible)? First I would like to say that since I have found your blog, I have been finding a lot more interesting blogs on scholarly communication/ OA/ repositories. So thanks.

Anyway, maybe I have been following the Stevan Harnad/ Alma Swan crowd too much, but I was under the impression that OAI-PMH, in the context of institutional repositories, was currently the best approach to achieve a high level of accessibility of eprints? And also that it is flexible enough to be extended with other metadata (such as comments)?

If not, can someone tell me about (better) alternatives for accessibility/ interoperability between institutional repositories? What other protocols are there like OAI-PMH (but better and more flexible)?

]]> By: Brad Czerniak http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-930 Brad Czerniak Thu, 24 Apr 2008 23:01:35 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-930 I'm not so sure that sitemaps are going to grow in importance. Their principal importance in the past was due to inadequete spidering. Nowadays, Google's spidering with additional HTTP requests (unfriendly URLs). The only good thing about being Google sitemaps-compliant is the ability to have the most important links exposed in results lists, but some sites get that without being particularly SEO. Also, harvesting over sitemap would be a difficult proposition for large collections, as the heft of the XML layer would be more pronounced than a single-object query via a web services protocol. There's nothing wrong with OAI-PMH necessarily -- there just isn't any "killer app" to really entice people. I guess this might be an instance where "If you can't beat 'em, join 'em" would be sound advice. I’m not so sure that sitemaps are going to grow in importance. Their principal importance in the past was due to inadequete spidering. Nowadays, Google’s spidering with additional HTTP requests (unfriendly URLs). The only good thing about being Google sitemaps-compliant is the ability to have the most important links exposed in results lists, but some sites get that without being particularly SEO.

Also, harvesting over sitemap would be a difficult proposition for large collections, as the heft of the XML layer would be more pronounced than a single-object query via a web services protocol.

There’s nothing wrong with OAI-PMH necessarily — there just isn’t any “killer app” to really entice people. I guess this might be an instance where “If you can’t beat ‘em, join ‘em” would be sound advice.

]]> By: Mike http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-929 Mike Thu, 24 Apr 2008 11:32:25 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-929 The lesson presumably isn't "Google don't do it therefore we shouldn't" but "the fact that only 200 sites signed up say something about the technology which we should listen to"? I keep coming back to the same old cracked record mantra: if a technology doesn't have a wide and fairly rapid uptake, if it isn't easy, if it can't demonstrate benefits to ALL (not just the geek community) within a reasonable time span, then it simply isn't a technology worth pursuing. Knowing not much about OAI The latest Ofcom report seems to back this up. http://tinyurl.com/4vrkcz The lesson presumably isn’t “Google don’t do it therefore we shouldn’t” but “the fact that only 200 sites signed up say something about the technology which we should listen to”?

I keep coming back to the same old cracked record mantra: if a technology doesn’t have a wide and fairly rapid uptake, if it isn’t easy, if it can’t demonstrate benefits to ALL (not just the geek community) within a reasonable time span, then it simply isn’t a technology worth pursuing. Knowing not much about OAI

The latest Ofcom report seems to back this up. http://tinyurl.com/4vrkcz

]]> By: Owen Stephens http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-928 Owen Stephens Thu, 24 Apr 2008 11:28:19 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-928 @pete cliff - but this is a good point, z39.50 has never been widely adopted outside the library sector, and now looks like an increasing barrier to access rather than enabler. I think there is a risk that OAI-PMH is in a similar situation - a sector specific solution, that is seen by many as too complicated and is never going to get widespread acceptance by the wider community. We've got to ask, what does OAI-PMH add - is it achieving it's goal of being "is a low-barrier mechanism for repository interoperability" (from http://www.openarchives.org/pmh/)? To understand this, we have to start dragging the definition of 'repository' apart, but if we (for example) say Flickr is a repository of pictures, you could argue that OAI-PMH does nothing for interoperability between an e-prints repository and Flickr. What OAI-PMH does is enable interoperability between two systems supporting OAI-PMH - and as support for this drops, the likelihood of OAI-PMH being successful also drops. We need to look really hard at this - I'm not convinced that OAI-PMH is 'the future' for repositories, and if not, we need to look at where the future is, and start heading in that direction. @pete cliff - but this is a good point, z39.50 has never been widely adopted outside the library sector, and now looks like an increasing barrier to access rather than enabler.

I think there is a risk that OAI-PMH is in a similar situation - a sector specific solution, that is seen by many as too complicated and is never going to get widespread acceptance by the wider community.

We’ve got to ask, what does OAI-PMH add - is it achieving it’s goal of being “is a low-barrier mechanism for repository interoperability” (from http://www.openarchives.org/pmh/)? To understand this, we have to start dragging the definition of ‘repository’ apart, but if we (for example) say Flickr is a repository of pictures, you could argue that OAI-PMH does nothing for interoperability between an e-prints repository and Flickr. What OAI-PMH does is enable interoperability between two systems supporting OAI-PMH - and as support for this drops, the likelihood of OAI-PMH being successful also drops.

We need to look really hard at this - I’m not convinced that OAI-PMH is ‘the future’ for repositories, and if not, we need to look at where the future is, and start heading in that direction.

]]> By: Stuart Lewis http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-926 Stuart Lewis Thu, 24 Apr 2008 08:34:19 +0000 http://blog.paulwalk.net/2008/04/23/google-gives-up-on-supporting-oai-pmh-for-sitemaps/#comment-926 The webmaster tools can be pretty useful for developers checking everything is OK (see: http://blogs.openaccesscentral.com/blogs/orblog/entry/how_many_how_fast for an example). Over time I suspect sitemaps are going to grow in importance. Ignoring repositories for a moment, most standard webmasters aren't really aware of them, and the benefits they can bring, for example being able to mark certain pages as having a higher priority than others. No doubt once sitemaps become part of your average webmaster's daily toolkit, their importance will grow, and repository people will catch on. Most repository managers have a set of tick boxes they like to check, one of them being inclusion in G and GS. If they know sitemaps will help them tick these boxes, I'm sure they'll be all for them. And from a development perspective, sitemaps are very easy to create (100x easier than writing an OAI-PMH interface) so there is no reason really for repository platforms not to support them out-of-the-box. The webmaster tools can be pretty useful for developers checking everything is OK (see: http://blogs.openaccesscentral.com/blogs/orblog/entry/how_many_how_fast for an example).

Over time I suspect sitemaps are going to grow in importance. Ignoring repositories for a moment, most standard webmasters aren’t really aware of them, and the benefits they can bring, for example being able to mark certain pages as having a higher priority than others.

No doubt once sitemaps become part of your average webmaster’s daily toolkit, their importance will grow, and repository people will catch on. Most repository managers have a set of tick boxes they like to check, one of them being inclusion in G and GS. If they know sitemaps will help them tick these boxes, I’m sure they’ll be all for them.

And from a development perspective, sitemaps are very easy to create (100x easier than writing an OAI-PMH interface) so there is no reason really for repository platforms not to support them out-of-the-box.

]]>