Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

May 24 2011

The search for a minimum viable record

Catalogue by Helga's Lobster Stew, on FlickrAt first blush, bibliographic data seems like it would be a fairly straightforward thing: author, title, publisher, publication date. But that's really just the beginning of the sorts of data tracked in library catalogs. There's also a variety of metadata standards and information classification systems that need to be addressed.

The Open Library has run into these complexities and challenges as it seeks to create "one web page for every book ever published."

George Oates, Open Library lead, recently gave a presentation in which she surveyed audience members, asking them to list the five fields they thought necessary to adequately describe a book. In other words, what constitutes a "minimum viable record"? Akin to the idea of the "minimum viable product" for getting a web project coded and deployed quickly, the minimum viable record (MVR) could be a way to facilitate an easier exchange of information between library catalogs and information systems.

In the interview below, Oates explains the issues and opportunities attached to categorization and MVRs.

What are some of the challenges that libraries and archives face when compiling and comparing records?

George OatesGeorge Oates: I think the challenges for compilation and comparison of records rest in different styles, and the innate human need to collect, organize, and describe the things around us. As Barbara Tillett noted in a 2004 paper: "Once you have a collection of over say 2,000 items, a human being can no longer remember every item and needs a system to help find things."

I was struck by an article I saw on a site called Apartment Therapy, about "10 Tiny Gardens," where the author surveyed extremely different decorations and outputs within remarkable constraints. That same concept can be dropped into cataloging, where even in the old days, when librarians described books within the boundaries of a physical index card, great variation still occurred. Trying to describe a book on a 3x5 card is oddly reductionist.

It's precisely this practice that's produced this "diabolical rationality" of library metadata that Karen Coyle describes [slide No. 38]. We're not designed to be rational like this, all marching to the same descriptive drum, even though these mythical levels of control and uniformity are still claimed. It seems to be a human imperative to stretch ontological boundaries and strive for greater levels of detail.

Some specific categorization challenges are found in the way people's names are cataloged. There's the very simple difference between "Lastname, Firstname" and "Firstname Lastname" or the myriad "disambiguators" that can help tell two authors with the same name apart — like a middle initial, a birthdate, title, common name, etc.

There are also challenges attached to the normal evolution of language, and a particular classification's ability to keep up. An example is the recent introduction of the word "cooking" as an official Library of Congress Subject Heading. "Cooking" supersedes "Cookery," so now you have to make sure all the records you have in your catalog that previously referred to "Cookery" now know about this newfangled "Cooking" word. This process is something of a ouroboros, although it's certainly made easier now that mass updates are possible with software.

A useful contrast to all this is the way tagging on Flickr was never controlled (even though several Flickr members crusaded for various patterns). Now, even from this chaos, order emerges. On Flickr it's now possible to find photos of red graffiti on walls in Brooklyn, all through tags. Using metadata "native" to a digital photograph, like the date it was taken, and various camera details, you can focus even deeper, to find photos taken with a Nikon in the winter of 2008. Even though that's awesome, I'm sure it rankles professionals since Flickr also has a bunch of photos that have no tags at all.

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

In a blog post, you wrote about "a metastasized level of complexity." How does that connect to our need for minimum viable records?

George Oates: What I'm trying to get at is a sense that cataloging is a bit like case law: special cataloging rules apply in even the most specific of situations. Just take a quick glance at some of the documentation on cataloging rules for a sense of that. It's incredible. As a librarian friend of mine once said, "Some catalogers like it hard."

At Open Library, we're trying to ingest catalogs from all over the place, but we're constantly tripped up by fields we don't recognize, or things in fields that probably shouldn't be there. Trying to write an importing program that's filled with special treatments and exceptions doesn't seem practical since it would need constant tweaking to keep up with new styles or standards.

The desire to simplify this sort of thing isn't new. The Dublin Core (DC) initiative came out of a meeting hosted by OCLC in 1995. There are now 15 base DC fields that can describe pretty much anything, and DC is widely used as an approachable standard for all sorts of exchanges of data today. All in all, it's really successful.

Interestingly, after 16 years, DC now has an incorporated organization, loads of contributors, and documentation that seems much more complex than "just use these 15 fields for everything." As every good archivist would tell you, it's better to archive something than nothing, and to get as much information as you can from your source. The temptation for us is to keep trying to handle any kind of metadata at all times, which is super hard.

How do you see computers and electronic formats helping with minimum viable records?

George Oates: MVR might be an opportunity to create a simpler exchange of records. One computer says "Let me send you my MVR for an initial match." If the receiving computer can interpret it, then the systems can talk and ask each other for more.

The tricky part about digital humanities is that its lifeblood is in the details. For example, this section from the Tillett paper I mentioned earlier looked at the relationship between precision and recall:

Studies ... looked at precision and recall, demonstrating that the two are inversely related — greater recall means poorer precision and greater precision means poorer recall — high recall being the ability to retrieve everything that relates to a search request from the database searched, while precision is retrieving only those relevant to a user.

It's a huge step to sacrifice detail (hence, precision) in favor of recall. But, perhaps that's the step we need, as long as recall can elicit precision, if asked. Certainly in the case of computers, the less fiddly the special cases, the more straightforward it is to make a match.

Photos: Catalogue by Helga's Lobster Stew, on Flickr; profile photo by Derek Powazek

This interview was edited and condensed.


  • The Library of the Commons: Rise of the Infodex

  • Rethinking museums and libraries as living structures

  • The quiet rise of machine learning

  • August 04 2010

    Four short links: 4 August 2010

    1. FuXi -- Python-based, bi-directional logical reasoning system for the semantic web from the folks at the Open Knowledge Foundation. (via About Inferencing)
    2. Harness the Power of Being an Internet -- I learn by trying to build something, there's no other way I can discover the devils-in-the-details. Unfortunately that's an incredibly inefficient way to gain knowledge. I basically wander around stepping on every rake in the grass, while the A Students memorize someone else's route and carefully pick their way across the lawn without incident. My only saving graces are that every now and again I discover a better path, and faced with a completely new lawn I have an instinct for where the rakes are.
    3. Stack Overflow's Curated Folksonomy -- community-driven tag synonym system to reduce the chaos of different names for the same thing. (via Skud)
    4. Image Deblurring using Inertial Measurement Sensors (Microsoft Research) -- using Arduino to correct motion blur. (via Jon Oxer)

    Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
    Could not load more posts
    Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
    Just a second, loading more posts...
    You've reached the end.

    Don't be the product, buy the product!