An Aquifer Perspective on Code4Lib Conf 2008

Code4lib
It's a small well run tightly focused forum for programming efforts in support of libraries. With a single track, primary talks limited to 20 minutes talks, then ending each day with an hour of 5 minute lightning talks you can't help feeling in touch with what's going on in the library programming community. See http://code4lib.org/conference/2008/
The trend that stood out most for me is the number of projects implementing catalogs in one form or another. The software tools necessary to create a catalog are more robust and easier to use, borne out by our own project. Very little of the presentation time on these things was spent bemoaning the difficulties of master the different metadata formats, the problems, whatever they are, seem secondary to the issues of getting real time information on holdings, circulation status, etc. My sense is that the continuing movement towards keyword and faceted searching via SOLR/Lucene and like engines is sufficient in the minds of this group (at least) to address concerns about how the quality and differences of the underlying metadata affect the ability to search and present it.
Not that metadata issues are dead, two rather pointed talks on the on-going RDA effort show that to be far from the case. My naïve view is that all metadata efforts would be immensely helped by employing programmers on the metadata team up front and charging them with delivering, upon release of the specification, a reference implementation, any reference implementation. Secondly, (and I can't quite believe I am saying this) more exuberant use of the specificity that xml can provide. As an example, Karen Coyle, one of the keynote speakers, described the inability to achieve closure on the issue of encoding the author and publisher when what appears on cover page is wrong. Since this seems to be important, I say, go for a little tag richness and provide an encoding for both pieces. What will make this work is that the programmer working on the reference implementation will need to know the preferred order for presenting this detail via DC or RSS which is all that most of the world will ever see of it. It's the right solution, the information is not lost but implementers will have a clue what the specification architects had in mind.
In short: great conference, lovely city, nice weather, a beer town of mythic proportions.

Chick bangs his head against a Date

With apologies to Mr Twain, I have always said, "I'd sooner date two jealous psychopaths than one MODS record." Nothing lately has been making me feel any differently. The question is how to take the date information from a MODS record and get it index usefully into an act_as_solr SOLR instance. Our approach is two-fold. Translate the date information in the originInfo element and run the result through the software package developed by the CDL that implements the erstwhile TEMPER date analysis standard.

Here is an example, say I have a record dated 1900-1920. I want to to index this in such a way that if a user chooses to find records between 1905 and 1910 it finds this one. So what do I do. I use the TEMPER software to expand t date range into a separate facet for each year. This wildly duplicates the records when faceted by year but satisifies that it will be found on any overlap with the desired date range.

And you are probably asking how I got the date range, take a look at this xsl snippet which is taken from an xsl transform within the context of an originInfo element

<xsl:element name="doc_date">
<xsl:for-each select="dateIssued|dateCreated|copyrightDate">
<xsl:value-of select=".">
<xsl:choose>
<xsl:when test="@point = 'start'">
<xsl:text>-</xsl:text>
</xsl:when>
<xsl:otherwise>
<xsl:text> and </xsl:text>
</xsl:otherwise>
</xsl:choose>
</xsl:value-of>
</xsl:for-each></xsl:element>
This code concatenates, all desired date elements and even nicely adds a dash after a date with the point='start' attribute (indicating that it begins a range, hopefully followed by the ending date in the next element)
E.G.
</xsl:for-each></xsl:element></pre></span>

This code concatenates, all desired date elements and even nicely adds a dash after a date with the point='start' attribute (indicating that it begins a range, hopefully followed by the ending date in the next element)
E.G.
<origininfo>
<dateissued>[no date recorded on caption card]</dateissued>
<dateissued encoding="marc" point="start">1900</dateissued>
<dateissued encoding="marc" point="end">1920</dateissued>
</origininfo>

becomes
"<doc_date>[no date recorded on caption card] and 1900-1920 and</doc_date>

Which is then nicely parsed into a range by the TEMPER software. I'd love to hear better theories on how to do this. Really! And I haven't even done all the tests on circa yet.

An update on Indexing MODS records with SOLR

My goal in the Aquifer project is to make the machinery behind the indexing as transparent as possible. Indexing is almost invariably a mapping between the source data and index representation. Here, MODS records are mapped to a SOLR input schema using a XSL transform mods-solr.xsl (visible here). The complexity of the MODS document is reduced to a list of fields, such as title, name, subject, etc. Many sub-elements may be concatenated together to create these fields.
Here is an example of what goes into SOLR

<add>
<doc>
<field name="id">1</field>
<field name="title">History of the Saginaw Valley,; its resources, progress and business interests</field>
<field name="name">Fox, Truman B</field>
<field name="type-of-resource">text</field>
<field name="origin">, Daily Courier Steam Job Print1868</field>
<field name="language"><field name="physical-description">text/xml;image/tiff; 80 p. 20 cm.; reformatted digital</field>
<field name="note">Advertising matter interspersed.</field>
<field name="subject-topic">History</field>
<field name="subject-geographic">Saginaw River Valley (Mich.)</field>
<field name="url">http://name.umdl.umich.edu/0885076</field>
<field name="access-condition">Where applicable, subject to copyright. Other restrictions on distribution may apply. Please go to http://www.umdl.umich.edu/ for more information.</field>
</field>
</doc>
</add>

There are problems with some of the fields having extraneous commas from concatenation rules. And more interesting problems like whether fields like access-condition should be indexed at all. But at least with the XSL, the rules are not buried within Java or some other language.

Futher details on how SOLR uses this input is controlled via it's schema.xml file (visible here)
The fields created by the transform must be defined as well as copy information which allows fields to be indexed in more than one way. For example the line

<copyfield source="subject-hierarchic" dest="subject">

causes a topic subject to also be indexed as a generic subject line. More lines like this can cause all text to be indexed under a single default which is used when users do not specify a field in a query. Additional configuration can control how fields are parsed, how sentences and punctuation and are handled on a document or field level.

These files need more work but they are not a bad starting point.

Harvesting and mapping

It's hard to proceed without data so yesterday and today were devoted to a simple ruby harvester. It is up an running and right now, painfully slow. It was also tested on sites that had no errors so robustness has not been broached. The good news is I can move on to indexing and display. Next specific task is to create a intermediate form of the mods record. This will be very similar to the BibClass implementation of MODS. Parsing the MODS data at runtime every time we want to show or index a record will be just to much overhead. I will probably go with either a simple ruby or json serialization of a an array of array of fields. There will be mapping tables and filters that will translate between the various MODS sources and this layout. I think I will put these in the database so administrators can try applying various ones to new sources as they appear. I am definitely leaning towards json as the serialization method because it will make it much easier for other applications written in other languages to mine the data directly.

The biggest trick to getting the harvester working was divining how the resumption_token was used and then figuring out that that token to list_records can not be accompanied by other tokens.

res = client.list_records( :metadata_prefix => 'mods' )
return if res == nil

entries = res.entries
while true do
entries.each do |entry|
puts entry.header.identifier
save_record entry
end
puts "done with first batch"

token = res.resumption_token
puts "token=#{token}"

res = client.list_records( :resumption_token => token )
break if res == nil
entries = res.entries

end

I have a problem that I did not figure out on this pass. It seems that getting the entries with entries = res.entries takes as long a the list_records call making me think that I am fetching twice. I could just iterate over list records but then I am not sure if the resumption_token is where I need it. A bit of poking will probably reveal all.