Go to:
Online Doc
Andrew Whitis's picture

Notes from the MW10 Discussion Forum "Discovery Systems: Solutions a User Could Love?"

MARS Local Systems & Services held a Discussion Forum on Sunday, January 17, 2010 at the Midwinter Meeting. 

Panelists included:

(1) Marshall Breeding, Director for Innovative Technology and Research,Vanderbilt University, "Overview of Discovery Systems."
(2) Cody Hanson, Technology Library, University of Minnesota, "User testing and feedback on Primo at the University of Minnesota."
(3) Frances McNamara, Director, Integrated Library Systems and Administrative and Desktop Systems,University of Chicago, "LENS Discovery System, based on AquaBrowser."
(4) Barbara DeFelice, Director Digital Resources Program, Dartmouth "Summon @ Dartmouth College: the User View."

Try the discovery tools first hand:

  • Dartmouth College's implementation of Summon
  • University of Chicago's Lens
  • University of Minnesota's MNCATPlus

Major points made by the panelists:

First, Marshall Breeding (Director for Innovative Technology and Research, Vanderbilt University) gave an "Overview of Discovery Systems." He discussed the “crowded landscape of information providers and pointed out that libraries should look to e-commerce sites for usability practices: Amazon.com, for example, is self-explanatory.

Breeding also discussed the shift from print to electronic in libraries. We are delivering different kinds of content (electronic) and serving a different kind of user (millennials), which must change the way that we expect to deliver content to users. Our websites now tend to be menus of disjointed silos: books, articles, local digital collections – all searched separately. Libraries also tend to lack a unified web presence within their websites – in other sites, the search function is part of the website, not something separate from the website.

Users have been saying they want a "single point of entry" to library content, while librarians continue to say we don’t want this. That’s part of our problem – we are reluctant to embrace these new systems.

Library websites are evolving and must move towards modernization, which includes features like type-ahead, "do you mean,"relevance ranking, one search box, visual components, and enriched content. This is the target, and such a target goes beyond the content that is in the OPAC. Therefore it must be decoupled from the OPAC. This is the future.

Breeding also talked about the possibilities afforded by social discovery: tags, ratings, etc., are all fodder for the infosearch process. Many systems no longer search merely thin metadata (like what is in the OPAC) but the whole item (the full text). Thus the item itself is its indexing, which can also inform search results.

Technologically, he says, the problem of the single point of entry has been solved by searching pre-populated indexes instead of using federated search technology. Now, he says, the problem is going to be convincing the information providers to let their information be used this way.

Next, Cody Hanson, Technology Librarian at the University of Minnesota, talked about "Primo User Testing: Feedback and Metrics." He began by giving an introduction to Primo.

The University of Minnesota was a Primo development partner with Ex Libris, which included a usability testing component. They conducted tests in 2007-2008 in their usability lab, which is set up to capture screen activity, do eye tracking, film the user, record sound, etc.

The tests involved users thinking aloud while completing scenarios and tasks, followed by a debriefing in which users also filled out a desirability matrix. Their goals were to see if users could be aware of and use the variety of information in Primo, find its availability, use facets, and more. The tasks included things like:

a)      find this article from this citation
b)      find books about this
c)      save the items you find
d)     find this book and tag it
e)      find this book and read its user review

They found that users had trouble finding the tabs and using the advanced search (though Hanson didn’t think this was a Primo problem).

Overall, University of Minnesota users liked Primo. On the desirability matrix, which included negative and positive adjectives, of which users were asked to circle 5, the most common chosen were organized, clean and straightforward.

Their use stats reflect most students finding something useful when they do a search, because most click through to view more about something in their results lists. They have also learned that most searches are for books, not many users have been adding tags, and not many are saving items. The advanced features in Primo were not heavily used at first.

Hanson also discussed user comments, which can sometimes be very negative, because feedback is rarely motivated by a positive experience. Staff, especially, had negative comments, but it’s important to remember that staff are not our audience.

Frances McNamara (Director, Integrated Library Systems and Administrative and Desktop Systems at the University of Chicago) then discussed the "LENS Discovery System, based on AquaBrowser." She gave us an overview of LENS, pointing out how it contains more information than had been in the OPAC, like tables of contents, covers, and Google book links.

They tested the facets in 2006 on 12 PhD candidates (a primary audience for them) which demonstrated users’ difficulties with the old catalog, and in 2008 did usability testing on LENS followed by other specific pathfinding studies.

In the 2008 study they asked 8 graduate students to do 10 tasks. The study compared the old catalog with the new (LENS). They found that the advanced search needed to be easier to find, and that users wanted clarification on what LENS was searching (transparency). Users weren’t 100% confident when using LENS that they were finding everything and some mistrusted the relevancy ranking. McNamara also said they discovered it was important to pay attention to what people liked about the old system as they continued to design the new.

One of their later studies was on finding journal articles; users found this was difficult. Another further study was on music researchers. At the University of Chicago, the developers use personae when talking about the design of their system in order to stay focused on their different user types.

Finally, Barbara DeFelice (Director, Digital Resources Program at Dartmouth College) talked about "Summon @ Dartmouth College: the User View."

They implemented Summon (Serials Solutions) at Dartmouth in April of 2009. They still have the catalog, federated search, and other silos, in addition to their Summon Beta.

They did usability testing on the beta before making it public by doing six one-on-one sessions with undergraduates and two focus groups. They used TechSmith UserVue software for their one-on-one sessions. They did not use tasks, but asked the students to work on real assignments while using the system.

They found that the students liked the look of the system. Students also liked limiting to known databases/sources – DeFelice pointed out that they learned from this that students bring prior knowledge to a new system and want a way to work that knowledge into their use of it. Users also have a low tolerance for not getting smoothly to full text: they have found that their users hate the openURL results page.

Dartmouth also talked to faculty one-on-one. They found out their faculty really like Google Scholar and use it, and also that they recognize the continued value of known subject indexes. They felt the new system was good for finding known items.

Panelist Q&A:

During some time for Q & A, DeFelice and Hanson both recommended getting as many people to observe your tests as you can because it can be very eye-opening for the observers, especially if those people are library staff. It’s a good way to create buy-in for a new system.

Also during the Q & A, one of our panelists (Breeding?) made the point that no ILS vendor is updating its OPAC: the OPAC is dead, he said. Precision will be the thing to work on next in these systems. Also, keeping both (a discovery system alongside an OPAC and separate article databases) won’t be feasible for very long.

Notes taken by committee member Amy Fry during the panelists discussion.

Table Discussion Comments:

Hanson's slides of MNCATPlus3.14 MB
Judith Carter's picture

Thank you Amy for writing these up. I had to miss the meeting so I came looking for any additional info.