Syndicate content

ACRL Assessment Discussion Group Community

In: Assessment and Evaluation
View:   Faces | List

File ACRLAssessmentDiscussionGroup_20170117

by Nancy B. Turner on Fri, Jan 27, 2017 at 10:58 am

DOCX File, 16.17 KB

File ACRL-Assessment-Discussion-Group

by Nancy B. Turner on Tue, Jul 5, 2016 at 12:32 pm

DOCX File, 21.35 KB

Online Doc ACRL Assessment Discussion Group Midwinter 2015 Notes

by Charla Gilbert on Tue, Feb 17, 2015 at 09:52 am

Notes from the midwinter ACRL Discussion Group 2015

Event Assessment Discussion Group

by ALAConnect Helpdesk (staff) on Mon, Jan 26, 2015 at 02:44 pm

Open session to discuss assessment activities among colleagues. Participants are invited to bring topics as well as engage in lively discussion.

More information about this conference session

Discussion Notes from the 2014 Midwinter Meeting of the ACRL Assessment Discussion Group

by Carolyn Radcliff on Wed, Jul 2, 2014 at 03:25 pm

ACRL Assessment Discussion Group
Saturday, January 25, 2014

Welcome and Introductions

Discussion #1. Closing the Loop - Implementing Changes Based on Assessment Results –Liz Mengel - Johns Hopkins  

Johns Hopkins has no formal assessment program, but they do have a user experience librarian. They are doing “action oriented research.” Liz compared the process of doing assessment research to Kubler-Ross’ stages of grief: 

ACRL Assessment Discussion Group
Saturday, January 25, 2014

Welcome and Introductions

Discussion #1. Closing the Loop - Implementing Changes Based on Assessment Results –Liz Mengel - Johns Hopkins  

Johns Hopkins has no formal assessment program, but they do have a user experience librarian. They are doing “action oriented research.” Liz compared the process of doing assessment research to Kubler-Ross’ stages of grief: 

  • Shock
  • Denial
  • Anger
  • Depression
  • Acceptance
  • Integration   

Assessment can lead to small, quick changes that can be made based on assessment results, such as changing service hours based on seat and gate counts, and changing staffing based on an assessment of reference questions using the READ analysis tool.

Participants discussed assessment in their own libraries and reported on small changes they had made based on assessment: 

  • Students like privacy in public 
  • Students wanted keyboard free area - books only
  • Lounge furniture works best when it is arranged in pod areas (two chairs)
  • Patron-driven acquisitions 
  • Students who wanted 24-hour access to coffee and tea were thrilled with a hot water dispenser (they bring the tea bag and mug)

Consider Rapid Iteration Prototyping, e.g. Small changes in staffing of info desk

Another kind of assessment can lead to large organization changes. Plan on seven years to implement these. Example provided was ClimateQual. This is “hard to do” – “you need to chunk the work into small changes.”

Liz recommends the book, Academic Library Value: The Impact Starter Kit by Megan Oakleaf.

  • Reflect
  • Decide
  • Act 
  • Share 

Key question to ask when doing assessment is “What types of decisions do you want to occur as a result of this assessment?”

See PDF of slides and notes at the bottom of this post.

Discussion #2: Conducting Ethnographic Research in Academic Libraries”

Mary Thill - Northeastern Illinois University

Mary was involved with the grant writing and participated in some of the research for ERIAL,  a collaborative project involving research at five academic libraries in Illinois (DePaul University, Illinois Wesleyan University (IWU), Northeastern Illinois University (NEIU), the University of Illinois at Chicago (UIC) and the University of Illinois at Springfield (UIS).). The project site is located at: http://www.erial project.org

The project started in 2008 and cost $300,000. Most of this cost went toward hiring two anthropologists and developing a tool kit for libraries who want to conduct ethnography at their own library.

The research question was, “How are students completing their research questions? What are the mutual expectations of students, instructors and librarians of who did what?”

Thill addressed the question of “Why ethnography?” She suggests that ethnography is a method for discovering “What students are actually doing rather than what they say they are doing.”

The group then discussed the different methods used - cognitive maps, mapping diaries, photographic survey.

We talked about how best to get participation in these activities, some of which were time-intensive. An incentive might be a pizza party, a gift certificate for coffee at the library café. Students don’t necessarily need incentives: “students do it because they want to.” These methods are “Hypotheses generating rather than proving something.”

Mary distributed a very useful table of Selected Ethnographic Research Methods. The source is: Asher, Andrew and Susan Miller. So You Want to Do Anthropology in Your Library? Or A Practical Guide to Ethnographic Research in Academic Libraries. ERIAL http://www.erialproject.org/publications/toolkit

___________

Nancy Turner and Carolyn Radcliff
February 3, 2014

More...

Discussion Notes from ACRL Assessment Discussion Group, 2014 ALA Annual

by Carolyn Radcliff on Wed, Jul 2, 2014 at 03:18 pm

ACRL Assessment Discussion Group
Saturday, June 28, 1:30 pm to 3:00 pm
Las Vegas Convention Center

Part 1: Best Practices in Survey Design

Presenter: Nisa Bakkalbasi, Assessment Coordinator at Columbia University Libraries.

How can we be sure that the measures we take our free of error?  

ACRL Assessment Discussion Group
Saturday, June 28, 1:30 pm to 3:00 pm
Las Vegas Convention Center

Part 1: Best Practices in Survey Design

Presenter: Nisa Bakkalbasi, Assessment Coordinator at Columbia University Libraries.

How can we be sure that the measures we take our free of error?  

  • All we can do is try to reduce the error
  • Survey development process is collaborative and iterative
  • Share draft with colleagues 
  • Write down clear and specific objectives

Be clear about what you want to know. Develop a matrix to map the relationship of objects (questions) with what you want to know. Pilot your survey with others. 

Principles for Question Wording

  • Keep questions short and easy to read - 8th grade reading level is what to aim for (even if participants are not in 8th grade)
  • Questions should be clear, specific and precise
  • Ask only one question at a time 
  • Avoid or define any acronyms, jargon or abbreviations 
  • Construct questions objectively and avoid leading language

Question types

  • Open ended – Used to ask for problems and solutions
  • Close ended
    • Multiple choice (exhaustive list and mutually exclusive)
    • Rating scale - Make it clear what the numbers mean - keep scale odd, 5 is usually sufficient

Content evaluation: Is this information we can use? Don't ask because we are curious.

Overall survey design  

  • Include opening statement and introduction to survey, with statement about confidentiality
  • Keep survey short and to the point
  • Keep required questions to a minimum
  • There should be a logical flow to questions;
    • Put demographic questions at the end unless you will be disqualifying people based on a particular demographic 
    • Use skip logic 
    • Thank users for taking the survey

If you plan to measure change over time, you need to:

  • Use exact question wording from survey to survey
  • Use exact sequencing to maintain context

Additional Resources related to Survey Design

 

Part 2: Large Effects With Small Effort: The Quest to Leverage Library Data

Presenter: Joe Zucca, Director for Planning and Organizational Analysis, Univ. of Pennsylvania Libraries.

Zucca asked us to think about creating a machine that could generate “business intelligence” about all kinds of library data that we are collecting. The machine would provide an environment for storing data to convert into information. What do we currently collect?

  • ILMS
  • Apache server
  • Counter use data, non-counter use data
  • ILLIAD
  • Atlas
  • BePress
  • Summon,
  • Ares
  • Aeon 

Our challenge is scope and complexity of the data and the diversity of its architecture.

Scenarios 

The ideal assessment machine would solve these kinds of problems/questions:

  • Audience penetration 
  • Modes of engagement 
  • Resource implications 
  • User satisfaction 
  • Library building use
  • Bench marking against peers
  • Circulation analysis - print/e
  • Impact of implementation of discovery
  • Instruction in information literacy on student success 
  • Faculty usage of library
  • Link to university KPI
  • Gate swipe logs 

There are so many different units of analysis, we need to make a distinction between data and statistics. Think in terms of an event with a star schema.

For example, "I logged into PsychInfo" – This is an event that has an environment and can be associated with: 

  • time stamp
  • location
  • demographic,
  • acquisition
  • academic credential
  • metadata about resource  

Using the “event” as the unit of analysis allows us to pull together disparate data.

There is “no shrink-wrapped solution to this problem.”  But what are some scalable solutions? 

The library is a lens of what faculty do and the administration needs to know this. For instance, the library has the business intelligence that the institution should want about what faculty are doing, the demographics of users cross-tabbed with resources they are using. And the data don’t need to be limited to one institution. Systems like Vivo (really a  researcher intelligence platform) are coming along that help to solve this problem of helping us understand collaborations of faculty and information use and consumption.

We need to:

  • Get mechanical advantage over what we already have
  • Think of this as a way of leveraging library services.

--Nancy B. Turner

More...

To provide a forum for assessment librarians –  and those with responsibility for, and interest in, library assessment – to discuss methods, training, results, impact, institutional needs and challenges, and seek solutions to common problems faced by the library assessment community.

Subscribe to ACRL Assessment Discussion Group