A key component of the EAST project has been researching aspects of "validation" - what is the statistical likelihood that a given item - which may be allocated for retention for EAST - is available and useable?  This issue is at the heart of building confidence and trust among the EAST member libraries and the various stakeholders at their home institutions.

Validation Sample Study #1

Validation Sample Study #1 [VSS1] had as its goal to provide an "availability metric" for each of the 40 EAST participating libraries.  This metric would provide an indication of the liklelihood that a given item would be available to be loaned to another EAST library thereby providing a level of confidence across the member libraries and their local constituencies.  Working with a statistical consultant and advised by the Validation Working Group, the EAST team developed the following methodology for the study:

  • Provide a sample of 6,000 items for each of the 40 libraries for a total of 240,000 items to be sampled
  • The sample to be drawn from the data extracts the library had provided as part of the collection analysis
  • Develop a data collection tool that will support easy and efficient collection of data by workers [expected typically to be student workers] in the library
  • Provide an automated mechanism to check the library's sample against its local circulation system to eliminate items known to be checked out before items were checked on the shelves
  • Develop an easy to implement mechanism to provide a cursory check of the physical condition of the item as it was being handled by the worker
  • Provide documentation to the libraries to faciliate training of their local workers.

Working closely with the Validation Working Group, which had been appointed by the EAST Executive Committee, as well as OCLC Sustainable Collection Services, the vendor working on the EAST collection analysis, the EAST team completed prepartion for VSS#1 by mid-February and the formal data collection kicked off February 22, 2016 with a planned completion within 8 weeks.

The Data Collection Tool for Validation Sample Study #1

To support efficient data collection for the first sample validation study, the EAST Data Librarian, Sara Amato, worked closely with members of the Validation Working Group to design a web-based tool.  The tool supports download of the random lists of titles selected from the holdings records for each of the 40 libraries participating in VSS#1.  Prior to using the tool, librarians ran the randomized list of 6,000 titles against their current ILS to mark any items currently in circulation.  Accounting for checked out volumes from the beginning ensured only items which are expected to be found present will be downloaded to the tool.

To use the tool, a student or staff worker specifies the number of items they wish to check and those are loaded in location and call number order.  Using a tablet, laptop or workstation with an attached barcode, the worker then scans the first title in the list.  If the item is present based on the barcode read, the tool automatically updates the status as present.  The worker is then asked to do a brief condition check of the item to determine whether its condition is poor, good/acceptable or excellent.  Both documentation and a short video were developed by the EAST team to provide training to the workers to simplify this check.

Although access to wifi is required for the initial download of the items to be checked and to upload the list once the checking is complete, the actual barcode scanning and condition checking can be completed completely offline.  The tool is also designed to support multiple users so that libraries can use their staff  as appropriate locally.

Because the data collection tool reported back results in realtime as groups of items checks were submitted, the EAST team had up-to-date access to the availability metrics across the EAST libraries.  Upon completion of the data collection, results [anonymized except for reporting to the local library] were shared across the EAST membership.  

Since the workers were handling each item to perform the check, they were also asked to perform a check of the condition of the physical item based on a three point scale: 1 - Poor 2 - Average 3 - Excellent.  This data was entered into the data collection tool following the scanning of the barcode and required only minimal additional time on the part of the workers.  A training video for the workers was made available to assist and to better ensure consistency across the libraries.  Of the items checked 90% were found to be in either excellent or average condition.

A sample display from the EAST data collection tool.  Further details on the tool, including access to the open source software are available on GitHub.

Summary results of Validation Sample Study #1

Overall 97% of monographs in the sample set were accounted for (mean: 97%, median: 97.1%, high of  99.7% and low of 91%).  2.3% of titles were in circulation at the time of the study.  All libraries completed data collection by the deadline of April 22, 2016.

90% of titles were in average or excellent condition.  Approximately 10% of titles reviewed were marked as being in ‘poor’ condition.  The mean score for condition was 2.2 (2=average condition, 1=poor, 3=excellent).

Additional analysis of the VSS#1 sample data set is being undertaken by the EAST team in consultation with the statistical consultant.  Some initial additional findings, based on the data set are:

  • Items publisher pre-1900 were in significantly poorer condition - some 45% of these items ranked "poor" on the condition scale
  • Items that had circulated more frequently were somewhat more likely to be missing - 4.3% of these items were unaccounted for as compared to 3% overall.

The EAST team expects to provide further details on any insights gained from additional analysis of the data set. 

Validation Sample Study #2

Details of the second validation sample study are still to be determined and will announced by the fall of 2016.