About the Toolkit

This Toolkit was originally developed by the Oxford Internet Institute in 2008 as part of a JISC-funded, Usage and impact study that explored the questions:  are digital resources succeeding at reaching their intended users? Are they having an impact on their community of users? How can impact be measured? The Toolkit was subsequently updated in 2011, and again in 2013, again with funding from JISC.

The Toolkit is regularly updated and maintained, with the most recent full revision in July 2013.

During the initial project, researchers carried out an impact analysis of five specific resources funded as part of the JISC Phase one Digitisation programme (2004-2007) by using the different methodologies outlined in the Toolkit.

The digital resources examined were:

Qualitative and quantitative methods

There are a number of challenges in assessing the use and impact of online digital resources: these include new methods, shifts in the way that people access resources, new audiences, and new forms of information-seeking behaviour among different audiences.

The project combined quantitative and qualitative indicators to measure the impact of online scholarly resources. Quantitative measures included webometrics, log file analysis, scientometric (or bibliometric) analysis, and content analysis. These were complemented by an array of qualitative measures (stakeholder interviews, resource surveys, user feedback, focus groups, and questionnaires) that captured information about the whole cycle of usage and impact.

The qualitative measures allowed us to examine the impact of the projects from the point of view of various stakeholders, starting with the host institutions (such as libraries and archives), the personnel at the host institutions responsible for implementation (including the developers and engineers of the systems and curators and archivists of the collections), and stretching all the way to the various types of end users and the uses they represent. We also included measures that involved programme funders, who are often overlooked as stakeholders.

The combination of multiple methods and data sources allowed us to move beyond quantitative measures such as number of site visitors or citations, which tend to provide a limited view of usage and impact.  Rather, we were able to take a holistic approach, gathering rich data to provide a fuller and more complicated picture of the community surrounding digital resources. 

The full report of our findings can be downloaded here.

 

The Toolkit for the Impact of Digitised Scholarly Resources (TIDSR)

 

A key part of the project was not simply to use the range of tools described above, but to assess each tool and consider the usefulness of each tool in measuring usage and impact. The TIDSR toolkit is the result of this work, comprising a best practices toolkit for the assessment of the impact of digitisation projects. This toolkit includes a number of elements which can aid other researchers and funding bodies to assess the impact of digitisation projects. The toolkit is designed to be a living resource, giving users of the toolkit and the methods described the opportunity to leave feedback, create new articles, recommend and review new tools and contribute to the debate about what constitutes impact and how best to capture this. 

The Toolkit was subsequently updated by the OII as part of the JISC Impact and embedding of digitised resources programme which ran between November 2010 and March 2011.

As part of this programme a further seven projects used the TIDSR toolkit to probe their usage and impact, and to develop or implement strategies to maximise both.  The seven projects were:

Crime in the Community: Enhancing User Engagement for Teaching and Research with the Old Bailey Online
Professor Robert Shoemaker, University of Sheffield
Crime in the Community assessed the ways in the Old Bailey Proceedings Online is currently used, and generated a series of new tools and online facilities that  allow educationalists and researchers to make more effective use of its content.

Dance teaching resource and collaborative engagement spaces (D-TRACES)
Professor Sarah Whatley, Coventry University 
The D-TRACES project investigated the Siobhan Davies dance digital archive patterns of usage and developed a model for embedding the archive within the Personal Development Planning (PDP) element of the undergraduate dance curriculum at Coventry University.

Embedding A Vision of Britain through Time as a resource for academic research and learning
Professor Richard Healey and Dr Humphrey Southall, University of Portsmouth
This project identified tools and facilities to allow better exploitation of A Vision of Britain through Time web site by academic users.

HumBox Impact
Dr David Millard, University of Southampton
The project  analysed the impact of the resources deposited in HumBox, a repository of Open Educational Resources for the humanities, on contributors and the wider teaching audience and developed strategies to ensure the increased use of the HumBox collection in Higher Education. 

The Impact and Embedding of An Established Resource: British History Online as a Case Study
Dr Jane Winters, Institute of Historical Research, University of London
British History Online, a digital library of medieval and early modern sources, conducted an in-depth analysis on how the collection is used for learning and teaching and assessed the requirements for a more effective embedding of the resource within UK Higher Education institutions.

Listening for Impact
Melissa Highton, Oxford University Computing Services
This project performed a thorough, rapid analysis of the impact of the public Oxford Podcast audio-video collection of 1800 scholarly items, launched in September 2008 and identified ways of increasing their impact.

SPHERE: Stormont Parliamentary Hansards: Embedded in Research and Education
Lorna Hughes, King’s College, London and CDDA, Queen’s University, Belfast
SPHERE adopted a number of measures to assess the use, value and impact of the digital scholarly resource, the Historical Hansards, and implemented practical approaches to embed the resource within teaching, learning and research.
This work was completed in May 2011, and was showcased at the “Digital Impacts” event hosted by the Oxford Internet Institute on 20 May 2011.  The presentations from this event were recorded, and can be viewed here.  In addition, each of the projects contributed a case study to the TIDSR toolkit, which can be found here.

For more information about the JISC digitisation programme, please visit the JISC website:
www.jisc.ac.uk/digitisation