Objectives

In a recent study (Hider 2017) the methodology previously employed by Gross et al. (2015) was used to gauge the “retrieval power” of the subject descriptors and identifiers assigned by professional indexers for a particular bibliographic database. Gross et al. (2015) had earlier found that about a quarter of hits would be lost, on average, if keyword searches on a university library catalog were not supported by Library of Congress Subject Headings. In the recent study, the mean percentage of hits that would be lost to keyword queries on the Australian Education Index (AEI) was found to match exactly that of the later Gross study, i.e. 27.0%, despite the different type of database and different controlled subject vocabulary (Australian Thesaurus of Education Descriptors).

The objective of this follow-up study was firstly to gauge the retrieval power added by professional indexing for other bibliographic databases, applying the same measure and similar methodology, as part of a larger program of research examining the search value of professional indexing across a range of contexts and databases.

Secondly, the author wished to investigate, given the similar results from the experiments on the library catalog and AEI, if the “retrieval power” of subject indexing in bibliographic databases and catalogs might be generalizable, to some extent, in the same way that Lotka’s Law has been found to apply, fairly accurately, to the frequency distribution of name headings in different library catalogs (Smiraglia 2002).

Methods

The three bibliographic databases chosen for the study were: ERIC (https://eric.ed.gov), PsycINFO (http://www.apa.org/pubs/databases/psycinfo; specifically, the 1987-2017 version) and SocIndex (https://www.ebsco.com/products/research-databases/socindex). They support different disciplines, and use different controlled subject vocabularies (ERIC Thesaurus, the Thesaurus of Psychological Index Terms, and Sociology Thesaurus). It should be noted that, in addition, the search interfaces used to interrogate the three databases differed in their functionality and indexing rules: for example, words were retrieved on separately in the case of ERIC, but as phrases in the case of PsycINFO and SocIndex. Perhaps most importantly, the SocIndex system provided an “apply related words” option, which resulted in queries being automatically expanded to cover additional words with variant spellings and word forms. The hosts of the databases used in the study were ProQuest, Ovid and EBSCOhost, respectively.

A range of sampling sources was also employed. For ERIC, 55 of the keyword search queries used for the earlier AEI study were re-employed (ERIC covers the same field as AEI, and is international in scope and outlook; the AEI queries might in some cases comprise Australian terminology). For PsycINFO, the “sixty most-frequent multiword terms” identified in a search log from PsycINFO (Yi et al. 2006) were tested, while for SocIndex, the sample of 96 terms was based on lists of “Research Topic Ideas” for anthropology, criminal justice and sociology students found on a webpage of the Frances Willson Thompson Library at the University of Michigan-Flint (http://libguides.umflint.edu/topics/socialscience). Each source has both methodological pros and cons: the sample for ERIC represented actual search queries, but on a different database; the sample for PsycINFO also represented actual search terms, entered on the same database, but was modal rather than random; the sample for SocIndex constituted terms not derived from actual searching, but was thus independent of the database’s controlled vocabulary. The sample sizes were not large, but allowed for indicative findings (with an estimated confidence level of at least 70% for a 4% margin of error).

The terms in the samples were searched for on the relevant database thrice: first, on the full set of keyword indexes; then on all the indexes except for the index with the controlled subject vocabulary; and finally on all the indexes except for the indexes with the controlled vocabulary and the identifiers (additional, uncontrolled terms assigned by the indexers for concepts not represented in the controlled vocabulary). It was hypothesized that the loss of the identifiers would not reduce retrieval power all that much, as they would tend to be terms present in the abstracts, and thus covered in the full search.

Main results and conclusions

The full results will be detailed in the final paper. The mean percentage of hits lost due to the omission of all assigned subject indexing (controlled subject vocabulary and identifiers) was 36.5% in the case of ERIC, 35.2% in the case of PsycINFO, and 17.2% in the case of SocIndex with the “apply related words” option utilized. The figure for SocIndex is quite low, but it would probably be considerably higher, and a fairer comparison, if the “search thesaurus” applied automatically also utilized the reference structure of the controlled vocabulary. For the typical bibliographic database with a more traditional interface, it would appear that without professional indexing retrieval power would be reduced by even more than the 27.0% loss recorded in the earlier studies.

The mean percentage of hits lost due to the omission of the controlled subject vocabulary alone was 35.0% for ERIC, 26.6% for PsycINFO, and 11.0% for SocIndex. This indicates that the identifiers significantly increase retrieval power in the case of two out of the three databases; the sources of these identifiers, and the precision of the additional resources retrieved, merit further research.

Given that the experiments on the ERIC and AEI databases used the same search queries and retrieved on their respective indexes in the same way, their results suggest that the increase in retrieval power attributable to assigned subject indexing is not uniform across bibliographic databases. This is likely due to a number of variables, including, perhaps, the size of the database and the amount of indexing, both assigned and derived, the scope of the controlled vocabulary, and the discipline and nature of the material covered. However, this does not preclude the existence of underlying “laws”, the results of which being modified by these variables. The relative proximity of the measurements produced by the experiments on the four databases, and the library catalog, suggests that further scrutiny of the relationship between the languages of authors, indexers and searchers could be profitably undertaken.

References

Gross, T., Taylor, A. G., & Joudrey, D. N. (2015). Still a Lot to Lose: The Role of Controlled Vocabulary in Keyword Searching. Cataloging & Classification Quarterly, 53(1), 1-39.

Hider, P. (2017).  The Search Value Added by Professional Indexing to a Journal Database. Presented at NASKO 2017, June 15-16, Urbana-Champaign.

Smiraglia, R. P. (2002). The Progress of Theory in Knowledge Organization. Library Trends, 50(3), 330-349.

Yi, K., Beheshti , J., Cole, C., Leide, J. E., & Large, A. (2006). User Search Behavior of Domain-specific IR Systems: An Analysis of the Query Logs from PsycINFO and ABC-Clio’s Historical Abstracts/America: History and Life. Journal of the American Society for Information Science and Technology, 57(9), 1208-1220.