Okapi '90: Okapi at City: Automatic Query Expansion in a library catalogue and an online database - live evaluation

In July 1989, Okapi and its researcher Stephen Walker moved to the Centre for Interactive Systems Research in the Department of Information Science. The 1990 study focussed on:

evaluation of query expansion in a live setting;
analysis of transaction logs to investigate patterns of use of the system over the network and the effectiveness of automatic query expansion in a comprehensively indexed database;
survey of frequent users to identify whether patterns of use could form the basis of an adaptive information retrieval system.

Okapi's search engine was enhanced for this project so that it could handle different database file structures, with a variety of searchable and indexing fields. Improved updating and indexing programs were also written.

Design

To provide access to a wider range of databases the INSPEC and LISA data files were converted to Okapi format, as was City University's library catalogue. Okapi was connected to a campus wide ethernet system in May 1990 and a terminal was installed in the library.

Evaluation

Extensive transaction logging was used to record users' searches. The data was analysed to identify patterns of use of query expansion. Where query expansion was available but not carried out, searches were replayed to see if it would have found more items that, in the experimenter's view, were relevant.

Besides logging, structured interviews and live evaluations provided further data.

Transaction logging showed that there were very few failed searches, with 3.9% being recorded as compared with the 40% found with some exact match keyword systems. It was found that query expansion was used particularly when the results of an original query were not very good or when Okapi had only found a partial match. Results showed that frequent users tend to carry out long searches, often repeating portions of their query which may indicate that they forget what they have already tried, suggesting a possible need for some kind of history facility.

Live evaluation results showed that only one third of users took up query expansion as an option. However, in almost half the searches where query expansion was used, at least one extra item was retrieved.

The next project,Okapi '93 examined users' searching behaviour with subject-enriched catalogue records.