The user interface reflected the more sophisticated hardware used for this project, with better user messages. The single keystroke commands were retained but they were based on the initial letter of the command names rather than coloured keys.
Messages were included within lists of brief citations to emphasise the degree of match. These thresholds were determined according to a maximum possible weight for a record containing all the query terms. Records achieving two thirds of the maximum were deemed to match the users' search well, while a second threshold of half the maximum weight identified records which matched less well. Records which scored less than half the maximum weight were identified by a third message which indicated that they matched, but not very well.
Controlled experiments were used to evaluate the query expansion facility. Three versions of Okapi were configured which offered subject searching only. One incorporated automatic query expansion using relevance feedback; another provided automatic query expansion and classification browsing; while a third contained no query expansion. The three systems were compared for effectiveness, efficiency and user acceptability.
The evaluation was based on both transaction logs and questionnaires. Results showed that a very low proportion of searches failed to find anything and that query expansion was deemed to be effective in finding more specific books, or in shifting the emphasis of a search. It was also effective when the initial search had not been very successful. Classification browsing was found to be less helpful, however. Users tended to scan lists of titles to find only one or two additional items.
Although the controlled experiments provided an insight into the effectiveness of query expansion, live evalution of the system was needed. This was carried out in the next project, Okapi at City.