Even where search results matched queries well, the overall number of documents viewed and judged relevant was small. Out of an average of around 13 and 10 items examined, just under three items were found to be relevant in LBS and INSPEC respectively. Clearly end-users do not undertake exhaustive searches. There is also evidence to show that in a ranking system they tend to view the top-ranking items only [12].
Unlike previous versions of Okapi, where results were first displayed as a hitlist of titles, ENQUIRE showed full document details automatically and viewing the hitlist was optional. The intention was to make a more obvious link between the initial query terms and the document on display. An unavoidable consequence was that searchers could avoid making relevance judgements on full documents by repeatedly switching back to the hitlist --- informal observations during the first field trial indicate that some people adopted this strategy deliberately. As a result, relevance judgements were not made on around 20% of viewed items, and consequently those items could not be used for term extraction, even though they were likely to have been high in the ranking and possibly relevant.
In general it remains unclear whether it is best to display a hitlist or a full document first. User preferences for viewing a hitlist may be simply due to the fact that the convention exists in other systems. However there is a danger that by designing an interface to suit those preferences we may be working against users' best interests by allowing them to by-pass the relevance feedback and query expansion mechanisms which are the principal strengths of our system.