Four measures were used to assess the success of a query: two user self-report measures of satisfaction and two system measures, both based on initial and final search results.
The two self-report measures were gathered by a very brief on-line questionnaire. Whenever the user selected either the New Search or Exit System button, a pop-up window was presented which posed the following questions:
How satisfied were you with the results of your first search?
How satisfied were you with the results of your final search?
Possible answers were:
Yes: Largely satisfied.
No: Largely dissatisfied.
Those who had not expanded their search were asked only the first question.
Matched to the self-report were two system measures, based on relevance judgements following a) the initial search and b) all subsequent searches. In each case, a success was recorded if at least one relevant document was chosen.