Several users complained about having to make relevance judgements on each book they saw, and a couple used the ``No'' button as a navigation device even when they really wanted a particular book. We attempted to rectify this problem by changing the on-line help and on-screen messages to put across the idea that relevance judgements are important for the user, and are not just for the benefit of the experimenter.
Two users were worried by the effect that ``No'' judgements would have. They were concerned that this would lead to documents being excluded.
Comments from six users indicated that they disliked looking through a lot of documents. At that stage, no algorithm was used to determine how many documents should be displayed, or to show their degree of match with the query. This was rectified for the next set of tests.