At TREC 3 there were two main categories of entries. In the routing section, where the search is assumed to be run regularly against incoming item, one Okapi run gave the best performance of all the results returned, on most performance measures. In the ad hoc section, where the search is taken to represent a new requirement on an existing database, the best Okapi run was second on most measures. The top run in this section, on the INQUERY system from the University of Massachussets, included some manual intervention, while the Okapi run was automatic.
TREC has stimulated a revival of research interest in information retrieval theory and system design, particularly in the context of large (by experimental standards) text datasets. The format encourages friendly rivalry while promoting a real exchange of ideas and techniques. Some groups collaborate in various ways (for example, in an earlier round, a group at the University of California Los Angeles used the Okapi software for its entry. The Okapi team at City, which has been supported in its efforts by grants from the British Library, certainly learnt from the other participants in the first two rounds. In the next round of TREC, the Okapi team will focus its efforts on devising appropriate test conditions for interactive searching, to simulate a more realistic environment.
The TREC conferences are reported (results of the tests, papers by the participants, system descriptions etc.) in the proceedings, edited by Donna Harman and published by NIST (Gaithersburg, MD, USA). Proceedings of TREC-3 have just been published. A special issue of Information Processing and Management will soon be published containing papers based on the first two TRECs, including one on Okapi.