The main ENQUIRE evaluation experiment involved collecting data about unsupervised search sessions, undertaken by users accessing the sytem remotely from library and laboratory terminals. In order to obtain accurate information it was necessary to be able to identify individual sessions and --- wherever possible --- users, for the purpose of analysis. Some of the associated tasks are described briefly below.
Methods of user identification differed according to location. At City University, users accessed ENQUIRE from a main menu supported by the local window-manager. The menu item had associated with it a shell script to perform a remote login to the ENQUIRE server, and register it as an X-host. This script took the user's login name from the appropriate environment variable and passed it down as a parameter, providing an accurate identification.
At the London Business School, users running the system under emulation from a PC were asked to type in their user name, since there was no way to get it automatically. This step was omitted for the installation in the LBS library, since it had been discovered from the earlier formative evaluations that most catalogue users were students from other colleges or LBS alumni, who could not in any case be contacted for follow-up studies.
It was anticipated that some users would walk away from the terminal once they had obtained their results, rather than ending the session in the orthodox manner. So a time-out mechanism is implemented: after a certain period without user-initiated action the system quits automatically and, at least in the LBS library environment where a dedicated terminal is in use, reverts to the ENQUIRE startup window. The actual timeout setting is a parameter sent by the interface script to the query layer --- its optimum value is partly dependent on the environment for the search session. There is a trade-off between making it too short and cutting off a user who is still thinking about his query, and leaving the terminal unattended for so long that several different undemarcated sessions could take place. The value selected for the ENQUIRE experiments was 10 minutes.
Detailed transaction logging is vital for the analysis and evaluation of an experimental IR system; it has been implemented in some form or other for all previous Okapi research projects. What is logged varies according to the nature of the investigation --- for some purposes it has been important to record term and document weights in order to monitor the performance of the underlying search algorithms, while studies focusing mainly on the interaction may be more concerned with user window selection, scrolling, and other surface activities. For ENQUIRE it was decided to concentrate on the task level by recording only those user actions (adding and deleting terms, displaying and judging documents, option selection) which might affect the stored query state.
Logging is performed by the query layer following the issue of an appropriate command by the interface script. Initially, messages are written to a sequential file, then periodically they are batched and reformatted for loading into an (Oracle) relational database for analysis. This provides a flexible structure for handling large and complex data sets, and a powerful high-level query language (SQL) to support selection and summary. The quantitative results presented in Chapter 5 were all generated by SQL queries. Appendix D sets out the structure of the underlying database tables.