First Formative Evaluation : Test Administration Script

1. Enquire of LBS Library users as they approach the LBS catalogue, ``Excuse me, I'm carrying out a study on behalf of the Centre for Interactive Systems Research at City University. I wonder would you mind if I asked you some questions?'' If they do not wish to participate: ``Thank you very much for your time.''
   • ``Are you a student here at the London Business School?''
   • ``Were you planning to search the catalog by a particular author or title, or were you planning to search by a topic/subject area?''

2. Determine suitability of possible participants:
   • For suitable candidate (an LBS student who intends to perform a subject related search), ``Would you mind taking part in a short experiment? It will take around half an hour and will involve you carrying out a library search on an experimental online catalogue. Everyone who takes part in the experiment will receive a small gift.''
   • If they are not suitable, or if they do not wish to participate further, ``Thank you very much for your time.''

3. Show participants to the test location: ask them to be seated at the terminal.

4. Explain the test goals: ``We are in the process of creating a new method for searching by subject in databases such as library catalogues, and we are trying to make it as easy as possible to use. We need your help in evaluating this system, since you are the type of person who will be using it. The set of database records that you will be searching is the same as the LBS TINMAN Library Catalog that you were about to use.''

5. Hand them the participant questionnaire and ask: ``So that we have a profile of those who have helped us with this evaluation, would you mind filling out this brief questionnaire?''

6. Explain the test process: ``What I would like you to do is search for the topic or one of the topics you were going to search for at the online catalogue in the library.''

7. Explain the think-aloud process: ``Because we are interested in what you are thinking, I would like you to speak aloud about what is going through your mind whilst you are searching. If you are silent for any length of time, I may remind you to continue thinking aloud. Describe what you are doing and what you expect to happen. If something is confusing or surprising, please comment on it and tell me why. Please feel free to comment at any time about anything on the screen that catches your attention. Think of this as you giving me a running commentary on what you are doing and thinking. I will be noting the problems you mention, but in case I miss anything I would like to record your comments. This recording will be anonymous and treated in confidence. Would that be okay?'' [Position recorder on table.]

8. Final instructions: ``Please remember that it is not you who is being tested, but our experimental database system and its interface. Take as much or as little time as you want. If you want to print any records for your own use later, you do so by clicking the Print button at any time. Let me know when you have finished. Afterwards, I would like to take about 15 more minutes to discuss the system with you and have you describe some of your impressions of it.''

9. Begin test: press Record button on tape recorder, note time. ``Before we start, could you tell me what topic(s) you are going to be searching for?'' [Make a written note of the participant's statement of the search topic at the top of the Data Collection Form.] ``You may begin searching for information on your topic, and remember to think out loud about each thing that you are doing, why you select certain things, what is confusing, etc. Do you have any questions? Okay, you may begin.''

10. During the test:
    • Data Collection. Using the Data Collection Form, record the original query as entered as well as instances instances of surprising or unanticipated events. These should be classified as far as possible into the stages of the search process. Make notes of unanticipated reactions and situations so that they may be discussed further in the interview.

    • Prompting. If a user is silent for more than a few seconds, prompt with such statements as ``Could you tell me what you are thinking?'' or ``Could you explain to me what you are doing now?'' If the participant is not speaking loudly enough, prompt with ``I could not hear what you said just then,'' or some such statements.

11. Ending the test: when the participant indicates that they are finished, record the time. [Leave microphone on --- go directly to Post-Test Interview.] ``Allow me one moment to print out the log of your search, and then we can discuss your overall impressions.''

12. Post-test interview:
    • ``I would now like to ask you some questions so that you can further describe what you liked, what you didn't like, what would have been helpful, etc. Feel free to make any comments that you think would be helpful for improving the system for future users.

    • Using the Data Collection Form, transaction log and screen images as aids, ask the participant the questions from the Interview Topics form. These questions should be tailored to specific instances that users' search session in order to create as much of a dialogue as possible. Direct answers should be recorded also on the Data Collection in a different coloured ink so that data collected during and after the test may be differentiated.

13. Finishing: ``Thank you very much for your assistance with our evaluation. You gave some very helpful information that should help us to improve our system. Here is a small gift in appreciation for your time time. I hope that you have a pleasant day.''