Instruments

The combination of multiple data collection instruments facilitates the capture of a comprehensive and intensive picture of complex phenomena. Even if each data collection method delivers only partial evidence on the phenomenon, several methods together could cover multiple aspects. In addition, multiple methods allow for cross-checking the results of each, thus increasing the reliability and validity of the data: giving rise to the term triangulation (Ingwersen & Jarvelin, 2005). People’s

64

Web search interaction is a sort of complicated activity involved with a lot of cognitive and behaviour efforts. The method of interview or survey could provide only a partial picture of the interaction because users may not be aware of or able to recollect what they did during the searching process. Triangulation through multiple methods revealed a more complete picture of users’ Web searching behaviour.

Saracevic, Mokros and Su (1990) used videotaping, search logging and observation to collect data on the nature of online searching interaction between users and intermediaries. Bystrom (1999) used multiple data collection methods including theme interviews, observation and diaries in the study on civil servants’

information seeking behaviour. In addition to a pre-search questionnaire, Wang, Hawk & Tenopir (2000) employed a process-tracing technique for recording individual user's processes and behaviours, specifically the transactions with timestamps and verbal reports as they interacted with the Web when searching for information. These studies provided good examples of using a variety of methods to collect data in order to explore users’ information behaviour.

In this study, a combination of multiple data collection techniques, including pre- and post-search questionnaires, think-aloud protocols, observations, Web search logs, and post-search interviews, were used to gather data. Multiple methods provided in-depth and comprehensive insights into the users’ Web searching behaviour.

Pre- and Post-Web Search Questionnaires

A questionnaire requires less effort but can be an effective technique to gather information directly from study participants.

65

Basic information of study participants was gathered through a pre-search questionnaire, including variables of age, gender, affiliation, student status, and Web using experiences. The following questions partly based on the questions raised in Spink, Park & Koshman’s (2006) study were designed to indicate each study participant’s Web using experiences:

• How long have you been using the Web to look for information (one year–

five years; six years–ten years; eleven years and over)?

• Which Web browser is used most frequently for information browsing (Ex.

Internet Explorer, Netscape, Mozilla, Maxthon, MyIE, and FireFox)?

• Which Web search engines do you use most frequently for information?

Both a post-Web search questionnaire and a pre-Web search questionnaire were used to capture the transition of information users’ state before and after the search (Spink & Dee, 2007). This allowed the measurement of cognitive shifts of study participants that resulted from their Web search.

The pre-and post-Web search questionnaires are included in Appendix B and Appendix C, respectively.

Think-aloud Protocols

Think-aloud or verbalization of a person's thoughts while undertaking a cognitive activity has been used for years as an assessment tool by psychologists and reading researchers to investigate the mental processes applied to various kinds of thinking, problem-solving, and reading comprehension tasks (Smith, 2006).

The advantages of think-aloud and recording lie in the real-time attribute that everything is recorded in real time, and that the often complicated cognitive tasks,

66

which take place over long periods of time, can be analysed. In addition, thinking aloud could produce valid verbal accounts of cognitive processes (Ericsson &

Simon, 1993). The disadvantage of the think-aloud method originates from its obtrusiveness, which may lead to a validity problem—there is uncertainty about the degree to which the protocols reflect actual thoughts or intentionality of actions, and bring forward reliable data (Ingwersen, 1992). In a comprehensive summary, Ericsson and Simon (1993) argued that the think-aloud method is an accurate and representative measurement of cognitive processes, particularly when subjects are reporting memory traces that are already in verbal form before they begin the process of verbalizing about them. This condition is presumably met in Web searching study because individuals will verbalise while solving information problems.

Yang (1997) used think-aloud and protocol analysis in the study on information seeking and retrieving in hypertext. The verbal data were recorded and analysed to find the goal-oriented and information seeking patterns in the subjects’ problem solving. Spink, Park, and Koshman (2006) asked subjects to think aloud as they searched and were encouraged to express the reasons for their Web search actions.

Think-aloud protocols provided rich data in the study, including both operational steps and cognitive moves. Study participants were required to verbalise everything related to the searches, including thoughts, motivations/reasons, and actions, as they were searching on the Web. The verbal stream was recorded through Camtasia Studio software. These think-aloud data uncovered important cognitive processes surrounding the Web searching actions and decision making processes.

Cognitive elements identified through study participants’ verbal reports included information problem identification, clarification, and query formulation. Verbal

67

reports also reflected how the study participants moved between cognitive states, for example, from relevance judgment to query formulation.

The recorded think-aloud utterances offered an excellent vehicle to the researcher to access the study participant’s thoughts and an opportunity to explain and confirm the Web search logs which logged their actions of interacting with Web search systems.

Observation

Observation allows real time data collection. The value of this method is that it permits researchers to study people in their native environment in order to understand "things" from their perspective. In the process of observation, the researcher uses all of his/her senses to gather information about the phenomena under study (Adler & Adler, 1994). While it may affect the search process because of obtrusiveness, it may also provide a rich data set for analysis. Non-participant observation requires a longer period of accommodation by those being observed to return to natural behaviour, but it also allows the observer to concentrate on the observation process (Krathwohl, 1997). Spink (2004) used observation as one of the data collection methods to explore from a distance the multitasking information behaviour by an information seeker in a public library during library visits. On-site observation was also used in Spink, Park and Cole’s (2006) research to examine how a business consultant conducted multitasking work-task behaviours.

Non-participant observation was used in this study to familiarise the researcher with the users’ Web searching processes by generating notes to supplement thinking aloud audio data. The observations were carried out as unobtrusively as possible.

Everything was done as it would have proceeded without the observation. The researcher interrupted study participants only when reminding them about how

68

much time was left or when asking the study participants to speak loudly as doing the searching. The observation notes and comments were manually recorded by the researcher at the same time. The observation notes and comments helped the researcher to construct certain interview questions for each study participant afterwards.

Web Search Logs

During Web searching, search logs can be collected unobtrusively and economically. Web search engine logs analysis has been employed successfully to reveal information about users’ search strategies, the products of users’ minds (Hider, 2007). Many search engine log studies have been performed since the late 1990s (Jansen & Pooch, 2001). For instance, Spink, Park, Jansen and Pedersen (2006) used transaction logs of AltaVista Web search engine to analyse two-query and three-or-more-query search sessions respectively, in order to examine the characteristics of users' task switching and multitasking information behaviour during Web searching sessions. Nevertheless, the data are poor if this method is used alone, since they lack most traces of a searcher’s intentions and thoughts during Web interaction (Ingwersen & Jarvelin, 2005).

It is understandable that study participants could verbalise only a subset of the thoughts occurring during the interactions because some thoughts were difficult to verbalise (Wang, Hawk & Tenopir, 2000). The method of Web search logs captured all of the screens and the moves that study participants made, including those which study participants were not aware of or were not able to verbalise. In this study, characteristics of actual searching behaviours of study participants and the duration of search sessions were recorded by the search logs. The Web search logs which recorded each study participant’s Web searching interactions were captured and saved by Camtasia Studio software. Analysis of the Web search logs

69

along with the verbal reports provided insight into the study participants’ Web searching behaviour and thoughts.

Post-search Interview

Interviewing requires more effort than questionnaires but provides the possibility for more thorough analysis of Web search behaviour. It may effectively augment and validate data collected through previous methods, such as observation, which may be incomplete and/or given incorrect interpretations. Therefore, a clarifying interview used as triangulation of data may be very helpful (Ingwersen & Jarvelin, 2005). Dervin (1992) and Schamber (2000) developed a micro-moment time-line interview technique for the sense-making approach, which involved asking the respondent in detail what happened in a problematic situation step by step. Spink, Park and Cole (2006) interviewed a business consultant about his multitasking behaviours. The goal of the open-ended interview technique was to discuss in detail the consultant’s information and non-information tasks and processes.

In the study, a post-Web search interview was conducted with each study participant, immediately after their Web search interactions, in order to allow further elaboration of their actions and underlying thoughts. The interview with study participants made the acquisition of retrospective data possible. We could seek evidence or further explanation of what was found in the think-aloud utterances.

Interview questions were formulated on the semi-structured interview outline and the individual-based observation notes. The researcher modified questions or added additional ones as deemed appropriate. The interviews were recorded using an MP3 player at the same time. The semi-structured interview questions are included in Appendix D.

70