IMPLEMENTATION & EVALUATION

3.4.1. Make learners try the Mobile App online (e.g., home) and/or offline (e.g., classroom)

The finalized app is tested via a pilot test in this phase. Several representative users use the app in a natural setting. Since the purpose is to develop an app for educational purposes, the representative users are learners and the setting is learning environment. Therefore, the app is tested by learners in the learning environment containing online (e.g., home) and/or offline (e.g., classroom) environment. However, if there is no such situation requiring using the app for learning purposes in the learning environment, then the learners are provided an artificial scenario by the tester.

3.4.2. Conduct field observation

During the app testing by the learners in the learning environment, field observation is conducted since field observation is an appropriate evaluation method of a software in the testing phase, as shown in Table 12.

The learners are observed to understand how they are using the app to

accomplish the tasks and what kind of mental model the learners have about the system (Nielsen, 1993). The learners should be allowed to use the app not only in a classroom environment but also outside of school.

3.4.3. Conduct usability questionnaire

After learners tested the app, usability questionnaire is conducted with the learners who attended to field test, in order to measure the usability score of the app. Questionnaires have long been used to evaluate user interfaces (Root & Draper, 1983) and it is also an appropriate method as the evaluation method of the testing app (Folmer & Bosch, 2004). There is a variety of proposed questionnaire instruments in order to measure the usability level of a product, such as widely used SUS (System Usability Scale) (Brooke, 1996), Questionnaire for User Interface Satisfaction (Chin et al, 1988), Perceived Usefulness and Ease of Use (Davis, 1989), Purdue Usability Testing Questionnaire and USE Questionnaire (Lund, 2001).

However, some studies state that traditional guidelines and methods used in usability testing of desktop applications might not be directly applicable to a mobile environment since mobile devices having unique challenges such as mobile context, connectivity, small screen size, different display resolutions, limited processing capability and power, and data entry methods (Zhang &

Adipat, 2005). Therefore, recent studies have focused on developing new usability questionnaire (Hoehle & Venkatesh, 2015) as well as creating a generic framework for developing and conducting usability testings for mobile applications (Harrison et al., 2013; Zhang & Adipat, 2005). Harrison et al. (2013) proposed a usability model called PACMAD (People At the Centre of Mobile Application Development), including the following seven

attributes:

Effectiveness:the ability of a user to complete a task in a specified context.

Efficiency: the ability of the user to complete their task with speed and accuracy.

Errors:how well the user can complete the desired tasks without errors.

Learnability: the ease with which users can gain proficiency with an application.

Memorability: the ability of a user to retain how to use an application effectively.

Cognitive Load:analyzes the impact that using the mobile device will have on the user’s performance.

Satisfaction:the perceived level of comfort and pleasantness afforded to the user through the use of the software.

Because mobile technologies have been evolving day by day, their characteristics have been changing as well, such as interaction with a multi touch screen, displays of different resolutions and dimensions, device orientation changes, and gestures like tap, flick, and pinch (Nayebi et al., 2012). Therefore, this study suggests developers utilize this PACMAD usability model when creating their usability questionnaire items that suitable for their apps and their technology. However, if there is a lack of time for creating own usability questionnaire, it is also possible to apply straight usability questionnaires such as SUS scale because they are also utilized for mobile apps. For instance, Kortum and Sorber (2015) has

successfully applied SUS scale in order to measure the usability of several mobile applications. If the SUS score of a system like mobile apps is above 68, then it is considered that the system is above the average of other systems.

Figure 16 shows the acceptable range of SUS scores (Bangor et al., 2009).

Figure 16. A comparison of mean System Usability Scale (SUS) scores by quartile, adjective ratings, and the acceptability of the overall SUS score (Bangor et al., 2009)

3.4.4. Conduct semi-structured interview with learners

After conducting field test and usability questionnaire, the next step is to make interviews with the learners who participated in the field test, in order to gather detailed information such as their likes and dislikes about the app. Interviews are an effective and appropriate evaluation method for evaluating the app in the testing phase (Folmer & Bosch, 2004). There are two methods of interviewing consisting of the unstructured and structured interview. While unstructured interviews do not have a well-defined agenda and are not concerned with any detailed aspects of the systems, structured interviews have a specific, predetermined agenda with specific questions (Nielson, 1993).

The interviewer can utilize the following guidelines when holding a

usability interview (Nielson, 1993). First, record the interview. Taking notes is a kind of distraction to the interviewee, who will have to restrain him/herself from looking the what is being written. Second, express the questions in an open or neutral way. In addition, encourage the user to reply with full sentences, rather than a simple "yes" or "no". Third, begin with less complicated issues and move to more complex subjects. Fourth, ask questions to disclose more information, not to approve the investigator's beliefs. Sixth, include instructions about the answer. For instance, answers can range from long descriptions to simple explanations, to identification or brief selection, to a simple "yes" or "no". Seventh, do not try to express to a subject why the system behaved in a particular way. Do not justify the design decision. Eighth, avoid using jargon and use terms that the interviewee can understand. Ninth, do not ask leading questions. A leading question implies that a situation exists and influences the direction of response. Tenth, do not agree or disagree with the user; remain neutral. Eleventh, use probes to acquire more information after the original question is answered. Probes are used to promote the subjects to continue speaking, or to guide their answer in a particular direction so a maximum amount of useful information is gathered.

However, this study suggests conducting semi-structured interviews with at least two learners participated in the field test, as suggested by Nielson (1993). Since the semi-structured interviews have some predetermined questions, the acquired information will be neither from a very broad view nor from very narrow perspective. Such questions might be

“What parts of the system did you think were well designed?”, “Which parts of the system did you think were inadequately designed?”, and “Do you have

any other comments about the system functions and regarding its usability?”, which are used by Georgsson and Staggers (2016) for the evaluation of a mobile system.