USER RESEARCH

Role 

  • User researcher 

  • Associate user tester with Matt Sia

Problem 

A research plan and testing was needed before game development got any further ahead than design.

Tools 

  • GSuite

Objectives

  1. Establish research outline for Argotian

  2. Conduct research for the VR game

  3. Present findings and suggestions to key stakeholders

 

Overview

We arranged research sessions at the downtown SFU campus and at a local high school. 

At SFU, we were only able to conduct observational research as the class time did not allow for interviews.

At the high school, we were able to work with the students on a group level and an individual level. 

Students were divided by their English listening, speaking, reading and writing skills into: High Basic and Low Intermediate.

Research Goal

 

Aim: get a deeper understanding of how Argotian could be used with mature ESL students (18+yrs) and in a high school classroom.

As the nature of both of these settings are quite different, different outcomes were expected allowing for a broader scope of research. The college play-testing gave a broad overview of how Argotian can be integrated into the classroom. For the individual play-testing at the high school, more specific and micro research gave a deeper understanding of user issues with the game.

 

Participants

For the SFU students, the nature of the research was quite passive (i.e. observational research) but each session still had a directive. Students would either have a "discovery" session where they could amble through the game at their own pace or they were tasked with completing a certain action (e.g. teleport throughout the space and find a certain character). It was important to see how students navigated the game with little to no interference. This way, obvious end user pain points arose would inform design iterations. 

As it was not always possible for me to conduct the on-site research, I drew up a structure that could be applied to each session at the college by other testers. The P.E.M. (people, environment, messages) structure ensured there was consistency across every testing session. 

For the “people” portion, it was important to record as much information as possible without noting down personal information as these were a mixture of high school and college students. The immersive nature of the game is largely dependent on where it is being played, so for the “environment” portion, it was important to note any factors that may inhibit players from getting immersed. The “messages” portion is arguably the most valuable section of the research. Here, testers were encouraged to take notes of organic conversations participants have when using the product. ​

Here is the document that was used in each observational session at SFU:

P
P

PEM-2
PEM-2

PEM-4
PEM-4

P
P

1/4
 

Results

The observational research conducted at SFU was extremely insightful. We learned that students would talk to themselves in their native tongue to explain certain game mechanics to themselves. Students exiting out of Argotian and exploring other VR games available in the headsets was an unfortunate common occurrence. The motivated learners tended to use more space available to them and found it easier to stand up. 

As for the high school students, here are some questionnaire highlights:

What students liked about Argotian

Screen Shot 2020-06-12 at 9.48.53 PM.png

As shown, students had varied responses. It was encouraging that they enjoyed the main premise of the game of talking to the characters. A huge bonus was they felt it improved their English skills in a safe practice space as shown by the following graph. 

Whether it improved their oral skills

Screen Shot 2020-06-12 at 9.49.49 PM.png

High basic students seemed to benefit the most. This could be that they felt more confident to talk compared to their low intermediate counterparts.

What they did not enjoy

Screen Shot 2020-06-12 at 9.50.08 PM.png

Students reported feeling unsure of what to say to the AI characters and frustrated that the characters did not respond. This informed changes to how we explained this game feature in the tutorial. The game team worked hard to further incentivise Argotian so that learners would be encouraged to get to the end of the game.

Reflection

Working with JOSS and the SFU ESL students was extremely insightful. Direct pain points from Argotian’s target end users helped to inform the overall direction of the game.

Through play-testing with the JOSS students, key features relating to accessibility (colours that were used), character’s aesthetic, and the environment changed based on these responses. It was clear that more focus needed to be put on the gamification of Argotian. Observational research with the SFU students and their teaching instructors provided valuable information on how the game could fit in the classroom context. 

The team at Virtro has since made exciting changes to the game that I am sure will make an impact on language learning in VR.  

I really enjoyed conducting user research for Argotian. I was able to see how the game functioned in classrooms and get a first look at what changes were required. It tested my communication skills as I had to synthesize my findings and present them to key stakeholders and teams in a palatable way. Aside from competitive analysis, user research is certainly engaging and a pivotal step in the design process. 

Other related projects I was involved with include the VR TUTORIAL where I provided game design, and the PORTAL that is a web-based display of gameplay metrics.