Continuously Modeling User Emotion in Interactive Play Environments: A Fuzzy Physiological Approach
Researchers are using emerging technologies to develop novel play environments, while established computer and console game markets continue to grow rapidly. Even so, evaluating the success of interactive play environments is still an open research challenge. Both subjective and objective techniques fall short due to limited evaluative bandwidth; there remains no corollary in play environments to task performance with productivity systems. This research presents a method of modeling user emotional state, based on a user's physiology, for users interacting with play technologies. We developed a fuzzy logic model that transforms four physiological signals (GSR, EMGsmile, EMGfrown, and HR) into arousal and valence. A second fuzzy logic model transforms arousal and valance into five emotional states: boredom, challenge, excitement, frustration, and fun.
The above figure demonstrates the quadrant display including: the camera feed of the participants, a screen capture of the physiological data, a screen capture of the game, and audio from the participants and the game.
Modeled emotions are powerful because they capture usability and playability through metrics relevant to ludic experience; account for user emotion; are quantitative and objective; and are represented continuously over a session, pinpointing moments in time when a user's emotional state changes. Researchers and developers can uncover individual moments when a user begins to get stressed, starts having fun, or becomes bored. This information could be used as an evaluative tool, or could be used to dynamically adapt game settings to keep players engaged.
We tested our model under three different collaborative play conditions: playing against a computer, against a co-located friend, and against a co-located stranger. Our modeled emotions show the same trends as reported emotions for fun, boredom, and excitement; however, the modeled emotions revealed differences between the three play conditions, while the differences between the subjective reports failed to reach significance.
The above figure shows frustration for one participant in three conditions. Examining the mean output may reveal differences between conditions; however, examining the entire time series reveals how a participant's emotional state changes over time.
Current and Future Directions
We are currently working on a tool to assist researchers with observation analysis from video. Using modeled emotions, researchers can scrub through the related video source, addressing interesting elements, and dismissing the bulk of the video. This will assist researchers and developers of game technologies by drastically reducing the time necessary to analyze video.
We are also looking at different modeling methods, besides fuzzy logic, and modeling more game-related emotional states such as schadenfreude or pride.
Publications
R.L. Mandryk, M.S. Atkins, and K.M. Inkpen (2006). A Continuous and Objective Evaluation of Emotional Experience with Interactive Play Environments, Accepted to the Conference on Human Factors in Computing Systems (CHI 2006), Montréal, Québec, Canada, April 2006. PDF
R.L. Mandryk, K.M. Inkpen, and T.W. Calvert. (2006). Using Psychophysiological Techniques to Measure User Experience with Entertainment Technologies. Journal of Behaviour and Information Technology (Special Issue on User Experience). Vol. 25, No. 2, March-April 2006, pg. 141-158. Preprint PDF (email Regan for a copy of the published paper)
R.L. Mandryk, and K.M. Inkpen. (2004). Physiological Indicators for the Evaluation of Co-located Collaborative Play, In Proceedings of CSCW 2004, Chicago, USA, pg. 102-111. PDF
R.L. Mandryk (2004). Objectively Evaluating Entertainment Technology (Doctoral Consortium Paper), in Extended Abstracts of CHI 2004, Vienna, Austria, pg. 1057-1058. PDF