Interactions based on Emotions

Whilst techniques for narrative generation and agent behaviour have made significant progress in recent years, natural language processing remains a bottleneck hampering the scalability of IS systems. Natural language communication is an essential element of interactive narratives in which user interaction is often based on natural language communication with virtual agents, either through isolated utterances or through dialogue.

To address this challenge, we introduce a novel interaction technique based solely on emotional speech recognition. It allows the user to take part in dialogues with virtual actors without any constraints on style or expressivity. This is achieved via recognition of emotional categories using emotional speech recoginser EmoVoice which are mapped to narrative situations and virtual characters' feelings.

The emotional category extracted from a user utterance can be analysed in terms of the current narrative context, which includes characters’ beliefs, feelings and expectations, to produce a specific influence on the target character. User interactions in the virtual world modify either the world state or the agent’s beliefs, so that when the virtual actor generates its next action as part of its role, it propagates the influence of the user intervention potentially altering the course of the story. This will then become visible through a change in the characters’ behaviour, thus achieving a high level of realism in the interaction. A limited number of emotional categories is sufficient to drive the narrative across multiple courses of actions, since it comprises over thirty narrative functions.

The last version of the system supports multimodal interaction in the form of emotional speech recognition (which allows the extraction of emotional categories from user utterances without placing any constraint on them) and user attitude detection (measuring user distance to a character, along with her body and head orientation to determine attention or interest).

Another interaction modality, which naturally derives from the story embodiment in a virtual world, consists in interacting with narrative objects: the user can grab and offer Emma flowers, steal the flowers Rodolphe intended to offer her or the book she wanted to give Rodolphe, all of which has consequences on the emotional state of one or more of the characters.

To evaluate the impact of interactions and how the generated narrative may depart from the baseline story, diagrams like the following one were designed to illustrate different story evolutions depending on the emotional categories recognised. In this portion of the novel, a dimensional model opposing duty to pleasure characterises the driving forces behind Emma’s behaviour.

Videos

Positive Speech Uterance	Negative Speech Uterance
Physical Interaction	Narrative Object

Videos available on our YouTube channels: VirtualEmma and JLTeesside