The Role of Interactivity in Human-Machine Conversation for Automatic Word Acquisition

Shaolin Qu and Joyce Chai

SIGDIAL Workshop on Discourse and Dialogue (SIGDIAL 2009)
Queen Mary University of London, September 11-12, 2009

Summary

Motivated by the psycholinguistic finding that human eye gaze is tightly linked to speech production, previous work has applied naturally occurring eye gaze for automatic vocabulary acquisition. However, unlike in the typical settings for psycholinguistic studies, eye gaze can serve different functions in human-machine conversation. Some gaze streams do not link to the content of the spoken utterances and thus can be potentially detrimental to word acquisition. To address this problem, this paper investigates the incorporation of interactivity in identifying the closely coupled speech and gaze streams for word acquisition. Our empirical results indicate that the automatically identified gaze-speech streams lead to significantly better word acquisition performance.