Interactive Gesture in Dialogue: a PTT Model

Hannes Rieser and Massimo Poesio

SIGDIAL Workshop on Discourse and Dialogue (SIGDIAL 2009)
Queen Mary University of London, September 11-12, 2009


Abstract Gestures are usually looked at in isolation or from an intra-propositional perspective essentially tied to one speaker. The Bielefeld multi-modal Speech-And- Gesture-Alignment (SAGA) corpus has many interactive gestures relevant for the structure of dialogue (Anon 2008, 2009). To describe them, a dialogue theory is needed which can serve as a speech-gesture interface. PTT (Anon and Traum 1997, Anon and Anon submitted a) can do this job in principle, how this can be achieved is the main topic of this paper. As a precondition, the empirical research procedure from systematic corpus annotation via gesture typology to a partial ontology for gestures is described. It is then explained how PTT is extended to provide an incremental modelling of speech plus gesture in an assertion-acknowledgement adjacency pair where grounding between dialogue participants is obtained through gesture.