PARADISE-style Evaluation of a Human-Human Library Corpus

Rebecca J. Passonneau,  Irene Alvarado,  Phil Crone,  Simon Jerome
Columbia University


Abstract

We apply a PARADISE-style evaluation to a human-human dialogue corpus. It was originally collected to support the design of a spoken dialogue system for library transactions. User satisfaction is not well predicted by measures of success and costs alone. We identify two qualitative characteristics that were somewhat predictive. The number of overlapping turns in non-task oriented segments of dialogue was predictive, presumably because the overlapping turns reflect the level of rapport between the speakers. The number of non-specific book requests, also predictive, indicate how often the librarian was able to identify a specific book to satisfy a nonspecific desire.