Title: A Toolkit for 3D-Gesture and Speech Dialog in Automotive Environments
Authors: Timo Sowa, Alexander Richter, Dietmar Fey
Abstract:
3D-gesture is a new input modality that complements touch-screens and
speech dialog systems for automotive infotainment systems. Yet, tools for im-
plementing multimodal interfaces including 3D-gestures are rare. We present a
toolkit for in-car human-machine interfaces (HMIs), that allows modeling 3D-
gesture recognition and integration of gestures with speech. Our approach focuses
on three interaction styles: Firstly, we demonstrate how dialogs containing isolated
gestures, sequences of speech and gesture, and alternative uses of different modal-
ities can be modeled based on the state graph paradigm. Secondly, we show how
continuous gestural interaction, such as gradually modifying a value with continu-
ous hand movements, can be designed by binding hand model parameters to HMI
model values. Thirdly, we present an approach toward modeling and processing
of multimodal utterances with semantics distributed over speech and 3D-gesture.
The approach is based on a semantic representation of the individual modalities as
typed feature structures which are collected in a short-term memory. An integrator
module unifies partial meanings until a fully specified command is created which
can be executed. The approach has been implemented in a demonstration system
using a Leap Motion controller.
Year: 2018
In session: Demo Session
Pages: 285 to 292