Paper Reading #21: A Multimodal Labeling Interface for Wearable Computing

Comments
Steven Hennessy 
Jeremy Nelissen

Reference Information
Title: A Multimodal Labeling Interface for Wearable Computing
Authors: Shanqing Li, Yunde Jia
When/Where: IUI 2010

Summary
This paper proposed a system for labeling objects in wearable environments using a multimodal interface instead of with the inconvenient mice and keyboard. The program uses visual tracking to allow the user to circle on object by pointing, then the user speaks the name of the object and speech recognition is used to get the label. The visual tracking and speech recognition is performed in real-time. The system hardware consists of a head-mounted display to track the user's circling of objects, a wearable computer, and a Bluetooth microphone to capture the user's voice.

View of the system in process. Source: Article.

Experiments were held with the system to evaluate it, involving a user circling various objects and speaking the name "circle". By comparing these results with those using a mice and keyboard, the evaluation showed that this approach was more efficient that mice and keyboard for wearable environments.

Discussion
I thought this system was pretty interesting. It makes sense that for this kind of environment, a keyboard and mouse would be inconvenient, so this proposed approach seems like it would be efficient. However, it seems like its effectiveness would depend largely on how well the speech recognition and hand-tracking recognition systems work. Also, most of the paper was largely technical and mathematical, making it rather hard to understand. However, I like the idea of changing the input methods for unique systems such that the method of input is adapted to the system itself.