Documentation of Natural Human-Computer Interaction in Ubiquitous Environments

Future user interfaces are going to be ubiquitous and seamlessly integrated into the world, offer a large variety of interaction modalities, facilitate situation-dependent adaptation to an increasing diversity of heterogeneous contexts of use and become more tangible, concrete and touchable again through connections with real objects.
Interaction designers has to document best practices for such natural interactions appropriately so that other practitioners can use them easily.
A common and problem-oriented method for capturing design knowledge is using interaction patterns. Standard pattern formats, e.g. Pattern Language Markup Language (PLML) [1], specify the upper layer with common elements like pattern name, problem and solution. However, interaction patterns are delivered mostly in narrative and unstructured text [2]. And in doing so, they mainly address the presentation of user interfaces.
Natural interactions comes along with special characteristics which go far beyond a presentation of user interface elements. Hence, it is a lack of structured and detailed information for supporting tangible, multimodal, and situation-dependent interactions of future user interfaces.

This thesis aims at developing an enhanced pattern format and investigating the aspects of characteristic natural interactions between users and interfaces. An important aspect are parameters for describing natural interaction behavior, for example movements and gestures and their characteristics like speed, duration, start or end postures. Further on, the interaction context like the users situation, social and environmental affects are an decisive factor for applying a pattern.
For investigating, three fields are focused:

  • Augmented Reality (AR): interactions with virtual objects in the real world
  • Device-based interactions: gestures with mobile devices held in the hand
  • Wearable interactions: sensors and actuators attached to the body

The findings will lead to a framework for creating more consistent patterns for natural human-computer interactions.
It is intended to give assistance in creating and applying patterns with an improved format. In addition, interaction catalogs for each of the interaction fields are developed to show concrete applications.

[1] Sally Fincher, Janet Finlay, Sharon Greene, Lauretta Jones, Paul Matchen, John Thomas, and Pedro
J. Molina, „Perspectives on HCI Patterns: Concepts and Tools“, CHI ’03 Extended Abstracts on
Human Factors in Computing Systems, CHI EA ’03, ACM, pp. 1044-1045, 2003.

[2] Ahmed Seffah, „HCI Pattern Capture and Dissemination: Practices, Lifecycle, and Tools,“ Patterns of HCI Design and HCI Design of Patterns: Bridging HCI Design and Model-Driven Software Engineering, Springer International Publishing, pp. 219-242, 2015.