On Adding Haptic Feedback to Interaction with Unconventional Display Devices

The sense of touch is important to human interaction as well as for interacting with our surroundings. Touch and tactile feedback are essential also when interacting with digital technology. The present doctoral thesis aimed to discover a profound understanding of how touch sensations could be added to unconventional display devices that do not inherently provide tactile feedback. Novel and less explored devices and use cases were studied in five carefully controlled studies.

Study I introduced a novel lightweight, environment sensing limitedly volumetric permeable display. It could be interacted with by swiping the display in mid-air to, for example, reveal slices of medical magnetic resonance images. The results of this study consisted of the introduction of a novel interactive display device and use cases for it.

Studies II and III investigated the use of ultrasonic mid-air haptic feedback with unconventional displays. In both studies, the participants were to enter numbers by tapping virtual buttons. Their hands were tracked, and the tasks were repeated with and without haptic feedback. In study II, the user interface was presented using a permeable display formed of flowing light-scattering particles. In study III, the user interface was presented using a head-mounted virtual reality display. The results showed that ultrasonic mid-air haptics can be a suitable method for tactile feedback in intangible user interfaces. In terms of task completion times and error rates, there were no statistically significant differences in entering the numbers with or without tactile feedback. On the other hand, the addition of ultrasonic feedback was uniformly preferred by the users.

Study IV investigated the effect of adding vibrotactile actuation to gestural interaction with a large permeable display. Tapping and dwell-based interactions were supplemented with feedback from a custom-built wireless wearable actuation device in addition to audio-visual feedback. The experiments were repeated with and without the haptic device. The results showed that while tapping on a target was more efficient than dwelling over it, limitations in tracking quality made the use of dwell-based selection methods more robust, as is often the case. Vibrotactile feedback was preferred by the users in dwell-based interaction over audio-visual feedback alone.

The aim of study V was to investigate the differentiation of six ultrasonic tactile stimulations that were varied by form (i.e., square and circle) and timing (i.e., movement speed and duration, and the frequency of tactile phases within a stimulus). In a stimulus familiarization task, participants were introduced to all stimuli in a specific order (1 to 6) and repeated four times. After this, participants were to identify the stimuli presented in random order by pressing the number key corresponding to the stimulus index used in the familiarization task. The results showed that stimuli formed of repetitive 400ms pulses were easier and more reliably identified than those with a shorter duration regardless of the number of repetitions. Post-experiment interviews revealed that the changes in form were not noticed.

In conclusion, the results showed that ultrasonic mid-air haptic stimuli are a well-suited method for feedback delivery in permeable and virtual displays. Further, the results showed that users uniformly preferred the addition of haptic feedback to interaction with intangible user interfaces. In addition, the results showed that mid-air tactile stimuli can be designed so that they were reliably identifiable after minimal familiarization and hence can be utilized for efficient information transfer on tactile displays. Taken together, the findings of this thesis suggest functional solutions for adding haptic feedback in interaction with displays that currently are classified as unconventional but will become more mainstream technologies in the future.