Timo Arnall

Mikael Fernström gave a lecture at AHO on sound objects this week. His work at IDC focuses on sound in ubiquitous computing, an area that is relatively unexplored in interaction design.

These are some of my notes from his lecture, and our discussion over lunch.

The aim of the Soundobject research is to liberate interaction design from visual dominance, to free up our eyes, and to do what small displays don’t do well.

Reasons for focusing on sound:

• Sound is currently under-utilised in interaction design
• Vision is overloaded and our auditory senses are seldom engaged
• In the world we are used to hearing a lot

• Adding sound to existing, optimised visual interfaces does not add much to usability

  Sound is very good at attracting our attention, so we have alarms and notification systems that successfully use sound in communication and interaction. We talked about using ‘caller groups’ on mobile phones where people in an address book can be assigned different ringtones, and how effective it was in changing our relationship with our phones. In fact it’s possible to sleep through unimportant calls: our brains are processing and evaluating sound while we sleep.

  One fascinating thing that I hadn’t considered is that sound is our fastest sense: it has an extremely high temporal resolution (ten times faster than vision), so for instance our ears can hear pulses at a much higher rate than our eyes can watch a flashing light.

  Disadvantages of sound objects

  Sound is not good for continuous representation because we cannot shut out sound in the way we can divert our visual attention. It’s also not good for absolute display: pitch, loudness and timbre are relative to most people, even people that have absolute pitch can be affected by contextual sounds. And context is a big issue: loud or quiet environments affect the way that sound must be used in interfaces: libraries and airplanes for example.

  There are also big problems with spatial representation in sound, techniques that mimic the position of sound based on binaural differences are inaccessible by about a fifth of the population. This perception of space in sound is also intricately linked with the position and movement of the head. Some Google searches on spatial representation of sound. See also Psychophysical Scaling of Sonification Mappings [pdf]

  Cartoonification

  ‘Filling a bottle with water’ is a sound that could work as part of an interface, representing actions such as downloading, uploading or in replacement of progress bars. The sound can be abstracted into a ‘cartoonification’ that works more effectively: the abstraction separates simulated sounds from everyday sounds.

  Mikael cites inspiration from foley artists working on film sound design, that are experienced in emphasising and simplifying sound actions, and in creating dynamic sound environments, especially in animation.

  A side effect of this ‘cartoonification’ is that sounds can be generated in simpler ways: reducing processing and memory overhead in mobile devices. In fact all of the soundobject experiments rely on parametric sound synthesis using PureData: generated on the fly rather than using sampled sound files, resulting in small, fast, adaptive interface environments (sound files and the PD files used to generate the sounds can be found at the Soundobject site).

  One exciting and pragmatic idea that Mikael mentioned was simulating ‘peas in a tin’ to hear how much battery is left in a mobile device. Something that seems quite possible, reduced to mere software, with the accelerometer in the Nokia 3220. Imagine one ‘pea’ rattling about, instead of one ‘bar’ on a visual display…

  Research conclusions

  The most advanced prototype of a working sound interface was a box that responded to touch, and had invisible soft-buttons on it’s surface that could only be heard through sound. The synthesised sounds responded to the movement of the fingertips across a large touchpad like device (I think it was a tactex device). These soft-buttons used a simplified sound model that synthesised impact, friction and deformation. See Human-Computer Interaction Design based on Interactive Sonification [pdf]

  The testing involved asking users to feel and hear their way around a number of different patterns of soft-buttons, and to draw the objects they found. See these slides for some of the results.

  The conclusions were that users were almost as good at using sound interfaces as with normal soft-button interfaces and that auditory displays are certainly a viable option for ubiquitous, especially wearable, computing.

  More reading

  Soundobject
  Gesture Controlled Audio Systems
  ICAD