These are some of my notes from Mikael FernstrÃ¶m’s lecture at AHO.
The aim of the Soundobject research is to liberate interaction design from visual dominance, to free up our eyes, and to do what small displays don’t do well.
Reasons for focusing on sound:
- Sound is currently under-utilised in interaction design
- Vision is overloaded and our auditory senses are seldom engaged
- In the world we are used to hearing a lot
- Adding sound to existing, optimised visual interfaces does not add much to usability
Sound is very good at attracting our attention, so we have alarms and notification systems that successfully use sound in communication and interaction. We talked about using ‘caller groups’ on mobile phones where people in an address book can be assigned different ringtones, and how effective it was in changing our relationship with our phones. In fact it’s possible to sleep through unimportant calls: our brains are processing and evaluating sound while we sleep.
One fascinating thing that I hadn’t considered is that sound is our fastest sense: it has an extremely high temporal resolution (ten times faster than vision), so for instance our ears can hear pulses at a much higher rate than our eyes can watch a flashing light.
Disadvantages of sound objects
Sound is not good for continuous representation because we cannot shut out sound in the way we can divert our visual attention. It’s also not good for absolute display: pitch, loudness and timbre are relative to most people, even people that have absolute pitch can be affected by contextual sounds. And context is a big issue: loud or quiet environments affect the way that sound must be used in interfaces: libraries and airplanes for example.
There are also big problems with spatial representation in sound, techniques that mimic the position of sound based on binaural differences are inaccessible by about a fifth of the population. This perception of space in sound is also intricately linked with the position and movement of the head. Some Google searches on spatial representation of sound. See also Psychophysical Scaling of Sonification Mappings [pdf]
‘Filling a bottle with water’ is a sound that could work as part of an interface, representing actions such as downloading, uploading or in replacement of progress bars. The sound can be abstracted into a ‘cartoonification’ that works more effectively: the abstraction separates simulated sounds from everyday sounds.
Mikael cites inspiration from foley artists working on film sound design, that are experienced in emphasising and simplifying sound actions, and in creating dynamic sound environments, especially in animation.
A side effect of this ‘cartoonification’ is that sounds can be generated in simpler ways: reducing processing and memory overhead in mobile devices. In fact all of the soundobject experiments rely on parametric sound synthesis using PureData: generated on the fly rather than using sampled sound files, resulting in small, fast, adaptive interface environments (sound files and the PD files used to generate the sounds can be found at the Soundobject site).
One exciting and pragmatic idea that Mikael mentioned was simulating ‘peas in a tin’ to hear how much battery is left in a mobile device. Something that seems quite possible, reduced to mere software, with the accelerometer in the Nokia 3220. Imagine one ‘pea’ rattling about, instead of one ‘bar’ on a visual display…
The most advanced prototype of a working sound interface was a box that responded to touch, and had invisible soft-buttons on it’s surface that could only be heard through sound. The synthesised sounds responded to the movement of the fingertips across a large touchpad like device (I think it was a tactex device). These soft-buttons used a simplified sound model that synthesised impact, friction and deformation. See Human-Computer Interaction Design based on Interactive Sonification [pdf]
The testing involved asking users to feel and hear their way around a number of different patterns of soft-buttons, and to draw the objects they found. See these slides for some of the results.
The conclusions were that users were almost as good at using sound interfaces as with normal soft-button interfaces and that auditory displays are certainly a viable option for ubiquitous, especially wearable, computing.
Notes on two related projects:
1. Time that land forgot
- A project in collaboration with Even Westvang
- Made in 10 days at the Icelandic locative media workshop, summer 2004
- Had the intention of making photo archives and gps trails more useful/expressive
- Looked at patterns in my photography: 5 months, 8000 photos, visualised them by date / time of day. Fantastic resource for me: late night parties, early morning flights, holidays and the effect of midnight sun is visible.
- Looking now to make it useful as part of more pragmatic interface, to try other approaches less about the abstracted visualisation
- info, details, research and source code
- time visualisation
2. Marking in urban public space
I’ve also been mapping stickering, stencilling and flyposting: walking around with the camera+gps and photographing examples of marking (not painted graffiti).
This research looks at the marking of public space by investigating the physical annotation of the city: stickering, stencilling, tagging and flyposting. It attempts to find patterns in this marking practice, looking at visibility, techniques, process, location, content and audience. It proposes ways in which this marking could be a layer between the physical city and digital spatial annotation.
Some attributes of sticker design
- Visibility: contrast, monochromatic, patterns, bold shapes, repetition
- Patina: history, time, decay, degredation, relevance, filtering, social effects
- Physicality: residue of physical objects: interesting because these could easily contain digital info
- Adaptation and layout: layout is usually respectful, innovative use of dtp and photocopiers, adaptive use of sticker patina to make new messages on top of old
Layers of information build on top of each other, as with graffiti, stickers show their age through fading and patina, flyposters become unstuck, torn and covered in fresh material. Viewed from a distance the patina is evident, new work tends to respect old, and even commercial flyposting respects existing graffiti work.
Techniques vary from strapping zip-ties through cardboard and around lampposts for large posters, to simple hand-written notes stapled to trees, and short-run printed stickers. One of the most fascinating and interactive techniques is the poster offering strips of tear-off information. These are widely used, even in remote areas.
Initial findings show that stickers don’t relate to local space, that they are less about specific locations than about finding popular locations, “cool neighbourhoods” or just ensuring repeat exposure. This is opposite to my expectations, and perhaps sheds some light on current success/failure of spatial annotation projects.
I am particularly interested in the urban environment as an interface to information and an interaction layer for functionality, using our spatial and navigational senses to access local and situated information.
There is concern that in a dense spatially annotated city we might have an overload of information, what about filtering and fore-grounding of relevant, important information? Given that current technologies have very short ranges (10-30mm), we might be able to use our existing spatial skills to navigate overlapping information. We could shift some of the burden of information retrieval from information architecture to physical space.
I finished by showing this animation by Kriss Salmanis, a young Latvian artist. Amazing re-mediation of urban space through stencilling, animation and photography. (“Un ar reizi naks tas bridis” roughly translates as “And in time the moment will come”.
Graffiti Archaeology, Cassidy Curtis
Street Memes, collaborative project
Spatial annotation projects list
Nokia RFID kit for 5140
Spotcodes, High Energy Magic
?Mystery Meat navigation?, Vincent Flanders
RDF as barcodes, Chris Heathcote
Implementation: spatial literature