This work explores the visual link between information and physical things, specifically around the emerging use of the mobile phone to interact with RFID or NFC. It was a presentation and poster at Design Engaged, Berlin on the 11th November 2005.
As mobile phones are increasingly able to read and write to RFID tags embedded in the physical world, I am wondering how we will appropriate this for personal and social uses.
I’m interested in the visual link between information and physical things. How do we represent an object that has digital function, information or history beyond it’s physical form? What are the visual clues for this interaction? We shouldn’t rely on a kind of mystery meat navigation (the scourge of the web-design world) where we have to touch everything to find out it’s meaning.
This work doesn’t attempt to be a definitive system for marking physical things, it is an exploratory process to find out how digital/physical interactions might work. It uncovers interesting directions while the technology is still largely out of the hands of everyday users.
Reference to existing work
The inspiration for this is in the marking of public space and existing iconography for interactions with objects: push buttons on pedestrian crossings, contactless cards, signage and instructional diagrams.
This draws heavily on the substantial body of images of visual marking in public space. One of the key findings of this research was that visibility and placement of stickers in public space is an essential part of their use. Current research in ubicomp and ‘locative media’ is not addressing these visibility issues.
There is also a growing collection of existing iconography in contactless payment systems, with a number of interesting graphic treatments in a technology-led, vernacular form. In Japan there are also instances of touch-based interactions being represented by characters, colours and iconography that are abstracted from the action itself.
Sketching and development revealed five initial directions: circles, wireless, card-based, mobile-based and arrows (see the poster for more details). The icons range from being generic (abstracted circles or arrows to indicate function) to specific (mobile phones or cards touching tags).
Arrows might be suitable for specific functions or actions in combinations with other illustrative material. Icons with mobile phones or cards might be helpful in situations where basic usability for a wide range of users is required. Although the ‘wireless’ icons are often found in current card readers, they do not successfully indicate the touch-based interactions inherent in the technology, and may be confused with WiFi or Bluetooth. The circular icons work at the highest level, and might be most suitable for generic labelling.
For further investigation I have selected a simple circle, surrounded by an ‘aura’ described by a dashed line. I think this successfully communicates the near field nature of the technology, while describing that the physical object contains something beyond its physical form.
In most current NFC implementations, such as the 3220 from Nokia and many iMode phones, the RFID reader is in the bottom of the phone. This means that the area of ‘activation’ is obscured in many cases by the phone and hand. The circular iconography allows for a space to be marked as ‘active’ by the size of the circle, and we might see it used to mark areas rather than points. Usability may improve when these icons are around the same size as the phone, rather than being a specific point to touch.
Work in progress
This is early days for this technology, and this is work-in-progress. There is more to be done in looking at specific applications, finding suitable uses and extending the language to cover other functions and content.
Until now I have been concerned with generic iconography for a digitally augmented object. But this should develop into a richer language, as the applications for this type of interaction become more specific, and related to the types of objects and information being used. For example it would be interesting to find a graphic treatment that could be applied to a Pokemon sticker offering power-ups as well as a bus stop offering timetable downloads.
I’m also interested in the physical placement of these icons. How large or visible should they be? Are there places that should not be ‘active’? And how will this fit with the natural, centres of gravity of the mobile phone in public and private space.
I’ll expand on these things in a few upcoming projects that explore touch-based interactions in personal spaces.
Feel free to use and modify the icons, I would be very interested to see how they can be applied and extended.
Oyster Card, Transport for London.
eNFC, Inside Contactless.
ExpressPay, American Express.
MiFare, various vendors.
Suica, JR, East Japan Railway Company.
RFID Field Force Solutions, Nokia.
NFC shell for 3220, Nokia.
ERG Transit Systems payment, Dubai.
Various generic contactless vendors.
Contactless payment symbol, Mastercard.
Open Here, Paul Mijksenaar, Piet Westendorp, Thames and Hudson, 1999.
Understanding Comics, Scott McCloud, Harper, 1994
I too have ditched my large iPod for the iPod Shuffle, finding that I love the white-knuckle ride of random listening. But that doesn’t exclude the need for a better small-screen-based music experience.
The pseudo-analogue interface of the iPod clickwheel doesn’t cut it. It can be difficult to control when accessing huge alphabetically ordered lists, and the acceleration or inertia of the view can be really frustrating. The combinations of interactions: clicking into deeper lists, scrolling, clicking deeper, turn into long and tortuous experiences if you are engaged in any simultaneous activity. Plus its difficult to use through clothing, or with gloves.
Music and language
My first thought was something Jack and I discussed a long time ago, using a phone keypad to type the first few letters of a artist, album or genre and seeing the results in real-time, much like iTunes does on a desktop. I find myself using this a lot in iTunes rather than browsing lists.
h3. Navigating through movement
Since scrolling is inevitable to some degree, even within fine search results, what about using simple movement or tilt to control the search results? One of the problems with using movement for input is context: when is movement intended? And when is movement the result of walking or a bump in the road?
One solution could be a “squeeze and shake” quasi-mode: squeezing the device puts it into a receptive state.
Another could be more reliance on the 3 axes of tilt, which are less sensitive to larger movements of walking or transport.
I’m not sure about gestural interfaces, most of the prototypes I have seen are difficult to learn, and require a certain level of performativity that I’m not sure everyone wants to be doing in public space. But having accelerometers inside these devices should, and would, allow for the hacking together other personal, adaptive gestural interfaces that would perhaps access higher level functions of the device.
One gesture I think could be simple and effective would be covering the ear to switch tracks. To try this out we could add a light or capacitive touch sensor to each earbud.
With this I think we would have trouble with interference from other objects, like resting the head against a wall. But there’s something nicely personal and intimate about putting the hand next to the ear, as if to listen more intently.
Things that are truly analogue, like volume and time, should be mapped to analogue controls. I think one of the greatest unexplored areas in digital music is real-time audio-scrubbing, currently not well supported on any device, probably because of technical constraints. But scrubbing through an entire album, with a directly mapped input, would be a great way of finding the track you wanted.
Research projects like the DJammer are starting to look at this, specifically for DJs. But since music is inherently time-based there is more work to be done here for everyday players and devices. Let’s skip the interaction design habits we’ve learnt from the CD era and go back to vinyl :)
Evolution of the display
Where displays are required, I hope we can be free of small, fuzzy, low-contrast LCDs. With new displays being printable on paper, textiles and other surfaces there’s the possibility of improving the usability, readability and “glanceability” of the display.
We are beginning to see signs of this with this OLED display on this Sony Network Walkman where the display is under the surface of the product material, without a separate “glass” area.
For the white surface of an iPod, the high-contrast, paper-like surfaces of technologies like e-ink would make great, highly readable displays.
So I really need to get prototyping with accelerometers and display technologies, to understand simple movement and gesture in navigating music libraries. There are other questions to answer: I’m wondering if using movement to scroll through search results would create the appearance of a large screen space, through the lens of a small screen. As with bumptunes, I think many more opportunities will emerge as we make these things.
These are some of my notes from Mikael FernstrÃ¶m’s lecture at AHO.
The aim of the Soundobject research is to liberate interaction design from visual dominance, to free up our eyes, and to do what small displays don’t do well.
Reasons for focusing on sound:
- Sound is currently under-utilised in interaction design
- Vision is overloaded and our auditory senses are seldom engaged
- In the world we are used to hearing a lot
- Adding sound to existing, optimised visual interfaces does not add much to usability
Sound is very good at attracting our attention, so we have alarms and notification systems that successfully use sound in communication and interaction. We talked about using ‘caller groups’ on mobile phones where people in an address book can be assigned different ringtones, and how effective it was in changing our relationship with our phones. In fact it’s possible to sleep through unimportant calls: our brains are processing and evaluating sound while we sleep.
One fascinating thing that I hadn’t considered is that sound is our fastest sense: it has an extremely high temporal resolution (ten times faster than vision), so for instance our ears can hear pulses at a much higher rate than our eyes can watch a flashing light.
Disadvantages of sound objects
Sound is not good for continuous representation because we cannot shut out sound in the way we can divert our visual attention. It’s also not good for absolute display: pitch, loudness and timbre are relative to most people, even people that have absolute pitch can be affected by contextual sounds. And context is a big issue: loud or quiet environments affect the way that sound must be used in interfaces: libraries and airplanes for example.
There are also big problems with spatial representation in sound, techniques that mimic the position of sound based on binaural differences are inaccessible by about a fifth of the population. This perception of space in sound is also intricately linked with the position and movement of the head. Some Google searches on spatial representation of sound. See also Psychophysical Scaling of Sonification Mappings [pdf]
‘Filling a bottle with water’ is a sound that could work as part of an interface, representing actions such as downloading, uploading or in replacement of progress bars. The sound can be abstracted into a ‘cartoonification’ that works more effectively: the abstraction separates simulated sounds from everyday sounds.
Mikael cites inspiration from foley artists working on film sound design, that are experienced in emphasising and simplifying sound actions, and in creating dynamic sound environments, especially in animation.
A side effect of this ‘cartoonification’ is that sounds can be generated in simpler ways: reducing processing and memory overhead in mobile devices. In fact all of the soundobject experiments rely on parametric sound synthesis using PureData: generated on the fly rather than using sampled sound files, resulting in small, fast, adaptive interface environments (sound files and the PD files used to generate the sounds can be found at the Soundobject site).
One exciting and pragmatic idea that Mikael mentioned was simulating ‘peas in a tin’ to hear how much battery is left in a mobile device. Something that seems quite possible, reduced to mere software, with the accelerometer in the Nokia 3220. Imagine one ‘pea’ rattling about, instead of one ‘bar’ on a visual display…
The most advanced prototype of a working sound interface was a box that responded to touch, and had invisible soft-buttons on it’s surface that could only be heard through sound. The synthesised sounds responded to the movement of the fingertips across a large touchpad like device (I think it was a tactex device). These soft-buttons used a simplified sound model that synthesised impact, friction and deformation. See Human-Computer Interaction Design based on Interactive Sonification [pdf]
The testing involved asking users to feel and hear their way around a number of different patterns of soft-buttons, and to draw the objects they found. See these slides for some of the results.
The conclusions were that users were almost as good at using sound interfaces as with normal soft-button interfaces and that auditory displays are certainly a viable option for ubiquitous, especially wearable, computing.
Notes on two related projects:
1. Time that land forgot
- A project in collaboration with Even Westvang
- Made in 10 days at the Icelandic locative media workshop, summer 2004
- Had the intention of making photo archives and gps trails more useful/expressive
- Looked at patterns in my photography: 5 months, 8000 photos, visualised them by date / time of day. Fantastic resource for me: late night parties, early morning flights, holidays and the effect of midnight sun is visible.
- Looking now to make it useful as part of more pragmatic interface, to try other approaches less about the abstracted visualisation
- info, details, research and source code
- time visualisation
2. Marking in urban public space
I’ve also been mapping stickering, stencilling and flyposting: walking around with the camera+gps and photographing examples of marking (not painted graffiti).
This research looks at the marking of public space by investigating the physical annotation of the city: stickering, stencilling, tagging and flyposting. It attempts to find patterns in this marking practice, looking at visibility, techniques, process, location, content and audience. It proposes ways in which this marking could be a layer between the physical city and digital spatial annotation.
Some attributes of sticker design
- Visibility: contrast, monochromatic, patterns, bold shapes, repetition
- Patina: history, time, decay, degredation, relevance, filtering, social effects
- Physicality: residue of physical objects: interesting because these could easily contain digital info
- Adaptation and layout: layout is usually respectful, innovative use of dtp and photocopiers, adaptive use of sticker patina to make new messages on top of old
Layers of information build on top of each other, as with graffiti, stickers show their age through fading and patina, flyposters become unstuck, torn and covered in fresh material. Viewed from a distance the patina is evident, new work tends to respect old, and even commercial flyposting respects existing graffiti work.
Techniques vary from strapping zip-ties through cardboard and around lampposts for large posters, to simple hand-written notes stapled to trees, and short-run printed stickers. One of the most fascinating and interactive techniques is the poster offering strips of tear-off information. These are widely used, even in remote areas.
Initial findings show that stickers don’t relate to local space, that they are less about specific locations than about finding popular locations, “cool neighbourhoods” or just ensuring repeat exposure. This is opposite to my expectations, and perhaps sheds some light on current success/failure of spatial annotation projects.
I am particularly interested in the urban environment as an interface to information and an interaction layer for functionality, using our spatial and navigational senses to access local and situated information.
There is concern that in a dense spatially annotated city we might have an overload of information, what about filtering and fore-grounding of relevant, important information? Given that current technologies have very short ranges (10-30mm), we might be able to use our existing spatial skills to navigate overlapping information. We could shift some of the burden of information retrieval from information architecture to physical space.
I finished by showing this animation by Kriss Salmanis, a young Latvian artist. Amazing re-mediation of urban space through stencilling, animation and photography. (“Un ar reizi naks tas bridis” roughly translates as “And in time the moment will come”.
Graffiti Archaeology, Cassidy Curtis
Street Memes, collaborative project
Spatial annotation projects list
Nokia RFID kit for 5140
Spotcodes, High Energy Magic
?Mystery Meat navigation?, Vincent Flanders
RDF as barcodes, Chris Heathcote
Implementation: spatial literature
My intention was to avoid the screen for the duration of the workshop, to concentrate on simple interactions between sensors and outputs entirely independent of a desktop computer. But I ended up staring at microprocessor programming languages like PBasic and JAL while making lots of LEDs blink.
A lot of it brought back memories of school; circuit diagrams, resistance calculations, it was great to refresh the memory. We spent a lot of time translating circuit diagrams onto breadboards, and programming both PIC and Basic Stamp microprocessors.
Erich is now setting up a Physcomp lab at Atelier Nord to support art/design projects in Oslo, maybe alongside some regular meetings (entitled Atelier Nerd :). There are many projects that I would like to pursue, this should be a great resource.
Outside In is a forum for involving new voices, media and practices in a discourse about the use and design of public space. It took place from 14 – 15 June 2004.
Roda Sten is amazing, below a suspension bridge, with huge concrete creations. Really windy, but calm inside the lecture space. Here are my notes and a few pictures.
Session 2: Hacking the streets (I missed the 1st workshop)
- Putting memories in spaces: spaces arent the same after having been disrupted. after ‘reclaim the streets’ or a ‘circle line party’ you can’t see the space in the same way.
- Distinction between public and private. What is it?
- Public space doesn’t exist anymore.
- Ken’s new city hall is half private half public (private investment was involved in the building, so protests cannot happen outside)
- Do we need institutions in order to do events, is that the only way to do it legally?
- What’s stopping people from doing these things is not necessarily capitalism, but the fear of looking like a pillock: self-regulation is a big factor. Can spark things to let down inhibitions or shackles. Uses example of the scooter, became a kids toy and then it wasn’t cool anymore.
- What’s the connection between anarchism and these spontaneous events. Emergent order is interesting, so much control over actions, and the ways people move through the city. How does this relate to anarchy? Is this anarchy?
- The city is a workshop: not just walls to tag
- Shadows of urban furniture: really good
- Visual kidknapping: Lavazza woman gets cut out of the frame
- Big poster with bleeding eyes
- Uses a high pressure water jet to clean the city, but also write at the same time.
- Digs at the notion of authorship, a site where people find work on the streets
- The work is anonymous, but there is the projection of authorial control behind it, its individual and definitely authored
- Would be interesting to explore more about Graffiti authorship: how do public artists want to be recognised?
- Managing the mystique around the work and the author.
- Difference between author/instigator
- Visual kidknapping
3D bombing: Akim
- Polystyrene models, matched to fit specific city spaces
- City of names: what if the writers are the ones who build the houses?
Session 3: Network experience
- Wants to deconstruct network context
- Context: physical and social situation in which computation sits
- How does the network affect the output and experience
- Companies are claiming ownership of space because of signal
- WiFihog: saps out all wifi bandwidth
- LAN party versus Flash Mob
- Simpletext: collaborative sms image searching on large screens
- re-mapping and changing the context of interfaces: what about
- Simpletext project: assigns an image search to inputted text
- Steven Levy quote on hackers
- Altering space by altering the body
- character of a space
- remnants of things, people, individuals
strength: strengthening signals to drown out free competion
shifting consequences: changing the input/output relationship.
messages, and displays via jitter/max on a large screen.
- put magnets on wrists and fingers and bodies to reveal the proximity of electronic devices: unexpected connections to other people and lampposts. Nice.
Data Climates: Pedro Sepúlveda Sandoval
- Living in a scanscape city
- electronic space, synthetic city
- Congestion charge as walled city, in electronic space
- London: highest density of cctv in the world
- will we decide to travel to areas based on the quality of electronic space
- A new architectural language for electronic space
- Houses without windows, just cameras. Can start to control life inside. Can also choose to use the weather channel as windows
- Pay a fee for personal surveillance: ask them to watch you all the way to the supermarket.
- The city of Yokohama was brought down by the coming of age party for 40,000 teenagers: the networks were overloaded with messages, because the teenagers didn’t want to talk face to face.
- Palm trees as cell towers (seen in south africa)
- Looked at a community in Hackney that were campaigning to not have a cell phone tower.
- Designed a house for them that would shield them from the signals, but they would have to give up cell phone connectivity. Designed it so that windows would open and close based on calls being made, or would give them 10 minute windows in which to make calls every 2 hours.
- Digital shelter: stand inside the line
- These presentations all use the strategy of showing ‘hypothetical products’ that are really non-products. They are doing this, rather than providing platforms or design methodologies, or distributing resources and infrastructures for people to design their own systems. I understand the need for designers as visionaries, but this could be made more valuable and useful.
- specialists in electronic space could be similar to lighting design specialists in the ‘70s. Will grow into a general field of understanding.
- Platforms and inftrastructure for technology is beyond architects, but understanding of the use and consequences is really important.
- Skateboarding as adaptive design: difference between skate parks and the street, skate parks become designed over time to mimic certain aspects of streets, but also according to innate, human skaters needs. A combination of factors go into making a good skateboarding space: free, alcohol, quality, location.
- New to NY: wanted to work outside gallery space, was inspired by collage of city streets. Not from a graffiti background, being a female, can do certain things outside the norms of graffiti.
- Changes billboards during the day, looks official.
- Open democratic visual space
- a visual direct democracy…
- Cuba used to have street art as a means of free expression, but outlawed by dictatorship
- Makes lightboxes with imagined cities, and mounts on the reverse side of construction site walls, with peepholes ‘peer here’
- Interesting mix of opportunism and ‘designed intervention’
- Sometimes driven purely by visual interest.
- Mike Davis: Public is phantom
- Bedouin as a model of sustainable nomadic communities
- Homeless use waste air from air conditioning (airvac exhaust ports) to stay warm and dry
- Homeless have receded to the peripheral vision of the public. Want to see and be seen.
- Seeing is important for living nomadically in the city.
- Started to map the heat and the power of the exhaust fans in the city. Found a high one at MIT plasma lab.
- Re-routed smell from from a bakery to an art gallery, to subvert a ‘high art’ re-appropriation of space
Workshop ‘Loop City’
- Dietmar Offenhuber & Sara Hodges
- Showed Rybczynski’s film New Book using 9 frames: a good way of mapping space in the city. Starts off and the viewer is not sure if each frame is occurring synchronously, or in the same space, but a bus passes between all of the frames and the spatial link is made immediately. There is also a point where a plane flies overhead and all the actors look up: showing time synchronicity too.
Looking at the city
- as a set of repeated actions
- as a playground: situationists
- as a balance of social as well as physical architectures
Some of our ambitions were:
- Investigate transformative use of space and place
- Address gaps in infrastructure: access to standards, material frameworks and technology
- Instigate a triangular network: tried and trusted network practice
- Pursue research and practice, less engineering
- Explore relationships between media, gaming, locative, mobile, visual media
The discussion is continuing, and the next informal meeting of participants is happening at ISEA 2004.