Early in 2005 I drafted a project together with the Oslo School of Architecture & Design that was designed to look at Near Field Communication (NFC) with an interaction design and user-centred perspective. In December 2005 the project was funded in full by the Research Council of Norway. So since March 2006 we have been setting up the project and conducting preliminary exploratory research work. You can see our ongoing process on the project weblog (and pick up the RSS feed too).
A year ago, Even and I played around for an afternoon with ARtoolkit, an open-source application for handling Augmented Reality objects: physical markings that when processed through a video camera can be augmented with 3D digital objects.
I’m really not a fan of the goggle/glasses/helmet variety of AR, where the user wears something in front of their eyes that superimposes 3D objects into the physical world. In my experience this has been slow, inaccurate, cumbersome, headache inducing, the worst of VR plus a lot more problems. But AR is really interesting when it’s just a screen and a video feed, it becomes somehow magical: to see the same space represented twice: once in front of you, and once on screen with magical objects. I can imagine this working really well on mobile phones: the phone screen as magic lens to secret things.
On that afternoon we didn’t have a printer handy for making the AR marks, so we took to drafting them by hand, stencilling them off the screen with a pencil and inking them in. This hand-crafted process led to all sorts of interesting connections between the possibilities of craft and digital information.
We had lots of ideas about printing the markers on clothes, painting them on nails, glazing them into ceramics, etc. We confused ARtoolkit by drawing markers in perspective, and tried to get recursive objects by using screen based markers and video feedback.
Now as it turns out there is an entire research programme dedicated to looking at just this topic. Variable Environment is a research programme involving partners like ECAL and EPFL. The great thing is that they are blogging the entire exploratory (they call it ‘sketch’) phase and curating the results online. The work is multi-disciplinary and involves architects, visual designers, computer scientists, interaction designers, etc. Check out the simple AR ready products, sample applications and mixed reality tests with various patterns.
This seems to be part of a shift in the research community, to publishing ongoing and exploratory work online (championed by the likes of Nicolas Nova and Anne Galloway). Very inspirational.
I’m collecting images from around the world depicting ‘you are here’ marks or ideo locators at Flickr. I’m fascinated by this mapping in context, in particular the relationship to local physical space. This is mapping with a a point of view, and maps as direct interface to the world. The best example to date is from Seoul, where 3D cross sections of a metro station are directly related to the point at which you are looking at the map.
For the last couple of weeks I have been experimenting with tagging personal space with NFC. This started by embedding RFID tags in my desk, to use it as an information surface for contacts, SMSes and links.
Underneath the desk I have stuck a grid of RFID tags, and on the top surface, the same grid of post-it notes. With the standard Nokia Service Discovery application it is possible to call people, send pre-defined SMSes or load URLs by touching the phone to each post-it on the desk. On the post-its themselves I have hand-written the function, message and the recipient. This is somewhat like a cross between a phone-book, to-do list and temporary diary, with notes, scribbles and tea stains alongside names.
Initial ideas were to spraypaint or silkscreen some of the touch icons to the desk surface, and I may well do that at some point. But for quick prototyping it made sense to use address labels or post-it notes that can be stuck, re-positioned and layered with hand-written notes.
This is an initial step in thinking about the use of RFID and mobile phones, a way of thinking through making. In many ways it is proving to be more inconvenient than the small screen (particularly with the occasionally unreliable firmware on this particular cover, I can’t speak for the production version). But it has highlighted some really interesting issues.
First of all it has brought to the forefront the importance of implicit habits. Initially, it took a real effort to think about the action of using the table as an interface: I would reach for the phone and press names to make a call, instead of placing it on the desk. But for some functions, such as sending an SMS, it has become more habitual.
SMSes have become more like ‘pings’ when very little effort is made to send them. At the same time they are more physically tangible: I rest the phone in a certain position on the desk and wait for it to complete an action. The most useful functions have been “I’m here” or “I’m leaving” messages to close friends.
I have had to consider the ‘negative space’ where the mobile must rest without any action. This space has potential be used for context information; a corner of the table could make my phone silent, another corner could change my presence online. Here it would be interesting to refer to Jan Chipchase’s ideas around centres of gravity and points of reflection, it’s these points that could be most directly mapped to behaviour. I’m thinking about other objects and spaces that might be appropriate for this, and perhaps around the idea of thoughtless acts.
If this was a space without wireless internet, I could also imagine this working very well for URLs: quick access to google searches, local services or number lookups, which is usually very tricky on a small screen. Here it would be interesting to think about how the mobile is used in non-connected places, such as the traditional Norwegian Hytte [pdf].
This process also raised a larger issue around the move towards tangible or physical information, which also implies a move towards the social. As I was making the layout of my address book and associated functions, I realised that maybe these things shouldn’t be explicit, visible, social objects. The arrangement of people within the grid makes personal sense; the placement is a personal preference and maps in certain ways to frequency and type of contact. But I wonder how it appears to other people when this pattern is exposed. Will people be offended by my layout? What if I don’t include a rarely called contact? Are there numbers I want to keep secret, hidden behind acronyms in the ‘Names’ menu?
It will be interesting to see how this plays out and changes over time, particularly in the reaction of others. I’ll post more about the use of NFC in other personal contexts in the near future.
The desk is made from 20 mm birch ply, surfaced in Linoleum. I stuck a single RFID to the underside, in the place that felt most natural. A 10 cm grid was worked out from that point, and the RFIDs were stuck in that grid, and the same worked out on top. If I were to re-build the desk with this project in mind, the tags should probably be layered close to the surface, between the ply and Linoleum. This would make them slightly more responsive to touch by giving them a larger read/write distance.
Rewriteable 512 bit, Philips MiFare UL stickers.
10 cm grid of tags on the underside of the desk.
Blank post-it notes on the surface, with the same grid.
Thanks to Matt and Nokia I’ve had a prototype 3220 NFC shell on loan for a few weeks. It’s the second Nokia phone to feature an RFID reader and writer for ‘Near Field Communication’ the technology that I’ve been getting excited about for mobile services, stickering and touch.
Overall the interaction between phone and RFID tags has been good. The reader/writer is on the base of the phone, at the bottom. This seems a little awkward to use at first, but slowly becomes natural. When I have given it to others, their immediate reaction is to point the top of the phone to the tag, and nothing happens. There follows a few moments of explaining as the intricacies of RFID and looking at the phone, with it’s Nokia ‘fingerprint’ icon. As phones increasingly become replacements for ‘contactless cards’, it seems likely that this interaction will become more habitual and natural.
Once the ‘service discovery’ application is running, the read time from tags is really quick. The sharp vibrations and flashing lights add to a solid feeling of interacting with something, both in the haptic and visual senses. This should turn out to be a great platform for embodied interaction with information and function.
The ability to read and write to tags makes it potentially adaptive as a platform wider than just advertising or ticketing. As an interaction designer I feel quite enabled by this technology: the three basic functions (making phonecalls, going to URLs, or sending SMSs) are enough to start thinking about tangible interactions without having to go and program any Java midlets or server-side applications.
I’m really happy that Nokia is putting this technology into a ‘low-end’ phone rather than pushing it out in a ‘smartphone’ range. This is where there is potential for wider usage and mass-market applications, especially around gaming and content discovery.
I had some problems launching the ‘service discovery’ application. Sometimes it works, sometimes it doesn’t and it’s difficult to tell why this is. It would be great to be able to place the phone on the table, knowing that it will respond to a tag, but it was just a little too unreliable to do that without checking to see that it had responded. The version I have still says it’s a prototype, so this may well be sorted out by the released version.
Overall there is a lack of integration between the service discovery application and the rest of the system: Contacts, SMS archive and service bookmarks etc. At the moment we need to enter the application to write and manage tags, or to give a ‘shortcut’ to another phone, but it seems that, as with bluetooth and IR, this should be part of the contextual menus that appear under ‘Options’ within each area of the phone. There are also some infuriating prompts that appear when interacting with URL, more details below.
The phone opens the ‘service discovery’ application whenever it detects a compatible RFID tag near the base of the phone (when the keypad lock is off). This part is a bit obscure: sometimes it doesn’t ‘wake up’ for a tag, and the application needs to be loaded before it will read properly. Once the application is open (about 2-3 seconds) the read time of the tags seems instantaneous.
The shortcuts menu gives access to shortcuts. Confusingly, this is different from ‘bookmarks’ and the ‘names’ list on the phone, although names can be searched from within the application. I think tighter integration with the OS is called for.
Shortcuts can be added, edited, deleted, etc. in the same way as contacts. They can be ‘Given’ to another phone or ‘Written’ to a tag.
There are three kinds of shortcuts: Call, URL or SMS. ‘Call’ will create a call to a pre-defined number, ‘URL’ will load a pre-defined URL, and ‘SMS’ will send a pre-defined SMS to a particular number. This part of the application has the most room for innovative extensions: we should be able to set the state of the phone, change profiles, change themes, download graphics, etc. This can be achieved by loading URLs, but URLs and mobiles don’t mix, so why should we be presented with them, when there could be a more usable layer inbetween? There could also be preferences for prompts: at the moment each action has to be confirmed with a yes or a no, but in some secure environments it would be nice to be able to have a function launched without the extra button push.
If a tag contains no data, then we are notified and placed back on the main screen (as happened when I tried to write to my Oyster card).
If the tag is writeable we are asked which shortcut to write to the tag.
When we touch a tag with a shortcut on it, a prompt appears asking for confirmation. This is a level of UI to prevent mistakes, and a certain level of security, but it also reduces the overall usability of the system. With URL launching, there are two stages of confirmation, which is infuriating. There needs to be some other mode of confirmation, and the ‘service discovery’ app needs to somehow be deeper in the system to avoid these double button presses.
Lastly, there is a log of actions. Useful to see if the application has been reading something in your bag or wallet, without you knowing…
This work explores the visual link between information and physical things, specifically around the emerging use of the mobile phone to interact with RFID or NFC. It was a presentation and poster at Design Engaged, Berlin on the 11th November 2005.
Download the icons (PDF, 721KB, Gif preview).
As mobile phones are increasingly able to read and write to RFID tags embedded in the physical world, I am wondering how we will appropriate this for personal and social uses.
I’m interested in the visual link between information and physical things. How do we represent an object that has digital function, information or history beyond it’s physical form? What are the visual clues for this interaction? We shouldn’t rely on a kind of mystery meat navigation (the scourge of the web-design world) where we have to touch everything to find out it’s meaning.
This work doesn’t attempt to be a definitive system for marking physical things, it is an exploratory process to find out how digital/physical interactions might work. It uncovers interesting directions while the technology is still largely out of the hands of everyday users.
The inspiration for this is in the marking of public space and existing iconography for interactions with objects: push buttons on pedestrian crossings, contactless cards, signage and instructional diagrams.
This draws heavily on the substantial body of images of visual marking in public space. One of the key findings of this research was that visibility and placement of stickers in public space is an essential part of their use. Current research in ubicomp and ‘locative media’ is not addressing these visibility issues.
There is also a growing collection of existing iconography in contactless payment systems, with a number of interesting graphic treatments in a technology-led, vernacular form. In Japan there are also instances of touch-based interactions being represented by characters, colours and iconography that are abstracted from the action itself.
I have also had great discussions with Ulla-Maaria Mutanen and Jyri Engeström who have been doing interesting work with thinglinks and the intricate weaving of RFID into craft products.
Sketching and development revealed five initial directions: circles, wireless, card-based, mobile-based and arrows (see the poster for more details). The icons range from being generic (abstracted circles or arrows to indicate function) to specific (mobile phones or cards touching tags).
Arrows might be suitable for specific functions or actions in combinations with other illustrative material. Icons with mobile phones or cards might be helpful in situations where basic usability for a wide range of users is required. Although the ‘wireless’ icons are often found in current card readers, they do not successfully indicate the touch-based interactions inherent in the technology, and may be confused with WiFi or Bluetooth. The circular icons work at the highest level, and might be most suitable for generic labelling.
For further investigation I have selected a simple circle, surrounded by an ‘aura’ described by a dashed line. I think this successfully communicates the near field nature of the technology, while describing that the physical object contains something beyond its physical form.
In most current NFC implementations, such as the 3220 from Nokia and many iMode phones, the RFID reader is in the bottom of the phone. This means that the area of ‘activation’ is obscured in many cases by the phone and hand. The circular iconography allows for a space to be marked as ‘active’ by the size of the circle, and we might see it used to mark areas rather than points. Usability may improve when these icons are around the same size as the phone, rather than being a specific point to touch.
This is early days for this technology, and this is work-in-progress. There is more to be done in looking at specific applications, finding suitable uses and extending the language to cover other functions and content.
Until now I have been concerned with generic iconography for a digitally augmented object. But this should develop into a richer language, as the applications for this type of interaction become more specific, and related to the types of objects and information being used. For example it would be interesting to find a graphic treatment that could be applied to a Pokemon sticker offering power-ups as well as a bus stop offering timetable downloads.
I’m also interested in the physical placement of these icons. How large or visible should they be? Are there places that should not be ‘active’? And how will this fit with the natural, centres of gravity of the mobile phone in public and private space.
I’ll expand on these things in a few upcoming projects that explore touch-based interactions in personal spaces.
Feel free to use and modify the icons, I would be very interested to see how they can be applied and extended.
Oyster Card, Transport for London.
eNFC, Inside Contactless.
Paypass, Mastercard.
ExpressPay, American Express.
FeliCa, Sony.
MiFare, various vendors.
Suica, JR, East Japan Railway Company.
RFID Field Force Solutions, Nokia.
NFC shell for 3220, Nokia.
ERG Transit Systems payment, Dubai.
Various generic contactless vendors.
Contactless payment symbol, Mastercard.
Open Here, Paul Mijksenaar, Piet Westendorp, Thames and Hudson, 1999.
Understanding Comics, Scott McCloud, Harper, 1994
Here are my raw notes from the three days of excellent conversation, urban exploration and brainstorming of Design Engaged 2005.
Over Easter I sketched out some ideas for navigating music on a portable player. I was frustrated with the iPod clickwheel, thinking about reducing the reliance on visual interfaces and how navigating music has a lot to do with language. I wanted to explore richer interfaces that combine movement, language and vision.
I too have ditched my large iPod for the iPod Shuffle, finding that I love the white-knuckle ride of random listening. But that doesn’t exclude the need for a better small-screen-based music experience.
The pseudo-analogue interface of the iPod clickwheel doesn’t cut it. It can be difficult to control when accessing huge alphabetically ordered lists, and the acceleration or inertia of the view can be really frustrating. The combinations of interactions: clicking into deeper lists, scrolling, clicking deeper, turn into long and tortuous experiences if you are engaged in any simultaneous activity. Plus its difficult to use through clothing, or with gloves.
My first thought was something Jack and I discussed a long time ago, using a phone keypad to type the first few letters of a artist, album or genre and seeing the results in real-time, much like iTunes does on a desktop. I find myself using this a lot in iTunes rather than browsing lists.
Predictive text input would be very effective here, when limited to the dictionary of your own music library. (I wonder if QIX search would do this for a music library on a mobile?)
Maybe now is the time to look at this as we see mobile phone music convergence.
h3. Navigating through movement
Since scrolling is inevitable to some degree, even within fine search results, what about using simple movement or tilt to control the search results? One of the problems with using movement for input is context: when is movement intended? And when is movement the result of walking or a bump in the road?
One solution could be a “squeeze and shake” quasi-mode: squeezing the device puts it into a receptive state.
Another could be more reliance on the 3 axes of tilt, which are less sensitive to larger movements of walking or transport.
I’m not sure about gestural interfaces, most of the prototypes I have seen are difficult to learn, and require a certain level of performativity that I’m not sure everyone wants to be doing in public space. But having accelerometers inside these devices should, and would, allow for the hacking together other personal, adaptive gestural interfaces that would perhaps access higher level functions of the device.
One gesture I think could be simple and effective would be covering the ear to switch tracks. To try this out we could add a light or capacitive touch sensor to each earbud.
With this I think we would have trouble with interference from other objects, like resting the head against a wall. But there’s something nicely personal and intimate about putting the hand next to the ear, as if to listen more intently.
Things that are truly analogue, like volume and time, should be mapped to analogue controls. I think one of the greatest unexplored areas in digital music is real-time audio-scrubbing, currently not well supported on any device, probably because of technical constraints. But scrubbing through an entire album, with a directly mapped input, would be a great way of finding the track you wanted.
Research projects like the DJammer are starting to look at this, specifically for DJs. But since music is inherently time-based there is more work to be done here for everyday players and devices. Let’s skip the interaction design habits we’ve learnt from the CD era and go back to vinyl :)
Where displays are required, I hope we can be free of small, fuzzy, low-contrast LCDs. With new displays being printable on paper, textiles and other surfaces there’s the possibility of improving the usability, readability and “glanceability” of the display.
We are beginning to see signs of this with this OLED display on this Sony Network Walkman where the display is under the surface of the product material, without a separate “glass” area.
For the white surface of an iPod, the high-contrast, paper-like surfaces of technologies like e-ink would make great, highly readable displays.
So I really need to get prototyping with accelerometers and display technologies, to understand simple movement and gesture in navigating music libraries. There are other questions to answer: I’m wondering if using movement to scroll through search results would create the appearance of a large screen space, through the lens of a small screen. As with bumptunes, I think many more opportunities will emerge as we make these things.
Designing for Shuffling
Thoughts on the iPod Shuffle
Bumptunes
Audioclouds/gestural interaction
Sound objects
DJammer
On the body
Runster
Timo Arnall is a designer working with interactive products and media. Timo leads an international research project on mobile technology, collaborates on interaction design work and lectures in design, physical computing and design methodology. More...
