Category: Interaction design

Making Visible

My PhD thesis called ‘Making Visible’ was submitted in December 2013 and successfully defended on 12 June 2014. The thesis reflects upon the design material exploration research from the Touch and Yourban projects. It uses these explorations to situate design research with technology as a cultural, material and mediational practice:

In Making Visible I outline how interaction design may engage in the material and mediation of new interface technologies. Drawing upon a design project called Touch, that investigated an emerging interface technology called Radio Frequency Identification or RFID, I show how interaction design research can explore technology through material and mediational approaches. I demonstrate and analyse how this research addresses the inter-related issues of invisibility, seamlessness and materiality that have become central issues in the design of contemporary interfaces. These issues are analysed and developed through three intertwined approaches of research by design: 1. a socio- and techno-cultural approach to understanding emerging technologies, 2. through material exploration and 3. through communication and mediation. When taken together these approaches form a communicative mode of interaction design research that engages directly with the exploration, understanding and discussion of emerging interface technologies.

The thesis is made up of four peer-reviewed journal articles accompanied by a ‘meta-reflection’ document that reflects upon and situates these publications through theory, concepts and models.

Overview

This document develops the concept of mediational material that focuses attention on the material and communicative practices in interaction design. These are used to explore, develop and share knowledge of technologies as design materials.

I’ve made it available in a number of different digital formats:

• Download as PDF (7MB)
• Download as ePub (14.4MB)
• Read it on Kindle (US, UK)

It will also be available through AHO’s open-access archive ADORA.

Articles

The four included articles have been published in peer-reviewed journals.

Article 1: Exploring ‘immaterials’: mediating design’s invisible materials

This article takes up the issues of so-called ‘immaterial’ and ‘seamless’ technologies and asks how designers might explore them in order to consider them as design materials. It situates interaction design as a sociocultural practice that is concerned with culture, critical approaches and with engaging the technocultural imagination. It concludes by analysing its mediational strategies, such as the use of documentary formats, online film and weblog writing, and the ways in which new material perspectives have been shared, discussed and developed by others.

Arnall, T. (in press). ‘Exploring ‘immaterials’: mediating design’s invisible materials’. International Journal of Design, 29. Will be available at the International Journal of Design.

Article 2: Visualizations of digital interaction in daily life

This article explores how visual signage may make aspects of ubiquitous computing technologies visible and how digital tools and platforms impact that visual design and semiosis. It looks at a range of ‘identification’ technologies such as barcodes and rfid, that only become ‘visible’ or ‘interactional’ through a designer’s intervention in physical or visual expression. It finds that designers should emphasize the bindings and distinctions between design processes and visual mediations, and symbols and signs, in engaging with emerging technologies as material for creative and communicative composition.

Morrison, A., & Arnall, T. (2011). Visualizations of Digital Interaction in Daily Life. Computers and Composition, 28(3), 224-234. Available at Computers and Composition.

Article 3: Satellite Lamps

Satellite lamps is a project about using design to investigate and reveal one of the fundamental constructs of the networked city: GPS – the Global Positioning System. It extends the concepts of ‘mediational materials’ to an investigation of the ways in which GPS technology inhabits urban spaces. The article takes up how a discursive and reflexive interaction design practice can contribute to new perspectives on networked city life. Importantly, Satellite Lamps is a multimediational web text, involving different media (film, media, notebooks and a host of images) allowing for the richness of the work to come to the surface in a way that would not have been possible in traditional means of academic publishing.

Martinussen, E, Knutsen, J & Arnall, T. (forthcoming 2014). Satellite Lamps. Kairos: A Journal of Rhetoric, Technology, and Pedagogy. Will be available at Kairos.

Article 4: Depth of Field: Discursive design research through film

This article is about the role of film in interaction and product design research with technology, and the use of film in exploring and explaining emerging technologies in multiple contexts. It concludes by looking towards the potentials for a discursive design practice, where the object of design and analysis is the discourse that is catalysed by new artefacts, and the emphasis of design research is on communication.

Arnall, T., & Martinussen, E. S. (2010). Depth of field: discursive design research through film. FORMakademisk, 3(1). Available at FORMakademisk

The Immaterials project

The Immaterials project is concerned with the increasing invisibility of interfaces and infrastructures. The systems we interact with everyday such as WiFi and 3G networks have a profound impact on how we experience the world. As Adam Greenfield says:

the complex technologies the networked city relies upon to produce its effects remain distressingly opaque, even to those exposed to them on a daily basis. […] it’s hard to be appropriately critical and to make sound choices in a world where we don’t understand the objects around us.

And as James Bridle has eloquently and disturbingly observed:

Those who cannot perceive the network cannot act effectively within it, and are powerless.

The project set out to expose some of the phenomena and mechanisms of technological infrastructures through visual, photographic, narrative, animated and cinematic techniques. Over the last five years I have worked with Einar Sneve Martinussen, Jørn Knutsen, Jack Schulze and Matt Jones towards a body of work that is now brought together in an exhibition for the first time.

From 2004–2008 I speculated about the ways in which wireless interactions inhabited physical space, through my work on a Graphic language for touch, and also through films such as Wireless in the world. Some of my students made beautiful but fictional speculations about the physical qualities of different kinds of radio.

Jack Schulze and I also made a short, playful film called Nearness about action at a distance. In the film, a series of simple reactions are set off by immaterial phenomena, such as radio waves, mobile networks, light, magnetism and wind.

In 2006 we ran a Touch workshop with BERG where we became concerned about the invisibility of RFID technology, and the effect that had on our ability to design with it. We found it extraordinary that a technology that was defined as a proximity or ‘touch’-based interface, was so opaque in terms of its physical, spatial, gestural materiality. How do we as designers make these materials visible, so we can have reflective conversations with them?

We developed Experiments in Field Drawing as a method of revealing, literally drawing, the physical presence of RFID interactions. We revealed these fields in a much richer, multi-dimensional way using photography, animation and light painting in the film Immaterials: Ghost in the Field.

Matt Jones coined the term immaterials to describe the project and gave a great talk about some ways of understanding the immaterials of interaction design. Matt and I also looked at machine vision, another phenomena that increasingly becomes a material for design in Robot Readable World.

In 2011 at AHO, as part of a research project called Yourban, we extended the investigations to WiFi, using similar light painting techniques we revealed the enormous scale and pervasiveness of ad-hoc WiFi networks in urban spaces in Immaterials: Light Painting WiFi.

Finally, over the last two years we’ve become increasingly interested in the Global Positioning System (GPS), that has become a central part of both the vision and the implementation of contemporary interfaces.

We have built a series of Satellite Lamps that sense the presence of the 24 GPS satellites in orbit. The lamps change brightness according to the strength of GPS signals they receive, showing how the technology itself is messy and unpredictable, and revealing how GPS is a negotiation between radio waves, earth-orbit geometry and the urban environment.

Satellite Lamps has so far been exhibited at Lighthouse and Dread, you can watch the film and read the extensive article detailing our process as well as a cultural history of GPS.

The visual languages that we’ve developed have ended up in advertising, on the BBC and Discovery Channel, and the techniques have been extended in research at MIT and CIID, and by many designers, enthusiasts and hackers. It’s exciting that both the subject and the methods are being taken up and used broadly by other people, and we’re looking forward to seeing more.

the truly pressing need is for translators: people capable of opening these occult systems up, demystifying them, explaining their implications to the people whose neighborhoods and choices and very lives are increasingly conditioned by them. — Adam Greenfield (2009)

The Immaterials project emerged from the humble preoccupations of a few designers dealing with some of the invisible, immaterial, intangible stuff we had in front of us. These small experiments led to larger and more visually and narratively communicative work. In the end what I think we’ve developed is an approach to technology that revolves around material exploration, explanation and communication. Because images and language, as well as materials, form our understandings of technology, Immaterials has shown how we can use ‘design and playful explorations to shape or stir the popular imagination’.

The exhibitions

All the Immaterials projects are on display at Lighthouse in Brighton from 5 September until 13 October 2013.

Satellite Lamps and Robot Readable World are on display at Dread in Amsterdam from 7 September until 24 November 2013.

No to NoUI

‘The best design is invisible‘ is the interaction design phrase of the moment. The images above are from my ever-expanding collection of quotes about how design and technology will ‘disappear‘, become ‘invisible‘ or how the ‘best interface is no interface‘.

The Verge has recently given both Oliver Reichenstein and Golden Krishna a platform to talk about this. This has spawned manifestos, films, talks, books, #NoUI hashtags and some debates about what it might mean. I’ll call this cluster of things ‘invisible design’.

I agree with some of the reasons driving this movement; that design’s current infatuation with touchscreens is really problematic. I’ve spent the last eight years rallying against glowing rectangles, studying our obsession with screens and the ways in which this has become a cultural phenomena. In response I have been researching and inventing interfaces for taking interaction out from under the glass.

But I also take issue with much of the thinking for a few reasons that I’ll outline below.

1. Invisible design propagates the myth of immateriality

We already have plenty of thinking that celebrates the invisibility and seamlessness of technology. We are overloaded with childish mythologies like ‘the cloud’; a soft, fuzzy metaphor for enormous infrastructural projects of undersea cables and power-hungry data farms. This mythology can be harmful and is often just plain wrong. Networks go down, hard disks fail, sensors fail to sense, processors overheat and batteries die.

Computing systems are suffused through and through with the constraints of their materiality. – Jean-François Blanchette

Invisible design propogates the myth that technology will ‘disappear’ or ‘just get out of the way’ rather than addressing the qualities of interface technologies that can make them difficult or delightful.

Intentionally hiding the phenomena and materiality of interfaces, smoothing over the natural edges, seams and transitions that constitute all technical systems, entails a loss of understanding and agency for both designers and users of computing. Lack of understanding leads to uncertainty and folk-theories that hinder our ability to use technical systems, and clouds the critique of technological developments.

As systems increasingly record our personal activity and data, invisibility is exactly the wrong model.

By removing our knowledge of the glue that holds the systems that make up the infrastructure together, it becomes much more difficult, if not impossible, to begin to understand how we are constructed as subjects, what types of systems are brought into place (legal, technical, social, etc.) and where the possibilities for transformation exist. – Matt Ratto (2007)

In other words, as both users and designers of interface technology, we are disenfranchised by the concepts of invisibility and disappearance.

2. Invisible design falls into the natural/intuitive trap

The movement tells us to ‘embrace natural processes’ and talks about the ‘incredibly intuitive’ Mercedes car interface. This language is a trap (we should ban the use of natural and intuitive btw) that doesn’t give us any insight into how complex products might actually become simple or familiar.

Invisible design leads us towards the horrors of Reality Clippy. Does my refrigerator light really go off? Why was my car unlocked this morning? How did my phone go silent all of a sudden? Without highly legible systems for managing and understanding all of this ‘smartness’ we are going to get very lost and highly frustrated. The tricky business of push notifications and the Facebook privacy train wreck is just the tip of the iceberg.

The example of the Nest thermostat invisibly ‘learning’ your habits to control your home temperature is a good one. But the Nest has a highly visible interface that reassures you as to its status, tells you when it is learning, and a large dial for adjusting temperature. Beautiful, legible microinteractions. A Nest without these visual and direct manipulation interfaces would be useless, uncanny and frustrating. Nest wants UI.

The discussion around invisible design often talks about using sensors and tangible interfaces instead of visual interfaces. But these systems are not inherently simpler or more familiar. They have their own material qualities with edges and ‘grain’ that need to be understood and learnt. Their literal invisibility can cause confusion, even fear, and they often increase unpredictability and failure.

In our work with interface technologies such as RFID and computer vision, we’ve discovered that it takes a lot of work to make sense of the technologies as design materials. So it’s not useful to say that UI is ‘disappearing’ into sensing, algorithms and tangible interfaces, when we don’t fully understand them as UI yet.

3. Invisible design ignores interface culture

Interfaces are the dominant cultural form of our time. So much of contemporary culture takes place through interfaces and inside UI. Interfaces are part of cultural expression and participation, skeuomorphism is evidence that interfaces are more than chrome around content, and more than tools to solve problems. To declare interfaces ‘invisible’ is to deny them a cultural form or medium. Could we say ‘the best TV is no TV’, the ‘best typography is no typography’ or ‘the best buildings are no architecture’?

Much of our work at BERG is not just about solving problems, but about cultural invention:

We’re not interested in this idea of the invisible technology in a modernist sense. Tech won’t be visible but only if it’s embedded into the culture that it exists within. By foregrounding the culture, you background the technology. It’s the difference between grinding your way through menus on an old Nokia, trying to do something very simple, and inhabiting the bright bouncy bubbly universe of iOS. The technology is there, of course, but it’s effectively invisible as the culture is foregrounded.” – Jack Schulze (in Domus 965 / January 2013)

We should be able to simultaneously celebrate the fantastic explosion of diversity in UI, and develop healthy critique around the use of interfaces like touch screens. But by calling for UI to disappear altogether so that things can be more efficient, we remain in the same utilitarian and rational mindset that produces inert technological visions like this, rather than seeing interfaces as part of the cultural landscape.

4. Invisible design ignores design and technology history

The movement ignores at least thirty years of thinking in design and technology. A few examples:

Much of the recent invisible design discussions repeat the thinking in Jared Spool’s ‘Great Designs Should Be Experienced and Not Seen‘ and Donald Norman’s ‘Invisible Computer. But a better reference point would be Don Norman’s earlier book, The design of everyday things, where he instead talks about the ‘problems caused by inadequate attention to visibility’ and supporting or managing our mental models of systems. We need a lot more thinking about our mental models of algorithms in particular.

Adam Greenfield has investigated the social and ethical issues around the development of ubiquitous computing systems, and is particularly concerned by its disappearance:

“Ubiquitous systems must contain provisions for immediate and transparent querying of their ownership, use, and capabilities. Everyware must, in other words, be self-disclosing. Whether such disclosures are made graphically, or otherwise, they ensure that you are empowered to make informed decisions as to the level of exposure you wish to entertain.” – Adam Greenfield (2006)

Some designers have talked about the actual qualities they want from ubiquitous computing interfaces, such as polite, pertinent and pretty:

“The vast quantities of information that personal informatics generate need not only to be clear and understandable to create legibility and literacy in this new world, but I’d argue in this first wave also seductive, in order to encourage play, trial and adoption” Matt Jones & Tom Coates (2008)

Matthew Chalmers has, more than anyone else, revealed the history of seamlessness. Seamlessness is ‘the deliberate “making invisible” of the variety of technical systems, artifacts, individuals and organizations that make up an information infrastructure. This work actively disguises the moments of transition and boundary crossing between these various parts in order to present a solid and seemingly coherent interface to users.’ (Ratto 2007). Although Mark Weiser is often thought of as an advocate of seamless systems, Chalmers found that:

Weiser describes seamlessness as a misleading or misguided concept. In his invited talks to UIST94 and USENIX95 he suggested that making things seamless amounts to making everything the same, reducing components, tools and systems to their ‘lowest common denominator’. He advocated seamful systems (with “beautiful seams”) as a goal. Around Xerox PARC, where many researchers worked on document tools, Weiser used an example of seamful integration of a paint tool and a text editor (Weiser, personal communication). He complained that seamless integration of such tools often meant that the user was forced to use only one of them. One tool would be chosen as primary and the others reduced and simplified to conform to it, or they would be crudely patched together with ugly seams. Seamfully integrated tools would maintain the unique characteristics of each tool, through transformations that retained their individual characteristics. This would let the user brush some characters with the paint tool in some artful way, then use the text editor to ‘search and replace’ some of the brushstroked characters, and then paint over the result with colour washes. Interaction would be seamless as the features of each tool were “literally visible, effectively invisible”. Seamful integration is hard, but the quality of interaction can be improved if we let each tool ‘be itself’. – Matthew Chalmers (2003)

Matt Ratto investigates the darker side of this drive towards invisibility, revealing that seamlessness encourages:

“a particular kind of passivity and lack of engagement between people and their actions and between people and their social and material environment” and that we must “critique the clean, orderly, and homogenous future that is at the heart of these modernist visions” – Matt Ratto (2007)

And Anne Galloway suggests that it is in the seams where the design work can be done:

“Although seamlessness may remain a powerful and effective metaphor to guide particular projects, when it comes to actually getting the work done—and the challenges of having to do it with people who can be very different from each other—then I suggest it is in everyone’s best interests to recognise the importance of seams and scars in marking places where interventions can be made, or where potential can be found and acted upon.” – Anne Galloway (2007)

In interaction design we need to look at the long history of Durrell Bishop‘s work, one of the strongest advocates for self-evident design, whether it is physical or virtual, through his teaching and design practice. Durrell’s ‘Platform 12’ in the RCA Design Products course attempts to see design as:

“a celebration of a model for how things work, where once again we can treat function as beauty, instead of merely treating design as form and image.”

Durrell’s work on the Marble Answering Machine (1992) is a brilliant piece of self-evident design, and remains a touchstone for all interaction design work.

Designers also need to look at the first four chapters of ‘Where the action is‘ by Paul Dourish which give a coherent account of the relationships between human abilities and computer interfaces over the last 50-60 years. Dourish shows how interfaces are not becoming invisible, but how they are increasingly social and tangible.

And finally, from a design perspective, there is a long tradition of making complex products legible and understandable. Industrial designer Konstantin Grcic talks about the relationship between the technologies and the use of an object:

“A machine is beautiful when it’s legible, when its form describes how it works. It isn’t simply a matter of covering the technical components with an outer skin, but finding the correct balance between the architecture of the machine… and an expressive approach that is born out of the idea of interaction with those using the object.” – Konstantin Grcic (2007)

And perhaps more famously, Dieter Rams has always talked of honesty and understanding in his product design practice. Making a product understandable is one of his Ten Principles of “Good Design”.

This drive for understanding needs to go further than physical form (as it has done at Apple) and start to inform the design of systems and UI.

Towards legible, evident interaction

We must abandon invisibility as a goal for interfaces; it’s misleading, unhelpful and ultimately dishonest. It unleashes so much potential for unusable, harmful and frustrating interfaces, and systems that gradually erode users and designers agency. Invisibility might seem an attractive concept at first glance, but it ignores the real, thorny, difficult issues of designing and using complex interfaces and systems.

We might be better off instead taking our language from typography, and for instance talk about legibility and readability without denying that typography can call attention to itself in beautiful and spectacular ways. Our goal should be to ‘place as much control as possible in the hands of the end-user by making interfaces evident‘.

Of course the interfaces we design may become normalised in use, effectively invisible over time, but that will only happen if we design them to be legible, readable, understandable and to foreground culture over technology. To build trust and confidence in an interface in the first place, enough that it can comfortably recede into the background.

Talk to Me

I have five works in MoMA‘s latest exhibition ‘Talk to Me‘ in New York. The works are Nearness, Immaterials, Media Surfaces, The Journey and Suwappu.

I have written more about the exhibition and the works at the Touch and BERG weblogs. The exhibition has also been reviewed by CNN, the New York Times, Fast Company and the Wall Street Journal amongst others.

Branding with badges

Many of the early ideas were fairly conceptual: Not a place or a thing, but an idea tied to the use of Foursquare itself (“10 check-ins”) or the kind of real-world social behavior the service was attempting to leverage (checking into the same place three times in one week, or checking in with two people of the opposite sex). The round shape and circular border directly referenced Boy Scouts merit badges. Beyond that, Sheibley says the relevant design context wasn’t logos, it was the familiar instructional iconography meant to signal ideas without words: “How do you communicate to people in an airport, who don’t speak the same language, where the bathroom is?”

Summify – Rob Walker: On Slate: Branding with badges

Connecting the digital world with print

A rare piece of writing from Durrell Bishop:

What if there was a generic tool to link the digital and the physical worlds? A way to touch an object, to select and see its digital augmentation? It would just send a message via your chosen device: laptop, mobile, home wifi et al. The equivalent of a digital finger. Passive objects would act as physical buttons to the digital world. All that these objects would require is that we perceive their purpose, and see how to act on them. We would need to have an accessible tool to make our selection, and carry out the link between the two worlds.

Read the whole thing: Connecting the digital world with print.

Science Fiction Film as Design Scenario Exercise for Psychological Habitability

“They deliver potentially fundamental insights (Schneider, 2005) into the interaction between humans and the constructed environment surrounding them, including the mundance aspects of everyday routine (Carroll, 2000), even the potential subversion of the system or setting through its agents (Blythe & Wright, 2006). The user is advanced into a character or specific persona placed in fictional but feasible settings (Nielsen, 2002). The representation of scenarios through prototypes, use of storyboards, video, rapid prototyping tools and stories, annotated sketches, cartoons, photographs, role-playing or live dramatization (Suri & Marsh, 2000), allows the vision conveyed through the scenario to be opened up to critique (Carroll, 2000).”

“Design, here, does not assume the traditional role of problem-solving, but acts as a critical agent in the enquiry about real human needs and values by evoking reflection (Carroll, 1995) and stimulating debate amongst designers, industry and the public (Dunne & Raby, 2001). This critical strand in design, termed speculative design, critical design or design fiction, has emerged as a field in its own right and is establishing itself with the main markers of a new discipline, such as seminal publications (Dunne, 2005; Antonelli, 2008), exhibitions and conferences.”

Science Fiction Film as Design Scenario Exercise for Psychological Habitability: Production Designs 1955-2009

A phone to save us from our screens?

Microsoft has two new ads, anticipating their upcoming Windows Phone 7 launch. The first is an almost post-apocalyptic vision of humanity stuck with their heads in their mobile devices:

Here’s David Webster, chief strategy officer in Microsoft’s central marketing group, explaining their anti-screen strategy:

“Our sentiment was that if we could have an insight to drive the campaign that flipped the category on its head, then all the dollars that other people are spending glorifying becoming lost in your screen or melding with your phone are actually making our point for us.”

The problem of glowing rectangles is a subject close to my heart, and Matt Jones has been bothered by the increase in mobile glowing attention-wells.

I think Microsoft & Crispin Porter + Bogusky’s advertising strategy stands out in a world full of slick floaty media. The only problem is that without any strategy towards tangible interaction, I’m not sure the ’tiles’ interaction concept is strong enough to actually take people’s attention out of the glass.

Olars: physical toy inspired by karl sims evolved creatures

A lovely piece of work by Lars M. Vedeler and Ola Vågsholm from the Tangible Interactions course at The Oslo School of Architecture & Design:

Olars is an electronic interactive toy inspired by Karl Sims’ evolved virtual creatures. Having thousands of varieties in movement and behaviour by attaching different geometrical limbs, modifying the angle of these, twisting the body itself, and by adjusting the deflection of the motorised joints, results in both familiar and strange motion patterns.

Olars on Vimeo

Negotiating futures. Design fiction

Swiss Design Network Conference 2010:

Designers see the world not simply as it is, but rather as it could be. In this perspective, the world is a laboratory to explore the contingency of the existing and the thinking in options. Imaginations of the contra factual are a key source for the creation of alternative political, technological, social, or economic constellations of artefacts, interfaces, signs, actors, and spaces. At the same time, strategies of materialization are pivotal to shift the boundary between the fictional and the real and to finally bring possible new realities into being. The conference addresses the questions of how fictions are designed and how the multiplicity of possible new futures is negotiated and realized.

Design Fiction, Negotiating Futures October 28-30, 2010.