| The Art of Interaction Gillian Crampton Smith Professor of Interaction Design, Royal College of Art, UK
article from: Interacting in Virtual Worlds (John Wiley, editors R A Earnshaw, John Vince)
Introduction
To be fair, the Desktop was an imaginative response to two new developments. First, computers began to be used not just by technological professionals, but by ordinary people, who were not interested in computers, just the job in hand. Second, high resolution graphic displays appeared with enough computing power to drive them. This allowed graphic images and representations to be generated which were dramatically more rich and complex than had been possible on alphanumeric screens. From being a specialist tool for professionals the computer has now become part of everyday life, like buildings.
A building's basic purpose is to keep the rain off. But a good building must do much more. We must be able to 'read' it: know where the entrance is, what kind of building it is. It must appear appropriate to its function and suit the activities and the social relationships of its users. So why should we not expect expect the electronic environments in which we now work and play to serve us as well as good architecture?
The Roman architect Vitruvius said buildings should have firmness, commodity and delight. By this principle, human-computer interfaces should be robust, useful and pleasurable. Their design should be as much an art as a science.
The study of human-computer interface grew out of the needs of safety-critical systems such as fighter cockpits and nuclear reactors where the concerns of art have been thought irrelevant, if not dangerous. But one need not have the creation of an artwork as one's primary goal in order to use the techniques of art. Far from being opposed to function, the aesthetic is part of function. A well designed artefact is by definition a pleasure to use, an ill-designed one awkward and annoying, a cognitive pea under the mattress, diluting our concentration. Aesthetic connotations, moreover, are inevitable: there is no such thing as `neutral' design.
Stroll down any city street, switch on the TV, read a magazine, visit a museum, and you will see images -- complex, subtle, expressive, moving, beautiful, witty. Although bit-mapped displays have been affordable for a decade or more, computer systems, however, show none of these qualities.
Ten years ago, a designer of computer interfaces, with only a text-based display, had few variables to manipulate: position on the screen, normal type, reverse type and flashing type. There was little potential for visual design, and in any case few of the designers involved had been trained in the visual arts (Figure 1). But modern bit-mapped colour displays allow the designer to use the techniques of typography and graphic information design to express qualities beyond the simple content of the words. Designers can combine symbolic communication (where the relation to the referent is arbitrary, as in words) with the rather more direct iconic form (which has some correspondence to the object represented). Full-colour digital images also allow designers to use the allusive and suggestive qualities of images, offering an almost unlimited range of representations.
The design problem now becomes far more subtle, however. Interface design has grown from the equivalent of laying out a typewritten page to the design of a cross between a complex book and a televison programme. To exploit its potential demands skills not usually in the software designer's repertoire. The interface designer needs to be someone who knows how to manipulate the power of graphic representation.
The Power of Images
This is a deep tradition, although it tends to be taken for granted. (It is also powerful: some religions forbid images, for example, judging them a usurpation of God's creativity.) Many people find it a source of rich meaning and great pleasure. So for these reasons, the design of computer systems needs to tap this long tradition, which can build for the ordinary user a bridge from the familiar world to the less familiar one of the computer.
Many paintings in the National Gallery have their roots in societies where, because literacy was uncommon, images were the main transmitters of society's shared meanings. Painting, sculpture and architecture were in this situation strong and dense carriers of meaning and myth, becoming in many cases (paintings of saints for instance) as revered as those whom they represented. After the Renaissance, painting began to lose this mythic quality and refer, not only to the scene depicted, but also to the art of painting itself. The unexpected and novel -- daring foreshortening, perhaps, or deliberate flouting of well-understood conventions -- gave viewers a visual and intellectual frisson.
Until the invention of photography, the main challenge was to see and then represent what one sees. Later, when some of the key problems of representation had been solved, particularly by the invention of linear perspective, artists looked to express more intangible qualities of experience and mood, experience that could only be implied, not described.
Types of Ambiguity
Take for instance Edward Thomas's Cock-Crow,, which evokes startlingly brilliant visual images:
Out of the wood of thoughts that grows by night
The metaphor of each couplet contains a different semantic turn. We are carried along a rhythmic chain of associated meanings: wood, axe; cleave, blow (to strike, to trumpet); heralds, coat of arms. Then, in the last line, we are brought down to earth with a thud of both rhythm and meaning: "The milkers lace their boots up at the farms."
A second level of ambiguity is that of allusions to knowledge outside the poem itself: cocks crowed at dawn to remind Peter he had denied Christ, trumpeters sound at the day of judgement, buglers sound reveille. And, when we recall that Thomas, a war poet, died in 1917, "The sharp axe of light" begins to resonate differently, as does "The wood of thoughts that grows by night".
Poetry's analogue, the visual image, can also suggest the non-visual, and allude to knowledge of the world outside itself. Allusion depends on the viewer, so its effect is difficult to predict, but the fact that it is not overt is what makes great art continue to move us.
In computing, the shift to the iconic allowed by improved graphic technology has expanded the possible range and power of expression of computer systems. The potential for implicit communication lends images a density of meaning invaluable at the interface. In contrast with words, not everything has to be spelled out or demands the user's conscious attention. And graphic representation, having no syntax or well-understood, repeatable building blocks, is splendidly open to multiple interpretation. An image can be scanned in any order, gathered in a glance, or perused for hours. It does not have to be complete, just suggestive enough for viewers to complete it in their imaginaton. This imprecision of course precludes any invariable rules of design, because every interplay of form and meaning has a different effect.
Layers of Meaning
The role of the single, multi-layered, multi-meaning image, such as the painting, has steadily declined as a conduit for society's values and myths; its only dominant use outside fine art today is in advertising. But interactive media could restore its former importance. The computer interface can allow users to concentrate on performing their work without needing to think consciously about how they are doing it. For, since our minds can process images in parallel, much implicit information can be absorbed subliminally while other tasks are tackled consciously.
Recent typographic design has been exploring aspects of layering, ways of differentiating parts of information in a single image by implying three-dimensional relationships, not necessarily consistent, between elements. This is not yet much exploited in interface design because the emphasis has been on precision rather than impression and suggestion. But the density of meaning which images can accommodate is a particular virtue. For one thing, screen real estate is at a premium. For another, users grasp information better the more rapidly they can shift mental attention between layers of meaning without having to shift their visual attention.
The Language of Images
The workings of mechanical machines are obvious -- pull a lever here, press another there, fill up the boiler, adjust the pressure. Actions have an overt relationship to what happens. Electro-mechanical machines operate more opaquely, but the actions needed to control them bear some relation to the physical changes necessary for operation: a contact here, varying resistors there.
A computer, on the other hand, offers few external clues to its operation and no necessary correspondence between the controls and their feedback. We can program the feedback to appear however we like. So how the virtual machine is represented to users determines their ability to use it. In a sense, the interface is the tool.
Some representations of information allow us to grasp its import more or less effortlessly. Others impede our understanding. The interface of the otherwise very useful desk accessory Suitcase, for instance, constrained perhaps by the conventions of the Macintosh interface guidelines to a textual approach, offers few clues as to what the program does and how you use it; it lacks narrative. (Figure 4). A student project to design alternative versions of the desk accessory (which sets up pointers to fonts anywhere on the system) centred on ways of showing which fonts are merely accessible and which are actually loaded. Some solutions labeled the lists of fonts `library' or `active'; others, more successfully, implied the relationship by making the connection visually, showing the movement of the fonts from the library to the font menu.
As allegory is to story-telling, so metaphor is to the interface narrative: the Desktop or humanoid agents are obvious examples. Metaphor allows the user to infer a great deal of information about what to expect from a system without having it tediously spelt out. It also implies a logic which enables the user to reason about the system when things do not go as expected.
Metaphors have their dangers. They rarely map completely onto the actions the system allows, leading users to make mistaken inferences about the way the system works. And if systems must behave like objects in the real world, they may be prevented from following their own logic, particularly if the representation depicts things that did not exist before the computer and have no real-world equivalent, like the spreadsheet, for instance.
Metaphors do not need to be literal, however. They can be abstract. Lakoff and Johnson, who consider thought to be almost impossible without metaphor, distinguish a range of abstract metaphors about movement, direction, and the way we act in the world which are not specific to particular objects or conventions in society.
It is often more useful to the user, incidentally, to represent the system in a way that masks its real operation. The Desktop trash can, for example, implies that the data is being discarded rather than simply dereferenced. This is a lie but, to the user, a mercifully simplifying one.
Symbolism and Allusion
The symbolic world varies according to social groupings and changes over time. The phrase `Covent Garden' once meant a convent garden, later a vegetable market; now it means either an opera house or a tourist shopping mall. Most people, asked recently in a survey what they associated with the word `Madonna', replied sex, not the Virgin Mary.
Connotations are powerful because they set up expectations in the user about the system at both rational and emotional levels. People identify powerfully with things that support their view of themselves: Guardian readers would not be seen dead reading the Sun, for instance -- or the Daily Telegraph. Marketers exploit this by designing products to appeal to very specific groups of people. The graphic design of the database we designed at the Royal College of Art for fashion designers made them feel it was visual and stylish, appropriate for them -- and they liked it. (Plate 1). Connotation cannot be avoided: even apparent neutrality is a style .
The Representation of Space
In the late nineteenth and early twentieth century, especially in Cubism, interest focused on the tension between the picture's surface and the illusion of depth. A painting's frame, traditionally but fairly arbitrarily a rectangle, marks the boundary of the canvas surface. It is also a metaphor for a window, an aperture through which we apparently see space in three dimensions.
The monitor screen is also a rectangle, imitating the painting, and is both a plane and a window. But the window metaphor is stronger here than in a painting because the screen emits light and its images can move.
What we `see' through that virtual window is necessarily convention-based. When Picasso was painting a woman, her husband complained that the portrait did not look like her. Asked by Picasso to describe what she did look like, the husband took a photo from his wallet and showed him. Picasso looked at the photograph and commented "Rather small, isn't she?" The moral is that, however realistic an image appears, our ability to understand it depends on our knowledge of the conventions by which it is represented.
Ernst Gombrich describes the many conventions, from Ancient Egyptian murals to the photograph, used to represent visual phenomena; even the camera, it appears, is a device for making images which imitate the perspectival conventions of Renaissance painting. We may not respond equally readily to every convention, but all are available to the designer, and it would be wrong to imagine that any was inherently `more realistic' than another.
How then do we choose between them? I'm afraid, once again, it depends -- this time on what aspects of reality we want to represent. After all, we cannot represent them all, and sometimes only one aspect will imply the rest. A Matisse drawing, for example, a few quick lines, sums up the form of a woman's body sufficiently for us to `complete the picture' and summon up her presence in our imagination. By forcing the viewer to work, the drawing makes the illusion more vivid.
In interface design, similarly, representational conventions must make the user focus on the ideas that sustain the overall story we want to communicate. Software designers, for instance, sometimes appear to believe that the illusion of three dimensions is always a good thing: two dimensions good, so three dimensions better. Yet it depends what the interface is trying to do. A GUI with widgets in aggressive relief, unless this supports its metaphor, is just a showy distraction from the job in hand.
Abstraction
The artists at the Bauhaus investigated these abstract qualities in their teaching. Wassily Kandinsky's Point and Line to Plane, Paul Klee's Pedagogical Notebooks, and Johannes Itten's Art of Colour later formed the basis of foundation courses followed by generations of British and American graphic designers, who used what they had learnt to manipulate viewers' attention as they read a book, scan an advertisement or choose which soap powder to buy. Other seminal books published in English included Gyorgy Kepes' Language of Vision, influenced by gestalt psychology, Emil Ruder's Typographie and Jan Tschichold's Asymmetric Typography. Studio Vista also published in the 1960s a series of small but influential paperback guides, many by tutors at the Royal College of Art.
Composition and the Picture Surface
The techniques of composition can invest complexity with hierarchy and order by indicating which things are important, which should be looked at first, which are similar and which different. Here are some compositional tools and topics.
Symmetry and Asymmetry The Frame The Impression of Space
It is also well-known that colours can encourage the illusion of space. Red and yellow appear to come towards us, cool colours to recede. This is both a retinal effect, particularly extreme when looking at a slide in a slide viewer for instance, and a product of atmospheric perspective, which turns distant mountains blue. A depicted shadow--behind type, say, or a GUI `button' -- also implies depth.
Importance Similarity Contrast
All the compositional tools in this by no means exhaustive list are means which artists and designers use to suggest rather than spell out significance. They contribute to the density of meaning carried by images, because viewers interpret them without conscious attention. But they are not a set of rules that can be used to transform requirements into a design. They are instead a way of making sense of how art seems to work, and they derive from practice and reflection, considering the designs one has made in relation to the principles. Every instance is different and depends on the interplay between each element of the composition -- and their meanings.
George Steiner has pointed out that aesthetic theories are not like scientific theories: they cannot be proved or disproved, nor can they be predictive. Critical theories, on the other hand, of which those listed above might be fragments, are taxonomies, allowing us to reflect on what exists.
The Artist-designer's Approach
Artist designers are distinct from both fine artists and `engineer-designers'. Fine artists set their own agenda in terms of aim and process, while engineer-designers are more concerned with how to make things work, than what is made to work. But artist-designers, while working to the agenda of others, use the approaches of painting and sculpture, and people's understanding of those languages, to design things which are both satisfying and useful -- reconciling, as David Pye neatly puts it, the requirements of use, economy and appearance.
Artist-designers need to feed their imaginations, both with outside influences and by making and reflecting on their own work. The product designer Bill Moggridge, a founder of IDEO, calls this brain food `creative soup'. The recipe is typically unpredictable. For artist-designers are divergent rather than convergent thinkers, working not by narrowing in to derive a solution, but by broadening out, constantly generating alternative ideas which in turn spark further alternatives. To work with this intentional uncertainty can feel vertiginous for designers from engineering disciplines, who have to think convergently to get results. If they can learn to tolerate each other, however, artist-designers and engineer-designers together make a powerful team.
It is often assumed that artist-designers only do visual design, the gift wrapping tied on after the system has been designed. This is a wasteful mistake. It does not exploit the artist-designer's skill which, as well as crafting the final artefact, includes analysing the needs of users and inventing ways of suiting them.
The design of the representation of a system, moreover, is central to the design of the whole software, not the final touch. The interface is all the user knows about the tool; its success or failure distinguishes a good tool from a bad. But an initial decision for representing the system, made before the interface designer is consulted, may turn out incapable of adequate represention on screen. This either prevents him or her from maximizing the effectiveness of the system as a whole, or necessitates its costly redesign.
In the Royal College of Art's prototype for a fashion designer's database, for instance, we had hoped to use a direct manipulation paradigm, with the elements represented iconically (Plates 2, 3). When we came to representing it on screen, however, we found we could not display all the information needed while keeping the icons legible. So we decided to use a more abstract, typographic representation. Had the interface design only been considered at the end, the structural changes necessary would have wrecked our timetable and our budget.
The first stage of interaction design is user observation and research. Artist-designers use this stage to clarify the user's needs and kick-start their imagination. Their solutions build on both the intellectual analysis of the tasks to be done and a trained creative intuition.
The second stage is to invent and evaluate alternative concepts for the interface's central organising idea. This might be metaphor based on real-world objects, as in the Desktop interface, or more symbolic, like the control panel of a VCR. The organising idea is vital to the interface's success; without it, a potent representation of the system to the user is precluded. It also helps guide otherwise arbitrary choices about the interface and makes them cohere. The organising idea synthesizes what would otherwise be a ragbag of unrelated functions.
At this stage, before designing screens in detail, interaction designers simulate alternative solutions, using sketches scanned in and roughly animated, to test if they can be made to work.
Though concept and surface design are distinct, they must be worked on in parallel. The designer switches between the two, having an idea and immediately making a sketch to see how it could be represented visually; seeing the sketch may generate new ideas which are fed back into the concept. When a concept evolves which both meets the requirements and seems visually representable, the designer starts to design the screens in more detail and consider other qualities of the interface: movement, pace, sound, and so on.
A constant round follows of simulations and adaptations. Designers can proceed only so far in their heads before needing to test a solution informally, first on themselves, later on colleagues, then on final users. When a solution seems relatively promising, it is tested more formally on a new set of users. Only then is it sensible to go to a prototype, to test with real data.
The process is not linear but raggedly cyclical. One seldom gets it right first off, however careful or experienced the designer -- and few people can as yet be very experienced in interaction design. So at each stage we need ways to test ideas quickly at an appropriate level of detail. Working from the interface back to the system, rather than the other way around, makes designers concentrate on the user's needs rather than the system's.
A New Medium
For the past half-millennium, ideas have been reproduced mainly through words. During this period, visual imagery as a driving force in culture has waned in favour of the verbal and, particularly in our own century, the numerical. In the alphabetic tradition, as opposed to the ideographic, images were costlier to reproduce than words and so rarer. It is only in my own lifetime, for example, that the development of photo-lithographic printing has made economic the mass production of generously illustrated books or full-colour posters. Before then, every illustration was engraved, originally by hand and more recently photographically, an expensive process. Litho-printing and photo-reproduction also gave birth to a new type of artist-designer: the graphic designer, as opposed to typographer, who communicates messages by combining words and images.
The invention of printing radically changed ways of thinking, not just how things are communicated, but what can be thought. As it became possible to own books, it became less necessary to memorise knowledge, so the codified arts of memory atrophied. Similarly the arts of the spoken word -- rhetoric, poetry, drama -- took second place to those of the book.
McLuhan realised that every medium enables some types of thought and expression and inhibits others. Serious and sustained debate, for instance, as Neil Postman points out, cannot be developed on television. Good TV allows no gaps, the action must flow fast to keep the viewers' attention; pauses for thought, essential in a complex argument, are just not entertaining.
Interactive computer graphics is a dauntingly new medium, potentially as radical as the invention of printing or television, and likely to change not only what is transmitted but also the cognitive skills needed to receive it. It will revolutionise how and by whom information is authored, compiled, sold, bought, understood and exploited. But it is too early to say in what way.
Similarly, we cannot yet predict the cultural effects of interaction design. But we can be sure it will draw on existing families of aesthetic languages:
We certainly seem to see a change in the balance of culture, from the symbolic, the verbal, to the iconic, the image-based. Western culture is already strongly oriented towards film and television. Interactive media will tip the balance further: colour pictures and animation can be displayed at little extra cost and text on screen appears destined to remain for some time less legible than on paper. Interactive media have the potential to be powerfully engaging, but only when we have developed the aesthetic languages to exploit their advantages. People will `read' displays in the same way as they read other complex visual images -- posters, record sleeves, paintings -- and will invest them with cultural meaning. Displays will also be judged by the same exacting standards: whether we intend it or not, they will become part of culture. Since the aesthetic dimension is inevitable, it must be exploited with sensitivity and skill.
Our challenge today is to discover, drawing on the tradition and techniques of the visual arts, how to forge a new language of interaction design. It is to incorporate in the design of software the judgement and skill of artist-designers to make user interfaces that are powerful, subtle and pleasurable.
I have not tried to offer recipes for interaction design. It is an art, for which there is no quick fix. I have not even attempted to cobble together a theory. The practice of our art is in its infancy and, as Kandinsky wrote in 1914, "In real art, theory does not precede practice, but follows her.' But I have tried to show that, far from starting from scratch, we can draw practical and philosophical sustenance from two and a half thousand years of patient and passionate visual research. We see further and go faster if we ride on the shoulders of the past...
I am indebted to Dr Philip Tabor of University College London for our numerous discussions during the development of this paper. |
|
url: DOORS OF PERCEPTION editor@doorsofperception.com |