Towards Seamless Collaboration Media
HIROSHI ISHII

We have been studying how computer and communication technologies can support people working together in groups. We have designed a variety of collaboration media. Today, I would like to introduce our design approach toward seamless collaboration media. Ideally, tools enable us to work smoothly and without interruption. Complex computer-based tools, however, often require our attention, distracting us temporarily from the work we are doing.

`Seamlessness' in the sense of eliminating unnecessary, obstructing, perceptual seams has been a key concept of our evolving media design. Collaborative work is particularly marked by spatial, temporal, and functional constraints that force us to shift among a variety of spaces or modes of operation. Seamless design undertakes to decrease the cognitive loads of users' as they move dynamically across different aspects of their work. My research field is called `CSCW'. Computer-Supported Cooperative Work emerged in the middle of the 80s as an identifiable research field focused on the role of computer technology in group work. CSCW examines how people work together in groups and how technologies can support them. Computers are the promising media for collaboration. We interact, not with computers, but through computers.

G r o u p w a r e

Computer-Supported Cooperative Work

CS + CW

Support

Understand, [Groupware]

`Groupware' is a label for computer-based systems explicitly designed to support groups of people working together. It is growing rapidly as a new application category in the computer industry. Familiar groupware examples include electronic mail, bulletin boards, group schedulers, group decision support systems, workflow systems, collaborative authoring tools, and screen sharing software.

E v o l u t io n . o f . O u r . M e d i a . D e s i g n

Today I would like to introduce the evolution of our design activities at NTT Human Interface Laboratories.

This diagram shows an idealized trajectory of our research. It shows the progression of iterative design from TeamWorkStation to ClearBoard. `Seamlessness' has been a key concept of our evolving media design. We would like to show the progression of our design from TeamWorkStation to ClearBoard. First I will present TeamWorkStation-1 design based on computer and video fusion architecture. I will talk the motivation, design techniques, and findings. Then I will show you a new version,TeamWorkStation-2 that is designed for Narrowband ISDN using ClearFace technique. Then I will present the ClearBoard that goes beyond TeamWorkStation. I will talk about the motivation of our ClearBoard design, and the evolution from the first prototype, ClearBoard-One, to the computer-based prototype, ClearBoard-Two.

B e y o n d . T a l k i n g . H e a d s

Beyond Imitation of Physical Proximity or F-to-F Meeting

CSCW + Video Telephony

Integration of Groupware and Realtime Video Communication

Broadband Digital Network. Most of the current groupware systems are devoted to computational support and are designed under the constraint of limited communication bandwidth.

However, the deployment of broadband digital networks opens a new future for multimedia collaboration environments that integrate realtime audio and video communication links with computer-based shared workspaces [Brittan 1992; Lyles 1993]. We have been exploring the future of collaboration media that make good use of broadband networks through the iterative design of various CSCW systems.

Our CSCW research started in 1988 and was motivated by the study on shared drawing space [Tang 1991] in the Media Space environment [Bly 1993] . The focus of our research is on the creation of new application concepts, rather than the development of technologies themselves. While most of the current video telephony has been designed to see `talking heads,' our goal is to go beyond this model and demonstrate new usage of video communication technologies.

A p p r o a c h

User-Centered instead of Technology-Driven Approach

Let's do smart things with stupid technology today, rather than wait and do stupid things with smart technology tomorrow. (Bill Buxton)

S e a m l e s s . D e s i g n

Seamlessness (continuity) - with existing work practices

Seamlessness (smooth transition) - between functional spaces)

People develop their own work practices after using a wide variety of tools and interacting with a large number of people.

We believe the continuity with existing work practices and everyday skills is essential. Groupware that asks users to abandon their acquired skills and to learn a new protocol is likely to encounter strong resistance. Collaboration requires us to shift among a variety of functional spaces or modes. Seamless design undertakes to decrease the cognitive load of users as they move dynamically across different spaces

T e a m . W o r k S t a t i o n

(by Hiroshi Ishii & Kazuho Arita; Special Thanks to Masaaki Ohkubo and Naomi Miyake). People develop their own work practices after using a wide variety of manual and computer tools. Groupware can only be accepted if it supports continuity with existing individual work environments. Seamlessness is the key issue because users frequently move back and forth between individual and collaborative modes.

We designed TeamWorkStation to realize a `seamless shared workspace'. It supports real-time and distributed collaboration.

K e y . I d e a :

Seamless Shared Workspace

Smooth Transition

Continuity with existing Individual Work Environments

S h a r e d . D r a w i n g . A c t i v i t i e s

The key design idea of TeamWorkStation is `translucent overlay' of individual workspace images. TeamWorkStation combines two or more translucent live-video images of computer screens or physical desktop surfaces by using a video synthesis technique. Translucent overlay allows users to combine individual workspaces, and to point to and draw on the overlaid images simultaneously.

Translucent Overlay of Individual Workspace Images

Workspaces: Computer Displays and Actual Desktops

The first prototype was implemented in 1989. Two CCD video cameras are provided at each workstation: one for capturing live face images of the member, and the other for capturing the desktop surface images and hand gestures.

(A Shared Screen)

(System Architecture)

(TWS-2 Design)

TWS-2 is designed to run on standard INS-Net 64 and 1500. The main target is INS-Net 64 rather than 1500 because INS-Net 64 is much more widespread in Japan.

T e a m W o r k S t a t i o n - 2 . f o r . N - I S D N

N-ISDN (INS-Net 64 or 1500)

H.261 video compression standard

Dyadic communication between two sites

Desktop-centered design

TWS-2 uses a video CODEC that supports the video compression standard H.261. H.261 is recommended by CCITT as the compression algorithm for transmitting moving pictures. Because of the adoption of H.261, TWS-2 can be used with any video CODEC supporting this standard. We decided to limit the number of sites to be connected at any time to two . Although the technology of multi-point connection is already available, it definitely increases system complexity and cost. Dyadic architecture makes the centralized video server unnecessary.

The main argument for this decision, beyond technical and economic arguments, is our observation that most daily realtime informal communication happen between two persons and the most telephone calls happen between two people. The target of dyadic communication seems to be most important and appropriate as the first target of TWS-2 usage. Although some pre-planned formal meetings are held between more than two points, the main target of TWS-2 use is the informal sessions established between a small number of users. We redesigned TeamWorkStation to make it easier-to-use. The system configuration was simplified based on the feedback from test users in NTT.

C o n c l u s io n . o f . T W S - 2

TWS-2 demonstrated the new usage of realtime video to provide a seamless shared workspace over N-ISDN.

TWS-2 was found to be useful in freehand drawing and gesture-intensive design sessions even with 2B+D.

TWS-2 demonstrated advantages over ordinary videophone as N-ISDN AP.

-Desktop is relatively static compared to face. This leads to better impression of video quality compared to that of videophone.

Today I introduced the design evolution from TWS-1 to TWS-2, and the initial findings gained from TWS-2 experiments over INS-Net 64.

To overcome the limitations of the first TWS-1 prototype, we designed TWS-2 to connect two sites with a desktop overlay service using N-ISDN and CCITT H.261 standard. TWS-2 is designed based on the desktop-centered service concept.

Through informal observations of the experimental use of TWS-2 over INS-Net 64, we confirmed that TWS-2 is useful in drawing and gesture-intensive design sessions. TWS-2 is a promising application of N-ISDN and is a good example of `beyond being there'.

S u m m a ry . o f . T e a m . W o r k S t a t i o n

Fusion of computers and desktop media by translucent video overlay

Seamless Shared Workspace

users can keep using their favorite individual tools and hand gestures while collaborating in a shared workspace.

TeamWorkStation provides a `seamless shared workspace' in which participants can freely use a variety of everyday media, such as computer tools, hand writing, printed materials, and hand gestures simultaneously. Through many experiments, we realized that TeamWorkStation achieves `cognitive seamlessness' by allowing users to keep using their favorite individual tools while collaborating in a shared workspace.

(by Hiroshi Ishii & Minoru Kobayashi; Special Thanks to Naomi Miyake and Jonathan Grudin)

C l e a r B o a r d

Shared Workspace

Groupware

A whiteboard is an example of a shared workspace that is common in face-to-face meetings. The major focus of groupware development has been the creation of virtual shared workspaces in distributed computer environments. SWS activities include sharing information, pointing to specific items, marking, annotating, and editing. These activities can be supported by computer-based groupware, including shared screen systems, shared window systems, and a variety of multi-user editors.

I n t e r p e r s o n a l . S p a c e

Interpersonal Space

Videophone

In face-to-face meetings, we speak, make eye contact, and observe each other's facial expressions and gestures. These verbal and non-verbal channels are important in building confidence. One focus of the videophone and video conferencing technologies has been the creation of inter-personal space that establish a sense of tele-presence through the visibility of gestures and facial expressions of remote members. In order to overcome these limitations of multi-screen and multi-window approaches, we started the ClearBoad project. Our goal was to create a collaboration medium that seamlessly integrated the IPS and the SWS. We designed ClearBoard to allow users to shift easily between these spaces by using familiar everyday cues such as the partner's gestures, head movements, eye contact, and gaze direction. Seamless integration of SWS and IPS: Natural and smooth transition between them using everyday cues

C l e a r B o a r d . M e t a p h o r

In 1990, Minoru came up with the new metaphor of `talking through and drawing on a big transparent glass window'. He named it `ClearBoard'. Let me show a video clip of `ClearBoard-0' which is a simple mockup of this concept for a co-located pair of users.

In ClearBoard, since the partner's face and drawings are closely located on the board, refocusing between the drawing and the partner's face requires very little eye movement. One problem of this metaphor is that participants have no common orientation of `right' and `left'. However, this problem can be easily solved in the remote version by mirror-reversing the video image. Talking through and drawing on a big glass board. But ... no common orientation of right and left

D e s i g n . R e q u i r e m e n t s

In order to implement a ClearBoard which supports remote collaboration, we identified the following three design requirements.

direct drawing on the display screen surface,

achieving eye contact, and

common orientation of the drawing space at both sites.

The last requirement is important, especially for writing. The partner must be able to read the text in its correct orientation. The strict implementation of the transparent glass metaphor does not allow this.I

A r c h i t e c t u r e . o f . C l e a r B o a r d - 1

The video camera located above each screen captures the drawings and the image of the user reflected by the half-mirror as a continuous video image. This image is sent to the other terminal through a video network, and projected onto the partner's screen from the rear. The partner can draw directly over this transmitted video image.

G a z e . A w a r e n e s s

Novel and important feature of ClearBoard is that it provides precise `gaze awareness' or `gaze tracking.' Let me show an example. Please look at my eyes moving. A ClearBoard user can easily recognize what the partner is gazing at on the screen during a conversation. Precise gaze awareness can not be easily achieved in an ordinary meeting environment using a whiteboard because both users stand on the same side of the board. The importance of eye-contact is often discussed in the design of face-to-face communication tools. However, we believe the concept of gaze awareness is more generalized and is a more important notion. Gaze awareness lets a user know what the partner is looking at, the user's face or anything else on the shared workspace. Eye contact can be seen as just a special case of gaze awareness.

P r o b l e m s . i n . C l e a r B o a r d - 1

Through using this ClearBoard-1 prototype, we found several problems. Clarity of images on the screen is not enough. Erasing marks is burdensome. Lack of recording capabilities is another problem to reuse the work results.

T e a m P a i n t . f o r . C l e a r B o a r d - 2

To overcome these problems of ClearBoard-1, we decided to design a new computer-based prototype, `ClearBoard-2'. Instead of using color paint markers, ClearBoard-2 provides users with `TeamPaint', a multiuser computer-based paint editor and digitizer pen.

Let me show you a video clip of a TeamPaint session.

TeamPaint is a groupware application that runs on networked Macintosh computers. TeamPaint provides an electronic sketch pad with color pencils. It is a simple bit-map paint editor based on a replicated architecture. TeamPaint offers several useful functions: recording and reuse of working results, easy manipulation of marks, and the use of data made with other application programs such as MacDraw.

Multiuser paint editor

Multi-layer architecture

Simultaneous drawing and gesturing

Gesture and process awareness

C l e a r B o a r d - 2 . A r c h i t e c t u r e

This diagram shows the architecture of ClearBoard-2. In addition to the simple shared video drawing architecture of ClearBoard-1, ClearBoard-2 provides transparent digitizer sheets covering the display surfaces, and Macintosh computers connected via AppleTalk network. TeamPaint running on the Macintosh supports the computer drawing with digitizer pens. The shared drawing image produced by TeamPaint is overlaid onto the video image of the partner, and projected to the screen.

S u m m a r y . o f . C l e a r B o a r d

We have presented a novel collaboration medium, ClearBoard. ClearBoard realizes the seamless integration of SWS and IPS based on the metaphor of talking through and drawing on a transparent glass window. We have presented our design evolution from ClearBoard-1 to ClearBoard-2. ClearBoard-1 is a simple video drawing prototype, and ClearBoard-2 is a pen-based computer drawing system. Through informal use and experiments, we have confirmed that ClearBoard facilitates making eye contact, monitoring a partner's direction of gaze, and shift focus between SWS and IPS. We found the most important feature of ClearBoard to be gaze awareness. This is a potentially useful capability that CSCW technology can greatly enhance.

Seamless integration of SWS and IPS based on a glass board metaphor

Design evolution from ClearBoard-1 (video drawing) to ClearBoard-2 (computer drawing) with TeamPaint.

ClearBoard supports eye contact, gaze awareness, natural gestures, and smooth transition between SWS and IPS.

F u t u r e . W o r k

Multi-user/multi-point ClearBoard

Effects of gaze awareness and eye-contact in collaboration

Virtual interpersonal distance

There are many interesting technical and behavioral issues to be investigated in the future. The challenging design question is how to extend the ClearBoard concept to support multiple users and multiple points.

This cartoon shows an interesting triangle tabel-tennis metaphor for three presidential candidates. However, this metaphor does not work because there is no common workspace for all three users.We are planning to conduct empirical studies to understand the effects of gaze awareness and eye-contact in collaboration.

I n t e r p e r s o n a l . D i s t a n c e

Face to Face

ClearBoard

Inter-personal distance is another interesting issue.

ClearBoard visualizes the partner as standing about one meter from the the user. When I use ClearBoard with Minoru, one meter is an appropriate distance. However, for the meeting with the Executive Director, I felt the distance was too short. Especially if the director has the authority to transfer me out of my laboratory, I would be very uncomfortable with this virtual inter-personal distance. I would like the media to provide me with some control over the inter-personal distance.

W h a t . i s . C l e a r B o a r d - 2 ?

Computer and video-communication technologies have evolved independently. Although they can be loosely coupled using arbitrary multi-window interface, they have never been integrated seamlessly from the users' cognitive point of view. ClearBoard-2 is the first system that succeeds in naturally integrating two different technology streams: groupware technology and video conferencing technology. We expect that the seamless integration of computer and video communication technologies will realize the next generation of collaboration media. Moreover, ClearBoard-2 can be seen as an instance of the paradigm shift from traditional HCI to HHI mediated by computers. We are interacting not with computers, but through computers. I believe the ClearBoard design is beyond the traditional desktop metaphor based on a multi-window system.

ClearBoard-2 is a new collaboration medium that integrates groupware and videophone technology seamlessly.

From HCI (Human-Computer Interaction )

to HHI (Human-Human Interaction) mediated by computers.

Beyond desktop metaphor: design based on everyday skills of human communication and collaboration.

S k i l l - B a s e d . D e s i g n

We have presented the evolution of our research activities. The key concept is `seamlessness'. We believe it is most important to respect the skills that people use in everyday life. We have focused on skills such as talking, gesturing, gaze reading, drawing, and using computers. Based on these skills, our design will lead to cognitive seamlessness.

Respect the skills that people use in everyday life. (Bill Buxton)

talking, gesturing, gaze reading, drawing, and using computers. Continuity with everyday workpractices

B e y o n d . ` B e i n g . T h e r e '

For long time, how to imitate a `Being There' situation overcoming spatial restrictions was a tacit goal of telecommunication R&D. For instance, video telephony tries to create an illusion as if all the participants sit in an imaginative conference room using broad band communication channel. The goal was the simulation as close as to the real face to face meeting, however, it is something like a mirage. It can never be achieved 100%. The gap between the recognizable reality and virtuality cannot be filled by technology. As long as that gap can be perceived by human senses and as long as we have face to face meeting, a familiar alternative, we would choose this even if time and cost are required. Even the most exquisitely created simulation cannot surpass the real one. Here lies a fundamental contradiction of traditional R&D on telecommunication, especially of video telephony, that sets `Being There' as its ultimate goal .

If the telecommunication tool is interpreted as an alternative medium to face to face meeting in terms of the distance, the barrier cannot be overcome. In other words, as long as we try to find the raison d'etre of communication media only in its being `tele', we shall never be able to solve this contradiction. Conventional logic that it saves travel expenses and time should be reviewed. In order to solve this problem, a new goal should be set; an exciting medium to let people use even if they are nearby each other. It should have some value-added so that even people within the same building who can talk face to face would be interested in using that medium. Naturally, that added value should offer something that an ordinary face to face conversation cannot provide. By having that, the medium could be the one to be enjoyed regardless of the distance and cost. Then, what is that value-added?

One of the values-added viewed from the point of CSCW (Computer-Supported Cooperative Work) is `virtual shared workspace.' Communication media will evolve into collaboration media fusing with computer technology, particularly, with groupware technology. To provide a shared workspace with flexibility which conventional conference rooms could not offer can be a means for a differentiation from such a strong alternative as face to face meeting.

I would like to suggest to deny the goal, `Being There', of which adequacy was seldom questioned because it was considered as an ultimate form of telecommunication and also I would like to suggest to focus the future R&D efforts on the creation of value-added which cannot be obtained from face to face conversation. It means to abandon the goal of simulating the physical proximity or the simulation of `Being There' through the improvement of image transmission technology such as high resolution or realism. I propose to set up a paradigm shift in search of creating value-added type new services aiming at `Beyond Being There'. This is the point of the second suggestion.

Tacit Goal of Telecommunication: Imitation of `Being There' over distance (e.g. video conferencing. But this goal is a mirage. It can never be achieved 100%.) New Goal: Creation of added value that an ordinary face to face meeting cannot provide (e.g. virtual shared workspace)

F r o m . M u l t i m e d i a . t o . S e a m l e s s . M e d i a

Multimedia = more than 3 cables behind a computer?

Multimedia = a premature label that represents a stage of media evolution from the mono-media to the seamless media.

`Multi-ness' is not main issue. Seamless integration of media hiding the low level representations, and complexity of underlying technologies is the core issue.