of the American Society for Information Science and Technology  Vol. 28, No. 6    August / September 2002

Search

Go to
Bulletin Index

bookstore2Go to the ASIST Bookstore

 

Copies

Scientific Collaboratories: Evaluating their Potential

by Diane H. Sonnenwald, Mary C. Whitton and Kelly L. Maglaughlin

Diane H. Sonnenwald, Mary C. Whitton and Kelly L. Maglaughlin are affiliated with the University of North Carolina at Chapel Hill. Sonnenwald may be reached at SILS, University of North Carolina, Chapel Hill, CB#3360, 100 Manning Hall, Chapel Hill, NC 27599; 919-962-8065; dhs@ils.unc.edu

Scientific collaboratories have the potential to be centers "without walls, in which researchers can perform their research without regard to physical location interacting with colleagues, accessing instrumentation, sharing data and computational resources, and accessing information in digital libraries" [Wulf, W.A. (1993). The collaboratory opportunity.  Science, 261, 854-855]. A number of scientific collaboratories have been developed, and a comprehensive discussion of these collaboratories can be found in Finholt, T. (2002). Collaboratories, In B. Kronin (Ed.) Annual Review of Information Science and Technology. Washington, DC: American Society for Information Science and Technology.

However, the evaluation of scientific collaboratories has lagged behind their development. So few evaluations of scientific collaboratories exist that fundamental questions regarding their potential have yet to be answered: Can distributed scientific research produce high quality results? Do the capabilities afforded by collaboratories outweigh their disadvantages from scientists' perspectives?  How does the scientific process change in the context of a collaboratory?

The answers to these questions are not obvious. Previous research in computer-supported cooperative work [e.g., Olson, G.M. & Olson, J.S. (2000). Distance matters. Human-Computer Interaction, 15(2-3), 139-178] and theory of language [Clark, H. (1996). Using Language. Cambridge, UK: Cambridge University Press] would predict that working remotely would lack the richness of collocation and face-to-face interaction such as multiple and redundant communication channels, implicit cues and spatial co-references, which are difficult to support via computer-mediated communications. This lack of richness is thought to impair performance because it is more difficult to establish the common ground that enables individuals to understand the meaning of each other's utterances.

Evaluation of the Context: The nanoManipulator Collaboratory

The collaboratory system we evaluated provides distributed, collaborative access to a specialized scientific instrument called a nanoManipulator (nM). The single-user nM provides haptic and 3D visualization interfaces to a local (co-located) atomic force microscope (AFM), providing a natural scientist with the ability to interact directly with physical samples ranging in size from DNA to single cells. The nM and its uses are described in Guthold, M., Falvo, M.R., Matthews, W.G., Paulson, S., Washburn, S., Erie, D.A., Superfine, R., Brooks, Jr., F.P., & Taylor, III, R.M. (2000). Controlled Manipulation of Molecular Samples with the nanoManipulator. IEEE/ASME Transactions on Mechatronics, 5(2), 189-198.

The collaboratory version of the nM was designed based on results of an ethnographic study from which we developed an understanding of the scientific research process, current collaborative work practices, the role of an nM as a scientific instrument and scientists' expectations regarding technology to support scientific collaborations [Sonnenwald, D.H., Bergquist, R., Maglaughlin, K.L., Kupstas-Soo, E. & Whitton, M.C. (2001). Designing to support collaborative scientific research across distances: The nanoManipulator example, In E. Churchill, D. Snowdon, &A. Munro, (Eds.), Collaborative Virtual Environments (pp. 202-224). London: Springer Verlag].

Hardware elements in the collaboratory system include two PCs. One PC is equipped with a Sensable Devices Phantom force-feedback device. This PC and its associated software provide haptic and 3D visualization interfaces to a local or remote atomic force microscope (AFM) and support collaborative manipulation and exploration of scientific data.  Scientists can dynamically switch between working together in shared mode and working independently in private mode.

In shared mode, remote, that is, non-collocated, collaborators view and analyze the same (scientific) data. Mutual awareness is supported via multiple pointers, each showing the focus of attention and interaction state for one collaborator. Collaborators can perform almost all operations synchronously. Because of the risk of damage to an AFM, control of the microscope tip is explicitly passed between collaborators.

In private mode, each collaborator can independently analyze the same or different data from stream files previously generated. When switching back to private from shared mode, collaborators return to the exact data they were previously using.

Another PC supports shared application functionality and video conferencing (via Microsoft NetMeeting) and an electronic writing/drawing tablet. This PC allows collaborators to work together synchronously using a variety of domain-specific and off-the-shelf applications, including specialized data analysis, word processing and whiteboard applications. Video conferencing is supported by two cameras. One camera is mounted on a gooseneck stand so it can be pointed at the scientist's hands, sketches or other physical artifacts scientists may use during experiments; the other is positioned to capture a head and shoulders view of the user. Collaborators have software control of which camera view is broadcast from their site. A wireless telephone headset and speakerphone connected to a commercial telephone network provides high quality audio communications for collaborators.

The Experimental Evaluation

The experimental evaluation study was a repeated-measures, or within-subjects, controlled experiment comparing working face-to-face and working remotely with the order of conditions counterbalanced. Twenty pairs of study participants (upper level undergraduate natural science students) conducted two realistic scientific research activities each requiring two to three hours to complete. Ten pairs of study participants worked face-to-face first and, on a different day, worked remotely (in different locations.) Another 10 pairs worked remotely first and, on a different day, face-to-face. When face-to-face, the participants shared a single collaboratory system; when collaborating remotely, each location was equipped with its own complete collaboratory system.

The scientific research activities completed by the participants were designed in collaboration with natural scientists. The tasks were actual activities the scientists completed and documented during the course of their investigations. To complete the tasks the participants had to engage in the following activities typical of scientific research: operate the scientific equipment properly; capture and record data in their (electronic) notebooks; perform analysis using scientific data analysis software applications and include the results of that analysis in their notebooks; draw conclusions, create hypotheses and support those hypotheses based on their data and analysis; and prepare a formal report of their work.

We collected a variety of quantitative and qualitative evaluation data, including task performance measures to compare the quality of scientific work produced in the two collaboration conditions, and post-interviews to gain, from participants' perspectives, a more in-depth understanding of the scientific process in both conditions.

Task performance was measured through graded lab reports. The information participants were asked to provide in the reports mirrored the information found in the scientists' lab notes created when they conducted their original research. Each pair of study participants collaboratively created a lab report under each condition, generating a total of 40 lab reports; 20 created working remotely and 20 created working face-to-face. The lab reports were graded blindly; the graders had no knowledge of the report authors or under which condition the report was created.

To further our understanding of participants' perceptions of the system, we conducted semi-structured interviews with each participant after each task. Study participants were asked what they thought about their experience, including the most satisfying and dissatisfying aspects of their experience. In addition, we inquired about work patterns that emerged during the experience, and the impact technology may have had on their interactions with their collaborator. After completing their second task, participants were also asked to compare working face-to-face and working remotely. To better learn each participant's perspective, participants were interviewed individually, for a total of 80 interviews, each lasting from 30 to 60 minutes.

Results

Task Performance: Analysis of Graded Lab Reports. The average lab report scores for the first task session were identical (70/100) for both the face-to-face and remote condition. Previous research would predict that scores from a remote first session would be lower because the remote session would lack the richness of collocation and face-to-face interaction, including multiple and redundant communication channels, implicit cues and spatial co-references, that are difficult to support via computer-mediated communications. This lack of richness is often thought to impair performance. Perhaps technical features such as seeing your partner's pointer and functions, optimized shared control of scientific instrumentation and applications, improved video that provides multiple views and high quality audio communications may be  "good enough" for scientific tasks focusing on collecting, analyzing and interpreting data.

The data further suggest that collaborating first remotely may have a positive effect. Using a multivariate analysis of variance (MANOVA) test, the differences in scores for the face-to-face and remote conditions were not statistically significant. However, when order is taken into account, participants who collaborated remotely first scored significantly higher on the second task than did those who collaborated face-to-face first. There was no statistically significant difference between face-to-face and remote lab scores for participants who collaborated face-to-face first.

In general, the literature suggests that participants would learn more about the system, science and each other when collaborating face-to-face and that this knowledge helps increase their current and future performance. Our performance data suggest collaborating first remotely does not negatively impact current performance, and may positively impact future performance for scientific tasks such as data collection, analysis and interpretation. We looked to our interview data for explanations of this result.

Participants' Perceptions of the Scientific Process: Post-Interview Analysis. As expected, participants reported disadvantages to collaborating remotely. However, participants also reported that some of these disadvantages are not significant in scientific work contexts and that coping strategies, or work-arounds, can reduce the impact of other disadvantages. Furthermore, participants reported that remote collaboration provided several relative advantages compared with face-to-face collaboration (see Table 1). 

Similar to previous studies [e.g., Olson, G.M. & Olson, J.S. (2000). Distance matters. Human-Computer Interaction, 15(2-3), 139-178], study participants reported remote collaboration was less personal than face-to-face collaboration. When comparing working face-to-face and remotely, participants reported collaborating face-to-face was more personal and it was easier to express themselves. However, participants also reported that lacking this type of interaction when working remotely did not seem to have a negative impact on their work. The impersonal nature of remote collaboration increased their productivity and facilitated collaborative intellectual contributions. As participants explained:

    If we were. . . working side by side, we might tell more stories or something like that. . . . [However] if you're trying to get something done, sometimes the stories and stuff can get in your way.

     I think that being in separate rooms helps a little bit because it's more impersonal. . . [You] just throw stuff back and forth more easily.

Participants also reported that when working remotely they received fewer implicit cues about what their partners were doing and thinking. The study participants explained that without these cues, it may be difficult to follow social interaction norms and assist your collaborators:

    [when collaborating face to face] it was a lot easier to ask questions of each other. . . since you have a feeling [about] when to interrupt them . . . if you're in the same room . . . you'll wait [to ask a question] until the other person is not doing as much or not doing something very specific.

     It is hard to get the context of any question that's asked because you're not paying attention to what the other person is doing because they're in a little [video-conferencing] screen.

To compensate for this lack of cues, several participants reported they needed to talk more frequently and descriptively when collaborating remotely. Participants reported

    Even though we were in separate rooms, it kind of seemed like there was more interaction compared to being face-to-face, which seems kind of strange. . . . It just seemed more interaction was expected. . . . Maybe needed.

     We had a really good interaction [when collaborating remotely]. . . .  You're conscious that you're not together and you can't see [some things, and] so you think more about [interacting. For example, you think] 'I need to let this person know that I'm about to do this' or 'this is what I'm seeing and I'm trying to let you know so, and you're like doing the same to me.' 

Thus to compensate for the absence of implicit cues in the remote condition participants provided explicit cues for their partner. When working remotely, it appears that individuals recognize they do not have a common shared physical reality and subsequently may not have a shared cognitive reality. However, humans are intrinsically motivated to develop a shared reality [Schutz, A., & Luckmann, T. (1983). The Structures of the Life-World, Vol. I. Evanston, IL: Northwestern University Press]. Subsequently, study participants developed a strategy, providing explicit cues to their partners, to develop a shared reality. These explicit cues, or joint actions, typically contribute to faster and more accurate formation of common ground and mutual understanding [Clark, H. (1996). Using Language. Cambridge, UK: Cambridge University Press].

In addition to receiving fewer cues from a partner, participants also reported that some physical tasks are more difficult when collaborating remotely. These tasks include drawing, e.g., creating sketches of scientific structures and manipulating mathematical equations, and sharing control of applications within NetMeeting. Some of these problems may be remedied with advances in technology, such as shared applications that support multiple pointers and concurrent floor control. Participants explained

    [when collaborating face to face] you could draw more easily, communicate diagrams more easily, and you could look at the other person and see their level of understanding more easily.

     The thing that frustrated me the most [collaborating remotely] was the shared applications [NetMeeting] . . . you could see the other person doing things but you couldn't do anything [simultaneously].

Although technology made some tasks more difficult, study participants also reported that the collaboratory system provides some advantages over collaborating face-to-face. These advantages include the ability to easily explore the scientific instrument and data and their own ideas both independently and collaboratively, having identical views of the data visualization, and working simultaneously with the data visualization.

    I liked that we were separate.  I think it gave a whole new twist on the interactions, and if one of us got snagged up with something the other could independently work and get it done rather than both of us being bogged down by having to work on it simultaneously.

     Sometimes when you're working side by side with somebody, you have to deal with 'Well, you're looking at [the data] from a different angle than I am, and so you're seeing a different perspective there.' Now [working remotely] we could both of us be straight on, having the exact same perspective from where we're sitting. It made it easier.

     [My partner] could be changing the light focusing somewhere, while I could be zooming or moving [the plane] around. And that was really helpful because you're thinking, 'OK, as soon as I'm done moving the light I want to go ahead and shift [the plane] . . . [to be able to] say to [my partner], 'Why don't you [shift the plane] while I'm shining the light,' was really cool. It was really helpful.

The participants in this study experienced disadvantages attributed to remote collaboration that have been previously reported in the literature. However, the study participants also reported that some disadvantages had minimal impact on their scientific work, and they used coping strategies to compensate for disadvantages. In addition, they perceived remote collaboration to provide some advantages relative to face-to-face collaboration. These findings corroborate our findings regarding task performance.

Conclusions and Future Research

The results illustrate the potential of collaboratories, allowing individuals who do not have specialized, state-of-the-art scientific instruments locally to access scientific instruments remotely and conduct scientific experiments in collaboration with other students, faculty and staff at institutions that have the instruments.

When collaborating first remotely, study participants were able to use the technology and conduct science as well as or better than if they were face-to-face. Working face-to-face before working remotely did not produce the anticipated positive impact on the scientific process or outcomes. These conclusions are supported by the similarity in lab report grades for the first task session and statistically significant higher lab report grades for pairs who first worked remotely, as well as interview data that illustrate that the technology provides unique advantages and, although the technology has disadvantages, many individuals can develop coping strategies to reduce the impact of these disadvantages. The evaluation data combine to illustrate the potential of the collaboratory system for adoption by scientists and how this technology mediates collaborative scientific work processes without negatively impacting scientific data collection and analysis task outcomes.

However, the tasks used in the study do not encompass the entire life cycle of the scientific process. For example, problem formulation, research design and research dissemination were not included in the tasks. Furthermore, the tasks in the first and second sessions differed. Although designed to be similar in complexity, additional investigation may uncover aspects of the tasks that are inherently impacted by an interaction condition.

Future work includes a longitudinal field study to investigate whether the results reported here hold for professional scientific contexts. In the field study, the technology will be provided to scientists who have expressed an interest in conducting scientific investigations using the system. We plan to investigate the similarities and differences between scientists' and study participants' perceptions and use of the technology to further our understanding of the impact of collaboratories on the scientific process and outcomes.

Acknowledgments

Our thanks to the study participants; to students who helped run the experiment sessions and assisted in data analysis; to the team who built the nM, including Frederick P. Brooks, Jr., Martin Guthold, Aron Helser, Tom Hudson, Richard Superfine and Russell M. Taylor II. The development of the nM and this work have been funded by the NIH National Center for Research Resources, NCRR 5-P41-RR02170.  The nanoManipulator project is part of the Computer Graphics for Molecular Studies and Microscopy Research Resource at the University of North Carolina at Chapel Hill.

How to Order


ASIST Home Page

American Society for Information Science and Technology
8555 16th Street, Suite 850, Silver Spring, Maryland 20910, USA
Tel. 301-495-0900, Fax: 301-495-0810 | E-mail:
asis@asis.org

Copyright © 2002, American Society for Information Science and Technology