Time: 1:15-2:30PM
Location: UTA 5.522 (1616 Guadalupe St., 5th Floor)
Biography:
Kevin Crowston is a Distinguished Professor of Information Science in the School of Information Studies at Syracuse University. He received his Ph.D. (1991) in Information Technologies from the Sloan School of Management, Massachusetts Institute of Technology (MIT).
His research examines new ways of organizing made possible by the extensive use of information and communications technology. Specific research topics include the development practices of Free/Libre Open Source Software teams and work practices and technology support for citizen science research projects, both with NSF support. His most recent project is a research coordination network on Work in the Age of Intelligent Machines.
He is currently Editor-in-Chief for the journals ACM Transaction on Social Computing and Information, Technology & People.
Abstract/Title:
Scaffolded Training with Machines: An Experiment on Participant Learning in an On-line Production Community
Training users in online communities is important for making high performing contributors. However, several conundrums exists in choosing the most effective approaches to training users. For example, if it takes time to learn to do the task correctly, then the initial contributions may not be of high enough quality to be useful.
We conducted an online field experiment where we recruited users (N = 386) in a web-based citizen-science project to evaluate the two training approaches. The project uses machine learning to teach users how to identify anomalies in a classification task. In one training approach, users received one-time training and were asked to apply their knowledge to the classification task. In the other approach, users were gradually exposed to classes of data that were selected by trained machine learning algorithms as being members of particular classes.
The results of our analysis revealed that the gradual training produced high performing contributors. In our comparison of the treatment and control groups we found users who experienced gradual training performed significantly better on the task (an average accuracy of 90% vs. 54%), contributed more work (an average of 228 vs. 121 classifications), and were retained in the project for a longer period of time (an average of 2.5 vs. 2 sessions). The results suggests online production communities seeking to train newcomers would benefit from training regimes that gradually introduce them to the work of the project using real tasks.
crowston@syr.edu
348 Hinds Hall, Syracuse, NY 13244
crowston.syr.edu
Requested by: Dona Kurtz; dona@utexas.edu, Matt Lease; ml@utexas.edu