A new, hybrid approach to surveys Princeton researchers have developed combines the data-gathering advantages of interviews with the lower cost and analytical simplicity of traditional surveys to yield insights that would be difficult to obtain with other methods.
These "wiki surveys" — inspired in part by the constant, user-fueled evolution of websites like Wikipedia — begin with questions and a set of potential answers as in a traditional survey. The wiki surveys then continue to change as participants offer new potential answers, which are shown to future participants.
"On the one hand, we have methods that are open to new information, like interviews, but these are generally slow, expensive and hard to quantify," said Matthew Salganik, a Princeton professor of sociology and one of the researchers. "On the other hand, we have methods that are good at quantification, like traditional surveys, but they tend to be closed to new information. So this wiki survey project combines the quantifiability you get from a survey with the openness you get from an interview."
The new approach to surveys and two examples of "pairwise" wiki surveys — in which participants are presented a question and two possible answers — are described in an article, "Wiki Surveys: Open and Quantifiable Social Data Collection," published in May in the journal PLoS ONE. The authors are Salganik and Karen Levy, who earned her Ph.D. at Princeton and is now a postdoctoral associate at New York University.
One of the examples is a survey created by the New York City Mayor's Office of Long-Term Planning and Sustainability as part of its effort to update the city's sustainability plan, PlaNYC 2030. City officials wanted to incorporate ideas from residents, so they created a wiki survey that asked: "What do you think is a better idea for creating a greener, greater New York City?" The officials generated a list of 25 possible answers.
Participants saw the question and were presented with two possible answers. They could choose whichever they preferred (or "I can't decide") and then would see another pair, continuing for as long as they liked.
Instead of choosing either supplied answer, though, participants could offer their own. After being screened by survey organizers for relevance, that answer would be added to the stock of possible responses and be shown to future participants. Almost 1,500 people took the survey over four months, and they submitted 244 alternative responses that other participants could choose.
In the end, eight of the top 10 answers were contributed by participants.
This screenshot shows a wiki survey created for the New York City Mayor's Office of Long-Term Planning and Sustainability. Click on the image to take a sample wiki survey.
"The most important finding that we have is that ideas uploaded by users frequently score better than the best initial ideas," Salganik said. "This is a repeated pattern, and it illustrates the value of being open. There are good ideas out there."
Sometimes, the new user-submitted answers represent ideas survey creators just hadn't thought of. In other cases, users are able to express existing ideas in ways that resonate better with other participants.
For example, the top-rated idea in the New York survey, which was contributed by a participant, was "Keep NYC's drinking water clean by banning fracking in NYC's watershed."
City officials said they wouldn't have referenced fracking directly.
"We talk about it differently. We'll say, 'protect the watershed.' We don't' say, 'protect the watershed from fracking,' " city official Ibrahim Abdul-Matin said in the journal article.
Another advantage, Salganik said, is that participants can respond as many times — or as few times — as they like. In that way wiki surveys are able to gather information that would be missed by traditional surveys that require participants to answer an exact number of questions.
As part of their research, Salganik and Levy developed a website, allourideas.org, where anyone can create their own wiki survey. About 6,400 surveys have been created since the free website was launched in 2010. So far, the biggest users have been governments, nonprofit groups and companies. Academic researchers have been slower to adopt wiki surveys, Salganik said, though he said he hoped the publication of the paper would help the approach gain acceptance.
Michael Hout, a professor of sociology at New York University who studies social change, said wiki surveys represent an exciting innovation in data collection.
"As a survey researcher myself, I often describe our job as 'putting words in people's mouths' — that is, we not only ask the questions but we frame the answers, too," said Hout, who was not involved in the work. "The wiki approach allows the people who participate in the study the chance to use their own voices, altering both questions and answers to better express their point of view."
Salganik is working on a book titled "Bit by Bit: Social Research in the Digital Age" that examines the future of social science, including concepts such as wiki surveys.
In the future, he hopes to continue to improve the wiki surveys.
Support for the research came from Google Inc. and Princeton's Center for Information Technology Policy. Salganik did some of the research while on leave from Princeton at Microsoft Research.
