Computers intersect with sociology to sift through 'all our ideas'

Catholic Relief Services of Baltimore faced a daunting challenge: getting advice from several thousand people at once.

To improve its hiring and training efforts, the charity, which assists poor and vulnerable people around the world, wanted to ask its employees what makes someone an effective relief-services worker. But surveying the organization's more than 4,000 employees, who work out of 150 offices worldwide and speak a range of languages, was no easy task.

"We were looking for a tool for mass collaboration," said Stephen Moles, a manager in the charity's human resources department. "Most of what we found allowed people to post comments and edit documents together, but neither of those functions exactly suited our needs."

Fortunately, an acquaintance referred Moles to Matthew Salganik. An assistant professor of sociology at Princeton, Salganik has teamed up with Princeton computer scientists to develop a new way for organizations to solicit ideas from large groups of people and simultaneously have those same people vote on the merit of the ideas generated by the group.

Called "All Our Ideas," the survey tool melds concepts from sociology and computer science to allow an organization to quickly set up a free website where large numbers of people can contribute and rank ideas. The system could help governments tap into public opinion and provide sociologists with a new research tool.

"This is a hybrid method that combines the quantitative power of surveying with the power of focus groups to generate ideas," Salganik said. "Recent advances in Web-based computing make it possible. It's something that probably wasn't possible just five years ago."

Nadia Heninger (above) and Bill Zeller, graduate students in computer science, provided technological expertise in developing the All Our Ideas project.

Coming from a sociology background, Salganik relied on the technological expertise of two Princeton computer science graduate students, Nadia Heninger and Bill Zeller, to develop All Our Ideas. He connected with them through the Center for Information Technology Policy, a joint venture of Princeton's School of Engineering and Applied Science and the University's Woodrow Wilson School of Public and International Affairs. The center seeks to inform public discussion of digital technologies and promote the use of technology in helping societies function.

In creating the research-based survey program, Salganik drew inspiration from a decidedly non-academic source: Kitten Wars, a website that allows site visitors to upload photographs of kittens and vote for the cats they think are the cutest. Visitors to the Kitten Wars site are presented with two photos at a time and asked to choose which kitten is cuter. In this manner, big-eyed fur balls rise to the top rank, while hairless breeds sink to the bottom.

Similarly, All Our Ideas allows survey creators such as Moles to present respondents with a question and let them choose which of two answers they find more appropriate. If none of the answers seems ideal, the user can submit a new idea, which will join the pool of answers that are voted on in pairs.

In 2008, soon after the first version of All Our Ideas was released, the Princeton Undergraduate Student Government used the application to canvas students about priorities to guide the efforts of student government officials. About 2,000 students cast 40,000 votes. Despite the fact that student officials had seeded the survey with about 200 potential priorities, two of the entries voted most important by the respondents were new submissions, ideas the officials hadn't considered.

"No matter how hard we try, we have blind spots," Salganik said. "Organizations want to know what they don't know. In the past, that's been difficult. If you ask for suggestions, especially online, you may get thousands of ideas that can take weeks to sort through. On the other hand, if you use a survey with preset questions and answers, you can get lots of data but few new ideas."

Dmitri Garbuzov, who graduated in June with a bachelor's degree in computer science, is one of two students working on the project this summer with Salganik, supported by a grant from the Google Summer of Code Program.

In preparation for launching its survey, Catholic Relief Services ran a pilot study in May, soliciting a subset of its 4,000 employees. Four of the top five ranked ideas were seed entries from the organization, but one was new, contributed by an employee. A user suggested that an exemplary relief services worker "Works with a clear priority of assisting the poor."

"It's something that is clearly a part of our mission, but it wasn't articulated in our seeded ideas," said Moles. "One thing that I appreciate about this tool is that it allows people to put things in their own words. Part of what we are trying to accomplish is to find out not only what is effective but what is meaningful."

Edward Felten, director of the Center for Information Technology Policy, which along with Google provided initial funding for the development of All Our Ideas, said the application could provide a valuable tool for governments to understand public opinion.

"The process by which governments get comment from the public is really coming under strain," said Felten, professor of computer science and public affairs. "Agencies solicit comments, but the Internet has meant that they get far more responses that they used to. They are required by law to sort through them, but often don't have time and resources for such a task."

Although other surveying applications have been developed, the Princeton team designed All Our Ideas to avoid pitfalls that have plagued some online systems. For instance, to address the issue of people gaming the system by voting on their favorite entry multiple times, the system randomly selects the two voting choices presented to respondents, making it more difficult for someone to repeatedly find and vote for a specific entry.

Site visitors also make their choices blindly, without having seen the rankings of the various ideas, which prevents them from being biased by other people's opinions. The system accounts for some ideas being in the system longer than others, preventing older entries from dominating the rankings because they've been voted on more times.

Other organizations that have launched their own idea marketplaces using All Our Ideas include Columbia University, Bowie State University in Maryland and Empower Our Neighborhoods, a New Brunswick, N.J., community-development organization. A number of other organizations are considering using the application, including large city governments and federal agencies.

Heninger, the computer science graduate student who worked on the theoretical underpinnings of All Our Ideas, said the algorithms behind the application continue to evolve. "As a theorist, it's a great source of problems, because it's something real," she said. "The hope is that we can adopt more sophisticated techniques, allowing us to generate a lot more information than you would think you could."

To support further expansion of the project, Salganik recently received a grant through the Google Summer of Code Program that allowed him to hire two Princeton students to work on the project this summer. Google also is allowing him to promote the application through its AdWords platform, providing him with the equivalent of $300 per day in online advertising.

The Princeton group also is working on a version of the system, called Photocracy, which would allow users to submit and vote on photos in response to a question.

In addition to providing organizations with a tool for communicating with their constituents, Salganik said that All Our Ideas and similar online platforms could revolutionize social science.

"In the past, if we wanted to conduct a sociology experiment, we could only study a few hundred people, which really limited us," he said. However, online social networking could allow research at a vastly different scale, involving hundreds of thousands of people. "The Web has changed the way people interact, but not the way we do sociology -- yet."