Open For Questions - A Critique of Idea Sharing Sites

President-elect Obama’s web site change.gov has a page (“open for questions” ) where people can submit and vote on questions, and where he promised to answer the top 5 questions in each category. Apparently a journalist encouraged readers to vote for one of the questions (on appointing a special prosecutor to look at Bush’s actions), which resulting in the question going for #6 to #1 in the foreign policy category. Change.gov seemed pretty successful in some ways. It's a really nice example of the Internet allowing a whole new set of players to deeply influence the political debate at a national level. The special prosecutor question was dead in the mainstream media. The questions that rose to the top seem substantive and important and have substantial legitimacy because of how active the voting was (100,000 voters, 70,000 questions, 4 million votes). One can argue that spreading power through society, more broadly than was possible before the Internet, is a great thing. It seems clear that excessive concentration of power in a few individuals and institutions (corporations, executive branch) has been at the root of some pretty serious problems in the States. Idea sharing sites have also been used, with apparent success, in other contexts. The Dell ideastorm site, for example, includes over 11,000 ideas (for Dell products and services) that have received a total of over 680,000 ratings and 84,000 comments.

There are some important problems, however, with such sites. One is redundancy. Even in change.gov. a large proportion of the questions, including many in the top 5, are minor variants of each other. Many submissions are word-for-word identical. If the administration is strict about the "top 5 questions" rule, that will impoverish the range of questions they consider. The redundancy can also distort the rating results. A question with focused voting may beat out a more important question that had it's votes spread over many redundant instantiations. With over 70,000 questions submitted, manually pruning this list to consolidate equivalent questions would be a massive undertaking. Finally, most people are only likely to look at the most highly rated posts. The result is that the vast majority of the submitted ideas are lost to obscurity.

Another issue involves dysfunctional ranking outcomes. The Better World Campaign used an idea sharing system with user rating to come up with a proposal for Obama's first actions upon entering office. After a year of debate, over 5,000 ideas submitted, and close to one hundred thousand votes, the result was the proposal that Obama plant an organic garden at the White House. That vast investment of collective effort resulted in selecting a tiny, purely symbolic, gesture. Or look at the ideas list collected at http://www.change.org/ideas/. Is legalizing marijuana and other drugs really the top priority for this apparently serious and highly diverse (judging from the list of sponsoring organizations) set of voters? One possible explanation for these effects is self-reinforcing ratings. When people are asked to rate a very long list of items sorted by their current average rating, one can expect that the system will quickly "lock" into a fairly static, and arbitrary, ranking. People are more likely to vote for ideas that are already higher in the list, since if there are thousands of ideas, people will in all likelihood stop looking after the first few. So the first few winners take all, even if they are inferior to many other ideas in the list. Duncan Watts observed this property in looking at music rankings (Matthew J. Salganik, Peter Sheridan Dodds, Duncan J. Watt (2006). "Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market" Science 311:5762, pps 854-856): when people could see each other's rankings before rating, much of the ranking results were essentially random.

A final issue is non-collaborativeness. These idea-sharing sites came up with hundreds or even thousands of disconnected, and individually quite short and simple, ideas, of which only a tiny fraction see the light of day. While just getting people to participate has its own value, no doubt, it still seems like an incredible waste of the collective intelligence and commitment of the people who participated in the process. Surely that amount of effort could have been used to compose a smaller number of more substantive proposals. Idea-sharing sites, however, do not provide any mechanism for allowing many people to collaborate on a single larger proposal, as opposed to making many small ones.

These weaknesses with current idea sharing techniques suggest that there is an important need for better methods and tools. A scheme to consolidate redundant questions should help. The problem is probably less technical than social, since computers aren't very good at detecting this kind of semantic redundancy. The challenge is to evolve rules that govern human filtering that work as well as the rules that govern editing practices on Wikipedia. A argumentation system, especially if we include a rating/aggregation scheme for arguments, *should* tend to encourage rankings that are based in reason rather than just being "popularity contests" driven by a few influential bloggers. It should also allow people to collaboratively develop more elaborated ideas, since an idea can be linked to issues that ask how to refine some part of the idea, which can in turn be linked to other ideas. Another possible approach is to make sure people are not be able to see rankings until *after* they vote. One trick, proposed by MIT Prof. David Karger, is that people be given a randomly selected set of ideas to rank, so each has an equal opportunity to get attention. This will probably only work, however, if the set of ideas is small enough relative to the number of viewers. Otherwise, most ideas will get too few viewings, and ratings, to represent a good picture of the "wisdom of the crowds". If we had 4 million votes spread randomly over the 70,000 questions submitted to change.gov, for example, then each question would receive an average of only about 60 views. That hardly seems adequate to get a good sense of what a nation’s priorities are.