WikiDebate: A Proposal for a Wiki-based Mechanism for Cooperatively Structuring Debates, and Collectively Discovering Truth

SourceForge.net Logo

Click here for example, mock-up website.

Click here for the Sourceforge project page.
Click here for Dave Zanni's homepage.


Introduction

Constantly we are implored by our politicians to engage in a “national dialog” over one subject or another, be it the future of social security, health insurance, the threat of climate change, etc. Of course, the extent to which people actually engage in such national dialogs is questionable. Opinions tend to ossify in the absence of a real understanding and appreciation of the other side's arguments. Even people on the same side of an issue will often not agree on the correct vocabulary or framework to use in understanding the problem. Understandably, this can lead to either frustration or apathy, especially as the problems confronted become more complex.

Fortunately, technology allows for the use of profoundly new ways of engaging in dialog. Take a look, for instance, at the success of Wikipedia. It is an on-line encyclopedia which, through the cooperative efforts of thousands of volunteer writers, is able to provide us with a very comprehensive and relatively objective source of information. Likewise, WikiDebate should provide us with the means of creating a credible source to view a variety of opinions on a given topic. The benefits of WikiDebate should be as follows:

  1. Like all Wiki implementations, WikiDebate allows users to edit content created by others. In the context of a debate, this means that in editing material, users first have to understand what the author is trying to say. This addresses a problem that commonly occurs in social dialog: tunnel vision. If a user is editing text written by someone with whom he doesn't agree, then – so long as the spirit of respect and truth is maintained – others are forced to truly understand the opponent's arguments. It forces one to try to understand arguments; it forces one to listen, because unlike in other processes, one can actually change what is said. And since one must understand before editing, learning is enhanced.

  2. WikiDebate reduces the amount of material that has to be read to understand the structure of argumentation on a given topic. It is normal in an academic paper for authors to summarize others’ arguments before developing their own. In WikiDebate, by contrast, reiteration should not be necessary since the other arguments are already available. Participants can simply reference them if necessary.

  3. Since it is possible to edit everyone else's work, it should not be possible to “spin” an opponent’s arguments in a way that is transparently dishonest. Again, this should reinforce a habit of respect for the opinions of others. If one wants the material one writes to remain unedited, then one would be wise to characterize their arguments respectfully and honestly.

  4. One of the frustrating things in communication – especially over complex issues – is that interlocutors often talk past one another. Because they are wedded to their own vocabularies, their discussions are often very hard to disentangle. For a spectator of such a discussion, it can be frustrating to decipher what is being discussed and to reconcile authors' distinctive paradigms. By contrast, since they can edit each others' work, WikiDebate should help provide incentives for such participants to converge on a common vocabulary and framework, and this should both increase the clarity of the issue for spectators and lead to enhanced learning.

  5. Assuming debating is done in the spirit of respect and honesty, it should eventually reduce itself to the fundamental points of disagreement. They will be either value differences or empirical problems for which there is no clear evidence. In the case of empirical disagreements, WikiDebate should help point the way toward new research. In the case of value-based or philosophical disagreements, WikiDebate should at least help participants understand the logic of others, and may help foster respect if not, ultimately, agreement.

  6. By implementing a voting system, there will be some means of objectively determining who is “winning” a given debate. Perhaps more importantly, this will provide incentive for debaters to answer opponents' arguments convincingly, and the problem of preaching to the choir should be reduced.   One of the problems that occurs in dialogs is that opponents don't truly engage each other. But if your opponent proffers an argument and you provide no good response, your opponent is more likely to win the debate. This provides a disincentive to simply ignore an opponent's argument and creates a level playing field. Thus, the more dominant side in a debate will nevertheless be obliged to address criticisms. The more a given WikiDebate site becomes the object of scrutiny the stronger the pressure will be on debaters to truly address opponents' arguments and, moreover, the more debaters will learn because they have been forced to deal with critiques.

  7. Most interestingly to me, the voting system should allow participants to analyze the logical structure of their opponent's (or their ally's) argumentation. It should help clarify how individuals or groups view the issues. It would provide a system whereby opinion could be well quantified, and as such, new arguments could be formed based on such critiques.

Design and Process

The logical design of a WikiDebate will be as follows. First, someone will propose a debate topic that is worded so that it is either true or false. Let's refer to the debate statement as 'A'.

A debate starts with a debate statement, A, which is a statement of truth. That is, it is a statement that can be said to be either true or false. For example, “An increase in the federal minimum wage will increase the income of the average unskilled worker.” Then an argument is proposed. In the above diagram, Argument 1 is proposed first by the affirmative side. The affirmative side argues both that there is a set of truths, B, and that these truths are linked to the truth of A (That is, B -> A). In my construction, all arguments can be characterized in the form depicted above: a statement of truths (B), and a link between these truths and the truth of the debate statement (B->A).

Please note here that the symbolism used above is not to be thought of as strictly logical. Contrary to its normal interpretation in mathematics and logic, B -> A does not necessarily signify “if B, then A”. Rather it means that the truth of B militates in favor of the truth of A, but does not guarentee the truth of A.

The affirmative side's argument would naturally induce the negative side's argument. The negative side can opposed the affirmative side's argument in two ways: 1) it can oppose the affirmative side statements of truth (~A), or it can say that even if the truth statements are indeed true, the link between these truths and the debate statement is false, or, at best, weak. (Additionally, the negative side could say that the set truths proposed by the affirmative side actually militates in favor of the untruth of A – that is, B -> ~A. This type of argument would have to be dealt with in a separate argument proffered by the negative side. Later, when I talk about the voting system, it should be clear why.)

Similarly, the negative side can also make an argument. Argument 2 above is the negative side's argument. It is similar in structure to the affirmative side's argument, the only exception being that the negative side links some set of truths with the untruth of the debate statement – that is, any negative side's argument concludes with ~A. Again similarly to the above, this induces a positive-side argument that says that the negative sides statements of truth are not, in fact, true, and/or that such truths are not linked to the untruth of the debate statement.

To distinguish between the affirmative and negative sides in any given argument, we need the idea of the “advocate”. The “advocate” side of a given argument makes a conclusion about the truth or untruth of the debate statement. Thus, the affirmative side is the advocate for argument 1 because the affirmative side concludes with A – saying that the debate statement is true. The negative side is not the advocate for argument 1 because it only says that the affirmative side has a bad argument, not that the debate statement is untrue. However, the negative side is the advocate for argument 2 since it concludes with the untruth of the debate statement (~A). The affirmative side is not the advocate for argument 2 because it only argues that the negative side's argumentation is faulty – that its truth assertions are wrong or that these truths are not well linked to the untruth of the debate statement – but it does not argue that the debate statement is true. I suppose, therefore, that it would make sense to call this non-advocate side the “critic”, since this side's job is to critique the opposing side.

The Voting System

The intent of the voting system is two-fold: 1) It is intended to provide participants with some incentive to make relevant and cogent arguments that are respectful of the truth. After all, opponent's arguments count. Thus, to the extent that one can make a convincing argument that will convince one's opponents, then the more likely one is to further the success of one's particular argument. 2) It is meant to be used as a means of making analytical assessments of the argumentation. Logical inconsistencies of various groups or individual participants could be critiqued and such critiques could themselves be made into new arguments.

In order to achieve these goals, it is not enough simply to have participants vote for or against particular arguments. Such a system would not be workable since it would not provide enough information. It would be impossible on such a basis to credibly and objectively assert one side to be winning a given debate. In order to do it correctly, arguments have to be divided along the lines I have already indicated. Arguments have to be judged along two dimensions: 1) the truth of the truth statements, and 2) the strength of the linkage between such truth statements and the conclusion (be it A or ~A).

For example, in the debate over the necessity of taking the drastic measure of privatizing the U.S. public pension system (that is, “social security”), one might make the argument that the system is heading for bankruptcy and that therefore drastic and immediate action needs to be taken. We could break this argument into two parts: 1) That the social security is heading for bankruptcy, and 2) that because it is heading for bankruptcy, something drastic has to be done immediately.

The first is a statement about what we will call the “truth” of the argument, and the second is a statement about the “importance” of the argument. These two aspects of any argument can form four combinations: 1) where an argument is true and important, 2) where an argument is true but unimportant, 3) where an argument is untrue but important, and 4) where an argument is untrue and unimportant. So, to use the above example, it might be that, respectively:

  1. Social Security is going bankrupt and therefore privatization is needed.

  2. Social Security is going bankrupt but privatization is hardly the logical conclusion.

  3. Social Security is not going bankrupt, but if it were, privatization would be the logical prescription.

  4. Social Security is not going bankrupt, and, in any case, privatization is a bad idea even if it were.

Thus, the idea would be to have participants vote twice for each argument. The first vote would measure the truth of an argument – whether the premises and facts used in the argument are actually true. The second would have each participant rank the arguments as to their potential importance. That is, assuming that the premises and various truth statements are indeed true, how convincingly is the link made between such truth statements and the conclusion of the advocate.

The truth values of arguments that are considered important would receive more weight in the (as yet uncalculated) formula, and arguments that (even if true) are considered unimportant will be weighed least heavily. After all, it might be irrefutable that 1+1=2, but this doesn't really have much to do with social security, does it? In terms of Social Security the 1+1=2 argument, while true, is unimportant.

A possible graphical user interface for voting

(Click here for example, mock-up website.)

The way I imagine it, the voting system could work something like the following: there would be a sort of summary page of all the arguments with brief descriptions of each argument listed in rows.


Brief Description of Argument 1

Brief Description of Argument 2

Brief Description of Argument 3

Brief Description of Argument 4

Brief Description of Argument 5


The advocate for each argument might be denoted by color coding. Let's say that the affirmative side has a white background, and the negative side has a light gray background. Each description would be the advocate's description.


Brief Description of Affirmative's Argument 1

Brief Description of Negative's Argument 2

Brief Description of Affirmative's Argument 3

Brief Description of Affirmative's Argument 4

Brief Description of Negative's Argument 5


Each row would then have a place where the user could vote on the relative truth value of each argument.


Affirmative is more true

Neutral or ambiguous truth value

Negative is more true

Brief Description of Affirmative's Argument 1



X


















Brief Description of Negative's Argument 2

















X




Brief Description of Affirmative's Argument 3







X














Brief Description of Affirmative's Argument 4












X









Brief Description of Negative's Argument 5

X





















Each user would then also be able to shift the arguments vertically, placing the more important arguments at the top.


Affirmative is more true

Neutral or ambiguous truth value

Negative is more true

Brief Description of Negative's Argument 5

X




















Brief Description of Affirmative's Argument 3







X














Brief Description of Negative's Argument 2

















X




Brief Description of Affirmative's Argument 1



X


















Brief Description of Affirmative's Argument 4












X










Thus, this particular user sees Negative's argument 5 as being the most important, but the Affirmative side as being by far the most convincing as to the truth of the argument. This user also sees argument 4 as being the least important, but the negative side winning on the truth of it slightly.

The cumulation of each user's votes would enter into some yet to be determined formula that would tabulate how many “points” each side of the debate has won.

Some Interesting Possibilities

Calculation of relative vs. absolute truth values

In calculating the negative vs. affirmative side's overall scores, we face something of a problem. There may be a tendency for individuals to have truth values that lean heavily toward one side of a debate. Though we should hope that individuals would be caught up in the spirit of honesty, objectivity and fidelity to the truth, it may be too much to ask. Thus, in debates where there is a schism between individuals – one identifiable group tending to vote one way, another tending to vote in the opposite direction – it may be useful to tweak the formula by which the debate scores are calculated. In particular, it may be useful to introduce relativistic voting scores. That is, in a completely relativistic setup, individuals truth values would be calculated only relative to themselves, not relative to the absolute neutral value. Consider the following two voting patterns:

User 1's voting pattern

Affirmative is more true

Neutral or ambiguous truth value

Negative is more true

Brief Description of Negative's Argument 5

X




















Brief Description of Affirmative's Argument 3





X
















Brief Description of Negative's Argument 2










X











Brief Description of Affirmative's Argument 1


X



















Brief Description of Affirmative's Argument 4







X















User 2's voting pattern

Affirmative is more true

Neutral or ambiguous truth value

Negative is more true

Brief Description of Negative's Argument 5

X




















Brief Description of Affirmative's Argument 3










X











Brief Description of Negative's Argument 2




















X

Brief Description of Affirmative's Argument 1



X


















Brief Description of Affirmative's Argument 4














X








Under a strictly relativistic calculation of truth values, the truth score from User 1's truth value pattern would be exactly the same as the score from User 2's truth value pattern. By contrast, under an strictly absolutist formula for calculating truth scores, User 1 would give the affirmative side a lot more points than User 2 would. This is because in a relativistic calculation, scores are derived by calculating the deviation of a given argument's truth value from an average of the particular user's truth values. In an absolutist calculation, truth values would be calculated by their deviation from some absolute neutral vote.i

The interesting thing here is the possibility of tailoring the formula for a given debate to the needs of a debate. Formula's would not have to be either relativistic or absolutist, but they could be weighted. Debates that are more controversial and divisive could be calculated more relativistic ally, whereas more technical, non-controversial debates could be more absolutist. Maybe we could allow each user to see for himself how the scores change as the calculations are weighed more relativistically or absolutely.

A bias indicator

Somewhat related to the above, one problem that we might have is that users would have a tendency to place a higher (or lower) truth value on arguments that are more (or less) important. That is, a user might correlate the truth of a set of arguments with the importance of those arguments. I'm still a little uncertain about this, but I think that it's correct to say that there should not really be any correlation between the truth and importance of a set of arguments: just because an argument is more (or less) critical does not make it more (or less) true, and just because an argument is more (or less) true, does not make it more (or less) crucial to the debate.

As an illustration, let's consider the two following voting patterns:

User 1's voting pattern

Affirmative is more true

Neutral or ambiguous truth value

Negative is more true

Brief Description of Negative's Argument 5

X




















Brief Description of Affirmative's Argument 3





X
















Brief Description of Negative's Argument 2



X


















Brief Description of Affirmative's Argument 1




X

















Brief Description of Affirmative's Argument 4


X





















User 2's voting pattern

Affirmative is more true

Neutral or ambiguous truth value

Negative is more true

Brief Description of Negative's Argument 5

X




















Brief Description of Affirmative's Argument 3


X



















Brief Description of Negative's Argument 2



X


















Brief Description of Affirmative's Argument 1




X

















Brief Description of Affirmative's Argument 4





X


















User 3's voting pattern

Affirmative is more true

Neutral or ambiguous truth value

Negative is more true

Brief Description of Negative's Argument 5
















X





Brief Description of Affirmative's Argument 3

















X




Brief Description of Negative's Argument 2


















X



Brief Description of Affirmative's Argument 1



















X


Brief Description of Affirmative's Argument 4




















X


Consider User 1's voting structure. While the user certainly appears to side with the affirmative arguments, we couldn't really call the user biased. The point of WikiDebate is in providing a process through which objectivity (and learning) can be achieved. It is not a process through which neutrality is imposed. In other words, we cannot presume that just because someone agrees on balance with one side or another that that person is biased. Indeed, if anyone wins the debate, logically there must be at least one user that does not have a completely neutral position.

However, there might be some reason to be skeptical of the honesty or dispassion of User 2, who looks to be affirmatively biased, giving higher truth values to the affirmative side on the more important questions. Since there should be no correlation between truth and importance, User 2's voting pattern is likely to be biased. There are three explanations for User 2's correlation between truth and importance:

  1. That User 2 is indeed being honest and dispassionate, and that his voting choices are due to mere chance of where his opinions lie, a scenario that becomes less likely the higher the correlation between truth and importance.

  2. That User 2 is not being dispassionate. He wants his side to win so much that he tends to be more forgiving of the side he is wedded to the more important the issue.

  3. That User 2 is being dishonest, purposefully gaming the system so that, while his voting may at first appear neutral, he is actually trying to get his side to win the debate – knowing as he does that more important votes are weighed more heavily.

Of course, User 3 seems much more likely to be dishonestly gaming the system. In a relativistic debate, even though User 3's votes lean toward the negative side absolutely, his votes will actually count for the affirmative side since in a relatively structured debate, absolute votes don't count.

Thus, bias indicators will be much more useful in relativistic debates. In absolutist debates, people can game the system simply by voting “the party line” as it were.

Restriction of participants 

Given the peculiarities of a particular debate, it may serve to restrict who may contribute and who may vote. Of course, having a closed policy would largely violate the open spirit of Wiki, but, especially if controversial debates generate destructive behavior, it may prove necessary to limit the group of participants.

Similarly, it may be useful to restrict participation in voting and/or in writing when issues are particularly esoteric – the province of the experts. This is probably not necessary. Because non-experts will refrain from participation to the extent that they feel they can't contribute – forced exclusion shouldn't be necessary. Nevertheless, there is an argument to be made that such a rule would prevent monkey-wrenching. Of course, such restriction would reduce WikiDebate to “neat software used by commissions”, not a new means of developing social dialog.

Linkage of debates and sub-debates. One intriguing possibility is the creation of sub-debates within a given debate, and the possibility of these debates having their own voting structures. My guess, however, is that such a system would prove over-complex. Nevertheless, it does raise the possibility of inserting related debates into the structure of another debate. That is, the truth value of an argument might depend on the truth of another debate statement, the debate over which could be inserted into that argument.

Problems and Tentative Solutions

The following is a list of potential problems that I foresee and some tentative suggestions for solutions.

Kitchen-sink arguments

One problem might be that there would be a tendency to advocate as many arguments as you can. After all, since losing an argument when you're an advocate can't actually hurt your standing in the debate, it might be strategically worthwhile to throw in arguments no matter how stupid they sound. This would lead to overly long debates. Parsimony would be sacrificed and the credibility of the entire debate would be compromised.

One way to fix the problem would be to create an argument insertion penalty. This would be something like the adjusted R-squared in regression analysis, which encourages parsimony by penalizing statisticians who like to throw in every possible explanatory variable. Thus, if a given argument doesn't meet some threshold level of truth value, then its inclusion would actually penalize the advocate's side.

Alternatively, if the argument doesn't meet a certain threshold of truth value after a certain time period, then it could be automatically removed from the debate.

Temporal mismatch between voting and argument evolution

One of the nice things about Wiki implementations is that they are always changing and reacting dynamically to new information. In the context of WikiDebate, however, this creates a problem with the voting system. As the content of arguments change, user's may not be able to keep up.

My grandest hope for WikiDebate is that it may serve as a learning mechanism; if it is to be such, it will be important to enable – and encourage – people's opinions to be changed. But what happens when, as an argument evolves, some ex-voters have become disengaged with the debate and have stopped paying attention. If we archived the debates forever, we may end up with a bunch of stale votes.

One solution to this problem would be to have votes expire after a given argument has been substantially altered. If WikiDebates has users' e-mail addresses, they could be notified and encouraged to vote again.

Another option would be do divide a debate up into periods: a writing period and a voting period. A debate would be proposed and people would write on the debate for a specified period, after which writing would cease. Then the debate would enter the voting period where votes on the now-frozen debate would be collected, the statistics calculated, and the entire debate archived. The debate would then be thawed and participants would again be able to add and edit information until the writing period is over. Then there would be a new voting period, and the cycle would continue.

Forum Shifting

One problem is that there may arise competition between related, but slightly different, debates. As professional pollsters would attest, the answer you get depends largely on the phraseology of the question. Similarly, advantage to one or another side of a debate may depend somewhat on how the debate statement is constructed. If one side sees that the debate statement could be reworded to their advantage, they may create a new debate and try to coax participants over to that forum.

I don't have a very good solution for this potential problem other than to say that each debate could have a “related debates” section that shows the level of participation. Participants would be likely, I think, to choose to involve themselves in the debates with more participants, all other things being equal, since it is these debates that have most exposure, and it is therefore these debates which will allow the participant most influence.

Redundant arguments

One problem would be that advocates may tend to repeat arguments several times in a debate, with perhaps only trivial differences. This would have the effect of counting an argument multiple times to the great and unwarranted advantage of the advocates.

I don't know exactly how to correct this, especially because this would likely often entail a judgement call as to whether a given argument is truly redundant, or whether it is merely repeating previous arguments without substantial new information. I don't know how to fix this problem without simply relying on the integrity of the participants.

Suggestions are welcome.



iNot to over-complicate things, but “absolute” could mean either the neutral value (e.g. 3 on a scale of 1 to 5) or it could mean the average or median value of all users.

eXTReMe Tracker