1

I'm putting this discussion topic forward, to get consensus on this issue.

If we examine the highly progressed proposals, we see that in most of them, the participants are gaming the upvoting example questions, as to quickly reach the required number of questions with 10 upvotes, rather than 'voting honestly'.

For example:

  • Buddhism (178 followers, in commitment phase) - 4 questions with > 12 upvotes.

    Game Theory (221 followers). - 4 with >12 upvotes, one has 18.

    Moderators (147 followers) 3 with > 12 upvotes.

    Wikis (135 followers) 1 with >12.

    System Z (113 followers) 4 with >12.

    Economics (131 followers) 0 with >12.

    Coffee (135 followers) 4 with >12. One has 21.

Compare this to some of the older launched sites:

  • Academia (149 followers) 9 with >12. Four >= 20.

    Workplace (134 followers) 6 with >12. 3 with >= 20.

    UX (191 followers) 9 with >12 5 with >= 20.

There are a number of discussion threads that suggest that users are actively gaming:

Newcomers: Please upvote example questions

https://area51.meta.stackexchange.com/questions/10283/how-can-we-get-our-last-3-questions-to-10?lq=1

Don't overvote!

https://area51.meta.stackexchange.com/questions/11213/voting-on-questions-with-lower-than-10-votes?rq=1

Pareto optimality of the distribution of upvotes

We need more up-voted questions and re-distributed votes

So, the question isn't about whether this is a good thing or a bad thing, the question is:

Do you agree that a significant chunk of proposal followers are voting as to maximise example questions with 10 upvotes, rather than 'voting honestly'?

6
  • 2
    There are 26,000 users register on A51. So no, I don't think a "significant chunk" of those users are doing this. It only requires a very small set of users to game the votes like that. What point about this is there to be made beyond what has already been discussed? It's already acknowledged that it's a problem, what are you hoping to achieve from your question?
    – JohnB
    Jun 10, 2014 at 4:01
  • @JohnB - Where is it acknowledged? The purpose of this discussion, is to acknowledge it.
    – dwjohnston
    Jun 10, 2014 at 5:13
  • @JohnB - Take a look at the coffee proposal. If you assume that 1) Gamers aren't going to vote on proposals with low vote counts. 2) They won't vote on proposals with with more than 10. Then I count ~50 votes that are 'non-gamey' - that's about 10 users. Compared to the 390 votes that make up 39 questions with scores 10 and up.
    – dwjohnston
    Jun 10, 2014 at 5:22
  • @dwjohnston There are also problems with people or gangs who are gaming the system by downvoting questions of proposals systemtically below 10 to prevent them from succeeding. Both attitudes, what you discribe and troll downvoting, are problems related to the specific metrics for success defined on Area51, and they should therefore be adressed at the same time. I would even say that the troll downvoting is worse because the number of upvotes is limited whereas the number of downvotes is not, and all suggestions to limit the number of downvotes too are rejected by SE.
    – Dilaton
    Jun 10, 2014 at 10:52
  • 4
    @Dilaton - I agree. Downvoting example questions to block a proposal, is also gaming.
    – dwjohnston
    Jun 10, 2014 at 11:07
  • It has been acknowledged here
    – JohnB
    Jun 10, 2014 at 13:39

2 Answers 2

6

If we examine the highly progressed proposals, we see that in most of them, the participants are gaming the upvoting example questions, as to quickly reach the required number of questions with 10 upvotes, rather than 'voting honestly'.

I think there's a disconnect between any idea of what it means for individuals to vote "honestly" and the reality of being limited to only 5 upvotes on any one proposal. Just one more example of the tension that exists on Stack Exchange around gamification - it's incorporated explicitly and by design as a way of improving the quality of the site, but for it to be beneficial, users should only game the system in permitted ways. That's ambitious as hell, and it takes a lot of tweaking.

What you noticed is not few bad apples (or even a lot of them) gaming the system; it's a community phenomenon, a direct result of being limited to five equally-weighted votes apiece. In other words, it's a consequence of the way the process is designed rather than a concerted attempt by individuals to circumvent implied rules and "get away with" something.

The discussion threads pointed to in the OP to are just the tip of the iceberg* in that they're visible expressions of a larger unwritten rule evident in the voting patterns. You can choose to interpret this unwritten rule as something like "don't rock the boat/let's get this party started" but I would express it more simply as "achieve consensus" - the very sentiment you expressed in starting this discussion, and a goal for which voting is a uniquely efficient tool.

Whatever subset of the A51 community (itself a subset of the SE community) becomes involved in a proposal determines collectively how questions are rated. I think any reference to the "best" questions is a red herring in that, individually, we have no way to distinguish "good" from "better" from "best" when given 5 equally-weighted votes. We're not even rating things that would be complete questions in an actual beta; rather, we're faced with a sea of hypothetical post titles.

Given that circumstance, the score is less an objective measure of quality than an empirical measure of consensus approval and/or popularity. That's true to some extent on all the network sites, but out on a "real" network site scores are also influenced by the actual problem being solved, and is it visible on search engines, and is it helpful to many people, and so on. Here, we're strictly focused on the consensus and popularity bits. And once a question reaches +10 net score, the rules say that it's "good enough" - consensus has been achieved, let's move on to the next agenda item. As long as upvotes are a limited resource, spending them on a question that appears "settled" is counter-intuitive.

Let's take a look at some data. I'm following the Engineering proposal, which has 69 example questions. Here's the current score distribution:

Net Score | Count
----------+---------------------------
+11       | ████ 4
+10       | █████████████████████████ 25
+9        | 0
+8        | ██ 2
+7        | █ 1
+6        | 0
+5        | 0
+4        | 0
+3        | ███ 3
+2        | █ 1
+1        | █████████ 9
+0        | ███████████████████████ 23
-1        | █ 1

You can see that most questions have either "arrived" at +10 or are sitting at +0. The +7/8 questions are surely on their way to +10; they'll be sorted immediately below the "arrived" questions by score, so any user who wants to use their votes solely to advance the proposal will be upvoting them automatically. The fate of the +3s is less easy to divine, but at least some of them will probably be elevated by that crowd to consensus status as well.

So how else are people voting, if not "honestly?" Well, there's a decided lack of middle-ground in this distribution, but that's not what I'm noticing. Here's a hint: that single -1 question was at +0 before I went through the list to tally up the scores. In general, users viewing this proposal just aren't downvoting, despite having unlimited freedom to do so. There are plenty of potential reasons, starting with:

  • Most users don't bother looking at proposals they aren't interested in;
  • Users who follow a proposal want it to succeed, and have no short-term interest in downvoting example questions;
  • Some users will follow, upvote x5, and then not return until commitment/beta/launch;
  • Engineering isn't as likely to attract drive-by downvotes as a more polarizing topic;
  • Users don't see a point in downvoting questions before they've received attention, unless they're truly awful enough to "deserve" a negative score;

...and so on. Come on, baby, speculate with me! Or, here's a fun idea - let's compare the above distribution to that of the Gamification proposal (quick, what do you expect - will Gamification followers show more of a tendency to "game" the 10-vote rule, or less?):

Net Score | Count
----------+---------------------------
(more)    | █1
+20       | █1
+19       | 
+18       | 
+17       | █1
+16       | ██2
+15       | 0
+14       | 0
+13       | 0
+12       | ████4
+11       | ███3
+10       | ███████████████████████████27
+9        | 0
+8        | 0
+7        | 0
+6        | ███3
+5        | ████4
+4        | 0
+3        | 0
+2        | █1
+1        | ███3
+0        | ██████6
-1        | ███3
-2        | █1
-3        | █1
(less)    | █1

Keeping in mind the difference in overall progress between these two proposals (Gamification is now in the commitment phase, so voting is done for now), I would say we're looking at essentially the same trend, albeit with the addition of outliers on either side. I expect that the closer this proposal got to satisfying its definition phase, the more free users felt to vote up those runaway questions with 16+. Or maybe all that's required is enough of a popularity signal to send one or two questions high enough to entice users to spend a precious upvote or two on the luxury of confirming their own good taste. Without being able to see the voting timeline (or individual vote counts, for that matter), I'm still speculating.

Let's hit one more, because Gosh We Love Histograms. If Gamification might be expected to attract punks and haxxors, surely Moderators (which is even farther along than Gamification, awaiting beta launch) is full of only the most upstanding, conscientious and "honest" voters - right?

Net Score | Count
----------+---------------------------
(more)    | 0
+20       | █1
+19       | █1
+18       | 0
+17       | █1
+16       | 0
+15       | 0
+14       | 0
+13       | 0
+12       | 0
+11       | ██2
+10       | ███████████████████████████████████35
+9        | 0
+8        | 0
+7        | 0
+6        | 0
+5        | 0
+4        | █1
+3        | ██2
+2        | ██2
+1        | █1
+0        | █████5
-1        | █1
-2        | 0
-3        | 0
(more)    | █1

Oh myyy.

Now, keeping in mind that none of these charts account for the likelihood that some example questions are going to be pretty great or pretty terrible,** and some proposals probably attract better or worse questions than others... This seems to back up your assertion pretty solidly.

Or at least, half of it, because there's that other bit about "honest" voting that I think is misplaced. If this pattern were truly "dishonest" then I would be really surprised to see these voting patterns emphasized rather than moderated (no pun intended, for once) in the Moderation proposal. For that matter, I would argue that the rarity or absence of downvotes on a proposal is far more "dishonest" than a user recognizing that they have a limited number of upvotes to "spend" and wanting to get the most bang for their buck. And that includes net downvotes - it's well known (I don't have time to dig up a reference, so feel free to challenge me on this) that the first vote cast carries an exceptional amount of weight in determining the ultimate outcome of a popularity contest.

So yeah, people are doing this. Is it a problem? I don't see how. Should something be changed? Well, I think it's not particularly balanced to have unlimited downvotes and limited upvotes, but if people aren't that interested in using those downvotes this is probably an ain't broke/don't fix kind of thing. If having a more "honest" evaluation of which questions are the "best" is really that important, then upvotes should probably be unlimited and the condition for moving from definition to commitment phase revised appropriately, but it looks to me like you don't have a lot of support on that count.

*I actually prefer an analogy here involving the fruiting bodies of large underground fungi but I think that would carry a more negative implication than I'd intend, given their association with things like infection and decay.

4
  • I like most of your answer but you have drastically understated the down votes here I discuss a proposal with over 200 down votes. Jul 25, 2014 at 23:23
  • 1
    @JamesJenkins Downvote gaming is tangentially related form of gaming, where people downvote definition questions because they disagree with the proposal.
    – dwjohnston
    Jul 26, 2014 at 0:28
  • 1
    @JamesJenkins I did mention in passing that more-polarizing topics are likely to attract much more active downvoting. Anything overtly political or religious certainly qualifies. But at the same time, it's true that I'm counting only net downvotes, because I lack the rep to view true vote counts, and that biases my analysis.
    – Air
    Jul 26, 2014 at 18:58
  • Still, I lean toward viewing those exceptions as proving the rule - clearly people can downvote when they want to.
    – Air
    Jul 26, 2014 at 19:08
4

Area 51 has devolved to the point where gaming is a requirement to move forward. If everyone voted without gaming no site would ever move out of definition phase. Most proposals would have maybe 10 questions with A LOT of votes and hundred more with less than 10 votes, a year would pass and the proposal would be closed.

Looking at sites that have been created it takes, maybe 160 question to have 40 naturally get 10 votes, with more visitors per day that an area 51 proposal gets in the first two phases; in it's current form, that is just not going to happen at Area 51.

But we have criteria for leaving definition phase 60 followers and 40 questions with 10 or more votes, the problem with setting criteria is the same as setting goals, people will strive to reach them, then stop. The only thing worse than have a goal is not having one...

A Next Generation of Area 51 has been in the works for sometime. I don't think anyone is 'happy' with Area 51 in it's current form, but understandably available resources are focused on the next generation of site development sometimes referred to as Area 52.

At the same time resource go to support beta sites like Ebook which probably should not have made it to beta (hindsight being 20/20 and all).

So what can we do?

We do the best we can with what we have. We hope that the next gen site, will be up and running soon. We try to make choices based on what we think will create a healthy site, and not let our desire to have a particular site move forward or back, influence our voting.

A couple days ago Robert Cartaino jokingly suggested a change to "raising the number of questions needed", after a bit of reflections I thought that might actually be a good idea. but... Area 51 is in the shape it is in because the community/enterprise has made one little fix after the other, on a system that just needs a complete do over.

I (think I) understand why gaming occurs as outlined above, and I believe most of us with some experience know why it is not for the best. I believe without doubt that if you need to post a message telling a groups of supporters how to vote, your site proposal is not doing well. If you can't get 80 people to figure it out on their own, then you don't have the group needed to make the site work.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .