I've recently been poking around the SEDE data for RPi, and I wondered how this site's voting culture influenced other users, and whether they stick around (perhaps related to the other concerns that people have about high-rep users not staying on the site).
In brief, I calculated the percentage of users who continued to contribute (i.e. asked another question after their first) and checked some stats on their first post, such as its score, the number of answers, and so on. This allowed me to compare how different actions might affect user behaviour on the site.
By far the most important thing to askers is whether they get an answer:
- Askers who got an answer returned 24.5% of the time
- Askers who got no answer returned only 15.4% of the time.
Closure doesn't seem to bother askers much:
- Askers with an open first question returned 22.6% of the time
- Askers with a closed first question returned 21.1% of the time.
Upvoting encourages questioners to keep asking:
- Askers with an upvoted first question returned 24.7% of the time
- Askers with a zero-scored question returned 18.9% of the time
But downvoting... also encourages questioners to keep asking?:
- Askers with a downvoted first question returned 19.9% of the time.
Remark: This is not actually as odd as one might have thought. How Community Feedback Shapes User Behavior states that "Not only do authors of negatively-evaluated content contribute more, but also their future posts are of lower quality, and are perceived by the community as such. Moreover, these authors are more likely to subsequently evaluate their fellow users negatively, percolating these effects through the community."
Although the data suggests downvoting might not have the desired effect, there is still value in the signal that it provides to other users that the question isn't useful. In fact, this might be more valuable than the encouragement/discouragement of the asker themselves to help our regulars find good questions to answer.
- 30.5% of users who had their first answer upvoted went on to post another answer
- 20.5% of users who received no votes whatsoever on their first answer went on to post more answers
- 14.9% of users whose first answer was downvoted posted another one.
This isn't terribly surprising: answerers generally hope to get recognition for their work (or at least know that someone saw what they wrote and thought it was useful). It certainly makes sense that users who get no response at all to their answers would go somewhere else instead.
I also tested to see what happens to users who get lots of upvotes for their first answer:
- Users who got more than 3 upvotes on their first answer returned 37.0% of the time to post more answers
- Users who got more than 10 upvotes returned 44.9% of the time
- Users who got more than 20 upvotes returned 59.2% of the time
Note: I suspect that these figures may be distorted a little due to my methodology — my query gets the score of a user's first post as it is now, so older posts which have acquired more votes will naturally appear more. I suspect that most of these old, highly voted first posts are from committed beta users, so they would naturally have a higher participation rate. Nevertheless, the figures seem very impressive.
It would seem that there are a few points to discuss here:
How can we make sure that we encourage new users with good contributions? Can we encourage people to vote more? How?
Equally, can we take advantage of these figures to discourage poor questions and keep site quality high? It seems that closing questions and allowing them to be answered does little to discourage askers (although the question-ban algorithm is another factor here). Perhaps we need to close questions faster? How can we do that?