How does voting, answering and question closure affect new users?

Question

I've recently been poking around the SEDE data for RPi, and I wondered how this site's voting culture influenced other users, and whether they stick around (perhaps related to the other concerns that people have about high-rep users not staying on the site).

In brief, I calculated the percentage of users who continued to contribute (i.e. asked another question after their first) and checked some stats on their first post, such as its score, the number of answers, and so on. This allowed me to compare how different actions might affect user behaviour on the site.

Askers

By far the most important thing to askers is whether they get an answer:

• Askers who got an answer returned 24.5% of the time
• Askers who got no answer returned only 15.4% of the time.

Closure doesn't seem to bother askers much:

• Askers with an open first question returned 22.6% of the time
• Askers with a closed first question returned 21.1% of the time.

Upvoting encourages questioners to keep asking:

• Askers with an upvoted first question returned 24.7% of the time
• Askers with a zero-scored question returned 18.9% of the time

But downvoting... also encourages questioners to keep asking?:

• Askers with a downvoted first question returned 19.9% of the time.

^{Remark: This is not actually as odd as one might have thought. How Community Feedback Shapes User Behavior states that "Not only do authors of negatively-evaluated content contribute more, but also their future posts are of lower quality, and are perceived by the community as such. Moreover, these authors are more likely to subsequently evaluate their fellow users negatively, percolating these effects through the community."}

^{Although the data suggests downvoting might not have the desired effect, there is still value in the signal that it provides to other users that the question isn't useful. In fact, this might be more valuable than the encouragement/discouragement of the asker themselves to help our regulars find good questions to answer.}

Answerers

• 30.5% of users who had their first answer upvoted went on to post another answer
• 20.5% of users who received no votes whatsoever on their first answer went on to post more answers
• 14.9% of users whose first answer was downvoted posted another one.

This isn't terribly surprising: answerers generally hope to get recognition for their work (or at least know that someone saw what they wrote and thought it was useful). It certainly makes sense that users who get no response at all to their answers would go somewhere else instead.

I also tested to see what happens to users who get lots of upvotes for their first answer:

• Users who got more than 3 upvotes on their first answer returned 37.0% of the time to post more answers
• Users who got more than 10 upvotes returned 44.9% of the time
• Users who got more than 20 upvotes returned 59.2% of the time

^{Note: I suspect that these figures may be distorted a little due to my methodology — my query gets the score of a user's first post as it is now, so older posts which have acquired more votes will naturally appear more. I suspect that most of these old, highly voted first posts are from committed beta users, so they would naturally have a higher participation rate. Nevertheless, the figures seem very impressive.}

Discussion

It would seem that there are a few points to discuss here:

• How can we make sure that we encourage new users with good contributions? Can we encourage people to vote more? How?

• Equally, can we take advantage of these figures to discourage poor questions and keep site quality high? It seems that closing questions and allowing them to be answered does little to discourage askers (although the question-ban algorithm is another factor here). Perhaps we need to close questions faster? How can we do that?

Answer 1

This is definitely interesting data, although in some respects I think that I disagree with the conclusions that you've drawn from what you found.

On the side of agreement, I definitely think that the voting culture on this site is an issue. (Thanks for referencing my earlier meta question on the subject!) People who are inclined to participate probably want to get or give help - that's part of it - but it's also a volunteer activity that's got a mechanism to make it a bit fun through the point system. Like any game, it's only fun if the other people play, if you feel that it's at least a little bit fair, and you have some success. There's definitely some support in the numbers that you found for success in the game part of it correlating with continued interaction.

On the other hand, it seems like there may be at least two implicit assumptions that might be faulty, at least as presented in the data section of your question. (In the "Discussion" section, you do seem to acknowledge these points, so maybe it's not disagreement after all?)

1. In a few cases it looks like you're equating correlation with causation.
2. You seem to assume that we want all users to come back.

The biggest instance of both #1 and #2 together appears to be your bullet on down-voting, where you've said "But downvoting... also encourages questioners to keep asking?" [Emphasis and punctuation original.] It seems to me based on anecdotal evidence that it's more common that down-voting fails to deter someone who asked one bad question from continuing to ask bad questions. I fully admit that I did not do the data dive that you've done, and I would yield to further evidence. My gut feeling, however, is that many of the first time posters who got down voted on their first question, returned to get down-voted on their second question as well.

I have less of a feeling for whether this is also true for closure. I think that at least some people who get closed try to refine their same question to get an answer. In theory that could be done through an edit, but I suspect that many of them go instead to a new question. Again, I'd yield to actual data on this either way.

As to your specific discussion points:

• I remain convinced, as I've touched on other metas, that a site like this that's small yet covers a lot of breadth, needs to have more participation from people who hang out in other SE sites but would drop by here from time to time. I still think that part of the answer is for this site to try to act like other SE sites. I'm not sure that there was ever any explicit agreement on this point, although I have notice that some of the more idiosyncratic approaches have died out in practice. I also think that it might be interesting for some of the regular users here to try to recruit from sites like EE, even if it was only trying to pull someone in for specific questions.

• With respect to voting, I don't think that a wide-spread campaign is necessary. If just the top few most active users voted regularly, that might be enough on such a small site. Ten up-votes is 50 (question) or 100 (answer) points, which could be easily had with, well, ten active voters. I think that would be a catalyst for others to also vote more.

• With respect to discouraging bad questions, I think there are limited gains to be had, especially with first time posters. Anyone with a question can post once at no cost and without any understanding of the site. That is the nature of the site and it is common to all SE sites that I've seen. The one possible gain is that I think the help could use an update on what's on and off topic. I don't think that a huge number of people read that, but it's an infrequent update that might "help" section of the site a bit.

• More generally, I think the keys are to close bad posts more quickly, and, again, to down-vote. The advantage to closing more quickly - especially with questions - is that it prevents bad questions from getting positive feedback in the form of an answer. It also has some long-term value in that all SE sites get a certain number of new off-topic questions now because users see previous off-topic questions that were answered and/or not closed. From that they infer that they will also get an answer, and even that they should get an answer because they believe they are on-topic. I personally believe that down-voting does work to some extent, although it's not perfect. If nothing else, it can lead to bad questions getting deleted, which cleans up the site and prevents them from becoming a template for new questions.

• As to how questions can be closed faster: In the short-term we're dependent on the mods to take action in many cases. In the longer-term, it's again a question of voting culture. If the community up-votes on active, high-quality users, they will get the privilege levels necessary to help in this regard. As it stands now, I'm not clear on how many active users we have on this site with sufficient rep to help closing. I do know that I almost never see a question closed by regular (non-mod) voting. If there were more high-rep users, I think that would become the norm (as it is on other SE sites).