Case study: improving the question closing experience at Stack Overflow
You’re stumped on a coding problem. You’ve tried everything: scoured the internet, pinged your colleagues for help, and troubleshot for hours. Finally, you decide to ask your question on Stack Overflow — surely someone in this community of 50 million programmers can help you, right? So you publish a question and wait hopefully for a response.
Within minutes, however, your question is closed by community members and marked with a large yellow banner, letting you and others know that there’s something wrong with the question. Closing also prevents your question from being answered by others.
About 20% of questions posted on Stack Overflow are closed every day. As you might expect, this is an oft-cited negative experience for users. Not only does it prevent them from getting the help they need, but it contributes to the perception that Stack Overflow is exclusive and deters people from participating further in the community.
So why does question closing exist if it causes distress to users? Why not get rid of it entirely and let people ask what they’d like?
Closing questions is an important part of content curation. It helps ensure a high standard for content quality on the site. This is critical for attracting and retaining people willing to answer questions. People answering questions need to be able to generally expect that the questions they see are well-written, easy to understand, and non-duplicative content. Therefore, they expect people to abide by specific rules when asking questions on Stack Overflow. Questions that don’t abide by those rules are often closed.
How can we ensure that people asking questions get the help they need, while also empowering those curating the site to maintain a high bar for content quality? Earlier this year, the Community team at Stack Overflow set out to answer this question. Here’s my take on our approach.
We set out with the following UX goals:
- To make the question closing experience friendlier and more useful for question askers; and
- To make it easier to edit and reopen closed questions.
We planned to measure progress toward these goals by tracking the question editing rate and mentions of “overmoderation” in monthly site satisfaction metrics.
We started out by conducting an audit of the current question closing experience, mapping out workflows and their underlying logic. This was important because the system contained intricacies not fully documented or understood by the team. We needed a detailed view of the system in order to make useful changes without breaking the parts that worked.
In addition to the audit, we dug into qualitative feedback archives, as well as question closing and reopening stats, to learn about the workflow’s current performance, where it worked, and where it didn’t.
We found that the question editing and reopening rate was very low, even though this was the recommended course of action for users whose questions were closed. We attributed this to a fragmented workflow with dense, confusing guidance. Question askers had to expend significant effort to understand what they could do about their closed question. It was not clear that reopening their question was a viable option.
We came to our design solution through several rounds of iteration and user testing. Below are the core changes we’re making.
When a question is closed, it’s hidden from the public. One of the primary reasons that getting a question closed is such a negative experience is because it’s embarrassing. Allowing this to take place in public increases stress for the question asker during an already stressful time.
Under the new experience, a closed question is hidden from the homepage and other public locations. Not only does this give the question asker time to parse feedback privately, but it also reduces noise for people seeking to answer questions. The homepage and other locations where people look for questions to answer are no longer cluttered by closed questions that can’t be answered.
Automated guidance is built directly into the workflow. Right now, there isn’t much readily available help for people who get their questions closed. If they want to find out more about what this status means and what they can do about it, they’ll need to dig through dense pages of documentation and user forums, or rely on the goodwill of a community member to guide them.
The new experience provides built-in guidance at key touchpoints: on the closed question, by email notification, and during the editing process. Users don’t need to leave their workflow in order to learn about how to address their closed question.
Automated guidance also reduces opportunities for user-to-user friction. Instead of two users potentially clashing due to frustration or miscommunication, there is clear, neutral guidance offered by the product. This supplements or replaces ad hoc guidance from community members, allowing them to redirect energy to answering questions or performing other curation tasks.
A new editor with detailed guidance for closed questions. Asking or editing a question properly on Stack Overflow requires knowledge and skill, but the previous editor offered no guidance and a cluttered, confusing interface. People hoping to edit their closed question would need to do their own research.
The new question editor offers a simplified interface, detailed guidance about how to edit a closed question properly, and tips for how to write a question that is likely to get answers.
Sufficiently edited questions are automatically reopened. The previous experience didn’t offer an explicit path to reopening a question. A small chunk of edited questions are manually reopened, but the rest languish unseen, even after being edited. Even if a question does get reopened, it is often left unanswered because by then, it’s no longer “new.” New questions are ranked at the top of question lists and are most likely to be seen by people looking for questions to answer.
Under the new experience, questions are automatically reopened once they’ve been sufficiently edited. If the system detects potential abuse, the question is kicked back to the manual reopening process. Reopened questions are also bumped to the top of question lists and are therefore more likely to be answered.
What do users think so far?
We tested this solution with two user segments: less experienced users who are likely to ask questions, and more experienced users who are likely to curate or answer questions.
Overall, we found that both segments saw clear benefits in these changes. This would help question askers get the answers they needed while reducing noise and workload for curators and answerers.
“It would be helpful to not feel like I have to write so much… guiding [question askers] myself. I believe most people will welcome this.” — Content curator
“A closed question becoming hidden is a good idea, gives you the opportunity to fix things offsite. It can be a bit embarrassing.”
Next steps and reflection
The new question closing experience is rolling out incrementally through a combination of A/B tests and phased releases. We’ll be closely monitoring performance and gathering qualitative feedback at key milestones.
Beyond these improvements, I hope that we can continue to invest in this experience by simplifying rules about posting a question and making our guidance smarter.
🙏Thanks for reading! If you have your own stories about building product for online communities, I’d love to hear. Please leave a comment or email me at email@example.com.