How Stack Overflow for Teams solved our knowledge sharing problem
After a successful 60-day POC, Skai (formerly Kenshoo) has signed with Stack Overflow for Teams — the private version of the extremely popular public Q-and-A site — to facilitate our internal engineering knowledge sharing.
In today’s post, I’ll share with you how we got there.
The Problem: so many questions, so little time
We’ve been thinking about the problem of knowledge sharing a lot. Skai is a 15-year old global technology company with multiple products and hundreds of code repositories that keeps growing. Our 300+ engineers are spread across 8 sites, 3 countries, and 2 continents (and recently during the COVID-era, practically 300+ sites). They work collaboratively on some older, shared, monolithic codebases as well as many newer, smaller, team-owned projects. This amounts to a lot of internal knowledge that has to be shared outside one team’s cubicle. Newer teams spend a lot of time trying to figure things out, and veteran teams find themselves repeating the same answers or chasing outdated documents.
Some examples for typical questions:
- While running tests for project X locally, I’m seeing this error, what could be the cause?
- What is Skai’s naming convention for monitoring metrics sent to Graphite?
- How do I delete a “KenshooProcess” (a proprietary legacy abstraction for a scheduled, runnable, abortable procedure)?
- How can I customize the default CI/CD flow (created automatically by Microcosm) for my microservice?
- Which Java HTTP client should I use? Seems like we have 4 different options across our code base.
- Which interface should I extend for a persistence-layer Validator that checks a Keyword bid but needs its parent Campaign’s budget as input?
You can see there are a wide variety of topics and domains to address: best practices, specific errors, usage of specific proprietary components, domain-specific questions, and so on.
Past solutions
Over the years, documentation and knowledge-sharing solutions came and went. From Google Sites to GitHub wikis, we’ve seen partial success and gradual improvements here and there. One class of knowledge, however, has always been the most elusive — and that is the longtail of very specific, “random-access” questions that one faces when writing code or troubleshooting a system. Developers faced with such questions usually went seeking for this “tribal knowledge” among the people sitting nearest to them or those who are unlucky enough to bump into them near the coffee machine.
When physical proximity was no longer relevant (due to global expansion and global pandemics), dedicated Q-and-A channels on Slack became the common solution. But those suffer from poor searchability, low signal-to-noise ratio, and the need to know about these channels to begin with. These caveats not only prevented developers seeking answers from finding them, but also wasted time for the subject matter experts answering questions on a given topic by forcing them to either repeat the answer or painfully search for their own past answers on multiple platforms (e.g.“I remember I answered that, but where was it?”).
In an attempt to identify the root causes of wasted time and inefficient communication, we ended up listing the following requirements for any knowledge-sharing solution:
- Centralized: a single entry point to search for all tech-related questions
- Q-and-A Formatted: all knowledge organized as questions and answers (as opposed to Wikis / Guides / Channels)
- Searchable: tags / keywords can lead me to existing answers
- Accessible: everyone has read and write access
- Moderatable: includes mechanisms to ensure quality (delete bad content, improve readability etc.)
- Updatable: the cost of adding / updating answers is low; low barrier for contributing content
- Code-First: good code formatting
With these requirements in mind, one familiar candidate seemed like a good fit.
Stack Overflow for Teams as a potential solution
Very few developers are not familiar with Stack Overflow (SO). Founded in 2008, it did not exist when I started my engineering career, but I can’t really remember coding without it. Nowadays it seems to be one of the first tools picked up by junior developers and remains synonymous with “troubleshooting code” for juniors and veterans alike. In other words, we all know how to find solutions on Stack Overflow (“search, go to the top answer, skip the text, copy the code”, right?).
Stack Overflow for Teams is a nearly-identical SaaS product for a company’s internal use. The familiarity and proven efficiency of the public SO site made it an appealing candidate. It also checks all the boxes listed above (centralized, Q-and-A formatted etc.).
But we did have some unanswered questions:
- Developers are used to finding answers on SO, but will they easily ask and answer questions? It’s a known fact that only a tiny fraction of SO users create an account and contribute content actively; Will this statistic bear out in a private instance?
- Will a 300-person organization be able to generate the “critical mass” of knowledge necessary to make SO the go-to destination for developers in search of answers?
To answer the first question, we started by gauging the level of engagement our engineers exhibit on the public site. We conducted an internal survey, which produced some promising results:
- ~15% of our engineers have an active account on SO
- Over 50% of those asked a question on SO
- Over 30% of those answered a question on SO
Assuming that posting a question or answer internally is far less intimidating than posting it publicly, these numbers suggest that we can find a large-enough group of champions to lead the charge, and with the confidence and experience that would make everyone comfortable sharing their knowledge.
As for the second question, we saw no alternative to just trying it out. We planned and executed a 60-day proof-of-concept that would give us some idea of whether this works or not.
Proof of Concept: Would our teams really use Stack Overflow for Teams?
With help from the Stack Overflow team, we started by identifying some key topics that produce the highest volume of questions and answers on Slack, as well as the key Subject Matter Experts (SMEs) that commonly answer them. We ended up enlisting ~15 SMEs to lead the charge and become the seed of this new online community.
Next, each SME was asked to find the top 3–5 questions they get asked most often and post them along with the right answer. Within a week, we had over 50 questions and answers and all validated & tagged with the right topics.
Then, we shared the news with the entire engineering organization which encouraged everyone to search for questions on SO instead of asking them on Slack. If the search came back empty, we gave them short instructions on “how to ask”. At this phase, SMEs made sure to be extra attentive to new questions on topics they know about so that every new question prompted a quick enough answer to give our newcomers a positive experience that would encourage them to come back next time. Many developers were easy to convince. They were excited to have this familiar, reliable tool available for their internal needs and were happy to share their knowledge.
A few other tricks and tools were used to help everyone adopt new habits, which began replacing Slack or email with SO:
- SO <-> Slack integration This integration provides nice tooling within Slack, which allows any user to respond to a question posted on Slack with “I suggest you ask this on SO” including a link to a draft question based on the original Slack message. This makes SO visible exactly where it’s needed the most
- Some healthy gamification and competitions. We set up 2-week competitions with a small prize attached to them for the person who asked the most questions, posted the most answers, or gained the highest SO reputation. This created some positive noise and gave recognition to early adopters
- Constant reminders from managers, SMEs and early adopters. “Ask me on SO and you’ll know ;)” was a slightly-annoying, yet effective response, I personally used a few times during the POC (I still do)
The Results
By the end of the 60 days POC, we saw some promising results:
- Over 350 questions posted and answered
- Over 50% of those were “organic”, i.e. not questions posted by SMEs or posted and answered by the same person just to create content, but actual questions with actual answers that unblocked a developer in need
- Median time to answer: 1:20 hours
- An average of ~60 unique daily visitors (about one in 5 developers)
- Over 200 daily searches
Out of all these numbers, the number of daily searches stood out as a healthy sign of people changing their behavior and coming to Stack Overflow for answers. Anecdotal evidence showed that even at this early stage, many actually found what they were looking for and therefore gained value without actively participating. Even the SMEs, who had to put in some extra effort up front, reported noticeable time savings, specifically when a “hot topic” had a good SO answer they could just point people to instead of repeating the answer on Slack.
Future Plans
It’s been a few months since the POC concluded, and content keeps growing nicely in our community. KPIs such as the number of searches and number of unique daily visitors are on the rise:
There’s still work to be done around backfilling older knowledge less actively used as well as growing the habit of searching on SO before asking on Slack, but these seem to be progressing organically.
On top of that, we’re dipping our toes into some other parts of the offering including Stack Overflow for Teams Articles, which can centrally host more long-form content that’s currently hosted on several different GitHub repositories and Google Drive documents.
Another enhancement we may try in the future is expanding beyond our core engineering organization and make SO available to Product, Client Success, Solutions, and even Sales teams who could greatly benefit from this form of Q-and-A platform for less technical (and yet highly specialized) knowledge about our products themselves. This expansion seems more and more attractive as the current usage keeps growing and proving its value.
