This article first gives a background about the thriving Developer Relationship (DevRel), then discusses vanity and potential North Star metrics for DevRel practices.
Background
As the software industry evolves, enterprises face increasing complexity in building software systems. A growing need for specialization across various system aspects and layers compounds this complexity. Consequently, many companies have shifted from producing all their software in-house to relying on diverse technology products to fulfill their software needs.
Apart from implementing the core business logic in-house, software platforms and services that support the business logic can and should be procured. Even the development of the business logic itself can be accelerated through purchased development tools and platforms. Examples of the former include traditional commercial software and cloud services, while the latter include Copilot and Retool.
In this trend, developers play a crucial role in the decision-making process of purchasing software products within companies. They not only influence the development of technology but also serve as users and creators of software products. As a result, the developer economy has flourished, and developers themselves have become significant market customers, leading to the emergence of a series of initiatives aimed at developers within companies. This is the background of the development of Developer Relations (DevRel).
Detailed discussions of developer relations are beyond the scope of this article. For more information, refer to Developer Relations: How to Build and Grow a Successful Developer Program.
Vanity Metrics
Before diving into a discussion on possible metrics, let’s address a common pitfall: vanity metrics.
The concept of vanity metrics was introduced in “The Lean Startup,” referring to indicators that reflect superficial data. These metrics often present impressively large figures that seem compelling at a glance but fail to convey the specific value they bring to a business.
Typical examples of vanity metrics include clicks and download counts. In the context of today’s thriving open-source movement, the number of stars on a GitHub repository is another vanity metric.
The fundamental issue with these metrics is their lack of meaningful information.
For instance, in attempts to inflate star counts, we’ve repeatedly seen operations personnel enticing developers at various events to click the star button in exchange for small gifts: a street marketing approach.
Similarly, the value of download counts is questionable, as I’m well aware of the noise automation pipelines can introduce, causing DevRel teams to be incapable of extracting any useful insight from data reflecting tens of thousands to hundreds of thousands of monthly downloads.
The insufficiency of information stems from the simplicity or low cost of the actions. Anyone, not necessarily a developer, might click the star button for a small gift. The same goes for undifferentiated page views and download counts; they offer little guidance for developing DevRel initiatives.
The metric of stars is less meaningful. Its imaginable value is no more than advertising comparisons to create a false impression of superiority. However, a more detailed examination can enhance the analysis of page views and downloads.
For page views, tools like Google Analytics can analyze the origins of clicks, referral sites, and the bounce rates of individual pages. More advanced tools like ReadMe offer comprehensive user journey analytics, even integrating API page interactions and feedback. Beyond numerical indicators, interactive feedback widgets, as seen on official sites like Vercel and GitHub, particularly in documentation, aim to optimize content organization and improve user experience.
For downloads, enriching the information involves distinguishing the sources and purposes of downloads, specifically:
- What artifacts exactly were downloaded?
- From which regions or companies did the downloads originate?
- Were the downloads manual or through automation pipelines?
SCARF attempts to offer analytics based on download data, but this approach is still exploratory and unproven in effectiveness. Additionally, enhancing download data through SCARF requires directing users to download artifacts via a SCARF-provided gateway, which is not easily achieved.
The enriched data collection capabilities of Google Analytics rely on embedding UTM parameters in URLs. Pursuing this direction raises various challenges requiring user cooperation, such as refusal to provide information, data falsification, and even potential conflicts with regional information security regulations.
Brand Volume
While the latter half of the previous section discussed ways to derive some value from “vanity metrics,” these simple behavioral metrics are still unsuitable as North Star metrics for DevRel. A North Star metric, also known as a singular key metric, should guide the overall direction of DevRel efforts.
DevRel is a broad term that, to some extent, serves as a catch-all for activities aimed at developers that have not yet been differentiated by specialization. It encompasses various functions, including developer marketing, evangelism, technical advocacy, support, developer experience, training, developer success, community management, and project management.
A significant portion of this work is related to marketing efforts. For instance, software services or development tools launched by companies must be recognized and adopted by developers. This necessitates generating sufficient market conversation around these tech products, leading to the metric of brand volume.
One challenge with the brand volume metric is its lack of a standardized definition. Defining it merely in terms of clicks or views falls into the trap of vanity metrics and can lead to distorted operational actions.
A straightforward example of a brand volume data source is the Google Trends index, but in modern social media, relying solely on Google Trends can be misleading, especially for startup projects where developers are unlikely to discover them through Google.
Some niche areas have established definitions of brand volume, like DB-Engines ranking in the database domain, which clearly outlines its scoring factors and provides rankings within subdomains.
Importantly, DB-Engines ranking is used as a reference for user comparisons within the database domain. For an emerging database product, identifying its niche and main competitors and setting goals to outperform certain competitors or achieve a top ranking within a specific timeframe can be strategic.
The scoring factors can be customized and used for competitive analysis in fields without well-known brand volume metrics. Among these, I find Twitter mentions or hashtag references to be the most valuable channels for driving other efforts, as they correlate to specific individual actions, potentially leading to impactful reactions.
Beyond horizontal competitor comparisons, brand volume metrics can also be used for vertical comparisons within specific domains.
Community tools like Orbit and CommonRoom can collect behavior data in specific domains related to the communities. With this data, custom brand volume formulas can be created for vertical comparisons in the timeline. Each behavior underlying these metrics is traceable in this model, allowing for DevRel efforts to be tailored based on changes in aggregate metrics.
Two reminders: (1) The goal of measuring brand volume isn’t limited to the software itself. Often, brand volume assessments need to be linked with marketing objects, especially the main topics being promoted for the year. (2) When making horizontal comparisons, the focus shouldn’t only be on competing products; further segmenting fields and platforms can lead to a more accurate assessment of efforts.
Community Members
DevRel is inherently developer-focused, with its core methodology aimed at enabling developers to succeed with software products, thereby ensuring the products' success. Given this developer-centric approach, the work naturally adopts a people-oriented philosophy. Consequently, once a product’s community is defined, the size of its community membership serves as an excellent North Star metric.
Apache SkyWalking’s creator, Sheng Wu, used this metric a few years ago. He presented the number of SkyWalking contributors instead of stars, for describing the project’s health and development progress at various events.
However, for DevRel efforts within companies, contributors to the open-source software might not be the focal point. This is largely because the major code of the product is not open-sourced, or if it is, the major development work is likely carried out by employees. Implementing an open-source collaboration model on top of a corporate software engineering framework is not only challenging but also of questionable value in terms of its periodic contribution to commercial success.
This is not to say that commercial open-source companies should entirely abandon the collaboration model. On the contrary, a well-functioning collaboration model and ecosystem development can be significant leverage for a software product's technological growth and user-driven expansion over the long term.
However, it is unsuitable as a North Star metric for DevRel efforts. To be honest, prioritizing code contributor numbers or engagement levels as the North Star metric can lead to counterproductive operations and friction with the corporate software engineering model, ultimately harming both.
For DevRel efforts within companies, the North Star metric should be the number of community members, encompassing a broader spectrum of participants in related communities. The reference model for this can still draw inspiration from community tools like Orbit and CommonRoom.
- Slack
- Discord
- Discourse
- Stack Overflow
- YouTube
- X (Twitter)
- GitHub
This does not mean a community effort must cover all the above channels. For a product just initiating its DevRel efforts, achieving breakthroughs on a few key channels before spreading experiences and content to others is a healthier approach, especially when scaling the DevRel team to manage the increased workload from more channels.
Key channels involve choosing one instant messaging tool and one forum. Operating on X (Twitter) is essential, while LinkedIn and Stack Overflow may be covered depending on the situation. The choice of channels for announcements should be based on where the target developers primarily get their information and the team’s content production capacity.
Different types of channels attract different participants. For example, Slack and Discord involve members who join, Discourse involves registered members, and various social platforms involve followers and those who engage in discussions or content creation. When defining the North Star metric, one approach is to exclude trivial activities initially to calculate a general total community size, then examine specific behaviors of individuals from various channels to define what constitutes an active community member.
Beyond simply summing the number of community members across all channels, refining the North Star metric involves identifying those community members who can concretely drive the company’s commercial success. This leads to the next section: DevRel Qualified Leads.
DevRel Qualified Leads
DevRel Qualified Leads (DQLs) was possibly introduced by DevRel Qualified Leads: Repurposing A Common Business Metric To Prove Value. The discussion below refers a lot to this blog.
First, it’s important to clarify that the number of community members is usually quite small when a community is just starting. If you use DQL metrics at this stage, you’ll encounter problems similar to those when using Code Contributors as a metric.
This isn’t to say DQLs are not valuable at this point, but in the early stages of a community, such individuals are rare and often come about through serendipity or specific circumstances rather than as a scalable result of a particular strategy.
Setting DQL as the North Star metric when the community is small might be unachievable. Even under successful DevRel efforts, only a handful of DQLs might emerge over several quarters, and their appearance is often irregular. Using it as a North Star metric could disrupt the direction of DevRel efforts. Only once the community has reached a certain size can DQLs be generated through specific content strategies and operational actions.
However, even when the community is small, DQLs can serve as an auxiliary metric to guide which individuals to focus on during community growth and what follow-up actions to implement.
Let’s clarify what DevRel Qualified Leads (DQLs) mean.
The concept of Leads is widely understood in the marketing domain. In marketing activities, target individuals are often invited to fill out a form or register their information. Once you register your information in such contexts, especially with specific expectations or feedback, you become a Marketing Qualified Lead (MQL) for that company. In other words, the marketing team creates engaging content to attract potential customers to register their information. They then verify this information to ensure these potential customers meet the company’s standards or expectations before passing them on to the sales team for future contact. The marketing team thereby completes its role in filling the sales pipeline. Their responsibility doesn’t include ensuring these leads become customers; that’s the sales team’s job.
Translating this concept to DevRel teams.
This doesn’t imply that the DevRel team should have sales targets. As mentioned, the core of DevRel efforts is to facilitate developer success and foster genuine relationships. Linking this directly to sales metrics could immediately complicate DevRel efforts, as marketing teams already focus on filling the sales pipeline, and introducing monetary relationships directly into developer relations is unsustainable. In past years, some companies have absurdly expected to exchange small gifts for developers’ technical preferences or significant time investments in the software, but all of them failed.
In the DQL model, Leads are individuals who can contribute to the company in some way, not necessarily as potential customers.
Returning to the main methodology of DevRel: fostering developer success with your products, thereby ensuring the product’s success. For companies to benefit from this model, someone needs to think from the developers’ perspective, understanding what constitutes developer success and how to mobilize the company’s various departments towards this goal. Typically, no department within the current corporate structure considers issues this way.
This presents an opportunity for DQLs: The DevRel team thinks from the developers’ perspective and informs other departments on collaborating with developers in the community, ultimately benefiting the company “in some way.”
Here are a few specific examples:
Marketing
Common DQLs for the marketing department are case studies or user-generated content.
Huawei Cloud was an early adopter and collaborator of Kafka on Pulsar (KoP), a technology product by StreamNative that is compatible with the Apache Kafka API and built on the Apache Pulsar system.
After successfully utilizing KoP to meet Huawei’s needs, StreamNative and developers from Huawei Cloud co-authored and published the blog post From Kafka to Pulsar: Creating A Comprehensive Middleware Platform to Power HUAWEI Mobile Services. This post demonstrated to developers considering KoP technology that it is indeed viable, making a direct contribution to the product’s market marketing.
Similarly, after successful launches with Flipkart and Discord, StreamNative also released specialized shares, proving the viability of the company’s technology products in the e-commerce domain and online machine-learning tech domain (Discord’s scenario is generalized online machine-learning, not about its core business logic).
Without deep communication with developers, these user stories might have been reduced to a simple testimonial, “We used this product, and it was good,” without detailing the specific technical challenges and application scenarios. It would struggle to convince developers of the technology product.
Product
Common DQLs for the product department include user feedback and beta testing.
TiDB’s user forum, AskTUG, receives many pieces of feedback on TiDB-related products daily. While much feedback concerns incorrect usage, the forum has become a volunteer firewall for TiDB support after developing a well-functioning moderator program. Moreover, some product feedback does highlight design flaws or optimization opportunities in the product itself. Many TiDB-related products benefit from issues raised in the user forum.
PingCAP’s community team established a moderator program, making AskTUG a self-service problem-solving forum without significantly occupying the core development team’s time to answer user questions. This is the premise for users to provide sufficient feedback. Otherwise, if a forum lacks responses, it won’t engage users, let alone gather valuable product feedback. Further, the team can build reports of user feedback issues and lists of problems for the product department to solve, encouraging developers to participate in improving the product.
Of course, PingCAP’s community team also guides developers in participating in beta testing, such as the TiDB v7.1.0 Honorary Experience Officer activity.
While there are also gifts, the prizes offered by PingCAP for such activities are much more substantial than those typical for star-exchange small gifts. Correspondingly, the work required of these “Honorary Experience Officers” is more complex than merely clicking the Star button, but guidance and technical support are available.
PingCAP does not expect developers who participate in beta testing to choose TiDB for their production environment afterward necessarily, nor does it expect them to invest additional time beyond what the test guide requires. Hence, the prizes and the outcomes exchanged are corresponding. In this process, developers spend time understanding products like TiDB, which, from a long-term perspective, makes it possible for them to invest more time and at least have a comprehensive understanding of TiDB when making decisions.
Engineering
Common DQLs for the engineering department include bug reports and problem fixes.
If it’s open-source software, contributors might directly resolve the issue. Otherwise, at least users can provide the environment information and steps to reproduce the issue, which are important inputs for the engineering department.
Generally, when software is in the early stages of development, the employed developers can handle inputs from developers outside the company. At this stage, there likely won’t be many people reporting bugs or volunteering to fix issues. However, as the software moves toward commercial products, the engineering department will need to spend more time and energy solving customer issues, and the voices of community developers can easily be ignored. Moreover, the communication issues concealed by concrete guidance from experienced employees will quickly become apparent, especially as the software becomes increasingly complex, leaving all who wish to report bugs and fix issues lacking the necessary knowledge to resolve issues correctly.
The DevRel team can improve this situation. I helped developers from Denmark complete the automatic release of the Apache Pulsar C# client, improving API documentation and release notes. I helped the developers of pulsar-rs in code reviews and eventually merged and released their patches.
The main challenge in this process is that the DevRel team needs to establish trust with collaborators in the engineering team or have some specific development knowledge themselves. Otherwise, acting merely as a loudspeaker, pushing the development department’s developers to “deal” with these inputs like a taskmaster, without being able to explain what these inputs are, their value, or what kind of support they require, then engineers will likely ignore the DevRel team just as they might ignore email alerts, seeing them as lacking additional value.
Business Development
Common DQLs for business development include integrations and partnerships.
TiDB’s Flink CDC connector is maintained in Ververica’s repository, and StreamNative’s pulsar-spark integration has involved developers from Databricks. The primary motivation for these partner developers to create integrations is their users’ desire to combine these technologies.
Airbyte’s success heavily relies on integrations created by third-party developers. A DevRel team can maintain communication with integration developers to ensure that the software’s extension points are easy to implement, documentation is comprehensive, and there are even ready-to-use test suites to support integration development. Not all these tasks need to be executed by the DevRel team, but the DevRel team can clarify the necessity and priority of these tasks to push for their investment and delivery by the company, thus helping integration developers and those who depend on these integrations to succeed.
Recruiting
This is quite straightforward. It’s especially common in commercial open-source companies to encounter developers from the open-source community who are fully versed in the technology. If they believe in the technology, they are likely to join the company to contribute. Many well-known commercial open-source companies have employed their founding members this way.
Sales
Returning to the meaning of Leads in the marketing domain, the DevRel team can indeed identify potential sales leads. For example, in a company I ever worked with, many orders began with client developers contacting commercial products through community channels.
Conclusion
The above discussion introduces several metrics that can be used in DevRel works. It must be said that there is currently no single, universally accepted metric for DevRel works across the industry: Those engaged in DevRel either emphasize too much in the long-term to the extent that there appear to be no deliverables within any given timeframe, or they vaguely state that DevRel metrics depend on the situation, essentially defining the metrics by whatever happens to be done. Both approaches have hindered the broad recognition and promotion of DevRel.
Aside from vanity metrics, which serve as cautionary examples, the metrics I’ve listed are not intended to suggest a singular indicator. However, these metrics are interconnected.
From the unique value of DevRel work, DevRel Qualified Leads can serve as the North Star metric. But based on my experience in implementing DevRel, if the community is beginning to be built, using community membership as a North Star metric is a good alternative to ensure a precise definition of DQLs is gradually established. Brand volume is a metric specific to developer marketing that can be used for horizontal comparison. If the company's current priority is to increase developer awareness and the reach of the products and philosophy, this metric can be used as a guide.
Comparatively, DQLs highlight the DevRel team's unique value in catalyzing connections between the company’s departments and developers, benefiting the company. DevRel work is often linked with community work because facilitating developer success and catalyzing connections among developers and between developers and company departments inherently involves building a developer community.
For more details on running a DevRel program, you can refer to Jono Bacon’s People Powered and follow his YouTube channel. If possible, I plan to elaborate on the specific implementation methods and details of the aforementioned DevRel Qualified Leads metrics.
