From Employee Voices to Expert Stars: Uncovering the Most Valuable Assets in Organizations

Modern organizations strive to assemble a team of knowledgeable, dedicated, and hardworking experts who help them achieve ambitious goals and drive the business forward.

To identify such experts, we at Exness decided to ask those who undoubtedly know the answer better than anyone else — our employees. We conducted a survey and, based on its results, algorithmically determined the most valuable experts at all levels. In this article, we will discuss our approach to the task of identifying experts, the peculiarities of conducting such a survey, selecting a suitable algorithm, and the details of its operation. We will also address the limitations and challenges we encountered in the process.

Furthermore, this article has a logical continuation where we provide more detailed information about visualizing the results in Tableau. Skillful visualization is key to the successful internal adoption of similar projects, so we also encourage you to check out our colleague Ludmila Romanova’s article for a more comprehensive understanding. We would be glad if you consider implementing something similar in your organization and join the discussion in both articles.

The business idea behind the task

All growing organizations experience what is commonly known as “growth crises,” and Exness’ rapid growth was no exception. In 2022, we effectively doubled our staff from 1000 to 2000 people by the end of the year. This led to various challenges, including:

• Too many intermediate management levels between the company’s leadership and frontline employees.
• Increased informational noise, causing the company’s management to lose the ability to see the complete and accurate picture across all projects.

Our organizational structure before and now

The situation of a large number of management levels leads to two types of errors:

1. Valuable ideas and information that emerge “on the ground” get lost in the informational noise and fail to reach the company’s leadership in a proper form (losses in upward information flow).
2. Part of the managerial decisions becomes distorted when communicated to the implementers and loses its original meaning (losses in downward information flow).

Furthermore, as the staff size grows, another effect becomes more prominent — not all employees in the organization are equally valuable to the business. This effect exists in small organizations as well, but it intensifies in larger ones due to various reasons. Here are just a few of them:

1. As the workforce expands, employees may feel less and less significant, and their work becomes less noticeable on a company-wide scale. This can lead to a sense of anonymity and reduce the engagement of certain employees.
2. The hiring process takes on a more industrial nature, and there is less tolerance for waiting time. While a startup may spend a year interviewing people in search of the right “star,” a corporation would prefer to hire a more “standard” employee quickly.

As a result, as the company grows and transitions from a “boutique” or “family” organization to a corporate one, the quality of employees and the degree of their influence on the business become more diverse.

The general idea was quite simple: to identify experts with better knowledge (and business expertise) and involve them in the decision-making process as facilitators or approvers. Working groups consisting of top management and these experts should be created for key streams of the company’s work.

This approach would also provide motivation for the employees themselves and, in the long run, make the career track of an “expert” with influence on the business a promising growth option for any employee.

If we identify which of our employees at the lower levels of the organizational structure are experts, we will be able to involve them in solving significant business problems

But the question arises: how do we identify the most significant experts among all employees, who are already the best and wonderful as they are?

First approach: Competency matrices.

The intuitive answer is to assess each employee based on a competency matrix for their position. Then, selecting an expert becomes a task of ranking the assessments across all competency matrices. However, upon closer examination, it turned out that this approach doesn’t work for us because creating an exhaustive, accurate, and complete matrix of competency matrices in such a diverse organization and keeping it up to date is practically impossible. Moreover, obtaining assessments would require all employees to undergo comprehensive tests for all the skills in their competency matrix, which would:

• lead to numerous objections from the employees themselves
• require an enormous amount of resources from all employees without exception

Second approach: Existing regularly collected data.

Existing structured data within the company can be utilized to determine who among the employees is an expert. The available data for this task can vary: performance ratings, data from JIRA tasks, documented positive customer feedback, successfully completed merge requests in GIT, and so on. It seems like all doors are open, but it’s not that simple. Unfortunately, performance rating data can only be suitable for linear comparison of employees within their functional area and heavily depend on whether ratings have been devalued in a particular team. Additionally, in our case, these data simply lack the necessary depth (after the rapid growth in 2022, up to 50% of the company consisted of employees who had been with us for less than a year). Consequently, performance data is not suitable for identifying experts, and none of the other corporate systems provide a complete representation of employees and their tasks.

Third approach: Survey and ranking.

An alternative option is to survey the employees themselves to determine whom they consider the most competent, and then calculate the final top employees. In this case, we need to conduct a survey, and then working with its results becomes a ranking task. The top N employees from the final list will be our sought-after list of experts. During the survey stage, we can completely disregard the organizational structure and positions, giving all employees the opportunity to choose any employee.

Of course, this approach has several expected drawbacks:

• Drawback 1. The main problem is that we would have to rely on human evaluations gathered from questionnaires, which are inherently subjective data. It can be reasonably assumed that each employee has certain parameters of their visibility (1) and professionalism (2) to others. The higher these parameters, the more likely the person will be chosen (provided their visibility is not infamous, and their professionalism is not low). A specific case of this characteristic would be a situation where employees may consider someone competent who is not actually competent. This situation can be metaphorically referred to as the “Gilderoy Lockhart effect” from the Harry Potter universe. Of course, we do not want such individuals to end up on the list of experts, although it’s worth noting that similar situations occur in real life as well.

In the storyline, Professor Gilderoy Lockhart turns out to be a swindler and a fraud, which didn’t prevent him from making a career as a writer, getting a job, and even teaching at Hogwarts for a while.

• Drawback 2. With this approach, leaders at all levels (who, like everyone else in our project, are both respondents and represented in the questionnaires) are likely to acquire excessive significance because, by default, they are more “visible” employees and often professionals in the area they now manage (spoiler: this effect was not observed in reality).
• Drawback 3. Employees who are highly skilled but interact with a small number of other employees will likely be marginalized by such an approach.

At first glance, these drawbacks seem significant, but upon closer examination, it turns out that some of them can be circumvented or their effects reduced:

Dependency on human evaluations is an essentially unresolvable defect, but its influence diminishes according to the laws of the central limit theorem when a sufficient number of evaluations are available. This means that our project requires a large number of responses and a high response rate.

Excessive “weight” given to managers due to their higher visibility can be eliminated by simply excluding managers from the final ranked list.

Insufficient “weight” given to employees with limited interactions can be partially addressed through a well-designed algorithm and customization, which will be explained in more detail below.

Excessive “weight” given to some incompetent employees (Gilderoy Lockhart effect) can be partially mitigated through a careful formulation of the question. We can ask employees to vote only for those whose competence they can assess and in whom they have confidence. The wording of the question is crucial.

In our approach, there is an important assumption — we assume that our employees are competent enough to evaluate the competencies of other employees. We ask employees in the question formulation and assume that they will choose people from their own area. While formally employees are not restricted from selecting employees from other departments, they are likely to choose those they are confident in, which means we will, on average, obtain a reasonable answer.

In summary, our questionnaire consists of multiple screens, each asking the employee to choose any other employee in the company, without any hints or limitations.

Question formulation

Pre-survey questionnaire screen

PageRank algorithm

Once the survey has been conducted, we have to evaluate the obtained results. We can consider that initially (before ranking), all employees are equal, and their answers from the questionnaires serve as references to other employees. In other words, each voter is an object that refers to other objects in their questionnaire. We can then “weigh” all these referencing masses, not simply by counting the number of mentions, but by “transferring weight” from the chooser to the chosen. If we perform this weighting procedure iteratively several times, we will take into account that other participants also voted for the voter.

The algorithm that precisely does this already exists and is called PageRank.

Illustration of Pagerank in action. The size of the circle represents the final “weight” of the object, while the arrows indicate the links. Object B is the only object referencing object C, but due to its high weight, object C also gains significant importance.

PageRank is believed to have been invented in 1996–1998 by Larry Page and Sergey Brin as an effective ranking mechanism for web pages in the Google search engine. The first article about the algorithm was published in 1998. PageRank is both the name of the algorithm and a numerical value that characterizes the “importance” of an object (and also a reference to one of the authors’ surnames). In its basic implementation, the algorithm is quite simple. The more links there are to object A, the more “important” it is, but the “weight” of object B, which refers to A, is also important. Let’s not delve into the algorithm itself and its history in detail — there are numerous articles available to read about it. What is important for us now is that PageRank is a method of calculating the weight of an object by assessing the importance of references to it.

Pros, cons, and limitations of using PageRank:

Pros:

• The well-known algorithm can be verified using external tools.
• No need to reinvent the wheel. There is documented experience of its application for expert identification tasks [1], [2].
• Custom weights can be added, potentially incorporating data from external algorithms or sources into the final evaluation (e.g., to consider performance results).

Cons and limitations:

• It is not possible to explicitly account for different groups of employees based on the organizational structure (but this is not necessary for our task).

Work Documentation (Fragment)

Implementation

Initially, we attempted to use the implementation of PageRank proposed in the basic article about the algorithm. This implementation assumes that the algorithm receives a matrix of all objects and their responses. It performed quite well on synthetic tests with up to 20 objects, but when we attempted to conduct tests with a larger amount of data, problems arose with it.

Code snippet implementing the algorithm in Python from the Wikipedia article

Firstly, the default implementation from the Wiki doesn’t handle various corner cases, even the simplest ones. Secondly, this implementation imposes very strict requirements on data preparation, specifically on the matrix passed to the pagerank() function. For example, the sum of all votes cast for an object must be equal to 1 (i.e., all votes for an object need to be normalized to 1 and when dividing, a floating-point number with a very high precision is used). Even slight deviations can significantly and unpredictably disrupt the results.

The implementation based on this basic function required extensive matrix transformations before execution. Ultimately, this approach implied that on top of this implementation, a series of non-trivial data transformations and checks would need to be written, essentially laying pitfalls for all subsequent potential runs. This felt kinda crushing but then, at one beautiful moment, it just came to mind…

Let’s simply try implementing PageRank using the ready-made NetworkX library.

PageRank from NetworkX works not with a matrix but with a graph (a set of objects and connections) that needs to be constructed first. This can be done from edges using the nx.from_pandas_edgelist() function by passing a group of individual links in the form (‘A -> B’, ‘A -> C’, ‘C -> A’) directly from pandas.

Building a small directed graph with weights from numeric or textual IDs and visualizing the graph

Using NetworkX, you can construct a directed nx.DiGraph (it’s crucial not to confuse it with an undirected graph, as the results in the next step will be incorrect).

Results of PageRank evaluation after running on a toy example. The sum of all scores is 1

Then, by calling the nx.pagerank function, you can obtain the final list of weights for each object, which you can simply sort and take the top-N. You can also add features for each object to this list. This way, you can, for example, obtain sub-rankings for each specific business “stream”.

Find a code example of our PageRank implementation on our GitHub.

Conclusion

Reminding you again, our goal was to identify recognized experts and later involve them in decision-making processes, giving them the opportunity to be heard and contribute to key company decisions.

In this article, we have covered the first part of the task, which involves conducting the survey and ranking the experts. Another important (and visually more appealing) output from the algorithm is the graph of all employees and their connections. Read an article by Ludmila Romanova to learn how to visualize it.

P.S. I would like to express my gratitude to Denis Nepomnyaschikh, my colleague, who led the idea to use PageRank in that project. Special thanks to Ludmila Romanova, Polina Lukina, and Ilya Lokotilov for their incredible passion and valuable impact.