Efficiency Improvement Strategies That Most Data Teams Disregard
Methods and Best Practices that Data Science Teams can Enhance their Productivity and Effectiveness in the Era of Tech Layoffs
According to a projection published by Gartner in 2019, only 20% of analytical projects were expected to deliver business value by 2022. This means that 80% of data projects were expected to end up without any meaningful results for the companies they belonged to. As of the beginning of 2023, I believe this projection is highly accurate.
Many data teams face challenges that can hinder their productivity, which can affect the perception of their value to others, regardless of the complexity and sophistication of their technology. When a data team is not productive in terms of delivering business outcomes, it may start to be perceived as an R&D only team. The results produced by these kinds of teams are often only used for public relations activities, and this can shift the team towards a more traditional marketing approach, and given the high cost of these teams, they can be easily disposable, particularly during times when technology layoffs are frequent. Therefore, the efficiency of data teams in terms of their contribution to the core business is critical to maintain the perceived value of their efforts.
The goal of this post is to explore strategies and best practices that data science teams can use to boost their efficiency and productivity, enabling them to streamline their workflows, make better use of their time and resources, and ultimately deliver better results for their organizations. By implementing these strategies, I hope that data teams can achieve the valuation they deserve.
Unclear Goals and Objectives
In the initial phase of data science team building, they are often considered as magicians, and people expect them to perform miracles. However, it soon becomes clear that such expectations are unrealistic. Data scientists face high demands even when they do not have access to clean and structured data. While they struggle to create the data ecosystem they need, they may have already disappointed some impatient managers.
In situations like these, two things are critical:
First, it’s important to set realistic expectations. Experience and knowledge are crucial here. You should negotiate with others to find a common ground that makes all stakeholders happy. Additionally, you should use your knowledge to guide the decision-making process and show the right and reasonable path to take when deciding on a strategy. For example, if you propose using advanced technologies that the team is not capable of implementing, you’re on the wrong track.
Secondly, it’s essential to measure the performance of your projects with control groups in all circumstances. Never underestimate or ignore measurement. The first point in the previous paragraph is about defense, whereas this is about offense. Showing the contribution of the team is crucial in protecting them from severe criticism. However, without clear metrics that present the value of the team’s efforts, everything remains vague and ordinary, which paves the way for disappointment.
Lack of Lean Thinking
Lean thinking is management philosophy that prioritizes efficiency, quality, and customer satisfaction. It aims to identify and eliminate waste in production phases while highly emphasizes continuous improvement through incremental changes.
When the scope and goals of a project are unclear, as is often the case with data science projects, it becomes more challenging to apply lean principles. It’s common to fall into the pitfall of chasing buzz technologies rather than considering the project’s conditions and choosing the best methodology that can make the highest incremental addition to the organization. Linear regression may be dismissed as a lame move while any deep learning model may be accepted as a cool technology, regardless of the business outcome.
For instance, let’s say you’re tasked with developing a prediction model, and there is no deadline. You start by using a very sophisticated algorithm and spend a month on data preprocessing, training, and tuning the models. Then your manager asks for the results, and if the outcome is not satisfying, you may have to start over again from scratch, which could take another month. A month later, you may have the same conversation again. The worst-case scenario is hearing that the project is no longer on the table. After working on a project for months, seeing your efforts go to waste can be demotivating.
Occam’s Razor & MVP Approach
Occam’s Razor is a principle attributed to the 14th-century philosopher William of Ockham. The principle is often described as “the simplest explanation is usually the best one.” In the context of lean methodology, Occam’s Razor can be used to reduce complexity and find the most efficient solution to a problem. By doing so, a solution can be achieved with fewer resources and in a shorter time. Moreover, a simple solution is generally more feasible to implement and less prone to problems.
The MVP (Minimum Viable Product) approach of lean methodology perfectly aligns with Occam’s Razor. The main goal of MVP is to create a useful product with minimum resources and complexity and then continue to develop it according to feedback in each iteration. This methodology helps teams focus on critical product features and reduces waste by optimizing resources and time.
For instance, in the context of data science, Occam’s Razor can be used to identify the most straightforward model that adequately addresses a particular problem, rather than attempting to build a complex model that may overfit the data or be difficult to maintain and update. By starting with a simple baseline model in the first iteration, data teams can more easily address issues with data quality, model performance, and other factors, before gradually improving the model’s performance over time. This can help reduce the risk of investing time and resources in a model that may not deliver the desired outcomes, and can increase the chances of success in the long term.
The Position of the Team in the Organization
The ideal position for a Data Science team within a company’s organizational structure has been a subject of debate throughout its history. Some companies place their Data Science teams within their technology organization, while others locate them within the business unit. Each approach has its own advantages and disadvantages, and neither solution is perfect. However, the chosen position can have a direct impact on the team’s productivity.
Teams that are closely aligned with the Technology unit enjoy the advantage of easy access to data and tools. They have no trouble obtaining technical assistance and can effortlessly integrate their solutions into the production environment. However, these teams often lack a comprehensive understanding of the business perspective due to their distance from the domain teams. As a result, there is always a question as to whether they are utilizing their technical capabilities in the best possible direction in favor of the company.
On the other hand, teams placed within the business unit possess a distinct advantage of being at the center of domain knowledge and can more easily align with the company’s objectives. This enables them to make valuable contributions to the organization’s strategic interests. However, being part of a different department with technology team creates a barrier that can hinder their access to essential data and tools.
Regardless of the position of the data science team, I believe that adopting a full-stack approach is an ideal solution to mitigate the disadvantages of both situations. By doing so, teams can build sophisticated data infrastructures and successfully produce data products without losing sight of the business perspective. The critical aspect of this approach is fostering collaboration among team members with different skill sets. While it is not a simple methodology to implement, it can be highly rewarding. However, to make it work, the team must be multidisciplinary, continuously learn and improve. In another article, I plan to dive deeper into this topic to provide a more comprehensive understanding of the concept.
Conclusion
I believe that data science is a highly sophisticated subject that deserves its fame, but the productivity of teams may not reflect the true potential of these individuals. I hope that the strategies outlined in this article will assist the data community in achieving successful outcomes. Please feel free to share your comments and feedback on the article as I am confident that they will contribute to my personal growth as well.
