DAYMN — 29 Aug 2021
Data Articles You Might Need. Subscribe & follow here!
I read a lot of informative and thought-provoking articles every week, and share them immediately in a piecemeal fashion, with friends and colleagues.
One, it is hard to find these articles again — when you need them — despite using a combination of Read-Later tools. And second, the nature of these fleeting shares are quite ephemeral making it hard to retain the insights in this age of information overload.
So, I thought it would be a better practice to batch and share the top 5 articles I read every week along with a short write-up on what was interesting in these articles.
Here are the top 5 articles from this past week — please do share your feedback & thoughts!
1. Building a Metric/Information Layer at Scale
Building an enterprise information layer that can provide metric consistency at scale, while at the same time providing both the business and the data users flexibility and reliability — has been a big search of mine for the past few months. Minerva from AirbnbEng (AirBnB) is such an amazing & promising attempt at solving this information/metric layer problem at scale.
https://medium.com/airbnb-engineering/how-airbnb-achieved-metric-consistency-at-scale-f23cc53dea70
Call out? Minerva — Airbnb’s metric platform — came onto the scene. Minerva takes fact and dimension tables as inputs, performs data denormalization, and serves the aggregated data to downstream applications. The Minerva API bridges the gap between upstream data and downstream consumption, enabling Data Engineering teams the flexibility to modify core tables while maintaining support for various downstream consumers.
2. A New Way of Thinking About Data @ An Organisation
Kindly shared by KADA’s Dean, this article below succinctly describes the current challenges with how data is approached in organisations and the dire need for a new way to think about structuring both data functions and the teams & tools that support data-driven decision making.
While the broader article delves in and out of the data mesh topic, I’d love to guide your attention to the philosophies described (check the Call out) on how to approach data thinking differently for today and tomorrow.
Call out? To me, the modern data experience…
…enables everyone to do their job rather than asking them to be an analyst.
…merges BI and data science.
…remembers what we’ve learned
…doesn’t communicate in only tables.
…builds a bridge between the past (read: Excel) and future.
3. Understanding Doordash’s Dispatch Problem — Making Complex Simple
As a big fan of optimisation problems, and more specifically route and resource optimisation problems, this wonderfully written and illustrated article from DoorDash’s Alex Weinstein on how Doordash approaches their tricky dispatch problems is a great read. It talks to how they build, deploy, calibrate and improve their dispatching algo over time, with the details.
Call out? Taking on such a complex problem was a two-stage process. First, we built a sophisticated dispatch service that utilizes a number of ML and optimization models to understand the state of the marketplace and make the best possible offers to Dashers to meet the needs of our marketplace. The second stage was to build simulation and experimentation platforms that would allow us to make continual improvements to our dispatch service.
4. Data Product — Do We Need A New Discipline?
A lot of organisations are looking to either build or repurpose parts of their data engineering and data ops teams into a Data Product team. While Data Product has different meanings in different orgs, this article from Nadiem von Heydebrand does a great job of justifying why such a role is needed and how it can complement the current data teams and what they aim to achieve.
https://towardsdatascience.com/data-product-management-ffa582f7e047
5. Understanding Hanlon’s Razor For Your Sanity
Thanks to a wonderful gift from a good friend, I managed to finish “Mental Models — Part 1” this past week, and the one thing that stood out for me was Hanlon’s Razor, which states,“Never attribute to malice that which can be adequately explained by stupidity/neglect”
This 8 min read will change your life on how you look at typical issues faced when you deal with non-adoption of your work or a lack of appreciation for what you do for a living.
Call out? When someone messes up around us, we forget how many times we too have done the same. We forget how many times we have elbowed someone in the street, knocked over a drink at a relative’s house or forgotten to meet a friend at the right time. Instead, the perpetrator becomes a source of intense irritation.
To assume intent in such a situation is likely to worsen the problem. None of us can ever know what someone else wanted to happen. The smartest people make a lot of mistakes. Inability or neglect is far more likely to be the cause than malice. When a situation causes us to become angry or frustrated, it can be valuable to consider if those emotions are justified. Often, the best way to react to other people causing us problems is by seeking to educate them, not to disdain them. In this way, we can avoid repeats of the same situation.
Have a wonderful week ahead everyone, hope at least one of these articles is exciting reading material for you, and made you think for a moment!
