Serverless ETL is becoming the future for teams looking to stay focused on their core responsibilities rather than running a large infrastructure to power data pipelines. Nextdoor’s mission is to build community, and I joined the Systems Infrastructure team to help support product engineers in achieving that goal. The team’s responsibilities include managing the data ingestion pipeline which services 2.5 billion daily syslog and tracking events. We found that, as our data volumes grew, keeping the data ingestion pipeline stable became a full time endeavor that distracted us from our other responsibilities.
Not every company makes physical products and needs a physical supply chain. But every company makes decisions. And the raw material of decisions is data. The data supply chain that powers the daily decisions made by managers and software requires every bit as much attention as a physical supply chain — — probably more. When companies recruit CDOs they should do so with the data supply chain in mind. And they should recruit CDOs who will bring the same competitive spirit to the management of the data supply chain as Tim Cook brought to Apple’s physical supply chain. A CDO whose primary orientation is toward creating a competitive advantage can generate vastly more shareholder value than one whose orientation is toward keeping the company out of trouble.