Slack’s Josh Wills: “Data is coming for your industry”
It’s refreshing to meet people in the tech industry like Josh Wills of Slack.
“I think all of software engineering is terrible,” he said during a recent visit to the offices of perhaps the hottest startup in enterprise technology. “I’d like to think that data engineering is uniquely terrible.”
Wills, who has been director of data engineering at Slack for four months following stints at Cloudera and Google, isn’t talking trash about the discipline or its practitioners, mind you. It’s just that behind what appears to be the magic of new types of software is often a lot of stopgap thrown-together chewing-gum reinforced code that may not be elegant, but that gets the job done in spite of the lack of a industry-standard approach. And while that’s often good enough to ship, problems can creep up later when changes need to be made and newer engineers, raised at a time when the solutions to those original challenges have become more obvious, can’t fathom the direction that was taken by the older engineers nor change things without creating chaos.
“You end up with a hodgepodge,” he said. This is especially true for data engineering, which is a newer field with fewer established best practices. “(You’re) hiring and everyone only knows one way to build data infrastructure, so you try to rebuild the way you know it.” The problem is that what worked for Google or Facebook might not work for a company like Slack.
Wills will be a very interesting presence on stage at Structure Data, scheduled for March 9th and 10th in San Francisco at the UCSF Mission Bay conference center. He’s quick-witted and forthright about the state of his profession without hiding behind bland talking points, paired with just the right amount of mild sarcasm and self-awareness that he’s lucky to work in such at such a fundamental time in data engineering for some of the world’s best companies.
Slack’s workplace collaboration app is one of the hottest things inside Silicon Valley companies, and it’s making headway into broader industries. With a revenue-generating model baked right into the app and a user-friendly design that still allows technically-oriented people to dork out for hours creating bots and using a command-line interface, things are going pretty well at Slack these days.
Wills trots out one metric I hadn’t heard about Slack’s usage that was a little surprising: 10 percent of Slack users spend more than eight hours a day engaged with the app. That is an pretty sticky userbase that is generating a ton of data as they interact with the app. And that data helps Slack engineers understand what’s working — or not working — fairly quickly. One recent addition to Slack that came out of data analysis was the “quick switcher” tool that allows you to search for a channel or conversation across multiple teams rather than trying to hunt it down manually.
Wills is thinking a lot about how to bridge data-driven insights that can boost user productivity with the design-oriented culture so common to Slack and some of its peers. That type of culture creates beautiful products that users find compelling, but can be skeptical of using data to influence a lot of design choices, something Wills watched at Google during its transition from a purely data-driven design ethos to one a little more artful.
Google figured this out, but younger companies have to manage these tradeoffs on their own, because the solutions of the past don’t necessarily correspond to the problems of the future. Wills said he spends way more time than he would have thought on organizational planning, making sure data engineers are organized in the best fashion to share what they learn with the product development, operations, marketing, and business development teams.
Fortunately for Wills, he doesn’t have to convince the upper management of Slack that data will be an important part of its business for years to come. This isn’t the case for a lot of other companies, who struggle to understand how data can make an impact on their bottom lines or their continued existence.
“Data is coming for your industry,” he said. “(The question) is what comes for you an existential threat or not?” If your company already has some data culture, like banks and financial institutions, you’ll be fine. Even if you’re just now trying to “bolt on data culture,” you’ll probably be ok if you’re a company like Coca-Cola, which probably isn’t going to use data to design the formulas for Coke but could use data to save lots of money on marketing or distribution. But “if someone is coming for you in an asymmetrical fashion using data (think Uber and taxi companies), no amount of money spent is going to save you.”
This probably isn’t going to happen to Slack for a while. In fact, when asked what keeps him up at night about his career, Wills invoked the business version of FOMO.
“Moving fast enough for the scale of opportunity at Slack is freaking me out.”
You can find more information on Structure Data 2016 here, and register for the event here. Check out event curator Derrick Harris’ post on some of the major themes that we’ll touch on during the event.
