Simplified Storage and Configuration — Tweaking your algorithm or model architecture is a complete waste of time unless you have high quality, labeled data. More than ever before, continuous improvement in machine learning relies on labeled data. And labeled data usually requires a human in the loop. Tools like Label Studio have become popular for…

How to Create Databases, Tables, Stages and Load Data from Local Storage into Snowflake Using the SnowSQL CLI Tool — In order to get your data into Snowflake, you’re going to need to figure out how to do a few things. I’m Harpreet, a data scientist and machine learning practitioner who is a bit late to hop on the Snowflake hype train. In this post I’ll show you how to…

The Why You have a bunch of friends who’ve gotten together in a hacker house ready to challenge the likes of Pixar with their indie Blender modeling and animation skills, everyone contributes some compute power to create a heterogenous makeshift render farm, but it’s on you to make a seamless queuing system…

Data has become the new source code, and we need a way to manage it. — Data has become the new source code, and we need a way to manage it. Data is so important that many of the leading practitioners in AI are pushing for data to be at the center of the ML workflow. For many years, code has been at the center of…

There’s a narrative popping up lately about an impossible tradeoff between version control systems — is it better to have a diff-based or snapshot-based architecture for a data version control system? But the whole narrative comes from seeing the approaches as separate. The real truth is people haven’t thought about…

ClearML + Pachyderm — “Move fast, think even faster,” is the ultimate goal of data science. We all want our AI models to do predictions faster and better than we can do them ourselves. Even more, we want to develop those models at lightspeed. But the reality is often a lot different. Running experiments…

Label Studio + Pachyderm — The key to building powerful machine learning models is learning “the right things from the right data.” Just as we humans constantly take in new information and update what we think about the world, ML models must continually learn from new data to keep their insights sharp and relevant. Continuous…

Breast cancer is a horrible disease that affects millions worldwide. In the US and other high-income countries, advances in medicine and increased awareness have significantly improved the survival rate of breast cancer to 80% or higher. …

I’ve been building machine learning and data processing pipelines with Pachyderm for a while now. It’s an incredibly powerful platform, but as with most things, there are some “gotchas” along the way. …