Data Careers: Getting Motivated, Working on Soft Skills, and Finding Your Way

If you’re trying to break into the data space, or if you’ve just started, it’s easy enough to find resources. There are countless road maps, learning plans, study guides, and lists (so many lists) of technologies (so many technologies). This article isn’t any of those: not because they’re not important, but because they’re not the only important part of building a career in data. Instead, I want to share the habits and practices that have worked for me. No two journeys are exactly the same, but there are enough common threads that I hope you can pull something useful from mine.

Some Personal Background

I’m a mid-career data engineer working for a SaaS company. I spend my days maintaining and developing data assets used by data analysts, data scientists, and business folks. I love what I do, and I like to think I’m pretty good at it, but it’s been a long way here.

Like a lot of people in data, I didn’t plan this career. Five years ago, I could barely explain what a pivot table was, and I would have struggled to piece one together in Excel. I was busy managing a nursing home and burning out rapidly. Between the stress and long hours, my heart wasn’t in my work anymore. I started finding respite in financial reporting and quality metrics. Compared to people and personalities and crises, numbers were vastly easier and more engaging.

This interest at work sparked my curiosity in my precious free time; how did streaming services know what I wanted to watch? How did Google decide which ads to show me? How did businesses forecast sales and costs? The more I read and learned, the more I wanted to do data professionally. Over the course of a year, I studied voraciously and finally worked up the courage to quit my job and take a contract as a data analyst.

That first job was a lot of data entry with only a bit of analysis and data cleansing, but it paved my way to a full-time analytics gig with a manufacturer. There, I started out on a long-deprecated BI system; its semantics were outdated, and I spent a lot of time creating new relations in Excel, VLOOKUP by miserable VLOOKUP. Over time, I taught myself SQL and Power BI, and I even got my feet wet with R.

Most importantly, I learned that, as data professionals, our jobs are mostly about pain. What does the business lack visibility into? What reports do executives manually stitch together in Excel every month? Why don’t the numbers ever agree? There are a lot of exciting, advanced analytics use cases out there, but by and large, data teams are trying to provide timely, reliable, and interpretable data so that the business isn’t flying blind.

The other major lesson I learned was that analytics is difficult without data management. Lacking good data models and reliable pipelines, analysts spend a lot of time on tedious but necessary data quality checks and cleansing. Frustrated by the limitations of our infrastructure, I picked up Kimball and Ross’s The Data Warehouse Toolkit, and from there, I was hooked; I wanted to build things.

My next role was as a BI developer for a wholesaler where I was supporting an actual, honest-to-goodness Kimball-style data warehouse. I was able to see facts and dimensions in practice, and while everything was on-prem (lovingly orchestrated through stored procedures and the RDBMS’s own scheduler), it was a mind-blowing upgrade. The analyst in me was delighted by the ease-of-use, and the budding developer in me was excited to learn how this thing was put together and how I could help extend it.

The technology improvement was the second-best part of the job; the best bar far was my team. In my prior role, I had great support from the business, but I had precious little contact with technical experts, so I had to learn almost everything on my own. In my new role, I developed quickly with excellent mentorship and the ability to bounce ideas off of my teammates.

The other major lesson I learned was that even as my job became more technical, the business still mattered. A lot. While developing technical solutions, I would identify business process issues that showed up in the data. Like water and electricity, the business is going to flow through the path of least resistance. If systems get in their way, they will work around them, and when data’s involved (which it always is), their pain is going to become your pain.

While I had a wonderful team and a great company, I wanted to work with modern architecture. I wanted the cloud. I wanted “real” orchestration. I wanted to treat data work like software development, with proper source control and peer review. And perhaps most importantly, I wanted to “see the world.” How did other data teams structure themselves? How did other businesses operate? I don’t want to discount the value of growing professionally with a business over time, but especially early on in your career, I think it’s important to expose yourself to a wide range of ideas and cultures.

That’s how I found my way to my current company. I’ve been inundated with new technologies and development methodologies, and as the shock of technical onboarding is starting to wear off, I’m beginning to wrap my head around how the business works and how my team supports it. I’m also starting to get a better feel for what I find interesting in data, and that’s helping me put some shape to my career goals.

Find a Purpose

If it’s not clear enough, let me be explicit; I love data. I’m nearing five years working in the industry, and I still think it’s the coolest thing: databases, data warehouses, file formats, encodings, observability, quality, modeling, analytics; there are a million different paths in this space, and they all have their own unique challenges and rewards. When I get to talk about data with people, it’s clear how I feel. This also comes across in interviews, and I attribute a lot of my success to my enthusiasm.

Follow your passion. It makes me incredibly sad when I read posts from newcomers asking what the most lucrative position is, or which represents the best opportunity. I’m a firm believer that if you love analytics, you’re going to be far better off in terms of career satisfaction being a great analyst than you are as a mediocre data engineer.

Titles are fluid and ill-defined, and what’s hot today is not going to be hot five or ten years from now; however, data as a space isn’t going anywhere. If you want to chase total compensation, that’s certainly your prerogative, but I would always choose sustainability over a truly mercenary career.

You don’t need to spend every spare moment of your free time reading articles, attending meetups, or building projects, but if you’re not interested enough to keep current on some level, I would reconsider whether this is what you want to do. Like most spaces in tech, data is evolving rapidly, and your career success is in part tied to your ability to understand, evaluate, and (if necessary) adopt new tools and technologies.

Find a Team (Be the Dumbest Person in the Room)

For a lot of folks wanting to break into data, there’s a huge focus on learning paths, certifications, and side projects. I don’t want to discount those things, since they can be valuable, but nothing is a substitute for working with a real-world team on real-world problems. Even if you’re using “real” data that needs to be cleaned, you’re still doing greenfield work with few or no restrictions. Your biggest challenges in a job are often going to be working inside or with legacy environments, as well as learning how to be functional on your team.

The other benefit of working on a team is that your mentors are easier to find (since you already have a work relationship), and they have a lot of extra context about your workplace. Try to get mentors who can help you with career growth as well as technical growth; they’re related but not the same.

It’s a common trope that tech workers will bemoan “politics,” a term often used for anything other than the technical workload of a job. You may find it less comfortable to work on soft skills than hard skills, but part of professional growth is becoming a well-rounded individual. You should play to your strengths, but don’t completely ignore your weaknesses.

The Business Matters

Another time-honored tradition for data professionals is giving pensive sighs when confronted by the business. Why do they generate bad data? Why don’t they notify us of changes in processes? Why can’t they understand that their request isn’t reasonable? We can’t onboard a new data source that quickly.

This may not be your job right now, but at some point in your career, you may find yourself interfacing with the business. You’re going to have to learn why they do the things that frustrate you, and you’re going to have to learn what you and your team’s doing to frustrate them. Great technical leadership can reduce the friction between the business and data teams; they can set expectations and produce buy-in on both sides. They can remind everyone that they’re all working toward the same goal: making your organization win.

At the end of the day, ideas without resources are worthless; you need buy-in from the business to transform your designs into solutions. That doesn’t mean that the business makes the rules and runs the show, but they deserve a seat at the table when making decisions that impact them, just as much as the data team does when the business is making those same, impactful decisions.

Expand Your Horizons (Taking Risks)

Quitting my long-term care career was scary for a whole lot of reasons. The day after I resigned, I got a call from the agency saying that my contract rate plummeted by nearly half due to a “misunderstanding.” After some back-and-forth, we reached a compromise, and then the worst thing possible happened; my father died unexpectedly. Despite the advice to limit change following a major loss, I went forward with the contract role. It was one of the best decisions of my life.

I’ve worked with fantastic people and great businesses, and it’s always been hard to leave them behind. Job searches can be draining and even demotivating. New roles are full of risk; what if the team’s not a good fit? What if the culture isn’t what you expected? What if you fail to meet expectations?

But because of all of those risks I’ve taken, I have a rewarding and lucrative career with a bright future. I’m far happier than I was in my previous career, and I’ve learned a tremendous amount from every new role I’ve taken. Learn your risk tolerance, and then leverage your ability to take risks to propel yourself forward.

You (Probably) Belong

You aren’t ever going to feel ready. There are people who have been working in data for years, sometimes decades, who feel like frauds. I don’t think I’ve ever had more than a few weeks at a time where I felt truly confident as a data professional.

Nobody wants to be on the wrong side of the Dunning-Kruger curve; who likes a know-it-all who doesn’t actually know anything? But being too conservative, too full of self-doubt has its own problems. You grow most when you’re learning new ideas and concepts, not merely perfecting and consolidating what you already know. Over time, you’ll learn the pace that works for you, but especially early on, err on the side of going a little too fast.

Why? There’s little cost to being wrong, outside of your pride. Embrace the discomfort of putting yourself out there. Embrace the discomfort of being corrected. Ask a lot of questions, especially questions about how someone learned an answer. The most exciting part about working in data (and perhaps tech in general) is that there are a million things to learn, and you’ll never know even a fraction of it.

I’m going to say that this goes even more so for underrepresented folks, for whom it’s often even easier to say, “I’m not qualified for that.” The reality is that you’re not going to really know how you’ll do until you’re actually doing it, and you aren’t going to know if you’ll like the work enough to do it professionally until then, either.

I wouldn’t give up a job you love on the chance that you’ll like data. And if you only have a passing interest, I wouldn’t even give up a job you like. But if you keep finding yourself coming back to articles and videos and projects, trying it out might be a good risk to take.

All that being said, there’s a chance that data isn’t for you. You might not like the messiness, the minutiae, the tooling, or the attitudes that most organizations have toward their data teams. That’s not as bad an outcome as you might think. Typically, when you can identify what you don’t like or find fulfilling, that gives you more information about what might be a better fit. If data isn’t your future, don’t despair! There are a lot of engaging, challenging, and lucrative careers outside of data in particular and tech in general. To reiterate an earlier point, you don’t have to love your job, but if you don’t at least like it, you’re going to struggle to excel.

Good Luck

In conclusion:

1. Find your interest and pursue it relentlessly
2. Prioritize getting on a data team over building up the perfect skill set
3. Develop some level of interest in the business
4. Learn how to take the right risks
5. Don’t let imposter syndrome keep you from moving forward