From back-office to back-end in 3 years
How I managed to move my career from no experience and zero background in computer science to becoming a data engineer in 3 years.
Hints: have a dream and go one step at a time. A long process of reflection and self learning, patience, frustrations and taking the risk of every opportunity.
Many people ask me how I became a data engineer without a CS or SWE major. In my last article I mentioned I started my career at CX operations, far away from computing. So I had neither education nor experience. Well, it didn’t happen overnight as magic, but like a job ladder. I switched positions thrice, each time to a more relatively technical role. No, I’m not officially a back-end engineer, but I did join a BE team and this is a fun pun!
But I’ve been sure about one thing during my whole life: I have a technical profile and want a technical position. I guess that because I was better at communication than programming, people would expect the opposite (since college at the Formula SAE team). I was told to be a good planner, presenter, leader and so on. But I didn’t want that and I stood up to get what I wanted. This matters more for me than the money or my ability to do the tasks.
My key decision was joining the Data team. Why? Looking up some jobs, there were many positions, well paid, not much specific in computing. It was a good chance to work closer to numbers instead of text (or computer instead of people?). At the time I thought Data members only performed analysis and they were the only ones to do so. I said I could combine my math and logic skills to the business background (that’s the differential) to empower decisions. Believe me, for years this was my real slogan to employers (and worked!).
1. Business to Business Intelligence (1 year)
When I saw headphones at my desk, I freaked out. Making and answering calls made me nervous, I struggled at negotiation with clients. I was back-office, so that was less common. But it taught me a lot of strategy, how to structure speech, argue, interpret minds. I knew the customer journey from the heart and each process detail. I was an expert with demand. I could handle a lot of tickets and multitask with many systems, help colleagues. I always pictured tasks as a logic flow and sketched the big picture in my head.
In 6 months, I asked my manager for an opportunity to do something different. She gave me an open problem: study rescissions from landlords (remember I worked at QuintoAndar, a tech startup in real estate). I could read the conversations and present anything. I categorised complaints and found out a shift from payments to support. That was an insight! In 2018 I saw the BAs working with PowerBI and bought an online course. I built some time-series and Pareto charts and they loved. It was beyond expectations. Yay I was first analysing data, transforming long boring texts into visual stuff! Loved.
I got the name of a Data Analyst and scheduled lunch with him. Later I told my boss I wanted to apply for Data. She gave me honest feedback that she did not imagine me there then, but I could try. I was very sad, but I knew it was true. It was too early for that move, but not impossible. But she offered me a performance analyst position, to take care of our team reports. It was not what I planned but I saw it as an intermediate step to my Data dream. I said yes, but made clear that I didn’t change my mind. I will go to Data later on. That was my first goal, to leave the Operations team (or “call center” like roles).
2. Business Intelligence to Data Analytics (6 months)
My scope was huge as a performance analyst. It was the most stressful period in my career, but maybe when I learned the most. I represented my former team and covered both analytics and projects. I was definitely the “jill for all trades” and specialist in rescission. Besides reports, I configured tools, wrote comms and even interviewed customers. The nice part is that I grasped metrics modelling and managing their data. I got closer to PMs and learned KPIs. I realised I was serving strategic data to import managers for the first time.
I saw two paths on the road: business (BA) or data analyst (DA). I had a peer from each and watched their routines to choose my future step. In a simple way, the BA was the face of the project and charged for the results. He would talk to stakeholders and make plans and presentations. The DA was his technical supplier, would understand the needs and support with data. The DA translated a real problem into a model to answer questions (this “model” was the SQL query, where we make assumptions and use logic for calculations).
Of course I tended to data, but BA was my natural progression (a.k.a. easier for the manager to approve). I like to think of this dilemma as the CEO and CTO of the company. Let’s say Apple, I admire Jobs a lot (he’s an ENTJ too!), but I’m more of a Woz. I’m not that creative, spontaneous to be in the front, but I rather work backstage. I became fascinated to collaborate with managers like that. I really felt like the “intelligence behind the business”. But please don’t say that the BA was the cool whereas the DA was just the nerd one!
A game changer was learning SQL. It was not required, but I got very empowered. Again, a bit online but 99% practice. I often requested validation from a friend and googled (like row number for deduplication). I was the only in the team and became able to handle Data tasks and prove my value. A cool thing was when I saw we were missing OKR because of timing and were not aware. I built a simple dashboard to manage conversions close to expiration, so analysts would take more focused actions. It was a success.
My manager was afraid of my move because of my “non seniority”. This “you’re too junior” sentence always scared me a lot in my career. I was lucky the firm opened internal positions, otherwise I couldn’t be in the market (I tried but failed). I texted the Data manager and he was very kind with my interest. I just nailed the SQL test with 100%. No wonder, I was already working as a DA! I just needed to move officially. I was unhappy with my previous job (pressure and no focus) and gave me a year to get into Data. I got it in 6 months!
3. Data Analytics to Data Engineering (1.5 year)
After 1.5 year, I joined the Data team in mid 2019. I celebrated a lot! The DA plan was done. I was very lost in the beginning. I knew the term “ETL” because of hearing it once at a BI/database course. I was nervous to give satisfaction in all those Agile rituals. Again I humbly requested advice (one was this amazing dataviz book!). But I knew SQL, just had to transport code from Metabase to an IDE. I attended CX squads so topics were very familiar. The core job of queries and dashboards there was pretty much the same, but less amateur and more organized. I learned good practices of performance and validation.
After some time, I was already wondering about the next level (DE or DS). I reached the comfort zone of SQL. Everything started to feel the same. I knew all the databases and metrics, tasks were easy. I got frustrated dealing with non technical people, sometimes they wouldn’t accept my solutions and prefer faster workarounds. I was tired of consuming built data and attending stakeholders. But I enjoyed other cross-team initiatives without clients. This made me think I needed to decide the next technical role in data for me.
I got the hype of Data Science. It sounded like it satisfied my intellectual appetite. I surrendered by challenges and salaries (hm almost no one can do that, so let’s invest). It happened when I was looking for a graduate course, so I enrolled at the first MBA from USP/ICMC. I wasn’t sure, but it was shorter, cheaper, easier (just transcription, all online, on Saturdays for just a year). I’ll write a review later about the course. I learned a lot and got good job offers!
At first I was excited but then it wasn’t my thing. It required a lot of energy to read Statistics books and I didn’t have it anymore. I got unused to calculus notation. I was frustrated and not confident about using models without understanding the scenes behind them. I thought I wouldn’t pass in Neural Networks. I think I would have liked this course at college. I was tired of studying by myself and working in two teams (later). For a while, I really wanted to give up but it would be a waste, so I worked hard for that degree. Seeing the bright side, knowledge would be useful even for a non scientist.
Meanwhile, I designed my first DW model. I loved the freedom for rules and names and drawing tables (nobody liked that). We had new micro services and there was the need to integrate them. It required business knowledge and modelling skills along with heavy queries, so I drowned on those waters. I switched AWS Redshift to Athena/Presto and my friends JSON and REGEX (fun fact: I gave a REGEX class without knowing, then applied it to models). I could develop a lot of models in a short time and apparently people liked them.
Creating tables was better than analysis and I asked my manager to shadow data engineers as we were close. I received the chance to work in an end-to-end integration with them! But the headcount was low and I had to be autonomous. I joined their rituals, installed PyCharm, took Python and Airflow courses. Each new word they said (incremental, full, reprocess) I would note and search articles and tutorials on Medium to get informed. I was very happy. I worked double and stayed late at night just to chat with DEs.
Good times have gone. I applied for two DE positions and got two no’s, as they were looking for seniors. I freaked out and blew up coding and Big Data architecture, and had never seen something like that (and tricky MySQL 5.7!). After the project, I had to return to the DA team and lost motivation. This was the saddest part of my career, I even cried to my boss. I got into Data but incomplete. I asked for feedback and made a huge study plan. Learning before Data was on-hands, after it was studying (I’ll write about my favorite online tools later!). I would stand up, wait another 6 months and try again.
I almost lost my hope, but put myself open to the market. I doubted my chances but saw some (rare) junior roles. I saw Wildlife Studios and we matched! I was comfortable even with live coding (my first)! Their hiring process was focusing on discussing cases together, I loved. I talked about tools I hadn’t used but studied (streaming). I felt very warm when a recruiter said “relax, we know you’re not a data engineer. YET!”. I took the challenge to be the first data engineer at the Creatives squad! Me, no major, no experience, alone. It was sad to say goodbye after 3 years and I was afraid, but I just did!
Now: to Data Platform
I joined Wildlife in the back-end team and learned a lot with the devs. I was mentored and took a Python course and a DE Nanodegree from Udacity (I promise to review it in another article!). I also participated at a Hackathon with one of the teams to propose a Python project framework (Never had I ever written a unit test and I built a DAG factory with Airflow tests!). I worked with data scientists and could understand their models and implement metrics from what I learned in the MBA! All pieces came together!
Their stack differs from my castle of DW and star schema. My first project was structuring KPIs to create the team’s first dashboard. I used all my DA skills at Looker and built nice things. I was invited to present some training on my models and storytelling and felt very valued. Also SparkSQL was replaced by PySpark in the Data Lake! Finally I won’t be working with dashboards soon and will handover to a new DA. Then I’ll focus on core data engineering. :)
Our focus on data grew so much that we formed a new team to attend all the Creatives Vertical. Now I’m part of the Creatives Platform team and will act as a classic data engineer with pipelines, tools, cloud and architecture. We will start building a whole Big Data pipeline from scratch based on awesome tools and the best data practices. I never imagined myself at Data Platform! Anyway I’m very excited about this journey and where it will take me later! Who knows? (I joke that after DE can only come SWE, but I don’t think so.)
Thanks for reading, I know this was a long one! I wanted to write about every detail. I hope it was inspiring for you as people were for me during my career. Please believe in our dreams, work hard and have fun too! Good luck!
Quoting Rock the JVM, I’m dying for feedback! Please share yours ❤
