Lessons from a Startup Pt. I
Capturing the lessons that I learned working at an early-stage startup
After graduating from Springboard’s Data Science Bootcamp, I was fortunate enough to land a part-time position with EQ Community. They were six months into their venture, and I was their first technical hire. It was the perfect arrangement — I was eager to gain practical experience, and they had “some data work” that needed to be tackled.
Luckily, the “data work” they needed was precisely what I wanted to be working on — Data Engineering (not Data Science). During my stint at EQ Community (EQ), I had the opportunity to engage in a variety of Data Engineering tasks, such as database model design and data pipeline construction with AWS to connect diverse SaaS APIs with the database.
Naturally, since my role at EQ was focused on Data Engineering, my balance sheet became stacked with hard skills and technical lessons that were incredibly valuable. However, the lessons that “hit hardest” were not technical at all. Drawing on my experience as a former high school teacher, I firmly believe that creation is the ultimate form of learning so I decided to write this article to reflect on, capture, and preserve those lessons into something that I can always look back on for future reference.
Writing Code is Overpowered
Prior to EQ, my coding experiences were limited to academic projects, personal endeavors, and skill-building exercises. At EQ, my code very quickly became one of the driving forces behind the business, automating tasks that would previously have taken one or two employees several hours to complete. The ‘army of robots’ in my data pipelines were now able to tirelessly handle these tasks with ease, all day everyday. The amount of leverage I was able to create with just a small amount of code was pretty amazing. But now, if my code were to fail, the entire business would be negatively impacted, which means that the power to create that much business leverage comes with a significant amount of responsibility… and paranoia.
Don’t Outsource your Code
Given the power and responsibility that comes with writing code, it’s crucial to handle it with care as a valuable asset that plays a key role in the business strategy. Your code helps improve processes, increase efficiency, and simplify the process of scaling your operations. The best approach is to have an in-house professional who is willing to take ownership of the codebase and work on enhancing, maintaining, and updating it. This way, as your business landscape evolves, you can make necessary changes and adapt quickly. In contrast, outsourcing can lead to slow development cycles that can limit your ability to iterate and refine your offerings.
When outsourcing is the only option, be sure to choose a partner who is committed to a long-term relationship.
The Never Ending Battle Between Completion & Correctness
Balancing the need for efficiency and quality is a constant challenge. While it is ideal to strive for both, in a fast-paced startup environment, tasks often need to be completed yesterday. This means making the decision to prioritize either speed or accuracy. The tightrope between getting things done and getting them done correctly requires careful consideration in order to achieve the best outcome. As an Engineer, I hold the belief that accuracy and completeness should always be prioritized as much as possible.
The pipelines we are responsible for are vital to the success of the business and must be functional, precise, and able to be sustained.
While this approach requires more time and effort, it results in a strong, dependable, and thoroughly documented codebase. However, there are instances where speed is a must and the challenge is to find a balance between the two.
Journal from the Job — Reflecting on my first task as a Data Engineer, I recall working with Airtable’s API to transfer data from another system. While the task was urgent and required immediate completion, I took the extra time to create an internal API toolbox for Airtable. Looking back, I am glad I made the decision to prioritize accuracy and efficiency over speed. Building the toolbox not only streamlined development but also made it more efficient for the duration of my time at the startup. This experience reinforced my belief in the philosophy of “working hard once.” This experience has taught me the importance of being mindful of the long-term impact of my decisions, even when under pressure to complete urgent tasks.
Failure is the Only Guarantee
It’s inevitable that at some point, despite your best intentions and efforts to do things “correct”, you will experience failure. This is especially true when it comes to coding, which is why it’s important to plan for potential disasters from the get-go. You should always design with the expectation that something might not work as intended. While the likelihood of failure may seem low, it only takes that 1% chance of catastrophe to potentially wreak havoc on your business. It may sound a bit overdramatic, but keep in mind that your code is one of the backbones of the company, and as such, requires careful consideration and planning.
Journal from the Job — I remember a situation where I had assumed that a specific table would always receive a specific combination of data, and built the pipeline without any checks or quality control in place. When the pipeline encountered data that did not match my assumption, it continued to process it without any interruption. Unfortunately, this led to business actions being triggered that did not make sense, causing confusion and frustration. This experience taught me the importance of thorough testing and quality control in the design and implementation of data pipelines. It also highlighted the danger of making assumptions without thoroughly verifying them, and the potential consequences of overlooking even small details in the design process.
Excitement can be Dangerous, Use a Fundamental Compass
When working at a startup, especially in early stages, you are trying to find your way in the world and there is no shortage of ideation. Combine that with the high drive to succeed and you’ve created an environment where it’s easy to get caught up in the excitement and rush to take immediate action without thorough evaluation
In order to build a something successfully, it’s important to first establish a solid foundation. Take the time to identify the key components that are necessary to create a strong base, and then focus your efforts on strengthening them — easier said than done.
Journal from the Job — When we discovered that one of the SaaS tools we were using had some Data Science abilities, we were eager to jump on the opportunity to utilize them. We jumped right into prototyping and integrating them into our existing data processing pipelines. Looking back, it’s clear to me that our eagerness to experiment with the new Data Science abilities we discovered blinded us to the importance of assessing its potential value for our organization. We failed to take a step back and evaluate whether we had the necessary data volume and variety required for a successful implementation. Our enthusiasm led us to believe that this new tool was the solution we needed, when in fact, it wasn’t suitable for our current situation. This experience taught me the importance of approaching new ideas with a critical eye, taking the time to evaluate their potential value and ensuring they align with our organization’s current needs and goals before diving in headfirst.
Data is Useless
Abandoning Data Science for Data Engineering was one of the best decisions I ever made, and my previous job was the perfect case study. The reason is simple — raw data isn’t useful to a business, and it is the Data Engineering team’s responsibility to transform it into something less useless. Working with actual data, you quickly learn that it is always messy in some way. It’s just an unavoidable fact.
Even though the Data Engineering team is responsible for making raw data not raw, it still needs to be analyzed to extract valuable insights. If no analyst is available to help, the tables set up by the engineering team can remain unused gathering dust. Processing data is just the first step; the real magic happens when someone turns that data into something meaningful. However, even after analyzing the data, it may not be immediately apparent what actions to take. As Beau Lotto noted,
There is no inherent value in any piece of data because all information is meaningless in itself. Why? Because information doesn’t tell you what to do.
For example, suppose your data shows you that sales of vanilla ice cream increase during the summer months. Cool, what steps will you take in response to this information? In the world of business, I view data as a form of potential energy.
It has the potential to be useful, but only when decision-makers take action based on the insights and findings derived from it.
Having a vast amount of data doesn’t necessarily guarantee success. Rather, it provides the opportunity to make informed decisions. Data shouldn’t be dismissed as useless since it holds tremendous value, but it is the actions taken based on the insights that ultimately drive success.
A3I Written (Accelerated & Augmented w/ Artificial Intelligence)
