3 Mistakes I Made as a Founding Software Engineer
Lessons learned after my first month of being the only engineer at a time series event processing company.
About a month ago I left Drift. Working there was one of the best things I could’ve done for my career, and leaving was a difficult decision. In the end, though, I wanted to work on something that helped the environment. I decided to take a leap and join Singularity Energy. A company with only one full-time employee: Wenbo Shi, the CEO.
In this blog post, I’m going to highlight three lessons I’ve learned. I learned these things earlier & applied them, stumbled on the right choice, or earned the lesson though sweat and tears.
1) Do not put your server code directly behind api.yourdomain.com
This was a lesson that I learned in my first job out of college. We had a whitelabel webapp that we sold. One customer wanted to hire an outside firm to build the website and use us to host the data instead. My first instinct was one of pride: point api.ourdomain.com directly at our search API.
“Let’s slow down there” a senior engineer told me. She explained that putting an API in the public domain has a lot of overhead. Overhead in things we don’t want to manage ourselves. Things like:
burst rate limiting (protect yourself from “spiky” traffic), request allowance (e.g. only allow the user to send 2,000 requests per month), API key validation & cycling (to make sure the right people are allowed to access your API), access control lists (only grant permission to access from certain IPs or block specific IPs), and many more.
We aren’t a company that specializes in making public APIs. We could do this, but would that be worth our time? We decided to roll out a Kong server to handle that traffic.
At Singularity, we’re keeping things as cheap as possible for as long as we can. So even though I know how to use Kong, I decided to use AWS API Gateway. It’s a bit of a pain to manage another tool to allow access to our API, but since our API is our product, it’s worth the tradeoff.
2) Timezones: always use them and always use UTC
This one was one of the hard won lessons I learned. I knew that I should use UTC and I knew that I should use timezones. PostgreSQL (which we use) even outlines it very explicitly in their “Don’t do this” page. Still, I didn’t take it seriously. It’s so easy to screw up and cause problems by accident with this. Let me illustrate an example:
Singularity processes “events”. We don’t know ahead of time what type of events we’ll be processing, we just know they have a certain shape. They’ll have “data”, some blob of information. They’ll have a “datetime”, “region”, and “event_type”. We want to make sure we don’t process duplicate data, so we also make a “dedup_key” for each different event. As you can imagine, this is essentially a primary key for our data.
All was well and good until we processed March and November. Those two days a year (depending on the “region”) changes the time. If you don’t store the timezone, you’ll miss an hour when the clocks change. Once we figured that out, we reprocessed all data with the local timezone. Then we realized we had duplicated data! What was going on? It turns out that we weren’t storing the timezones consistently on those days. We switched between daylight savings timezone and standard timezone.
(╯ರ ~ ರ）╯︵ ┻━┻
We reprocessed all data with the UTC timezone consistently, no matter the region.
3) Don’t be afraid to change your tools
This one wasn’t easy or intuitive. Almost every post of a tech blog out there says to keep your stack the same. Using Java? Use Java for all services. Using Python? Use Python for all services. Working at Drift taught me that was useful until it caused problems. Especially if using a different tool would fix the problem. An example: Conversational Landing Pages.
I launched this product initially the same way we did all others: Java API with a React frontend. It worked fine. It did its job. Once we started paying attention to metrics, though, it wasn’t doing too well. Forget the metrics, take a look at the UX. It took 6 seconds on average for the conversation to start. 6 SECONDS! When the rest of your metrics are sub-second, this is horrendous.
By the time we were measuring that, I had changed teams. Will Fallows took over in my place. Instead of tuning existing tools, taking weeks to shave a second off, he realized that a different tool would fit the mold much better. Instead of a Java API with a React frontend, he switched to use NextJS. It was a great call. It was the exact right tool to use, since the experience lined up exactly with what the tool was made to provide.
At Singularity, we have a service that is highly CPU intensive. I wrote and deployed the service in Python and it gets roughly 6,000 requests per minute. Soon after deploying, I noticed the memory climbing steadily. I took about 4 hours to debug the memory leak. I had other things to do and half a day of work only remedied part of the memory leak. I took the other half of the day to rewrite the service in elixir. I’m not an expert at elixir, but I was trading ease of programming with better memory management. Once deployed, the problem vanished. The server was using much less CPU and memory for the exact same throughput.
I wasn’t sure about changing tools. I was nervous because I don’t know how to debug elixir as well as I do python. In the end, though, it was a stateless API and the change of tools was well worth it.
In summary, I’ve learned a lot at Singularity. It’s been less than a month and I’ve made a lot of mistakes that helped me learn. I’m excited for the next stage of growth over the coming months. If you’re interested in working on problems like these, reach out to me!
