The Zen of a Full Stack at a startup

Practical tips for full stack developers working at startups

I’ve been working as a Full Stack developer at HYPRBrands for almost two years now.
Roey Shamir and I joined HYPR together, and we were the first developers after our CTO and CO-Founder Guy Tamir had been working alone on the project for quite a while.

After we joined, we started following some guidelines and methodologies that we drew from our collective experience; and because they made the product development better and faster, they stuck and became our de-facto “Zen of Startup Full Stack” (borrowing from the zen of python, which in itself is a must read).

I know some of these might seem like DONT DO’s, so please keep an open mind, read the explanations, and of course, share your thoughts!

The Zen of a Full Stack at a startup:

Before anything else, code should work.
Aim at 80% completeness, not 100%.
Review, debate, and democratize PR’s.
Always doubt best practices.
But do try and follow them.
Removing code is better than adding.
Avoiding problems is better than solving them.
Using Saas is better than rolling your own.
Before rolling your own, try and adjust your requirements.
When you do roll your own, try and make it OpenSource.
Prefer the most starred / maintained package.
If not the most starred, the least likely to break.
Don’t be closed minded when working with 3rd party packages.
When hacking ugly patches, make sure they really are ugly.
Test , but not at any price.
Automate as soon as possible.
Don’t write platform / infrastructure specific code.
But don’t ignore your infrastructure, use it to your advantage.
Now is better than never.
Although never is often better than *right* now.

Each of the steps above has proven itself valuable, and i’ll try to explain each line with real life examples:

Before anything else, code should work.

At the end of the day ,every startup is still a business, and still has deadlines.
It may seem obvious at first, but when working at a small growing start-up, starting pretty much from scratch, it is very easy to lose yourself in a constant effort to do better, write cleaner code, build better infrastructure, refactor endlessly.
Those are a developer’s bread and butter, but are also very time consuming and should be done only AFTER some code is production ready.

Aim at 80% completeness, not 100%

For me, this is the hardest of them all, stepping away from the the holy grail of making your code perfect, bullet-proof… Masterpiece.
When under a tight schedule, with frequent releases, you need to plan within your limits and embrace them.
In software development there is always a trade-off between cost, time and quality.

For example, say you want to build an airplane. After some initial planning you find out that building a working plane will cost you 100 hours.
Ten hours in, you find out that you missed something, and the plain will only be able to reach a certain altitude, after which it will break. Fixing it will cost you another 200 hours, and now you're invested. It is very hard to let go of sunk costs.

My suggestion is simple, try to think small, plan to build a very limited machine, that only needs to carry X weight to Y altitude, and so on.
When the need arises, rebuilding it might cost a bit more, but in the meantime, you have made a working product, that earns you money and experience, and maybe, when tackling the problem again, you’ll understand that building an airplane wasn’t the right way to go at all.

Review, debate, and democratize PR’s

No one is perfect. PRs and code reviews are essential to high quality code. Reviews help both sides, the committer gets new ideas and is tackled with questions they didn’t expect, whereas the reviewer must re-affirm their beliefs before commenting, and of course keeping him up-to-date with the code base.
Make sure you create a safe and respecting environment. Be willing to hear other’s ideas and thoughts, but also explain yourself, while making sure you truly believe in what you are saying.
One special rule here is democracy. If a reviewer and the committer don’t agree, we call up a vote. This guarantees that a real debate happens, and knowledge is spread when the question is debatable.

Always doubt best practices

Best practices are just practices that people feel comfortable with, and pass them along. The problem arises when people start following these ideas like dogma, and don’t ask themselves why the practice came to be.
Sometimes things seem clear to us, but how many times, when tackled with easy questions, you really think of the answer before throwing some best practice at the question? (This is, byt the way, true for product questions as well, not just code)
Question: I need to store some data for a long time.
Answer: DB. There are Oracle, Postgres, MySql and so on.
Why: Wait why? What do you mean by a long time? How accessible do you want it to be? How about a File on the Local FS? Or on S3?

Question: I wrote a piece of code, but it runs at X ms. I need it lowered to X/2 ms.
Answer: Let’s profile the code, find the weak spot and fix it.
Why: Wait, why? What kind of hardware is it running on? What version of code framework? How about upgrading? Scaling up? It’s much faster and costs less money (a developer’s time costs more than hardware).

Knowing to ask the questions, and sometimes give unorthodox answers, is part of being a Full Stack developer.

But do try and follow them

That being said, you should try and follow best practices, as they are usually a result of a very long and rigorous process of learning from mistakes. Don’t re-do those same mistakes.

Removing code is better than adding

We have a standing contest who of manages to remove more code than they add to our repo’s. Removing code usually means one of 3 things:
1. Removing old code that isn’t being used anymore — very good!
2. Refactoring away bloated code, making things simpler — excellent!
3. Removing something that is needed and causing errors — tests can reduce the risk greatly.
So all in all, removing code usually means a higher understanding of the code and a more finely tuned, lean product.

Avoiding problems is better than solving them

This is kinda similar to the 80% rule, but the point of this is that developers are motivated by hard challenges. So, sometimes when we see a hard problem, we feel obligated to solve it. But, when working at a start-up, you can’t afford to fix every problem that exists. You need to focus on what you do best, and what your company is doing.
If you get to a point you understand you need to invest 200 extra hours building that plane from before, you need to ask yourself, why am I building a plane? Won’t a boat be cheaper? Besides, if my company is a traveling agency, why do we need planes? Maybe I can just rent a plane. Voila’. Problem solved.

Using Saas is better than rolling your own

When you have limited resources, and don’t want to have lot’s of dedicated teams to handle different parts of the architecture, SAAS (or IAAS / PAAS) is your friend.
We use https://www.elastic.co/cloud instead of maintaining our own elasticsearch cluster, AWS EMR instead of building our own Hive setup, https://www.cloudamqp.com/ for our RabbitMQ, and so on.
This makes it easier to use a service without fully understanding it at first, and then gradually learning it as we go. This principle goes well with “end of the day code should work” and “aim at 80%”.

Before rolling your own, try and adjust your requirements

This happened to us several times. Sometimes before reviewing existing solutions, you write up requirements or expected set of features, but you do so from your own specific point of view. This can lead to “re-inventing the wheel”.
Often, trying to adjust your needs to what is already out there, makes for a leaner, more straight forward solution.
Plus, you can always roll your own at a later stage if the need arises. Trying to overlook our initial set of restrictions (product and technical alike), and adjusting to what is already out there, proved valuable for us, every time we got stuck at such a wall of not finding what we are looking for, it shortened the development time and taught us some coding lessons along the way.

When you do write your own, try and make it OpenSource

Thinking about others using your code, making it readable, testable, and modular are pretty much the core aspects of good and robust code.
We often take shortcuts, create coupling, and skip proper namespaces or hierarchy just to get things done in time, and as I mentioned before, it is important to get to production as fast as possible and evaluate and refactor later. But even simply trying to plan ahead and make your code ready to be put out there, makes things go much smoother for future developers.

Prefer the most starred / maintained package

As cliche as it may sound, the wisdom of the crowd works.
Even if at first that package was poorly architectured or badly written, starts mean it is battle hardened and probably has gone through several development iterations in order to get to a stable state.

If not the most starred, the one that is less likely to break

It happens sometimes that a package doesn’t fit, due to lacking features, critical bugs and so on, it’s ok. Next up should be the one that probably won’t break any time soon. For example, packages at version 0.x.x will most likely break, as version 1.x.x is inevitable (first stable version) and not all developers care about breaking changes. Also, check package dependencies, as this can be a major pain point as well.

Don’t be closed minded when working with 3rd party packages

The best example I have here is Django Rest Framework, although it is true to a lot of tools we use. I come from C# and seeing the chaos that goes on with the serializers was terrifying at first. Only when we stopped trying to fight it, did we understand the framework’s underlying design patterns and choices. It took some a few months to get used to it, so don’t give up too fast when things don’t add up the way you're accustomed to.

When hacking ugly patches, make sure they really are ugly

Every once in a while we need to do a quick fix for some use-case we haven’t thought of. For example we have written some framework to abstract usage with elasticsearch (yes, I am familiar with haystack, but we decided to go with something leaner), and we have written a query builder. It satisfied our need for quite a long time, until a new requirement came along, and it didn’t fit the existing code, we were left with two options; 1. major rewrite, 2. ugly hack.
Rewrites are always dangerous, as they expose you to the unknown, a new set a problems you have to deal with, where current code is always at some degree of stability. So we chose the easy way out. It’s ok. But the important thing is to make sure it is written as explicitly and naively as possible with a low of red signs around it. It is important that the next time a developer will glance at it, the code will be screaming for a re-write, which will make the subject float in the next stand-up meeting for discussion.

Test, but not at any price

This is I think the most controversial clause here.
Testing is important, TDD, functional, unit, integration and all other kinds are equally important. BUT, when you are a small, yet strong team of professionals, you learn what is a little bit more valuable to you as R&D.
We don’t have the resources to write all kinds of tests for every feature we develop, not to mention we can’t afford spending time on performance improvements for our CI server. So we cherry pick. We test at a higher level, (for example we make sure the API responses are correct and valid, but we don’t necessarily write tests for every function in that flow).
Also, we don’t test every edge case, just the obvious 80%.
This is not suitable for enterprise level companies with huge dev departments, but we believe having a strong team, allows you to have very high quality, with a little less sisyphic work.

Automate as soon as possible

This is, I think, the most important point to help keep your people happy.
One of the first things we did at hypr was automating the deployment process (even though the entire dev team was just 3 people back then). We were able to do so thanks to AWS and Travis and it saves us countless development hours (that we don’t have) and the trouble of remembering how to do things :)
In fact, the one time that Travis was down, we had to work so hard to deploy a version, it was clear we should automate away everything possible.
Whenever we write a one time script, or a new package or a workflow, we ask ourselves, is this gonna be a repeated concern of us? If the answer is yes, we have no doubt.

Don’t write platform / infrastructure specific code

Sometimes we tend to rely on infrastructure and write specific code for it. A good example for this is how ORM came to be. One of the problems it tried to solve was abstracting that infrastructure (Database) specific code. This can creep up on you when relying on specific protocols, response times, failure handling and etc. Docker has made a lot of these concerns go away for us, but still, keep in mind what features are platform specific vs. The general standard of that technology, and try to make your code robust enough (or at least abstract it away in a layer you can later throw away).

But don’t ignore your infrastructure, use it to your advantage

This can save you time and money when building software. For example you can spend a lot of time making you server fail-safe (Node.js forever, Python supervisor and such), but then you understand you can simply use Docker and make sure it is restarts (Docker Cloud), or that you can use Elastic Beanstalk by AWS which does all the heavy lifting for you.
Not to mention, knowing your infrastructure, can sometimes help you get a away with bugs, if you know what to use and how to tune it to your scale.

Now is better than never, Although never is often better than *right* now.

This is a quote from The Zen Of Python, but I thought it was too important to be left out. It is crucial as a developer at a startup to know when to stop doing something and take a step back.
It is at times obvious that if you don’t finish a part of a task, whether it be a feature, automation task, testing or any other part of the development cycle, it probably won’t be written by anybody else.
This is where you can choose either to save yourself tons of work and the headache of missing every possible deadline, or to miss on a feature that would make your life easier in the future.
I have made the wrong choice many times, costing me a lot of annoying work, or costing me a lot of late nights at the office fixing my own mistakes.
To chose correctly, you need to develop the skill of separating the important from the not, and although it may sound easy, it sometimes is the hardest part of our job, so don’t ignore it, and spend some time reaching a thoughtful decision.