15 Lessons From Re-Writing Big Project From Zero

Have you ever rewritten your project from zero? Like, big project. I mean, putting 6–12 months of work and releasing new version. We’ve done it recently with Laravel QuickAdminPanel and learned a few valuable (and painful) lessons.

Some of the lessons will be applicable to creating any bigger project, not just rewriting it. Also, I assume that you’re not a solo coder/founder and the project is big enough to have a few developers working on it, as it was in our case.

My thoughts will be divided into two “groups” — perspective of me as a developer, and as a founder/owner of the product. I’ve personally financed all the development for the project (partly reinvesting current revenue from old version, partly using money from client work), and took more of a project management role, although still coded 5–10% of the codebase myself.

Part 1/2. Lessons from developer’s perspective

First of all, why the hell would you rewrite from scratch?

Every project story is different, but what tends to happen to codebases of a few years old — technical debt reaches high level. It becomes harder to implement new features, a lot of time is spent on bugfixing of just keeping project “alive”, some architectural decisions turn out to be not ideal, some code works with comments like “please don’t touch, no idea what this part does” etc.

Random image from Reddit

Similar thing happened to us. Current QuickAdminPanel, after surviving and growing for 2.5 years, reached the level when it was painful to maintain. Also, when creating this code generator, we didn’t expect customers to create such big projects with it, demanding resources and optimized generator mechanism — the whole idea was “quick” panel generator.

Of course, we tried a lot of “local” optimizations — caching stuff, disabling/rewriting some heavy functions, hardware upgrade, but it helped only for a while. The more 504s we saw from the server, the more we realized that rewriting is inevitable.

Luckily, our situation was quite good financially — our financially bootstrapped product was generating $2,000–3,000 every month (before taxes) so we could reinvest a little. Actually, that’s a really common thing I see for client projects, too — if the project is successful and starts generating income, then client comes back for version 2.

Fresh start also often comes with new perspective on the whole product idea — there is some feedback from existing customers, some ideas of where the future should lead, some painful lessons from unsuccessful features etc. So the whole incentive of complete rewrite should be a combined effort from both founder(s) and developer(s). That’s exactly what happened for us — after a few long conversations with a team over coffee/beer, I’ve given a green light on rewriting.

Lesson 1. Clear sprints -> Frequent releases -> Quick wins.

The main problem with bigger projects is huge scope, which is hard to remember and get into one’s head. It’s overwhelming to think both about the function you need to code today, and at the same time about thing you may be doing in a month.

Also, it’s quite discouraging for the team to not see the light of day for the project for upcoming 6–9 months.

So, divide the whole project into pretty clear “sprints” of 1–2 weeks each, and let the team focus only on the next upcoming tasks. After each sprint, analyze its result, grab a beer to celebrate and discuss next sprint.

Lesson 2. Avoid “bus factor”, present your work to each other.

If you’re like me, you would probably divide the work for colleagues to code on separate parts of the projects. Side effect — they mostly don’t know what others are doing. In a way, it is good, so everyone can focus, but the problem is that only one person knows their own code/function/logic. Ideally, at least two people should have seen any part of the codebase.

So-called “bus factor” is a question: what happens if person X is hit by bus tomorrow? Would anyone be able to take over their work?

One way of keeping the team in the loop is do a few quick presentations at the end of each sprint, every developer explaining the solutions they implemented, at least on the basic ideas level, without necessarily digging into the details of the code.

Lesson 3. Prepare for the bug before they attack you.

In big projects, inevitably you will discover a lot of bugs, especially at the stage of integrating different parts of the app, coded by different developers. It’s normal, don’t panic. But what you need to make sure is that you catch, log and be notified about them.

1. Make sure the bugs are actually logged (log folders are writeable etc)
2. Setup integration with your communication or project management tools — whenever bug appears, be notified via Slack, email, SMS or by automatic new card on Trello board.
3. Use bug management tool like Bugsnag (alternatives — Sentry, Rollbar) — it will give you all the details and massively help to fix.

Set it all up and configure before the “bug-storm” starts, otherwise you may miss important failures in the code.

My video about Bugsnag

Finally, come up with a clear process of who is responsible for fixing bugs and in what time period. Often it’s not clear from error message who is responsible for that part of the code, so someone needs to take (or be given) responsibility.

After you release first “alpha” version to the public, decide who would be “on duty” for fixing bugs. Quick customer support is one of the best ways to show customers you actually care, and ensure they will stay with you and purchase the product.

Lesson 4. Readme.md: make it easy to setup the project.

During the long development, there will be multiple cases when someone needs to setup project from zero. Someone new joins the team, someone buys new computer, someone switches operating system or web-server, someone accidentally deletes the code from his computer, you need to set up staging server etc. So make sure it can be recovered by procedure like this (example of Laravel project):

1. Clone repository
2. Configure .env parameters, looking at .env.example file
3. Launch “composer install”
4. Launch “php artisan migrate — seed” (you do have some seeded data for testing, right?)
5. Launch whatever extra commands needed, if they are written in readme.md instructions

If any extra credentials or commands are needed, please write them down in readme.md.

Also, if some additional web-server configuration or libraries are required, they need to be in readme, as well. But in ideal scenario you would setup a typical machine (via Docker?) and make sure that everyone has the same environment to work in. No surprises.

Lesson 5. Code like everyone else will try to break your code.

There is a phrase:

“Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live”.

You would think it’s irrelevant for friendly team within the same company, but here’s a twist: you will be hated if you break someone else’s code. And vice versa.

For example, if you’re trying to use a class written by your colleague, you expect it to handle various parameters and situations, including missing data, typos etc. And imagine if you don’t get proper error messages when something does go wrong. Frustrated, right?

Same goes the other way around. Try to write your code, imagining someone else actually trying to use it, potentially without proper knowledge about it. Help that person.

The most typical example is API written by back-ender and then front-end client with, for example, Vue.js. If some parameter is missing or misspelled, API should return proper validation errors instead of just meaningless 500 code. (side note — I’ve written an article about handling API errors in Laravel)

Lesson 6. Clear process on how to deliver code result.

You’re probably using some code versioning system like Github or Bitbucket, and that’s cool. But make sure that everyone follows the process related to:

• Commits and their messages
• Branches for features and bug fixes
• Releases and deployments to testing and live server
• Labels or statuses of tasks when they’re done

In general, there should be a clear definition of “done”, and it’s not that simple. “Done” can mean:

• I’ve finished the feature and committed it for someone to test;
• Feature is tested and approved for deployment;
• Feature is deployed on testing server;
• Feature is actually live on the production server.

Finally, it needs to be clear, who is responsible for the approval and deployment of tasks. In my opinion, the best idea is to have one “gatekeeper” or CTO who would be ultimately responsible for whatever goes to production.

Lesson 7. Documents, comments or just readable code?

The code should be readable and understandable, and not only to your colleagues. I’ll tell you a “secret” — you won’t remember your own code you have written 6 months ago.

There are various ways to tackle this problem: some teams write a lot of documentation, some focus on the comments and docblocks of the methods, but I prefer to emphasize the readability of the code itself: clear variable/method/class names, standard patterns where to look things up, avoiding large code blocks in one method, structuring classes with SOLID principles etc.

It seems like a self-explanatory thing, but countless times we’ve bumped into “Hmm, what does this old method ACTUALLY do? Can I touch it?” situations.

Lesson 8. Avoid coding when possible.

Weird advice, right? The thought here is to come up with some creative solutions for problems, knowing some feedback from previous version.

Try to re-code the features with smaller scope or more simple architecture. Or, in extreme examples, remove some less popular features. Or, maybe something changed in the market since version 1?

We have two examples of this:

1. We had a feature of importing SQL file and making a panel out of it. We noticed that it wasn’t reliable because SQL file can have different structure which our generator is unable to parse. So we removed the automatic import and instead provided a form to send SQL file to us for analysis, and we would advise on necessary changes for that file to work. So, providing consultancy in this case. No one complained.

2. Another example is function “Push to git” — in old version we had integration with Github and Bitbucket, and the latter one was particular pain in the neck. Different logic of saving temporary Auth tokens, separate logic of team accounts, some other glitches etc. And, while we were working on version 2, Github announced their free private repositories plan! So, Bitbucket quickly declined in usage, which allowed us to not create integration at all. So we saved time and went with only Github integration.

So, these are my technical lessons as a developer. Now, let’s put on the founder’s hat and see what I’ve learned as a product owner, or you can call me a “client” in this project.

Part 2/2. Founder’s perspective

Lesson 9. Are you keeping a backlog of what people say about version 1?

First, for those of you who are not planning version 2 immediately, but maybe someday down the road. Please, still keep all the important conversations with your customers (from email, live-chat, phone, wherever) and don’t forget to categorize them, adding new conversations to the same topic.

From the beginning of version 1, we had a Trello board with cards of feature requests by customers. For example, if someone wanted integration with Gitlab, we either added such card, or, if it already existed, added a new comment on that card, with customer’s details and exact words they were saying.

This will massively help you to plan and prioritize version 2, and you will see how many customers were actually requesting certain feature.

Also, you should analyze the usage of the features in version 1 and not be afraid to REMOVE some unpopular functionality. Don’t waste time on a brilliant module that was used only by 1% of customers. If they do still need that in version 2 (if they even notice it’s gone), they will contact you later and you may or may not bring it back.

Not only features, but also keep the log about common questions, complaints and unclear UX pieces. Then, in new version you can make some buttons more visible, and rearrange stuff visually to avoid those questions in the future.

Lesson 10. You can plan and estimate the scope. Sort of.

It seems that if you have version 1, then version 2 should not be that different and you can plan the same features with some new parts of code, right? And you would think it would take less time than previous version?

It depends on the project, but in our case the feature set was indeed similar. So we created a whiteboard (both in live-mode in the office and on Trello) and tried to estimate how long would it take.

But the problems came from unknown things. From those new features we planned, new and more flexible architecture and then re-thinking some of the old features, which actually means creating them from zero.

Realistically I can say two things:

1. You don’t know what you don’t know and can’t plan those “surprised” ahead.
2. Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.

So, in Summer 2018 we planned about 4 months of work, and only now released “beta” version in April 2019. So we missed the estimate by 2x. And yes, the budget also grew significantly (more on that below).

To be honest, I don’t have a good advice here, except for double any estimate you have in mind, cause things in software take time. Don’t be too optimistic and don’t plan for the perfect scenario.

Lesson 11. You can still stop version 2 at any time.

Related to the previous problem of bigger time/budget, if you see that the numbers are too big for you and you don’t want to invest that much money/time, it’s ok to cancel version 2. At any point — in the planning phase, or even when coding begins.

If you see that it’s a “suicide mission”, it’s never too late to stop. Then, instead of re-creating from zero, think about creative ways to solve old version problems cheaper. Doing something manually? Using 3rd party service for something? Investing in hardware or reconfigure infrastructure? Partner with someone?

In my case, I personally wanted to cancel the project at around 50% phase (didn’t even tell that to the team, shh). The only way forward was to find extra budget, so what I decided in the end was to take more client and consulting work myself, and also started online courses to bring more income and compensate the expenses while the guys are still focused on finishing QuickAdminPanel. It kinda worked, but it was a difficult winter.

Lesson 12. Actively communicate with your customers.

This should be done in all phases of version 2:

• While planning, you need to talk to customers — not only for you to decide the priorities, but for them to expect that something is coming. It’s like the first announcement of a movie sequel.
• When you release some features internally, post some screenshots and screencast GIFs on social media, that you’re creating something, and discuss those features with the audience, gather first feedback.
• Build list of alpha-testers (10–50 people) that will be the first to know when something “clickable” is released;
• Release SOMETHING as early as possible — it may be only 20% of features, but with new architecture in mind so you can already discuss with the loyal customers, or maybe even change direction on something.

Keep in mind — usually most people won’t care about your new product until you actually release its full version and they would be able to use it fully. So don’t be discouraged if only 10% of alpha testers actually try something out. People are busy. They really don’t care about your product that much, they have their lives. But you still have to communicate, so when they DO need your product, they will remember you.

Lesson 13. Constantly stop and reprioritize.

The beauty of working in agile way and having sprints every 1–2 weeks is that you can change your mind on something after every sprint.

Life is always changing, so are software priorities, it’s normal. You may come up with some new ideas, you may see some bottleneck to be solved ASAP, you may gather some useful information from customers, you may notice that something happened in the market (like new competitor product) etc.

It’s ok to change your mind, if it makes sense. But it’s not ok to constantly add new ideas to the scope which is already 6–9 months long. Keep less important ideas somewhere in the backlog, or just briefly discuss with the team and then put them in “Someday” or “Version 2.x” section.

Lesson 14. You need TWO project/task management tools.

While working on a big project, you constantly have to switch between two mindsets — local current sprint, and global vision for where product is heading. And, in my experience, there should be two different tools to manage those.

One tool should handle task list for upcoming weeks, and another tool would have massive list of all features planned in version 2. In ideal scenario, you would just take the tasks from that second tool and put into current sprint, but what happens more often is one “global” feature is actually divided into multiple small daily/weekly tasks, so it’s not one-to-one connection between tasks.

These two “views” may actually be within the same project management tool (Trello/Asana/JIRA or whatever you use), but in our case we decided to use Trello for global project management, and Github issues for weekly sprints and daily tasks.

One of our sprints.

It made developers’ life easier while committing and deploying the code — you can reference the exact issue number and focus only on dev-world in Github, where as Trello was mostly for myself as client.

Important thing is to make sure it doesn’t take too much time to handle the task management. Don’t force developers update two tools manually every time — as a product owner, you should be the one managing the global board and maybe even creating tasks in development board for them, so at all times developers are focused on the tasks at hand. Their time, attention and focus are very valuable.

Lesson 15. Prepare to support both version for a while.

However great is your new version, people hate change. So be prepared for the scenario that only small part of your customers will switch to version 2, even when it’s fully released. For various reasons — some will be used to version 1 workflow, some won’t have new projects to create with version 2, some will just not find time to try it out.

Don’t overpush it, and totally not discontinue version 1 immediately (unless that’s the only way). Every software should have a proper end of life, and “sunsetting” it is also an important part — you should at least fix the bugs on version 1 if they appear.

Finally, would I do it again, knowing what I know now?

To be honest, not sure. Probably I would have done it differently, cutting the scope of some features that took too long.

Also, I would then plan the budget accordingly and plan to get another source of income (maybe take on more client work and hire a few more developers).

Finally, I wouldn’t announce the exact dates and wouldn’t promise to the clients when it would be released. Cause it feels embarrassing to promise December, and release in April.

All, in all, what matters are first customers impressions, like these ones I’ve got:

Final advice for you — remember, version 2 is not the final one. There will be version 2.1, 2.3.45 etc. There will be always time to fix stuff and add more features. Just adapt to your customers and follow your ultimate goals.

Bonus. I recommend to watch a video by Kyle Racki from Proposify, who told their own story of version 2: