The way we choose our priorities and goals has huge impact on our day-to-day work. With this in mind, we decided to set very simple and clear goals for our engineering team. We wanted to achieve superior up-time of our cloud application, decrease number of known bugs and speed up software delivery.
Finally, we achieved:
- An up-time of 99.99% with frequent releasing every day
- A decrease of 30% in the total number of known bugs and 50% in the average weekly number of new bugs
- A decrease of 40% in the average lead time and 50% in the number of overdue tasks
In this post, I’d like to share how we got to these results, what is behind the numbers and how we work here at Slido engineering team.
To find the perfect development process for us in the current stage, we’re constantly looking for inspiration from various sources. We don’t follow any particular methodology by book, but the main influence is from Agile and Lean values, principles and practices (Scrum, Kanban, xP, TPS and gtd).
Our main goal is to discover & deliver the right product features as soon as possible with quality & security that our customers require and at the same time learn and develop our skills & habits in order to improve and adapt over the time. In other words: deliver our product at a speed that keeps users happy, developers energized and the helpdesk calm.
We experiment and use practices that make sense for us. This means that sometimes we have to break existing schemas and habits to create new ones. At the end of the day, we focus on these areas:
- System awareness and visibility of the value flow
- Empowered, self-organized, cross-module and cross-functional product (feature oriented) teams
- Continuous product discovery & product delivery mindset
- Technical excellence, clean design & simplicity
- Open culture based on trust, courage, feedback and ownership
We develop our own product and the idea behind each new feature come from four typical channels:
- Customer facing people (Customer Success, User Research, Sales …) collect ideas, needs and pains from our customers and log it into Slack channels (#customer-love, #customer-insights, #customer-pain)
- Internal Slack channel (#product-ideas) where everybody can contribute with their ideas in order to influence the product
- Feedback: Net Promoter Score feedback and regular user survey data
- Market expectations and trends we are monitoring
All these inputs are monitored by the product managers (PM) . They regularly identify the most valuable ideas and work on the plan for the next quarter or adjust the current one by steering to the golden path.
One product team are usually: PM, User researcher, Designer, Tester (QA) and Developers (back-end, front-end or full-stack) who are able to deliver all necessary code. One developer who is responsible for a whole team delivery is a Tech Lead.
To have a plan for the next quarter ready, team has to cooperate on:
- Identify the things that are not in our current vision or strategy and should be postponed or removed
- Deeply understand why we want to do this feature
- Suggest how much effort we want to invest
- Do user research, experiments and prototyping in order to find the best UI/UX and technology (if necessary)
- Discover dependencies or possible bottlenecks which could affect development
- Describe solution with identified milestones, constraints and options (very high level specification, usually 1–3 pages)
- Transform feature into user stories or technical tasks with relative estimates
- Write down QA acceptance criteria
This is usually creative, iterative and incremental process with a lot of feedback and review sessions to ensure the feature will be done well. In the end, we like to have it specified very simple as: Feature = Sprint(s) = Team.
Each sprint is planned with a clear delivery (outcome). The length of each sprint is flexible (1–3 weeks) and we do not measure velocity. Instead, we focus on the seamless feature flow through the development (cycle time).
To achieve this, developers are working on planned sprint driven by self-discipline. We are trying to enable cross-functional but also cross-module (100% of code is shared) organization with T-shaped people.
It is very important that people in the team know context and direction; they are aligned and enabled to achieve the sprint goal. The developers in the team agree on who will deliver which task. To accomplish a task, they need to work on:
- Source code
- Unit tests
- Integrations tests
- Code review
- CI configuration
- Application artifacts deployment scripts
- Database upgrade/migration scripts
- Data analytics hooks
Team members have full ownership and responsibility to deliver it to the production in the best way they can, and we prefer flexibility over local efficiency.
They closely cooperate with the PM, UX and QA guys to make sure that they develop according to the specification. Devops guys help them to automate and prepare infrastructure, which enables easy and fast development and deployment. If an issue is not solvable by the team directly, Tech Lead is responsible for the coordination of the appropriate action.
Product teams are vertically oriented to deliver particular feature end-to-end (E2E), but there is also a horizontal aspect to our teams: functional or module/platform expertise. Therefore each function or module/platform has an Architect or Leader who is responsible to develop skills, minimal standards, and simple ground rules and control technical debt. This creates necessary standardization and alignment even when a developed feature goes across multiple modules.
Developers follow Gitflow Workflow or Truk Based Development. Every pull request is automatically integrated and tested by CI platform (Bitbucket Pipelines):
- Automated regression tests (Cypress)
- Static code analysis (SonarCloud)
- Security analysis (Snyk)
The team communicates about actual progress and impediments on daily syncs. When a story/task is ready, it’s merged into the development branch, which is automatically deployed on staging. There, it’s available for testing and internal demo purposes. Demo session usually takes place every 2 weeks for all product teams.
When a whole feature or some releasable part is done, it is checked manually by a tester who is responsible for:
- Functional tests
- Exploratory tests
- Feature feedback sessions
When QA gives the green light, pre-production is automatically updated in a way similar to the staging and we execute tests against production DB. If tests are okay, the code goes into production. This is the last step of our definition of done and we can close the task/sprint. By this time, E2E test automation is being developed by the QA team.
Release notes are communicated by the Product marketing team. Feedback and usage is measured via analytics tools (Google Analytics, Exponea) and value expectations are analyzed by our Data team.
If Something Goes Wrong
The staging and production systems are deployed on AWS by the Devops team using Terraform (infrastructure as code) and monitored via various tools (CloudWatch, New Relic). In case of any issue, a new alert is triggered into the dedicated Slack channel. If we are in serious trouble, PagerDuty will ring on our phones to ask for immediate action.
Some issues are still not detected by us, and our customers may run into buggy operation or strange errors. In such cases, they usually contact our helpline. If it is a real bug and it is not known yet, the QA team takes care to create it in Jira. If there is a higher number of bugs reported or negative feedbacks received, we’ll plan a short cleaning sprint (3–5 days) in order to fix them.
In urgent cases, a hotfix is delivered ASAP within an hour or workaround is suggested to help quickly.
Make It Better Every Day
We know we’re not perfect and that’s why we meet every two weeks for the Retrospective to identify:
- What we did well and we should keep doing
- What was not so cool and needs to be improved
- What did not add any value and should be removed
- What we learned and should help us in the future
This is not a simple task and therefore, each team member is responsible and empowered to take care of improvements he/she can bring in. Making things better and identifying the root cause of problems or various bottlenecks is a difficult process, which involves a lot of experimenting and, of course, mistakes and failures.
We know that we have to fail and learn very quickly and also be able to confirm learning by short iterations and feedback loops to feel our progress and accomplishments. Therefore, everyone on Retrospective can talk very openly, with respect and safety.
We use Slido to collect ideas before the meeting and we review and discuss them together, so yes, we eat our own dog food. We always try to find out simple and smart solutions or at least one next possible step. We do not hide our troubles, but in hard times we keep the motivation and belief that we are running a winning race.
Our business is growing very quickly and we have to face all the typical challenges connected with it:
- Scalability, Performance & Usability: infrastructure scaling, monitoring & testing, standardization, maintaining of tech debt & refactoring, UX improvements, increasing speed of feature/value delivery loop with higher quality, data enabled decisions
- Organization & Culture: growing the team, personal learning, just-in-time information radiation & alignment, balancing chaos vs. bureaucracy, processes vs. good habits, avoiding cargo cult effect
There is no silver bullet or sorcerer’s stone that solves everything by magic. It is just hard work we have to do to make our history.
It All Comes Down to the People
The whole product is done by the great people who are Slido’s most valuable asset and who have a nice and open culture. We only look for people who fit into our culture and share similar values.
We use a lot of tools and procedures that help us to be organized, productive and make good decisions. Still, in some cases, we rely on intuition or gut feeling. That’s why people are the key to our success and we put in extra effort to have the right ones at Slido.
Do you see yourself fitting right in? Send us your work examples or CV and we’d be happy to talk about the opportunities here at Slido. https://www.sli.do/careers