QCon 2017 Takeaways

After attending the QCon 2017 in NYC this past June, it seemed fair enough to me to share with my colleagues an extract of what I got to see there.

Even though the organizers set up a variety of topic threads throughout the conference days, most of the sessions I attended, not by chance, went around three main topics: Modern Software Development, Agile Teams, and Microservices.

Modern Software Development

Modern software development is based on 4 principles: Small teams, TDD and Continuous Delivery, Microservices and DevOps

• Multiple tech adoptions are well supported only by standardizing these topics: RPC, Discovery, Registration, Runtime, OS, Configuration, Metrics, Logging, Tracing, Dashboards, Alerts, Stream Processing.
• Developers should be able to get a pre-assembled platform (like a Framework) with all the topics above integrated. If a tech stack is new, then a new Framework should be built to integrate that new tech stack for future usage. Benefits:
  - Consistent applications, system, and runtime metrics & logs reduce issues detection and repairing times.
  - Allows teams and engineers in charge to use what they think is the best tool set for the solution requested.
  - Makes feasible a culture of Freedom and Responsibility.
  - Enhances Product Innovation, Velocity, and Reliability in the company.
• Software companies should always be prepared for scale at every level:
  - Functional scalability: the ability to enhance a system by adding new functionality with minimal effort.
  - Load scalability: the ease with which a system or component can be modified, added, or removed to accommodate changing load.
  - Organizational scalability: the ability for an organization to easily add people and domain responsibilities in response to increased work and complexity. Also, the ease with which an organization or team can adapt to shifts in business strategy.

Agile Teams

“In order to encourage people to innovate and build, you need to give them the freedom to do so.” — Randy Shoup

• The main goal for an agile team is to minimize the distance between “hello, world” and production. Create value as quickly as possible, without losing quality. Build an organization and architecture that allows you to deploy change frequently, swiftly and safely to production, and own the impact of the change.
• Developers hierarchy of needs:
  - Basic (must have): laptop, wifi, desk, etc.
  - Perks: non-necessary but important compensations
  - Self-actualization: get relevant stuff done.
• Ideal size of teams (Pizza team size): 3–7 members
• Being agile is about removing friction. Friction is when it feels like the work is fighting you. Friction goes away primarily due to a systematic habit of “refined annoyance”, it means refusal to accept friction of any kind.
• Friction can be explained with an analogy of Newton’s laws of motion, where the body in motion is that feature wanted in production:
  The main force applied = people motivation, it is mainly based on the autonomy, the mastery and the purpose acquired by the people.
  Gravity force = complexity and innate difficulty of the work being done
  Friction force = reactive force resisting motion given by the system itself
• Main friction forces in software development:
  - Staging/testing environments: it is preferred to test in a production-like environment
  - Forced tech choices: voluntary decisions are preferred (consensus)
  - Fear of breaking all the things: move to microservices/serverless, since small pieces are easier to change
  - Forced team choices: self-selection is preferred, i.e. Internal job offers
  - Distractions: reinforce the notion that coding is the primary activity. Delegate a red hot engineer that absorbs all the distraction within daily operations. Meetings must be time constrained (2.75–5 hrs/week). After each meeting review whether it was useful and whether it should happen again.
• It is important to measure the friction at every time (Team Health Checks): are we delivering value? are we having fun? how easy it is to release changes? how healthy is the codebase? are we learning? is Mission(product) clear? do we feel like Players or Pawns? how speedy is the process? how suitable is the process? how is the Support? how is the teamwork?
• Spaghetti Diagrams make poor layouts and wasted motion obvious. Types of waste in software:
  - Intellect: spending time building and debugging environments instead of adding value
  - Overproduction: encourage fewer ‘big bang’ releases
  - Waiting: “can’t get my stuff deployed”
  - Motion: going back many times doesn’t allow to move forward
  - Transportation: multiple handoffs between Engineers, QA and Ops
  - Inventory: lots of commits held up in the pipeline
  - Rework: “it works in dev environment, but not in test environment”
  - Overprocessing: features tested and rested in different environments
• Agile teams should be constantly watching out for these issues:
  - Zombie projects: Insufficient staffing not really “alive” making no real progress, time filling activities which make the project not really dead making people unavailable for other projects, resulting in months of delay.
  - Delivery accountability: unresolved decisions and unclear accountability results in months of delay
  - Large launches: the allure of large changes, it feels better but the result is worse.
• Why teams tend to fall into this issues? Because all the poor approaches defer pain:
  - Zombie projects defer having to say no.
  - Not making a decision means not choosing wrong.
  - Large launches defer feedback.
  - Integrating later feels good now.
• Give your teams secure, unfettered control over their own infrastructure. Segregate and apply command-and-control where you need it most.

Microservices

The traditional 3-tier system (presentation/application/persistence) lacks one extra tier: State Changes (Events).

• An event is a significant change in state that other components in a system might be interested in. These can subscribe to be notified, typically asynchronously.
• A microservice interface includes any mechanism for getting data in or out: synchronous request-response, events produced, events consumed, bulk reads and writes.
• In the journey of breaking the monolith into microservices, there is always a common obstacle: creating microservices with isolated persistence. This means, that all external access to data store can be only done through the published microservice interface.
  Step 1: Create a microservice that will represent that independent part of the system that is going to be decoupled from the monolith.
  Step 2: Make applications use the microservice.
  Step 3: Move data to a private database, only accessible by the microservice.
  Step 4: Rinse and repeat.
• How to share data between microservices? The main principle to have in mind is that there should always be a Single System of Record, that means that every piece of data is owned by a single service, which is the canonical system of record (source of truth) for that data, and every other copy is a read-only, non-authoritative cache.
  Approach 1: synchronous lookup, services call each other when they need their data in real time.
  Approach 2: Async event + local cache, each service is in charge of notifying events on their property (owned data), and dependent services listen to those events and locally maintain a cache with the needed data. Dependent services operate based on their own cached data.
  Approach 3: shared metadata library, this is mainly thought for basically immutable data (i.e. US states, colors, etc), and it is done through exposing an accessible database for those services that depend on that data.
• How to join data from microservices in client applications?
  Approach 1: Join in client application (application mashup), best for 1:N joins. i.e. the orders history page queries one customer from the microservice A and then queries all matching orders for that customer from the microservice B.
  Approach 2: Materialize the view, best for M:N joins. The client app listens to events on the microservices it depends on and maintains its own denormalized join of data in a local storage. The client app operates based on their own data.
• How to manage transactions (Commits/Rollbacks) across multiple microservices?
  Using Workflows and Sagas, modeling any transaction as a state machine of atomic events. Implement transactions as workflows:

A -> B -> C -> D 

where each letter is an step/action within the workflow. Then implement rollbacks by applying compensating operations in reverse:

A <— B <— C <— D

Ideal implementations of this are based on Function as a Service (Serverless) technologies.

• Recommendations to make easier microservices architectures evolution over time:
  - Technology: avoid duplicating capabilities between components, have in mind that the assumptions will change over time, think about capabilities, consider relative costs (mid-long term instead of only short term), consider scalability, capacity planning is important (throughput), pursue independent & minimal coordination deployments, do not have machine-local state, PAAS infrastructure simplifies evolving a microservice architecture.
  - Monitoring: collection, storage, stream processing, visualization and ubiquitous service monitoring can help identify operational risks.
  - Culture: alignment — “organizations … are constrained to produce designs which are copies of the communication structures of these organizations” (Conway’s law), evolving an architecture might require re-evaluating organizational responsibilities.
  - Documentation: knowledge lives with teams, teams change over time, so the documentation needs to be highly available.

Tooling

Netflix Ribbon for Client Side Load Balancing
Bugsnag for Error tracking
Zookeeper & Consul for Configuration and Service Discovery
Kubernetes for Containers Orchestration
Cloud Foundry for Platform as a Service
Jenkins & Jartifactory for Continuous Integration
Graphite & Graphana for data analytics
Zipkin for Distributed Systems Tracking
Telegraf for Collection, InfluxDB for storage, Kapacitor for Stream Processing, Cronograf for Visualization of Services Metrics.