QCon London 2018: five trends to watch

I was recently fortunate to attend QCon 2018 in London, where more than few thousands people gathered to listen to some of the most recent developments/innovations in the world of software engineering. Below are five of my highlights from the conference and why I think they might have a big impact in time to come.

Why ML is essential for software engineers

The day 1 keynote was a great talk presented by Rob Harrop, CEO of @Skipjaq & Co-Founder of SpringSource. Harrop used his talk to claim that you should only use the term Machine Learning (ML) when referring to the topic.

Harrop laid out some points as to why he believes ML is important and how engineers can embrace and learn to use it in their everyday working lives. These include:

• That ML is a competitive advantage and shouldn’t be ignored.
• ML should be considered a part of software engineering.
• The languages used are Python or R; although he suggested to start with Python if you are new to ML or a new developer.
• There are various frameworks you can learn, including PyTorch, TensorFlow, Keras and mxnet. PyTorch can be a good start because of its features, e.g. good debugging, which you need when you are experimenting.
• Some good learning books for ML include:
• Doing Data Science
• Hands-On Machine Learning with Scikit-Learn &TensorFlow

Aside from ML, Harrop also highlighted the advantages of good software development practices. This included using a single team, composed of various skill sets, to deliver features end-to-end, instead of one team doing development and then handing over deployable packages to the other team, which then can cause chaos if a production incident happens. At The Telegraph, where software developers can be DevOps if required, is already following these practices.

Teaching ML to schoolchildren

Dale Lane, a software developer at IBM, demonstrated how we can teach children about artificial intelligence by feeding data to buckets, using ML model creations and making machines that learn from the information.

He shows the children how a machine’s accuracy will improve when you feed it more data, and how it can overcome a challenge by being shown different examples. But rather than doing this using dry data without context, he captures pupils’ imagination by encouraging them to generate data for these ML challenges by playing games and recording their journeys.

Anyone who would like to know more about how they can do this can visit this page.

Logistics as a service: Building the Ocado Smart Platform

Paul Sweetman and Alex Harvey, the respective head of architecture and head of robotic autonomous systems at Ocado, gave a fascinating session showing warehouse simulations.

Sweetman explained Ocado’s platform and Harvey explained their complex warehouse system using robotics and how the technology is reducing overall delivery time. It was an awesome experience to listen to them.

Ocado Technology has built a scalable, AWS-based microservices architecture and Google Cloud-powered data analytics, combined with a swarm robotics grid, controlled by an intelligent air traffic system running on a private cloud that is unlike anything currently available on the market.

The Ocado technology stack is divided in 3 parts

1. Operational Space
2. Data Platform
3. Real time control System

They have used AWS as their cloud provider, where they have got about 250 microservices running. They are using various AWS elements such as Elastic Beanstalk, DynamoDB, and API gateway for exposing endpoints. They are using kitemarks for best practices & platform standardisation.

They achieved autonomy by assigning microservices to different teams, have managed to do 500 daily deployments on a prod using a single team, and they have found their teams to be happy and productive. I especially liked their golden rule for developers: “Don’t deploy infrastructure into the cloud if the cloud provider offers a similar managed service”.

Ocado’s warehouse system is called the real time control system. It is very complex, but it was such a mesmerising experience to see the videos they presented, showing how their grid system is implemented to automate everything in the warehouse! When designing their logistics they use emulators that they can use to model the results of real-life hardware. They run tests on these until their designs meet their expectations. At that point, they invest in hardware.

How software Events are reshaping modern systems

Jonas Bonér, the founder and CTO at Lightbend and creator of Akka, gave a great session on the importance of events in the modern era and why we should care about events. He believes they drive autonomy, helps reduce risks, help to move faster, as well as offering further advantages.

Events are facts and facts are immutable & accurate, he said, and I liked following quote: “An autonomous component can only promise its own behaviour. Autonomy makes information local, leading to greater certainty and stability.”

He said the following definitions were important to bear in mind when designing systems:

Difference in State, Events/Facts & Commands

State: Our current present i.e. inside data.

Events/Facts: Blast from the past i.e. outside data

Commands: Hope for the future i.e. between services.

Event sourcing allows us to model time

• Event is a SNAPSHOT in time
• Event ID is an INDEX for time
• Event log is full HISTORY — The database of our Past & the Path to our Present.

In summary, Event First Design helps us to:

1. Move faster towards a Resilient architecture

2. Design autonomous services

3. Balance Certainty & Uncertainty.

4. Reduce risk in modernizing application.

At the Telegraph, we’re already in the world of events; our main CMS is Adobe Experience Manager, which is event driven and asynchronous in nature.

Serverless and Java in the real world

It was a relief to hear John Chapin, a cloud technology consultant and serverless computing expert, say that not all people want to set up servers and the required software to get a machine ready to run java code. This is where serverless Lambda comes into the picture.

I’ve been lucky to have already worked on Serverless Lambda using Java 8 at The Telegraph, and found his session similar to the work I did few years back here. Chapin showed us how to set up projects, how various Lambda references can be made and how to set up a Maven-dependency project. A number of useful Java libraries can also be used.

He also described “The Lambda diet” and other basic configuration:

• Fewer classes = faster startup
• Ruthlessly cull dependencies
• Beware — AWS libraries can be bloated!
• Command to dig out dependencies

mvn dependency:tree, sbt dependencyStats

You can find other useful resources here:

https://github.com/aws/aws-lambda-java-libs/ — log4j2 support

https://github.com/symphoniacloud/lambda-monitoring — Logging and metrics and PR can be opened

How to set up logging

• System.out/err goes to CloudWatch Logs
• One “log group” per Lambda (by default)
• Within “log group”, one “log stream” per container
• From CloudWatch, can aggregate/forward

This link is an example of a Lambda using SLF4J and Logback.

How to monitor a Lambda using CloudWatch Metrics:

• No build-in business metrics
• Lambda platform metrics
• Native metrics collection approach is dangerous!

Cloudwatch has account-level API limits.

Lastly, Chapin took us through a Flash sale demo using Lambda architecture, which autoscales based on demand and then scales down as soon as the sale ends. For an example, visit this page.

Rachana Mehta is Senior Software Engineer at The Telegraph. Follow her on Twitter @Rachna81185836.