Design methodologies for Deep Learning

How design tools can help solve the major challenges of deep learning

Our previous article ‘Why Deep Learning is a Design Challenge’ is the preface to this article.

Every experienced engineer and designer will tell you that process is everything. Instead of expecting a Eureka moment to develop the best possible product, projects require a lot of methodologies, experiments and rigorous problem solving. Design tools offer solutions to these challenges, by building an understanding of processes and users. For deep learning specifically, design methodologies can lead to enhanced explainability of processes and overall improvement of their performance.

In The State of Artificial Intelligence, Andrew Ng describes true AI companies to have strategic data acquisition, unified data warehouses (accessible to the whole company), pervasive automation and new job descriptions. In describing the design process of an autonomous car, he states that product managers need to understand how to create a dataset and set requirements for the engineers to utilise that data.

This is the perfect example of where design and deep learning complement one another. New workflows need to be put in place for these two disciplines to work together more intensely. An interdisciplinary approach between design and deep learning will bring this technological revolution forward .

Feature design

A common misconception that many people have about deep learning is, that all you have to do is feed the model with loads of data; terabytes of images or recorded speech will solve all your problems. While this is a possibility for some applications, most models will require intensive data preparation. Most often, the images or sound are transformed into data, many columns and rows of numbers. A particularly difficult aspect of developing deep learning models is deciding which variables exactly these should be.

For example, if you are training a model to predict the path of a self-driving car, do you want the model to take into account the weather as well? Roads will be more slippery, so probably you’ll have to take turns at lower speeds and less tight when it’s raining. If the only data preparation you do is labelling roads and sidewalks, you’ll be waiting a very long time for it to understand that cars should behave differently in the rain. Adding a ‘rain’ feature to the software stack will allow you to improve the performance much faster and does not require deep learning-intensive research to achieve. Interaction with humans — Andrew Ng

The role of psychology

Every product that interacts with humans will need to have some psychological understanding of its user. Understanding the requirements that people have while using the product is essential to building intuitive products. In the case of deep learning, data is very often generated by humans. Applying general psychological models to this data allows you to explain why certain data behaves the way it does and possible misconceptions a neural network might have. It also allows you to define the type of data that you want to feed the network. Self-driving cars are here by Andrew Ng gives a good example of the need for human-centric solutions and a roadmap that is heavily influenced by design methodologies.

Let’s say you are building a deep learning model to recognise pedestrian crossing behaviour in both London and Mumbai. Using common sense you can expect the London model not to be transferable to Mumbai and the other way around. By analysing pedestrian behaviour from a psychological standpoint, you can predict the differences. Once you know these differences you can transfer learn this new behaviour, instead of training a completely new model. All you have to do is gather and prepare specific data focusing on that exact behaviour, to update the model for the new environment.

User experience improvements

In more digital product experiences, data analytics on existing users can identify issues in the user experience. The main power of deep learning is discovering patterns in the data that you feed it. When your users generate a lot of data, this offers massive opportunities to improve their experience. User journey mapping is an applicable design method that visualises the complete experience of a user, in detail. It is very useful for understanding interaction points, finding opportunities to retrieve data and defining data that would be valuable. For a good article on user-centred design with AI, read Erica Virtue’s article Designing with AI.

User journey map — UX Mastery

Reinforcement learning

An exciting development for product improvement is deep reinforcement learning. It is a combination between deep learning and reinforcement learning, where a deep learning function learns dynamically from each newly generated datapoint. This allows designers to develop deep learning solutions that learn while a user interacts with it. An example of a design that allows for human feedback in deep learning was published by OpenAI. It shows the impact that human feedback can have in developing the behaviour of a virtual 3D model. Applying this method to other problem-solving products is not difficult to imagine.

Learning to backflip with human feedback — OpenAI —

An example where deep reinforcement learning could be interesting is the interaction between an autonomous vehicle and pedestrians. As pedestrians learn the behaviour of autonomous vehicles, they will change their own behaviour accordingly. If you know that a car will stop every time you put one foot on the road, you’ll do it more often when you want to cross. With a traditional deep learning algorithm you’ll have to retrain the full model or at least several layers, to recognise this new behaviour, while a deep reinforcement learning model will automatically add the behaviour to its repertoire. It is important to mention that there are many risks with this method and you would not want to deploy such a newly trained model in the path planning component of the software stack without rigorous testing. However, for the situation analysis component it would be extremely valuable.

Visualisation of processes

A large part of the problem-solving process in deep learning is to explain exactly what is happening at which point of the data flow. Words can only illustrate so much and they generally aren’t good at giving structured guidelines of mapping information. The model can be visualised both on its architecture level and in the inner workings, allowing for a more fruitful discussion between all parties involved in the product development.

Data flowchart

Deep learning products generally consist of multiple components, you’ll have some data preparation to be done, define a couple of hyperparameters and possibly have several logical formulas and neural network models that work together. When trying to detect problems it is useful to have an overview of all the components that work together. A skill shared by most designers is abstracting and organising processes. A discussion between the tech team and design allows for a visualisation of all the full model, after which it becomes much more efficient to define which parts to improve, measuring the performance of each component and adding capabilities.

Deep learning flowchart (source)

Model understanding

Although it might not seem immediately obvious from a technical perspective, it is extremely useful to visualise the output of the separate parts of a model. Applying this methodology to the flowchart above, would let you recognise which CNN is not performing well, or whether it is processing the images in a way that makes sense. Within a neural network you can visualise individual layers. The images below are layers of a neural network that has learnt to recognise faces. Not only does it show you which features of a face it recognises, it can also indicate whether there is a bias in your network. Being able to explain how the neural network works, is incredibly valuable. It allows you to improve the functionality of the product, for example to understand how people might trick your facial recognition software or seeing that your product will only be relevant to people from a specific ethnicity.

Visualisation of the layers of a neural network (source)

Business-driven design

There are several business-driven design methodologies that are applicable to deep learning solutions specifically. To define the access to the data a company has, the data a company might lack, future opportunities and threats that will influence the data that is driving the models, a good overview of the market needs to be made, with the current situation of the company. SWOT analyses are a useful method for this, including internal company and external market analysis. In these, the Strengths and Weaknesses of a company (department) can be defined, together with the Opportunities and Threats of the exterior environment. This information leads to a better understanding of short- and long-term impacts on the product. With this analysis, a product team can then make better use of the strengths, counter weaknesses, tackle opportunities and avert threats. The SWOT analysis in particular is an interesting tool for identifying existing and future data sources for a deep learning model.

Once these opportunities are defined, the solution process that is probably most common in business was developed at the Center for Studies in Creativity, New York State University in Buffalo:

  1. Fact finding: finding information based on the expected customer problem
  2. Data finding: sorting and analysing the data of the previous information
  3. Problem finding: discovering the cause of the problem and settling on a problem definition
  4. Idea finding: generating solutions that might solve the problem
  5. Solution finding: developing further the most promising solutions
  6. Acceptance finding: testing the acceptance of the solution in the market

This process can be useful in both directions. Deep learning can be applied to most of these steps to generate valuable insights for design. Applying this design method is useful for deep learning development in deciding which data you feed the network and how you measure the accuracy of your model in real-life situations.

Getting unstuck in technical difficulties

Often the engineering departments of a company get stuck in details. Switching subject completely for a short time can help offer creative ideas or shine a light on other opportunities that might be more urgent. An appropriate methodology for developing design solutions from a technological standpoint is the Pendulum of Design Innovation. The pendulum is a metaphor for switching between the extremes of a specific development phase.

To apply the pendulum to developing deep learning solutions, can be to alternate between defining the capabilities that the data offers you, and its applications. Or to define the properties of the data that you need, and the method in which you are going to obtain it. The advantage that the pendulum offers is that you do not get stuck on a specific task without taking a new perspective. This can bring refreshing solutions and allow for subconscious problem-solving. To go through an exercise like this with both deep learning engineers and designers will give more insight in the usefulness and development of a product from multiple standpoints.

Pendulum of design innovation — Takram

Speculative design

An interesting part of deep learning is that most of the applications are very innovative, so with many of these the general public will experience them for the first time ever. It is important to make sure that the general public is actually ready for such a product. Speculative design is the most future-driven version of design. Based on trends and signals of future developments, an overview is created of how society could develop. Within that future you start designing products and services that might be needed. Those designs are then critically analysed to determine how plausible or implausible they are.

If you look at the Innovation Horizons by Terwiesch and Ulrich, they define three horizons of opportunities, based on the knowledge of technology and the knowledge of the market. The further out of current understanding of technology and market the opportunity is, the further the horizon is. Right now deep learning is being used in a very narrow set of products, which means that most of the products that will come out of the design process will at least be in Horizon 2. By defining products that are based in Horizon 3, it is ensured that the solution will be innovative. Through experiments, the product can be brought back to Horizon 1, where you can build a product that is necessary in the current market with the current state of deep learning technology.

Innovation Horizons — Terwiesch & Ulrich

Using speculative design, you can test a product, before putting too many resources in building something that nobody might want to use. An example of a speculative design experiment is the visual below. By projecting this visual on the corner of a vehicle, showing that the car had seen you as a pedestrian, we could test the response of the general public. We wanted to see whether it made people more trusting of autonomous cars if they could see that the car had seen them. Going through these kind of processes allows both designers and deep learning engineers to understand the requirements well in advance of building an actual product.

Speculative design — Humanising Autonomy


Putting deep learning and design in a silo would be a mistake. The two fields have invaluable knowledge that complements one another. General data management can be improved by design methodologies in terms of gathering and preparing data (as described in Why Deep Learning is a Design Challenge). Visualising dataflows and layer functionality can offer great insights for both product and tech teams. Switching between design and engineering perspectives allows for efficient progress, instead of hitting roadblocks in one specific field. Speculative design allows a product to be tested before it has to be fully developed. These are just some of the overlaps that can improve application and performance of deep learning solutions. Integrating a proper design process into a deep learning development will bring better products to market, more quickly.

This piece was written by Leslie Nooteboom, CDO and Co-Founder of Humanising Autonomy. We develop natural interactions between people and autonomous vehicles. 👋🤖🚗

Get in touch with us at 📩