MSc Notes: Deep Learning

What is Deep Learning (DL) or Artificial Neural Networks (ANN)?

Imitation is the highest form of flattery, often known as biomimicry in the engineering community. Mosquitoes, birds, fireflies, mussels etc. inspired surgical needles, airplanes, LED light bulbs, adhesives respectively (more); Nature has inspired numerous other innovations. Therefore, it makes sense to model an intelligent computer after the architecture of the human brain. Deep Learning Network or Artificial Neural Networks is a type of machine learning model that draws inspiration from the organic network formed by neurons in human brains.

Note: Via this and related articles in this series, I hope to record and share my learning (obtained recently during and after pursuing Data Science with Business Programme at the University of Exeter, UK) in a simplified manner so that you could use this resource to quickly refer to or grasp concepts of varying complexity. This may be of interest to you if you are a ML/DL practitioner or researcher who, like me, requires a reliable go-to place for quick reference every now and then, or if you are a student attempting to develop a solid intuition of relevant concepts in this field.

Building Blocks of ANN

• Artificial Neuron or Node
• Weights & Biases
• Activation Function
• Layer (Input, Hidden, Output)
• Training or Learning Algorithm

Note: To delve deeper into what each bulleted topic (above and throughout the article) implies, I will be linking respective articles in due course.

Why and When to use Deep Learning over Classical Machine Learning?

As a result of significant advancements in the field of ANNs over the past several years, businesses from all sectors are now implementing deep learning solutions as part of their AI strategy. Deep learning has opened the door to a wide range of sophisticated new AI applications, including chatbots in customer service, picture and object identification in retail, and much more. But is it always the best course of action?

When is deep learning required, and when is it not?

Each use case is obviously highly unique and will depend on factors such as your specific business goals, the maturity of the AI, your timetable, your data, and your resources.

The general factors to think about before selecting whether or not to utilise deep learning to address a particular problem are listed below.

• Need for Human Intervention (in current context, level of domain understanding for Feature Engineering): Classical machine learning requires relatively more ongoing human intervention to get results. Deep learning is more complex to set up but requires minimal intervention thereafter.
• Problem Complexity or Nature of Data: Deep learning has the ability to tackle complex problems that include finding hidden patterns in the data and/or a thorough comprehension of complex interactions between a large number of interdependent variables as one of its key benefits. Deep learning algorithms are able to extract hidden patterns from the data on their own, combine them, and create decision rules that are far more effective, specially when it comes to difficult problems that frequently involve dealing with a large amount of unstructured data, such as picture classification, natural language processing, or speech recognition, among others. However, traditional machine learning might be a preferable choice for easier problems that need more straightforward feature building and don’t require processing unstructured data.
• Availability of Time & Compute Resources: Deep learning is still a challenging and expensive endeavour in reality, despite its increasing accessibility. Deep learning models, for example, take a long time to train and use a lot of computational resources because of their inherent complexity, the numerous layers they have, and the enormous amounts of data they need, which makes them incredibly time- and resource-intensive. GPUs, or graphics processing units, are now essentially necessary to run deep learning algorithms. Even though GPUs are very expensive, it would not be practicable to train deep networks to achieve high performance without them. Without needing the most advanced technology, classical ML algorithms can be trained just well with a good CPU (Central Processing Unit). One can iterate as well because they don’t require as much computer power. They also allow for faster iteration and the ability to test out a variety of methods in a shorter amount of time because they are less computationally expensive.
• Amount of available quality data: Deep learning should not be regarded as a model of learning in itself. You still need a lot of properly labelled data! Deep learning’s main strength is its ability to handle more complex data and relationships; however, this also implies that the algorithms used in deep learning will be more complex. And, in order to extract general enough complex patterns from data, you will need a large number of examples (far more than a traditional ML model) — typically millions of labelled data points for a classification task. One of the main reasons why deep learning can produce disappointing results in some business cases is a lack of a sufficiently large corpus of precisely labelled high-quality data. When it comes to cleaning, preparing, and labelling data, organisations face a slew of challenges, and these processes frequently consume a massive amount of time and resources from teams that could be better spent building the next machine learning model or pushing models into production. Active learning is one method for addressing the time-consuming task of data labelling, but in cases where having very large amounts of efficiently labelled data is not possible (or necessary), opting for more traditional ML algorithms and models may be a better use of your data team’s time, effort, and resources. On the hindsight, Transfer Learning is one

Whether you choose classical machine learning or deep learning for a specific data project, a common thread is that human oversight, evaluation, and decision-making should be involved at every stage of the process. The key to truly responsible and transparent AI is human-in-the-loop augmented intelligence.

HOW to apply Deep Learning (Project Pipeline)?

Because Deep Learning is a type of Machine Learning, the overall process and steps remain the same as those of a typical machine learning project, exactly as stated here. To highlight the differences, the following steps represent what happens in an end-to-end Deep Learning project:

• Big picture or Business Understanding
• Sourcing Data
• Exploratory Data Analysis (EDA) and Visualisations
• Data Preparation
• Building and Testing Model(s) ~ Algorithm(s) Selection and Training
• Fine-tuning the Model(s)
• Presenting Solution
• Launching, Monitoring, and Maintaining the System

Other crucial aspects of how Deep Learning applications are developed and maintained include:

• Tech stack(s) & ML/DLOps
• Team structure

Note: To delve deeper into what each bulleted topic (above and throughout the article) implies, I will be linking respective articles in due course.

Key Deep Learning Architectures

• Perceptron - the simplest ANN
• Multi Layer Perceptron(MLP) or Feed Forward Neural Networks
• Convolutional Neural Networks (CNN)
• Recurrent Neural Networks (RNN, LSTM, GRU) or Attention Network (Transformers)
• Autoencoder Networks (Linear AE — PCA, Non-linear AE; Stacked AE; Convolutional AE, Recurrent AE, Denoising AE, Sparse AE; VAE)
• Generative Adversarial Networks (GAN)
• Graph Neural Networks (GNN)

Deep Learning Challenges

• Data Collection — finding, augmenting, or creating; labelling (annotations); cleaning existing data
• Data Preparation
• Managing large Data Volumes
• Data Security and Privacy
• Model Interpretability

Note: I will continue to build, aggregate, and link content in this article such that it could become the finest go to resource for anything Deep Learning related. Further, I’d love to hear from and engage with you in this journey.