What is Deep Learning?

 

A lot is happening in the world of AI at the moment. Some of you may be wondering how machines have the ability to do what they can do. How can they recognise images, understand speech, and even reply to my requests???

Welcome to the world of Deep Learning. 

Deep Learning is a subfield of machine learning concerned with algorithms inspired by the structure and function of the brain called artificial neural networks.

Yes, I understand, that sounds very technical and overwhelming, right? 

If you are just starting out in the field of deep learning or you had some experience with neural networks some time ago, you may be confused. I know I was confused initially and so were many of my colleagues and friends who learned and used neural networks in the 1990s and early 2000s.

The leaders and experts in the field have ideas of what deep learning is and these specific and nuanced perspectives shed a lot of light on what deep learning is all about.

But what is deep learning? What is it all about? What did deep learning mean to the pioneers and thought leaders of today? If you’re thinking about these questions, then you’re in the right place. 

This article will explore what deep learning is by hearing from a range of experts and leaders in the field.

If you’re really keen to learn about deep learning, kick-start your project with my new book Deep Learning With Python, which includes step-by-step tutorials and the Python source code files for all examples.

Let’s dive in.

Andrew Ng on the Essence of Deep Learning

Renowned for his contributions to the field, Andrew Ng, founder of DeepLearning.AI and other platforms such as Coursera formally founded Google Brain. This eventually resulted in the productization of deep learning technologies across a large number of Google services.

Andrew Ng has frequently spoken and written a lot about what deep learning is, making him a great starting point for those who wish to learn more about the field.

In the early stages of deep learning, Andrew described deep learning in the context of traditional artificial neural networks. In the 2013 talk titled “Deep Learning, Self-Taught Learning and Unsupervised Feature Learning” he described the idea of deep learning as:

Using brain simulations, hope to:

– Make learning algorithms much better and easier to use.

– Make revolutionary advances in machine learning and AI.

I believe this is our best shot at progress towards real AI

Later his comments became more nuanced.

As time progressed, Andrew Ng’s insights into deep learning became more refined and nuanced.

According to Andrew, the core of deep learning is the availability of modern computational power and the vast amount of available data to actually train large neural networks. When discussing why now is the time that deep learning is taking off at ExtractConf 2015 in a talk titled “What data scientists should know about deep learning“, he commented:

very large neural networks we can now have and … huge amounts of data that we have access to

He also commented on the significance of scale in the world of deep learning. As we construct larger neural networks and train them with more and more data, their performance continues to increase. Unlike many traditional machine learning methods that reach a plateau in performance, deep learning stands out. 

Crafting bigger neural networks and furnishing them with increasing volumes of data will lead to a rise in efficacy. 

Andrew noted:

for most flavors of the old generations of learning algorithms … performance will plateau. … deep learning … is the first class of algorithms … that is scalable. … performance just keeps getting better as you feed them more data

Here is a nice cartoon explaining this from one of his slides:

Why Deep Learning?
Slide by Andrew Ng, all rights reserved.

Another point that Andrew Ng highlights is the importance of supervised learning within deep learning. Speaking at the 2015 ExtractConf, he pointed out:

almost all the value today of deep learning is through supervised learning or learning from labeled data

Echoing similar sentiments, during a 2014 lecture at Stanford University titled “Deep Learning,” he stated:

one reason that deep learning has taken off like crazy is because it is fantastic at supervised learning

Andrew often mentions that we should and will see more benefits coming from the unsupervised side of the tracks as the field matures to deal with the abundance of unlabeled data available.

Jeff Dean: The Architect Behind Google’s Deep Learning Infrastructure

Jeff Dean, a driving force behind the Systems and Infrastructure Group at Google, has now been appointed Google’s chief scientist and played a pivotal role and is perhaps partially responsible for the scaling and adoption of deep learning within Google. Jeff was involved in the Google Brain project and the development of large-scale deep learning software DistBelief and later TensorFlow.

In his 2016 presentation “Deep Learning for Building Intelligent Computer Systems” Jeff commented in a similar vein, that deep learning is really all about large neural networks.

When you hear the term deep learning, just think of a large deep neural net. Deep refers to the number of layers typically and so this kind of the popular term that’s been adopted in the press. I think of them as deep neural networks generally.

He has given this talk a few times, and in a modified set of slides for the same talk, Jeff emphasizes the scalability of neural networks indicating that results get better with more data and larger models, which in turn require more computation to train.

It seems like Andrew Ng and Jeff Dean were definitely having the same conversations.

Results Get Better With More Data, Larger Models, More Compute
Slide by Jeff Dean, All Rights Reserved.

Deep Learning: The Art of Hierarchical Feature Learning

Another characteristic of deep learning models is their ability to perform automatic feature extraction from raw data, also known as feature learning.

Yoshua Bengio: A Pioneer’s Perspective

Yoshua Bengio is another significant figure in the deep learning domain. Starting out with an interest in automatic feature learning capabilities that large neural networks are capable of achieving.

In his 2012 paper, “Deep Learning of Representations for Unsupervised and Transfer Learning” Bengio commented:

Deep learning algorithms seek to exploit the unknown structure in the input distribution in order to discover good representations, often at multiple levels, with higher-level learned features defined in terms of lower-level features

He then further expands on this idea in his 2009 technical report titled “Learning Deep Architectures for AI” where he emphasizes the importance of the hierarchy in feature learning.

Deep learning methods aim at learning feature hierarchies with features from higher levels of the hierarchy formed by the composition of lower level features. Automatically learning features at multiple levels of abstraction allow a system to learn complex functions mapping the input to the output directly from data, without depending completely on human-crafted features.

Bengio, in collaboration with Ian Goodfellow and Aaron Courville, published a book titled “Deep Learning” which defines deep learning in terms of the depth of the architecture of the models.

The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones. If we draw a graph showing how these concepts are built on top of each other, the graph is deep, with many layers. For this reason, we call this approach to AI deep learning.

This book has become a definitive resource within the field, presenting multilayer perceptrons as a core algorithm in deep learning, suggesting that deep learning has effectively integrated artificial neural networks.

Peter Norvig: Google’s Take on Depth and Abstraction

The quintessential example of a deep learning model is the feedforward deep network or multilayer perceptron (MLP).

Peter Norvig, the Director of Research at Google is well-known for his textbook on AI titled “Artificial Intelligence: A Modern Approach“.

In his 2016 presentation “Deep Learning and Understandability versus Software Engineering and Verification” Norvig resonates with Bengio’s views on deep learning. He defined deep learning with a focus on the power of abstraction permitted by using a deeper network structure.

a kind of learning where the representation you form has several levels of abstraction, rather than a direct input to output

The Evolution of the Term “Deep Learning”. Why Not Simply “Artificial Neural Networks”?

Geoffrey Hinton, a pioneer in the field of artificial neural networks co-published the first paper on the backpropagation algorithm for training multilayer perceptron networks.

In 2006, Hinton co-authored “A Fast Learning Algorithm for Deep Belief Nets” in which the term “deep” signified networks with multiple layers, particularly restricted Boltzmann machines.

Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.

This, along with another seminal paper Geoff co-authored titled “Deep Boltzmann Machines” on an undirected deep network were well received by the community as they were shown to be successful examples of greedy layer-wise training of networks, allowing many more layers in feedforward networks.

In another co-authored article in “Reducing the Dimensionality of Data with Neural Networks“,  the term “deep” persisted in describing their approach to developing networks with many more layers than was previously typical.

We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.

Echoing Andrew Ng’s sentiments about the fusion of computational power, using vast datasets, and optimal weight initialization, the article conveys:

It has been obvious since the 1980s that backpropagation through deep autoencoders would be very effective for nonlinear dimensionality reduction, provided that computers were fast enough, data sets were big enough, and the initial weights were close enough to a good solution. All three conditions are now satisfied.

Talking with the Royal Society in 2016 titled “Deep Learning“, Geoff commented that Deep Belief Networks were the start of deep learning in 2006. This success, especially in speech recognition, prompted both the neural network and speech recognition sectors to take heed in 2009 titled “Acoustic Modeling using Deep Belief Networks“, achieving state of the art results. 

It was the results that made the speech recognition and the neural network communities take notice of the use of “deep” as a differentiator on previous neural network techniques that probably resulted in the name change.

The descriptions of deep learning in the Royal Society talk are very backpropagation-centric as you would expect. Interestingly, Hinton gives 4 reasons why backpropagation (read “deep learning”) did not take off last time around in the 1990s. The first two points resonated highly with Andrew Ng’s comments about datasets being too small and computers being too slow.

What Was Actually Wrong With Backpropagation in 1986?
Slide by Geoff Hinton, all rights reserved.

Deep Learning as Scalable Learning Across Domains

Deep learning has shown to particularly excel in scenarios where inputs and often outputs are analog. This means that rather than relying on a few quantities values in tabular format, deep learning has shown to thrive when dealing with pixel data from images, documents of text data or files of audio data.

Yann LeCun: The Visionary Behind Convolutional Neural Networks (CNNs)

Currently serving as the Vice-President, Chief AI Scientist at Meta, Yann LeCun is the father of the network architecture that excels at object recognition in image data called the Convolutional Neural Network (CNN). This technique has and continues to see great success because like multilayer perceptron feedforward neural networks, the technique scales with data and model size and can be trained with backpropagation.

This biases his definition of deep learning as the development of very large CNNs, which have had great success on object recognition in photographs.

During a 2016 presentation at Lawrence Livermore National Laboratory titled “Accelerating Understanding: Deep Learning, Intelligent Applications, and GPUs” LeCun described deep learning as the pursuit of hierarchical representations and further goes on to say that it as a scalable approach to building object recognition systems:

deep learning [is] … a pipeline of modules all of which are trainable. … deep because [has] multiple stages in the process of recognizing an object and all of those stages are part of the training

Deep Learning = Learning Hierarchical Representations
Slide by Yann LeCun, all rights reserved.

Jurgen Schmidhuber: The Innovator Behind Long Short-Term Memory Networks (LSTMs)

We bring another father to the discussion, Jurgen Schmidhuber, celebrated for his creation of the Long Short-Term Memory Network (LSTM), a type of recurrent neural network.

Schmidhuber has expressed his reservations about labeling the domain as “deep learning” in his 2014 paper titled “Deep Learning in Neural Networks: An Overview“. He comments on the problematic naming of the field and the differentiation of deep from shallow learning. He also interestingly describes depth in terms of the complexity of the problem rather than the model used to solve the problem.

At which problem depth does Shallow Learning end, and Deep Learning begin? Discussions with DL experts have not yet yielded a conclusive response to this question. […], let me just define for the purposes of this overview: problems of depth > 10 require Very Deep Learning.

Demis Hassabis and the Rise of DeepMind

Demis Hassabis, the visionary behind DeepMind, which was later acquired by Google, heralded a revolutionary fusion of deep learning and reinforcement learning. This combined breakthrough was able to handle complex learning problems like game playing, famously demonstrated in playing Atari games and the game Go with Alpha Go.

In keeping with the naming, they called their new technique a Deep Q-Network, combining Deep Learning with Q-Learning, subsequently dubbing the expansive domain as “Deep Reinforcement Learning”.

In their 2015 nature paper titled “Human-level control through deep reinforcement learning” they comment on the importance of deep neural networks and how it played a pivotal role in their breakthrough and highlighted the need for hierarchical abstraction.

To achieve this,we developed a novel agent, a deep Q-network (DQN), which is able to combine reinforcement learning with a class of artificial neural networks known as deep neural networks. Notably, recent advances in deep neural networks, in which several layers of nodes are used to build up progressively more abstract representations of the data, have made it possible for artificial neural networks to learn concepts such as object categories directly from raw sensory data.

Now coming to bring great minds together, what stands as a landmark paper in the realm of deep learning, Yann LeCun, Yoshua Bengio and Geoffrey Hinton published a paper in Nature titled simply “Deep Learning“. In it, they open with a clean definition of deep learning highlighting the multi-layered approach.

Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction.

Later the multi-layered approach is described in terms of representation learning and abstraction.

Deep-learning methods are representation-learning methods with multiple levels of representation, obtained by composing simple but non-linear modules that each transform the representation at one level (starting with the raw input) into a representation at a higher, slightly more abstract level. […] The key aspect of deep learning is that these layers of features are not designed by human engineers: they are learned from data using a general-purpose learning procedure.

Although it is a nice and generic description, this holistic description encapsulates the essence of most artificial neural network algorithms, offering a good note to end on.

Summary

In this post you discovered that deep learning is just very big neural networks on a lot more data, requiring bigger computers.

Although early approaches published by Hinton and collaborators focus on greedy layerwise training and unsupervised methods like autoencoders, modern state-of-the-art deep learning is focused on training deep (many layered) neural network models using the backpropagation algorithm. The most popular techniques are:

• Multilayer Perceptron Networks.
• Convolutional Neural Networks.
• Long Short-Term Memory Recurrent Neural Networks.

I hope this has cleared up what deep learning is and how leading definitions fit together under the one umbrella.

If you have any questions about deep learning or about this post, ask your questions in the comments below and I will do my best to answer them.