Brief History of Machine Learning

Brief History of Machine Learning

1946 – 2017

Before we deep dive into Machine Learning as a due diligence we should know where we have been, before we know where we are going.
As you will see it is not different from any other histories, it has it’s ups and downs, in spite of the current boom and as we go along I’ll let you decide if it’s a hype or a paradigm shift of epic proportions.

The Wikipedia Timeline of machine learning starts at 1763, with the Discovery of The Underpinnings of Bayes’ Theorem, but we’ll start our journey with a more practical 20th-century date, the invention of the computer in 1946.


The Invention of the computer might be as important in human history as the invention of the fire, but AI has a great chance to take its place.
ENICA (Electronic Numerical Integrator and Computer) was the first electronic general-purpose programmable computer, built by the US Army and enhanced by my fellow Hungarian, John von Neumann (Hungarian: Neumann János Lajos), to make it programmable. He wrote programs to verify the feasibility of the H-bomb design, developed by another fellow Hungarian, Edward Teller (Hungarian: Teller Ede)



The press heralded ENIC as a “Giant Brain” and it motivated Alan Turing to design the so-called Turing-Test to detect artificial intelligence. To pass the test, the machine should make a human believe that it is another human being, instead of a computer. It still has not passed.


Arthur Samuel wrote the first computer program that was a learning machine, which learned to play checkers. His algorithms used a heuristic search memory to learn from experience.
By the mid- 1970’s his program was beating capable human players.


At The Dartmouth Workshop Marvin Minsky, John McCarthy, Claude Shannon, and Nathan Rochester proposed the term “Artificial Intelligence”.


Frank Rosenblatt designed the first artificial neural network, the Perceptron, commissioned by the US Office of Naval Research to solve visual recognition tasks.

The New York Times reported the Perceptron to be: “the embryo of an electronic computer that will be able to walk, talk, see, write, reproduce itself and be conscious of its existence”

Mark I Perceptron
Mark I Perceptron


Cover and Hart wrote the Nearest Neighbor algorithm, a huge milestone and the birth of computer pattern recognition.

KNN Classification
KNN Classification

“Winter Is Coming!”

1974 – 1980, The First AI Winter

Research hype led to funds being dried up and caused little to no progress.


Stanford students build Stanford Cart, the first mobile autonomous robot capable of moving around and avoiding obstacles.

Stanford Cart 1979
Stanford Cart 1979

1980 – 1987, The AI Summer

The introduction of the rule-based Expert Systems, which was quickly adopted by corporations and generated renewed interest in Machine Learning.


Explanation-Based Learning (EBL) was introduced. Analyzed training data and created general rules allowing the less important data to be discarded. 

1985 – Neural Network Breakthrough

Researchers (re)discovered the Backpropagation algorithm, this allowed more complex and powerful neural networks with hidden layers to be trained. As a result, it resurrected research and got everyone excited about Neural Nets as the model for the brain. And yet again, hype and pipe dreams went into overdrive.

Neural Network
Neural Network

“Winter Is Coming!”, again

1987 – 1993, Second AI Winter

Neural networks went out of favor (again), due to a lack of good theories and a tendency to overfit.


A Statistical Approach to Machine Learning, Support Vector Machine (SVM) is the popular new kid on the block. SVM is a good candidate for rigorous mathematical analysis and achieves a state-of-the-art performance.

Support Vector Machine (SVM)
Support Vector Machine (SVM)


IBM’s Deep Blue vs Chess Grandmaster Gary Kaspárov. Machine Wins!

IBM’s Deep Blue vs Chess Grandmaster Gary Kaspárov
IBM’s Deep Blue vs Chess Grandmaster Gary Kaspárov


Present Day, Explosion and Commercial Adoption
Big Data, fast computing, matured Neural net models reinvigorated the interest for Machine Learning. Digital Transformation, Big Data with Machine Learning is incorporated into company processes, products, and services in order to get ahead of the competition. 

Geoffrey Hinton, who has a great Neural Networks for Machine Learning course on Coursera, invented the phrase Deep Learning as a buzzword, or a re-branding of neural networks to explain the new architectures of profound neural networks which are capable of learning much better models.

Geoffrey Hinton - Deep Learning
Geoffrey Hinton – Deep Learning


The IBM Watson computer won the Jeopardy TV show, in which participants answer questions in natural language.


Google’s Jeff Dean and Andrew Ng from Stanford, who has a great Machine Learning course on Coursera, started the Google Brain Project. They used the entire Google infrastructure to develop a deep neural network which detected patterns in images and videos.

Google Brain Project
Google Brain Project

Geoffrey Hinton’s team wins the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) contest, by a large margin, using a Deep Neural Network (DNN) that has reached maturity and getting the attention it deserves.
This lead to the current explosion of Machine Learning based on DNNs.


Google X Laboratory built a DNN, running on a cluster of 16,000 computers, that recognized a cat based on 10 million images from YouTube.

GoogleX Cat with Andrew Ng
GoogleX Cat with Andrew Ng


DeepMind, a British deep learning startup, designed a Deep Reinforcement Learning model to play Atari games that can beat human experts.

Atari Games
Atari Games


Facebook develops DeepFace DNN, which can recognize people like humans do.

Google buys DeepMind.


Amazon introduced Amazon Machine Learning.

Microsoft announced the Distributed Machine Learning Toolkit (DMTK), a framework to solve machine learning problems efficiently on a cluster of GPUs and computers.

Elon Musk and Sam Altman, among others, found the non-profit organization OpenAI, providing it with one billion dollars with the objective of ensuring that artificial intelligence has a positive impact on humanity.


Microsoft releases Computational Network Toolkit (CNTK), its open source deep learning toolkit.

Google’s AlphaGo AI beats professional Go player Lee Se-dol. The algorithm can make creative moves that they had never seen before.


Google announces TensorFlow.

Microsoft releases Microsoft Cognitive Toolkit (previously known as CNTK), its open source deep learning toolkit.

Today, we are experiencing a third explosion in artificial intelligence.
It is driven by the massive amount of data and challenging problems that require entirely new approaches for the effective solutions.
It is creating whole new markets and producing great changes in the strategies of small, medium, and large businesses alike.
The increasing amount of data will keep scientists and researchers busy for a while and they will keep coming back with groundbreaking new ideas till we reach the singularity.
Is the third winter coming? I highly doubt it, why? Technology adoption and tangible results have reached critical mass.



One Reply to “Brief History of Machine Learning”

Leave a Reply

Your email address will not be published. Required fields are marked *