62,666
edits
Changes
no edit summary
|-
| 1943 || || || "The first case of neural networks was in 1943, when neurophysiologist Warren McCulloch and mathematician Walter Pitts wrote a paper about neurons, and how they work. They decided to create a model of this using an electrical circuit, and therefore the neural network was born."<ref name="dataversity.net"/> "In 1943, a human neural network was modeled with an electrical circuit. In 1950, the scientists started applying their idea to work and analyzed how human neurons might work."<ref name="javatpoint.comu"/>
|-
| 1949 || || || "First step toward prevalent ML was proposed by Hebb, in 1949, based on a neuropsychological learning formulation. It is called Hebbian Learning theory. With a simple explanation, it pursues correlations between nodes of a Recurrent Neural Network (RNN). It memorizes any commonalities on the network and serves like a memory later."<ref name="erogol.comt"/>
|-
| 1950 || || Turing's Learning Machine || [[wikipedia:Alan Turing|Alan Turing]] proposes a 'learning machine' that could learn and become artificially intelligent. Turing's specific proposal foreshadows [[wikipedia:genetic algorithms|genetic algorithms]].<ref>{{cite journal|last1=Turing|first1=Alan|title=COMPUTING MACHINERY AND INTELLIGENCE|journal=MIND|date=October 1950|volume=59|issue=236|pages=433–460|doi=10.1093/mind/LIX.236.433|url=http://mind.oxfordjournals.org/content/LIX/236/433|accessdate=8 June 2016}}</ref> "Alan Turing creates the “Turing Test” to determine if a computer has real intelligence. To pass the test, a computer must be able to fool a human into believing it is also human."<ref name="forbes.com">{{cite web |title=A Short History of Machine Learning |url=https://www.forbes.com/sites/bernardmarr/2016/02/19/a-short-history-of-machine-learning-every-manager-should-read/#756b4b2615e7 |website=forbes.com |accessdate=20 February 2020}}</ref><ref name="javatpoint.comu"/>
|-
| 1970 || || Automatic Differentation (Backpropagation) || [[wikipedia:Seppo Linnainmaa|Seppo Linnainmaa]] published the general method for automatic differentiation (AD) of discrete connected networks of nested differentiable functions.<ref name="lin1970">[[wikipedia:Seppo Linnainmaa|Seppo Linnainmaa]] (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis (in Finnish), Univ. Helsinki, 6-7.</ref><ref name="lin1976">[[wikipedia:Seppo Linnainmaa|Seppo Linnainmaa]] (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2), 146-160.</ref> This corresponds to the modern version of backpropagation, but is not yet named as such.<ref name="grie2012">Griewank, Andreas (2012). Who Invented the Reverse Mode of Differentiation?. Optimization Stories, Documenta Matematica, Extra Volume ISMP (2012), 389-400.</ref><ref name="grie2008">Griewank, Andreas and Walther, A.. Principles and Techniques of Algorithmic Differentiation, Second Edition. SIAM, 2008.</ref><ref name="schmidhuber2015">[[wikipedia:Jürgen Schmidhuber|Jürgen Schmidhuber]] (2015). Deep learning in neural networks: An overview. Neural Networks 61 (2015): 85-117. [http://arxiv.org/abs/1404.7828 ArXiv] </ref><ref name="scholarpedia2015">[[wikipedia:Jürgen Schmidhuber|Jürgen Schmidhuber]] (2015). Deep Learning. Scholarpedia, 10(11):32832. [http://www.scholarpedia.org/article/Deep_Learning#Backpropagation Section on Backpropagation]</ref>
|-
| 1970 || || || "There had been not to much effort until the intuition of Multi-Layer Perceptron (MLP) was suggested by Werbos[6] in 1981 with NN specific Backpropagation(BP) algorithm, albeit BP idea had been proposed before by Linnainmaa [5] in 1970 in the name "reverse mode of automatic differentiation"."<ref name="erogol.comt"/>
|-
| 1979 || || Stanford Cart || Students at Stanford University develop a cart that can navigate and avoid obstacles in a room <ref>{{cite web|last1=Marr|first1=Marr|title=A Short History of Machine Learning - Every Manager Should Read|url=http://www.forbes.com/sites/bernardmarr/2016/02/19/a-short-history-of-machine-learning-every-manager-should-read/#2a1a75f9323f|website=Forbes|accessdate=28 Sep 2016}}</ref> " Students at Stanford University invent the “Stanford Cart” which can navigate obstacles in a room on its own."<ref name="forbes.com"/>
|-
| 1981 || || Explanation Based Learning || Gerald Dejong introduces Explanation Based Learning, where a computer algorithm analyses data and creates a general rule it can follow and discard unimportant data.<ref>{{cite web|last1=Marr|first1=Marr|title=A Short History of Machine Learning - Every Manager Should Read|url=http://www.forbes.com/sites/bernardmarr/2016/02/19/a-short-history-of-machine-learning-every-manager-should-read/#2a1a75f9323f|website=Forbes|accessdate=28 Sep 2016}}</ref> "Gerald Dejong introduces the concept of Explanation Based Learning (EBL), in which a computer analyses training data and creates a general rule it can follow by discarding unimportant data."<ref name="forbes.com"/>
|-
| 1981 || || || "There had been not to much effort until the intuition of Multi-Layer Perceptron (MLP) was suggested by Werbos[6] in 1981 with NN specific Backpropagation(BP) algorithm, albeit BP idea had been proposed before by Linnainmaa [5] in 1970 in the name "reverse mode of automatic differentiation"."<ref name="erogol.comt"/>
|-
| 1982 || Discovery || Recurrent Neural Network || [[wikipedia:John Hopfield|John Hopfield]] popularizes [[wikipedia:Hopfield networks|Hopfield networks]], a type of [[wikipedia:recurrent neural network|recurrent neural network]] that can serve as [[wikipedia:content-addressable memory|content-addressable memory]] systems.<ref>{{cite journal|last1=Hopfield|first1=John|title=Neural networks and physical systems with emergent collective computational abilities|journal=Proceedings of the National Academy of Sciences of the United States of America|date=April 1982|volume=79|pages=2554–2558|url=http://www.pnas.org/content/79/8/2554.full.pdf|accessdate=8 June 2016|doi=10.1073/pnas.79.8.2554}}</ref><ref name="dataversity.net"/><ref name="import.ioe"/>
| 1985 || || NetTalk || A program that learns to pronounce words the same way a baby does, is developed by Terry Sejnowski.<ref>{{cite web|last1=Marr|first1=Marr|title=A Short History of Machine Learning - Every Manager Should Read|url=http://www.forbes.com/sites/bernardmarr/2016/02/19/a-short-history-of-machine-learning-every-manager-should-read/#2a1a75f9323f|website=Forbes|accessdate=28 Sep 2016}}</ref> " Terry Sejnowski invents NetTalk, which learns to pronounce words the same way a baby does." "In 1985, Terry Sejnowski and Charles Rosenberg invented a neural network NETtalk, which was able to teach itself how to correctly pronounce 20,000 words in one week."<ref name="javatpoint.comu"/>
|-
| 1985–1986 || || || "There had been not to much effort until the intuition of Multi-Layer Perceptron (MLP) was suggested by Werbos[6] in 1981 with NN specific Backpropagation(BP) algorithm, albeit BP idea had been proposed before by Linnainmaa [5] in 1970 in the name "reverse mode of automatic differentiation". Still BP is the key ingredient of today's NN architectures. With those new ideas, NN researches accelerated again. In 1985 - 1986 NN researchers successively presented the idea of MLP with practical BP training"<ref name="erogol.comt"/>|-| 1986 || Discovery || Backpropagation || The process of [[wikipedia:backpropagation|backpropagation]] is described by [[wikipedia:David Rumelhart|David Rumelhart]], [[wikipedia:Geoff Hinton|Geoff Hinton]] and [[wikipedia:Ronald J. Williams|Ronald J. Williams]].<ref>{{cite journal|last1=Rumelhart|first1=David|last2=Hinton|first2=Geoffrey|last3=Williams|first3=Ronald|title=Learning representations by back-propagating errors|journal=Nature|date=9 October 1986|volume=323|pages=533–536|url=http://elderlab.yorku.ca/~elder/teaching/cosc6390psyc6225/readings/hinton%201986.pdf|accessdate=5 June 2016|doi=10.1038/323533a0}}</ref><ref name="slideshare.netr">{{cite web |title=A brief history of machine learning |url=https://www.slideshare.net/bobcolner/a-brief-history-of-machine-learning |website=slideshare.net |accessdate=24 February 2020}}</ref>|-| 1986 || || || "At the another spectrum, a very-well known ML algorithm was proposed by J. R. Quinlan [9] in 1986 that we call Decision Trees, more specifically ID3 algorithm."<ref name="erogol.comt">{{cite web |title=Brief History of Machine Learning |url=http://www.erogol.com/brief-history-machine-learning/ |website=erogol.com |accessdate=24 February 2020}}</ref>
|-
| 1989 || Discovery || Reinforcement Learning || Christopher Watkins develops [[wikipedia:Q-learning|Q-learning]], which greatly improves the practicality and feasibility of [[wikipedia:reinforcement learning|reinforcement learning]].<ref>{{cite journal|last1=Watksin|first1=Christopher|title=Learning from Delayed Rewards|date=1 May 1989|url=http://www.cs.rhul.ac.uk/~chrisw/new_thesis.pdf}}</ref>
|-
| 1992 || Achievement || Machines Playing Backgammon || Gerald Tesauro develops [[wikipedia:TD-Gammon|TD-Gammon]], a computer [[wikipedia:backgammon|backgammon]] program that utilises an [[wikipedia:artificial neural network|artificial neural network]] trained using [[wikipedia:temporal-difference learning|temporal-difference learning]] (hence the 'TD' in the name). TD-Gammon is able to rival, but not consistently surpass, the abilities of top human backgammon players.<ref>{{cite journal|last1=Tesauro|first1=Gerald|title=Temporal Difference Learning and TD-Gammon|journal=Communications of the ACM|date=March 1995|volume=38|issue=3|url=http://www.bkgm.com/articles/tesauro/tdl.html}}</ref>
|-
| 1995 || || || "One of the most important ML breakthrough was Support Vector Machines (Networks) (SVM), proposed by Vapnik and Cortes[10] in 1995 with very strong theoretical standing and empirical results. That was the time separating the ML community into two crowds as NN or SVM advocates."<ref name="erogol.comt"/>
|-
| 1995 || Discovery || Random Forest Algorithm || Tin Kam Ho publishes a paper describing [[wikipedia:Random forest|Random decision forests]].<ref>{{cite journal|last1=Ho|first1=Tin Kam|title=Random Decision Forests|journal=Proceedings of the Third International Conference on Document Analysis and Recognition|date=August 1995|volume=1|pages=278–282|doi=10.1109/ICDAR.1995.598994|url=http://ect.bell-labs.com/who/tkh/publications/papers/odt.pdf|accessdate=5 June 2016|publisher=IEEE|location=Montreal, Quebec|isbn=0-8186-7128-9}}</ref>
|-
| 1995 || Discovery || Support Vector Machines || [[wikipedia:Corinna Cortes|Corinna Cortes]] and [[wikipedia:Vladimir Vapnik|Vladimir Vapnik]] publish their work on [[wikipedia:support vector machines|support vector machines]].<ref name="bhml">{{cite web|last1=Golge|first1=Eren|title=BRIEF HISTORY OF MACHINE LEARNING|url=http://www.erogol.com/brief-history-machine-learning/|website=A Blog From a Human-engineer-being|accessdate=5 June 2016}}</ref><ref>{{cite journal|last1=Cortes|first1=Corinna|last2=Vapnik|first2=Vladimir|title=Support-vector networks|journal=Machine Learning|date=September 1995|volume=20|issue=3|pages=273–297|doi=10.1007/BF00994018|url=http://download.springer.com/static/pdf/467/art%253A10.1007%252FBF00994018.pdf?originUrl=http%3A%2F%2Flink.springer.com%2Farticle%2F10.1007%2FBF00994018&token2=exp=1465109699~acl=%2Fstatic%2Fpdf%2F467%2Fart%25253A10.1007%25252FBF00994018.pdf%3ForiginUrl%3Dhttp%253A%252F%252Flink.springer.com%252Farticle%252F10.1007%252FBF00994018*~hmac=133f5211871b237411d6dcc05047fc16cdc99abc25ab4e74be863808ea53bfd7|accessdate=5 June 2016|publisher=Kluwer Academic Publishers|issn=0885-6125}}</ref>
|-
| 1997 || || IBM Deep Blue Beats Kasparov || IBM’s Deep Blue beats the world champion at chess.<ref>{{cite web|last1=Marr|first1=Marr|title=A Short History of Machine Learning - Every Manager Should Read|url=http://www.forbes.com/sites/bernardmarr/2016/02/19/a-short-history-of-machine-learning-every-manager-should-read/#2a1a75f9323f|website=Forbes|accessdate=28 Sep 2016}}</ref>
|-
| 1995 1997 || Discovery || Support Vector Machines LSTM || [[wikipedia:Corinna CortesSepp Hochreiter|Corinna CortesSepp Hochreiter]] and [[wikipedia:Vladimir VapnikJürgen Schmidhuber|Vladimir VapnikJürgen Schmidhuber]] publish their work on [[wikipedia:support vector machines|support vector machines]].<ref name="bhml">{{cite web|last1=Golge|first1=Eren|title=BRIEF HISTORY OF MACHINE LEARNING|url=http://www.erogol.com/briefinvent Long-history-machine-learning/|website=A Blog From a Human-engineer-being|accessdate=5 June 2016}}</ref>short term memory recurrent neural networks,<ref>{{cite journal|last1=CortesHochreiter|first1=CorinnaSepp|last2=VapnikSchmidhuber|first2=VladimirJürgen|title=SupportLONG SHORT-vector networksTERM MEMORY|journal=Machine LearningNeural Computation|date=September 19951997|volume=209|issue=38|pages=273–297|doi=10.1007/BF009940181735–1780|url=http://downloaddeeplearning.springercs.com/static/pdfcmu.edu/467pdfs/art%253A10.1007%252FBF00994018Hochreiter97_lstm.pdf?originUrl|doi=http%3A%2F%2Flink10.springer1162/neco.com%2Farticle%2F101997.1007%2FBF00994018&token2=exp=1465109699~acl=%2Fstatic%2Fpdf%2F467%2Fart%25253A109.1007%25252FBF009940188.pdf%3ForiginUrl%3Dhttp%253A%252F%252Flink.springer.com%252Farticle%252F10.1007%252FBF00994018*~hmac=133f5211871b237411d6dcc05047fc16cdc99abc25ab4e74be863808ea53bfd7|accessdate=5 June 2016|publisher=Kluwer Academic Publishers|issn=0885-61251735}}</ref>greatly improving the efficiency and practicality of recurrent neural networks.
|-
| 1997 || Discovery || LSTM || [[wikipedia:Sepp Hochreiter|Sepp Hochreiter]] "Little before, another solid ML model was proposed by Freund and [[wikipedia:Jürgen Schmidhuber|Jürgen Schmidhuber]] invent Long-short term memory recurrent neural networks,<ref>{{cite journal|last1=Hochreiter|first1=Sepp|last2=Schmidhuber|first2=Jürgen|title=LONG SHORT-TERM MEMORY|journal=Neural Computation|date=Schapire in 1997|volume=9|issue=8|pages=1735–1780|url=http://deeplearningprescribed with boosted ensemble of weak classifiers called Adaboost.csThis work also gave the Godel Prize to the authors at the time.cmuAdaboost trains weak set of classifiers that are easy to train, by giving more importance to hard instances.edu/pdfs/Hochreiter97_lstmThis model still the basis of many different tasks like face recognition and detection.pdf|doi"<ref name=10.1162/neco.1997.9.8"erogol.1735}}<comt"/ref> greatly improving the efficiency and practicality of recurrent neural networks.
|-
| 1998 || || MNIST database || A team led by [[wikipedia:Yann LeCun|Yann LeCun]] releases the [[wikipedia:MNIST database|MNIST database]], a dataset comprising a mix of handwritten digits from [[wikipedia:American Census Bureau|American Census Bureau]] employees and American high school students.<ref>{{cite web|last1=LeCun|first1=Yann|last2=Cortes|first2=Corinna|last3=Burges|first3=Christopher|title=THE MNIST DATABASE of handwritten digits|url=http://yann.lecun.com/exdb/mnist/|accessdate=16 June 2016}}</ref> The MNIST database has since become a benchmark for evaluating handwriting recognition.
|-
| 1999 || || || "Computer-aided diagnosis catches more cancers. Computers can’t cure cancer (yet), but they can help us diagnose it. The CAD Prototype Intelligent Workstation, developed at the University of Chicago, reviewed 22,000 mammograms and detected cancer 52% more accurately than radiologists did."
|-
| 2001 || || || "Another ensemble model explored by Breiman [12] in 2001 that ensembles multiple decision trees where each of them is curated by a random subset of instances and each node is selected from a random subset of features."<ref name="erogol.comt"/>
|-
| 2002 || || Torch Machine Learning Library || [[wikipedia:Torch (machine learning)|Torch]], a software library for machine learning, is first released.<ref>{{cite journal|last1=Collobert|first1=Ronan|last2=Benigo|first2=Samy|last3=Mariethoz|first3=Johnny|title=Torch: a modular machine learning software library|date=30 October 2002|url=http://www.idiap.ch/ftp/reports/2002/rr02-46.pdf|accessdate=5 June 2016}}</ref>
|-
| 2004 || || || "The second is the decrease in the cost of parallel computing and memory. This trend was discovered in 2004 when Google unveiled its MapReduce technology"<ref name="medium.comw">{{cite web |title=History of deep machine learning |url=https://medium.com/mindsync-ai/history-of-deep-machine-learning-1842dc3a4507 |website=medium.com |accessdate=21 February 2020}}</ref>
|-
| 2005 || || || " The 3rd rise of NN has begun roughly in 2005 with the conjunction of many different discoveries from past and present by recent mavens Hinton, LeCun, Bengio, Andrew Ng and other valuable older researchers. "<ref name="erogol.comt"/>
|-
| 2006 || || The Netflix Prize || The [[wikipedia:Netflix Prize|Netflix Prize]] competition is launched by [[wikipedia:Netflix|Netflix]]. The aim of the competition was to use machine learning to beat Netflix's own recommendation software's accuracy in predicting a user's rating for a film given their ratings for previous films by at least 10%.<ref>{{cite web|title=The Netflix Prize Rules|url=http://www.netflixprize.com/rules|website=Netflix Prize|publisher=Netflix|accessdate=16 June 2016}}</ref> The prize was won in 2009. "In 2006, Netflix offered $1M to anyone who could beat its algorithm at predicting consumer film ratings. The BellKor team of AT&T scientists took the prize three years later, beating the second-place team by mere minutes"<ref name="cloud.withgoogle.com"/>
===What the timeline is still missing===
* [http://www.erogol.com/brief-history-machine-learning/]
* [https://samsungnext.com/whats-next/a-brief-history-of-ai-and-machine-learning/]
