Three Eras of Machine Learning and Predicting the Future of AI

Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). Researchers studied trends in the most readily quantified factor – compute. They show : before 2010 training compute grew in line with Moore’s law, doubling roughly every 20 months. Deep Learning started in the early 2010s…
Three Eras of Machine Learning and Predicting the Future of AI


Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). Researchers studied trends in the most readily quantified factor – compute.

They show :


before 2010 training compute grew in line with Moore’s law, doubling roughly every 20 months.

Deep Learning started in the early 2010s and the scaling of training compute has accelerated, doubling approximately every 6 months.

In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute.

Based on these observations they split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era . Overall, the work highlights the fast-growing compute requirements for training advanced ML systems.

They have detailed investigation into the compute demand of milestone ML models over time. They make the following contributions:


1. They curate a dataset of 123 milestone Machine Learning systems, annotated with the compute it took to train them.


2. They tentatively frame the trends in compute in terms of three distinct eras: the Pre Deep Learning Era , the Deep Learning Era and the Large-Scale Era . They offer estimates of the doubling times during each of these eras.


3. They extensively check their results in a series of appendices, discussing alternate interpretations of the data, and differences with previous work

They studied trends in compute by curating a dataset of training compute with more than 100 milestone ML systems and used this data to analyze how the trend has grown over time.


The findings seem consistent with previous work, though they indicate a more moderate scaling of training compute.


In particular, they identify an 18-month doubling time between 1952 and 2010, a 6-month doubling time between 2010 and 2022, and a new trend of large-scale models between late 2015 and 2022, which started 2 to 3 orders of magnitude over the previous trend and displays a 10-month doubling time.

One aspect they have not covered in this article is another key quantifiable resource used to train Machine Learning models — data. They will be looking at trends in dataset size and their relationship to trends in compute in future work.

Read More

Total
0
Shares
Leave a Reply

Your email address will not be published.

Related Posts
SpaceX Starlink Rapid Software Updates Beat Russian Jamming Attacks
Read More

SpaceX Starlink Rapid Software Updates Beat Russian Jamming Attacks

SpaceX Starlink was able to rapidly upgrade whenever a new Russian jamming threat appeared. This enabled Starlink to maintain Ukraine’s communications. Dave Tremper, the Pentagon’s director of electronic warfare said that the US military has to adopt the speed of SpaceX response. “We have to be able to change our electromagnetic posture, to be able…
Fact Mix 831: Clemency
Read More

Fact Mix 831: Clemency

A “disorientating and frenetic” mix from the co-founder of Manchester’s Mutualism party and label. Clemency’s music is a blend of heads-down club music and experimental sonics, a reflection of both her teenage background in improvised live electronics and her current role in Manchester’s club scene. Her debut EP, References, released on Finn’s 2 B Real…
Heinali performs live from a bomb shelter in Lviv, Ukraine
Read More

Heinali performs live from a bomb shelter in Lviv, Ukraine

The composer, sound artist and modular synthesist will perform new material based loosely on the on 12th century organa of the Notre-Dame school, which is taken from a new album he was unable to finish due to the Russian invasion of Ukraine. Heinali is the musical moniker of Oleh Shpudeiko, a composer, sound artist and…
Tesla Virtual Power Plant in the USA
Read More

Tesla Virtual Power Plant in the USA

Those with Tesla Solar Power and Tesla Powerwalls who have opted into Tesla’s Virtual Power Plant program get $2 per kwh when their power utility uses the Tesla Virtual Power plant. CORRECTION: Matt Smith has indicating that these VPP power emergencies will not be that frequent and Tesla does not take a cut of the…