Many AI industry watchers have been speculating about the exponential performance increase of new AI models for years. Some of that AI has been re-released recently. “scaling law” The optimism of the past has been replaced with fears that we are already at a plateau when it comes to the abilities of large language models.
A weekend report by The Information summarized the fears of a number insiders within OpenAI. The Information reported that OpenAI’s unnamed researchers said Orion, its codename for the next full-fledged release of the company’s model, shows a smaller performance leap than the one seen in recent years between GPT-3 to GPT-4. In fact, on certain tasks, the upcoming model is actually faster than its predecessor. “isn’t reliably better than its predecessor,” According to OpenAI researchers who were not named in the article.
Ilya Sutskever (co-founder of OpenAI, who left earlier this year) added to the concern that LLMs had reached a plateau with traditional pre-training. Sutskever told Reuters “the 2010s were the age of scaling,” Adding additional computing resources and data to the same basic training methods can lead to impressive improvements.
“Now we’re back in the age of wonder and discovery once again,” Sutskever said Reuters. “Everyone is looking for the next thing. Scaling the right thing matters more now than ever.”
What’s next?
According to experts and insiders quoted in these and other articles, a large part of the problem is a lack new, high-quality textual data on which new LLMs can train. The model makers have probably already selected the easiest to use texts from the vast amount of information available on the Internet and in published books.