LLM Scaling Has Hit a Wall; What’s Next For ChatGPT?

Nov. 13, 2024

While OpenAI chief Sam Altman is drumming up hype thatAGIis just around the corner, new reports suggest that LLM scaling has hit a wall. The predominant view in the AI field has been that training larger models on massive amounts of data and compute resources will lead to greater intelligence.

In fact,Ilya Sutskever, former chief scientist at OpenAI and founder ofSafe Superintelligence Inc., has been a strong advocate for scaling models as the path to unlocking intelligence. Responding toReuters, Sutskever now says, “results from scaling up pre-training – the phase of training an AI model that uses a vast amount of unlabeled data to understand language patterns and structures – have plateaued.”

Recently,The Informationreported that OpenAI has changed its strategy as its next big “Orion” model didn’t deliver better results as anticipated. The jump from GPT-3.5 to GPT-4 was huge, but OpenAI employees who tested the upcoming model say that the improvement from GPT-4 to Orion is marginal. In tasks like coding, it doesn’t outperform prior GPT models.

OpenAI is now focused on inference scaling as a new way to improve model performance on ChatGPT. Noam Brown, a researcher at OpenAI, says that inference scaling improves the model performance significantly.

Recently, hetweeted, “OpenAI’s o1 thinks for seconds, but we aim for future versions to think for hours, days, even weeks. Inference costs will be higher, but what cost would you pay for a new cancer drug? For breakthrough batteries? For a proof of the Riemann Hypothesis? AI can be more than chatbots.”

Google and Anthropic are also working on a similar technique to improve model performance through inference scaling. However, François Chollet, a researcher at Google, argues that scaling LLMs alone won’t lead to generalized intelligence. Yann LeCun, chief AI scientist at Meta, similarly says that LLMs are not sufficient for achieving AGI.

As companies run out of data to train larger models, they are looking for novel techniques to improve LLM performance. Now whether AGI is genuinely around the corner or it’s simply hype is something only time will tell.

Passionate about Windows, ChromeOS, Android, security and privacy issues. Have a penchant to solve everyday computing problems.