Core Automation
Jerry Tworek, OpenAI’s former VP of Research who left the company earlier this month, is raising for his new start-up, Core Automation.
The start-up’s roadmap is ambitious, and starts with nothing less than developing a new AI architecture to be used in lieu of the transformer. Standard methods for training models “up to and including gradient descent” will go out the window. A new model named Ceres (after the Roman goddess of fertility) will be trained using these new methods. The training process will be hyper-efficient, using 100x less data than today’s frontier models. And Ceres will be able to learn through real-world experience - because Core Automation also intends to crack continual learning.
As if that weren’t enough, Core Automation’s goals after developing Ceres include automating development of future AI products, constructing self-replicating factories, and “potentially building biomachines to automatically create custom designs - or even terraform planets”.
We know that Ilya Sutskever thinks that he’ll be able to crack continual learning in 5 to 20 years. Jerry doesn’t have that kind of time. After all, his former employer intends to fully automate AI research by March 2028, which might lead to recursive self-improvement (RSI). With RSI also squarely on Core Automation’s roadmap (what did you think “automating development of future AI products” meant? vibes? papers? essays?), the company will need to execute on its goals very quickly - potentially within months! - lest the likes of OpenAI and Anthropic achieve RSI first and scupper its plans.
After all, the bet on automated AI research ultimately means betting on the bitter lesson - i.e., that “general methods that leverage computation are ultimately the most effective, and by a large margin”. Once the GPUs powering OpenAI’s and Anthropic’s automated AI researchers start humming, it appears quite possible that human-led AI research will quickly fall by the wayside and that the big unsolved problems of AI (such as continual learning) will become much more easily solvable if desired. Viewed from this perspective, the key dilemma emerges: given the race between the frontier labs to fully automate AI research and potentially achieve RSI, does it still make sense to “front-load” human-led research of new AI architectures and transformative ideas, or is it more prudent to instead curtail these initiatives and spend the freed-up resources on reaching the goal of automating AI research even faster?
Jerry Tworek’s bet is on humans for another few years. His former employer’s bet is exactly the opposite.
We shall soon find out which one of the two is right.
I am not privy to what research is occurring inside the frontier labs, but if I were to wager, I imagine that the human research must lead until the architectures are built in such a way that recursive self improvement (at huge scale) can be employed to uplift the rest of the way. Yann LeCun is likely correct about auto-regressive models not being the way to AGI, and other research approaches, such as world models, are likely key. But the continuous learning model applied to inference time compute at massive scale, may lead us to the next level. My two cents.