François Chollet believes that the current superintelligence benchmarks are too easy to pass.

François Chollet believes that the current benchmarks for superintelligence are too easily surpassed

Abstract

François Chollet created Keras and is currently doing AI research at Google. He recently criticized the way we measure progress towards artificial superintelligence. What he means is: if your AI only needs to beat a small group of humans—who may not be very good—that doesn’t indicate much.

Chollet’s definition of ASI is quite straightforward: to defeat all humans in a particular domain. By this standard, we have already achieved this in chess and Go.

The significance of this criticism is that it challenges the industry’s claims regarding AI capabilities and what constitutes true progress.

Analysis

Chollet’s perspective comes from his years of experience in deep learning and AI assessment. He is concerned that flashy demonstrations that beat a few humans may mislead people into thinking we are close to genuine superintelligence. This could skew investment and research.

Chess and Go are excellent references. AlphaGo didn’t just beat certain humans—it defeated all humans. This is exactly the standard Chollet hopes to see adopted in other domains as well.

This relates to a larger discussion: how to transition from narrow AI (which excels at only one task) to systems that surpass humans in all aspects. If this viewpoint gains more recognition, labs working on cutting-edge models may face pressure to adopt stricter benchmarks.

There is also an ongoing tension: researchers like Chollet from the open-source world hold more cautious views, while industry hype cycles tend to exaggerate.

Impact Assessment

• Importance: Medium
• Category: Technical Insight, AI Research, Industry Trends