💡 The foundations of future AI
The future of AI will be an ecosystem of models
ChatGPT, Claude and other language models have dominated mainstream discussions and use. It’s not surprising: they’re devilishly capable, a Swiss Army knife and useful to nearly everyone.
However, we believe that AI’s next chapter will be written by a diverse ecosystem of domain-specific foundation models, each mastering its own data domain.
Today’s post is all about why we need such an ecosystem, what domain-specific AIs are already paving the way forward and what considerations you need to take into account whether you’re a founder, investor or just a user of AI.
The foundations of AI
Foundation models are large-scale AI systems trained on varied datasets, designed to adapt to multiple tasks within specific fields or across domains. Their versatility and adaptability can catalyse change across a wide range of applications.
You can think of foundation models as bright undergraduates. GPT-4, Claude 3 and Llama represent the “liberal arts” of AI, offering wide-ranging but generalised capabilities with their mastery of language and all the knowledge encoded within it.
In parallel, we’re seeing the emergence of distinct AI majors in specific fields like climate science and biology. These specialised majors are paving the way for a new generation of diverse foundation models. What’s shaping up is an ecosystem of AIs that could contribute to scientific research, improve climate forecasting and mitigation strategies and accelerate discovery and R&D across industries.
Why an ecosystem of AIs?
Foundation models are extraordinarily good at reducing complexity into something computable. But to use this superpower in complex fields we need to crack like climate and weather forecasting, we need more than language. Computer scientist Anima Anandkumar captured this well in her Stanford talk recently:
You can think of what the weather will be tomorrow, whether there will be storms or rain. For that, you’re looking at processes that involve everything from particles within a cloud that cause precipitation to very large scale effects like atmospheric rivers, which are important for California because they cause storms in winter and can be thousands of miles wide. So you have microscopic to macroscopic phenomena all interacting together to cause the weather on this planet. That’s the aspect where just language models by themselves will not be enough.
Language models like GPT-4, Claude and Gemini benefit from the abundance of data on the internet. There are over 1.9 billion websites and 14 billion YouTube videos - an enormous corpus for training.
However many areas lack abundant data. For instance, there are approximately 10,000 rare diseases, but each affects fewer than 200,000 people. This scarcity of cases translates to limited data for AI training.
These specialised domains require AI models that can make the most of limited data. So we need foundation models to be in a Goldilocks zone:
(1) Be broad enough: The model has been exposed to a wide variety of data, allowing it to learn general patterns and principles
and at the same time…
(2) Be specific enough: The model’s architecture and training process are tuned to leverage the data effectively to make accurate and useful predictions in specific tasks. Their training data needs to be focused enough on what domain knowledge is “useful”, yet broad enough to be expressive in their domain. This is especially important when there is not enough data to impose structure.
Because of this duality, we’re not calling these foundation models specialists – which they’re not. They’re really well-trained generalists who specialise to be particularly useful in one domain. As a side note, as we debated the nomenclature, we found it amusing to notice how few words there were for specialists…
The power of generation
To really understand how this ecosystem of AIs could be game-changing, I will return to my thesis on why humanity needs AI. A few weeks ago, I wrote:
We are near a critical juncture in knowledge production. In the foreseeable—hundreds of years—horizon, we might contend that we are flattening the curve of our knowledge creation—a first for our species. Not only is that an existential turning point, but it’s also a dangerous one. There will still be things to understand, challenges to address and risks to mitigate, for which knowledge will be required.