Generalizing AI

 

The big picture

When OpenAI revealed ChatGPT in late 2022, people clambered to test it. They asked complicated questions, requested poems and got precisely what they wanted: in one case, instructions for removing a peanut butter sandwich from a VCR written in the style of the King James Bible.

And before ChatGPT, the internet was flooded with AI-generated art. Text-to-image generators like Stability AI’s Stable Diffusion and OpenAI’s DALL-E 2 stunned people by responding to written prompts with photorealistic images.

This generated content is part of one of the biggest step changes in the history of AI: the introduction of pretrained models with remarkable task adaptability.

It began with a landmark innovation in AI model architecture by Google researchers in 2017. Since then, tech companies and researchers have been supersizing AI by increasing the sizes of models and training sets. The result? Powerful pretrained models, often called “foundation models,” that offer unprecedented adaptability within the domains they’re trained on.

With foundation models, businesses can start to approach many tasks and challenges differently, shifting focus from building their own AI to learning to build with AI.

A foundation for intelligence breakthroughs

OpenAI’s GPT-3, released in 2020, was the largest language model in the world. It taught itself to perform tasks it had never been trained on and outperformed models that were trained on those tasks. Since then, companies like Google, Microsoft, and Meta have created their own large language models.

To define this new class of AI, researchers from the Stanford Institute for Human-Centered Artificial Intelligence coined the term “foundation model.” They generally defined them as large AI models trained on a vast quantity of data with significant downstream task adaptability.

Some are working to expand foundation models beyond language and images to include more data modalities. Meta, for instance, developed a model that learned the “language of protein” and accelerated protein structure predictions by up to sixtyfold.

Many efforts are underway to make building and deploying foundation models easier. Rapidly growing compute requirements—and the associated costs and expertise needed to handle this scale—are the biggest barriers today. And even after a model is trained, it’s expensive to run and host its downstream variations.