🔮 Grounding AI; social bank runs; Bye-du; redesigning Wikipedia ++ #414

Mar 19

Hi, I’m Azeem Azhar. I advise governments, some of the world’s largest firms, and investors how to make sense of our exponential future. Every Sunday, I share my view on developments that I think you should know about in this newsletter.

In today’s edition…

Why grounding is the key to Microsoft’s success with GPT-4,
How OpenAI changed the debate about safety and openness,
Is the collapse of Silicon Valley Bank the sign of social media’s madness, or intelligence?

⏱️ Today’s read time: 10 minutes

Sunday chart: GPT-4 breaking records

Share this chart

What a week. OpenAI releases an LLM that can easily pass the American Bar exams, create recipes from a picture of the inside of a fridge, and code a website within minutes. The chart above shows the percentile rank that GPT4 achieved on various exams. It is still doing badly in English literature, but SAT-level Math is within reach.

But we shouldn’t ignore Anthropic’s launch of Claude, another LLM designed with a different safety architecture to GPT-4, as well as Google opening up its PaLM LLM via an API. The first apps using these technologies are already available.

But one of the most important product announcement was Microsoft’s reveal of Copilot. Copilot brings a magical interface to the office apps we use every day: natural language to conduct data explorations in Excel; a Word processor that writes drafts for you and ‘smart agent’ sitting with you during meetings. The product demo is worth watching. Of course, no demo survives contact with actual usage. Reality will be more messy. But this tooling could have a massive productivity impact as I wrote earlier in the week. (If you haven’t read this analysis, I recommend it.)

How will Microsoft prevent Copilot from going rogue, like LLMs are wont to do? One way is through “grounding”. Because Copilot has access to the data from your Microsoft apps, Microsoft can construct a local graph of entity relationships that are related to you. This graph can be used iteratively to ground (that is shape and filter) the responses from the LLM. I guess the theory is that LLM will be bound, in some sense, by your own view of the world.

Grounding is an important general concept in AI research. It is about connecting the abstract symbols that an AI manipulates to real-world entities. This should lead to more robust AI models. Copilot seems to help GPT-4 anchor back to the representations of the real entities that matter to the user.

But GPT-4 itself made some strides to tackle grounding. It is multimodal, in the sense that it can deal with image and text inputs to generate text outputs. This is a big deal as one of the world’s foremost AI thinkers told me privately. It’s a step towards addressing the symbol grounding problem, a long-standing challenge in AI research: how do words get their meaning? What gives a dog the qualities that make us understand it is a dog?

Multimodality can connect those real world attributes, the slinkiness of a cat, the tangle of cables, the gait of a flamingo, to the representations in the model. This is a critical step in building AI systems that are better at understanding human conversation as well as navigating the physical world.

My key reads

Bye-du: China tried to ride the ChatGPT wave to great disappointment. Baidu’s shares tumbled as the company announced its own version of ChatGPT, a dry and unimpressive unveiling. Chinese companies suffer from a lack of free speech. Bing declared its love, threatened, and happily explored all sorts of political theories. These bots make mistakes, the models are adjusted, and they (hopefully) become safer. Chinese chatbots don’t have the luxury of free experimentation and public feedback. Their makers need tight guardrails to ensure the bots avoid certain topics. See also, the UK government plans to double its investment in quantum computing, and set £900 million aside to build the “BritGPT” supercomputer.

OpenAI vs closed AI: OpenAI was founded as a non-profit, consecrated to the open and ethical development of AI. On Wednesday, as it unveiled GPT-4, it didn’t disclose model size, training data, or very much about the technology at all. Should you open your models to democratic scrutiny, while running the risk of unforeseen consequences that cannot be undone with the rolling back of the model? Or do you keep as much information as possible under wraps, protecting the technology from bad actors, and limiting the appearance of black swans? Stability’s Emad Mostaque, for example, takes the view that openness affords more social good. I’ll explore this question in future issues of the newsletter, but my tl;dr is that effective social licences and democratic oversight are vital.

Collective intelligence or group panic: The Atlantic’s James Surowiecki argues that the Silicon Valley Bank run was the first bank collapse in which social media has been a major player. He does make the valuable point, however, that social media enables panic, and for that frenzy to have real-world effects. It’s true that because of the scale and speed at which communication happens online, mediated by clever(ish) algorithms, a collective reaction to a story happens very fast (remember, Gamestop?), which might be bad news for the next financial crisis. Then again, this might just be markets being the most efficient version of themselves.

Some like it hot: Investment firm Ambienta published a great report showing that electrifying industrial heat is not only a great way to decarbonise this hard-to-abate sector; it’s also a trillion euro investment opportunity. Heat can be electrified directly (by converting electricity into heat) or indirectly, by using electricity to produce green fuels like hydrogen. They argue that the direct method is better, 1 TWh of electricity generates over twice as much heat when it is converted directly. It is the 21st century problem: investment needs to flow into deep tech, especially climate deep tech. Industrial heat is not sexy but it is important. (The best discussion I have heard on industrial heat is between Michael Liebreich and Silvia Madedu.)

Dig deeper: Expert commentary

This week, EV member Rodolfo Rosini brings us the provocative argument about the end of (computing) history. Essentially, we may be hitting the physical limits of computation, at the dawn of the AI revolution.

The future that machine learning promises simply cannot be delivered without a continued increase in number-crunching and data scaling. With the growing ubiquity of ML models in consumer technology, heralding a ravenous, and likely hyperbolic demand, for compute in other sectors, cheap processing is becoming the bedrock of productivity. The death of Moore’s Law could bring the consequences of the Great Computing Stagnation to bear.

Read the commentary here.

Market data

The entry price of Twitter’s API has gone from free to $500k a year pricing out academics.

Nearly half of all German bike sales are e-bikes. The market has quadrupled in 10 years to €7.4bn in sales.

First quarter VC funding for generative AI has nearly tripled to $2.3b in 2023, compared to 2022. This is represented in the current batch of companies at Y Combinator, where 50 out of the 218 companies are working on generative AI.

Bitcoin surged by over 30% this week in light of the SVB fallout.

Short morsels to appear smart at dinner parties

⚖️ AI could help us bypass the influence of lobbyists by writing some of our laws.

🇰🇷 South Korea is looking to develop the world’s largest chip-making base with Samsung leading investment with a $230 billion commitment over 20 years.

🍎 Regenerative education could be how to future-proof our schools, and life-long learning.

🐘 The challenge of redesigning the 15-year-old, decentralised Wikipedia.

🫀 Your heartbeat could impact your time perception, suggesting that rather than the sole task of the brain, it may come from a network within the body.

🌱 Reevaluating 1.5°C.

🧠 How do you make a brain (asking for a friend)?

End note

What a week. Yes, there were a load of huge announcements from the tech industry which will change the shape of the industry. But more than that, these announcements encapsulate a set of huge societal questions that we’ll need to grapple with. One couldn’t do justice to them in a single issue of the wondermissive, however wonderful.

Through Microsoft and Google, billions of people will soon have access to tools that dramatically reduce the cost of knowledge activities. As I argued in my book, and regularly in the newsletter, when you dramatically reduce the cost of an activity, you fundamentally change behaviours and the institutions predicated on that cost and those behaviours. A rocky ride, indeed.

I haven’t really been able to touch on the safety dimensions of these LLMs, or the questions of governance, or the issue of existential risk, or the question of whether these LLMs risk becoming agents (or whether someone will bolt a policy network to one of them and create an agent). Or, how the dramatic reduction in the cost of knowledge activities might kickstart productivity across economies.

But we’ll be returning to these questions, especially in the members-only commentaries.

In the meantime, you can enjoy some of my experiments with Midjourney, the image generator’s new release. It’s a nightmarish rendition of Da Vinci’s hands. And its attempt to visualise Stringfellow Hawke from 80’s cop show Airwolf. (Midjourney replaced Jan-Michael Vincent with a poor facsimile of Tom Selleck, mustache and all.)

Have a great week

Exponential View by Azeem Azhar