🔮 Llama’s second coming; unreliable infrastructure; polysilicon; partying monks, quantum chaos ++ #432

Your insider guide to AI and exponential technologies

and

Jul 23, 2023

Hi, I’m Azeem Azhar. As a global expert on exponential technologies, I advise governments, some of the world’s largest firms, and investors on how to make sense of our exponential future. Every Sunday, I share my view on developments that I think you should know about in this newsletter.

Latest posts

If you’re not a subscriber, here’s what you missed recently:

Sunday chart: Open-source strikes back

Meta has introduced a new cog in the global AI machine: Llama 2. It is a high-performing model, deemed generally more helpful than its open-source (OS) counterparts and on par with GPT-3.5, OpenAI’s last-but-one closed-source offering from mid-2022.

That puts this open-source model just a year behind the best closed-source models. Given the licence terms, Llama 2 is available to developers provided they don’t have more than 700m monthly active users (which excludes Meta’s biggest rivals). One way of thinking about Llama 2 is as a public good for developers around the world to research and build applications upon. It will likely spread very quickly, further strengthening the OS movement in the competition between different approaches to developing AI. When I had drinks with Yann LeCun back in April, he emphasised the point that most internet infrastructure software, from DNS to web servers and databases, is open, not closed-source. In May, an employee warned Google that OS was the ultimate competition for both Google and OpenAI.

In return for deploying the model free of charge, Meta gets to own the infrastructure on which brilliant developers build. Plus, not only do they build, but they also test, debug, and give feedback. This crowdsourced oversight might be a decisive way to make a model safe and reliable: “given enough eyeballs, all bugs are shallow”1. Meta is already reaping the rewards of its earlier OS development, with PyTorch to be further scaled on Microsoft’s Azure.

If the release spurs enough innovation, this could prove an inflection point not only for Meta and open-source, but also for AI more broadly. Hopefully, the ecosystem it creates will make AI safer, more democratically accessible, and more accountable.

Key (AI) reads

Microsoft and the unreasonable effectiveness of natural language. Microsoft has announced Bing Chat enterprise, an AI ecosystem to surround their productivity tools. While Bing will remain free, Copilot will cost $30/month per user. In his keynote speech, CEO Satya Nadella brilliantly explained the changes that LLMs introduce to digital life. First, it is rapidly making natural language the primary interface to using computers. Second, AI adds a layer of reasoning over data that is different from automations we’ve been using for several decades, such as Excel calculations.

GPT-4 may be random. There have been several reports of the outputs from GPT-3.5 and GPT-4 varying a lot across time, as well as rumours of a drop in performance. Some researchers have found that ChatGPT’s performance and behaviour can vary greatly in short periods of time, but it’s difficult to tell whether the outputs really are getting worse. The variability, along with the opacity around the actual architecture of OpenAI’s models, makes it hard for developers who are used to an even, reliable output from APIs. As I described in January, our current AI moment is marked by how simultaneously high-performing yet inconsistent LLMs are; they are unreliable infrastructure.

See also: McKinsey and Cohere have announced a collaboration. I’ve spoken a couple of times with Aidan, Cohere’s CEO and one of the co-authors of the original BERT paper, and I’m quite impressed with him.

Market data

The price of polysilicon (a key input in solar modules) has dropped by 78% over the past year.

Energy received the most VC funding (by sector) in Q2 2023, followed by healthtech.

In 2022, Tesla reported an impressive 17% operating margin, a level of profitability only surpassed by Porsche among non-speciality automakers. However, now in Q2 2023, Tesla’s operating margin has fallen to 9.6% due to price reductions aimed at staying competitive with rivals.

How safe are EVs? In Norway, where 21% of cars are electric, ICE cars were responsible for 35 per 100,000 passenger car fires. Electric? Only 4 per 100,000.

Meta has 689 publicly accessible models on Hugging Face, a platform for open-source machine learning models. Google is not far behind with 591 models, while Microsoft ranks third with 252 models.

Short morsels to appear smart at dinner parties

🧭 Finally, a trial for a drug to slow down memory loss and cognitive decline in Alzheimer’s patients has been successful.

🐣 A brave new world? New ways of procreating are nearing.

🛸 Newton’s first law of motion seems to break down in the quantum realm.

🇨🇦 Canada launched a work permit aimed at skilled foreign workers in the US and got a whopping 10,000 applications.

🚗 More details on Tesla’s $1bn supercomputer to manage the data produced by autopilot and full self-driving modes. (Elsewhere, Cerebas announces a 4 exaflop AI supercomputer in partnership with a UAE-based AI service. The rig has already been used to train Arabic LLMs.)

🪨 A new hope for geothermal: a breakthrough suggests it may be a viable energy source after all.

🥀 The quest to resuscitate extinct plants.

🎉 Mediaeval monks used to party hard: they incarnated the original carnival spirit.

End note 🎶

I was travelling this week and stayed in a hotel without a Bluetooth speaker. Wanting a bit of music, I looked for an easy alternative and popped into the Apple Store round the corner to pick up a HomePod Mini. Now, I wouldn’t normally bring this up, but my experience was such that it deserves a public service announcement.

The HomePod Mini is a dreadful product. The sound quality is indescribably awful, I can’t quite describe how poor the rendition is but easily the worst sound I have heard. That would be sufficient for this product to receive “nul points”. But the device (it isn’t a speaker) is a Trojan horse for the Apple ecosystem, not just Apple Music but also Siri and a panoply of weird home services. The best part of the experience was the hassle-free return which took less than two minutes the next day. This is a product to avoid.

Annoying also to stay in a hotel without a sound system. A Marshall Acton II only costs a couple of hundred bucks, much less than a pointless TV. It fills a room with full, but not overpowering, bass and realistic mid-range and treble frequencies. The soundstage is expansive and decent. Reasonable hotels ought to make such an investment.

Cheers,
A

Exponential View

Discussion about this post