🤦🏽♂️ [Correction] EV#412
Ooops! A small correction and additional comment.
Well, it isn’t just large language models that make mistakes. Humans can too! Including this human.
In EV#412, I missed out a single word which inverted the meaning of a paragraph. In the section on the horse race between OpenAI and Google (including Deepmind), I meant to write:
Horse race. Given the feverish uptake of LLMs like GPT-3, why did Deepmind not get out of the gate first? Jonathan Godwin, a former Deepmind engineer, argues it comes down to company culture. OpenAI is an engineering organisation, whereas Deepmind is a research organisation. My view is this is credible: the engineering company ships products, such as ChatGPT. Products are evolving answers to open-ended questions. A research company builds answers to more well-defined research questions. AlphaFold addresses the protein folding problem. Product/engineering companies ship, learn, ship, learn, ship, learn. A far cry from the discipline of scientific research.
I left out the word “not” (in bold above). My point was to ask, how come OpenAI released ChatGPT before Deepmind released something similar?
I will expand on my take on the argument in this correction and add another wrinkle.
Culture matters a great deal, true. But it may also be that Deepmind is more cognisant of the risks of putting these models out there and into products than OpenAI is. Or it could be that OpenAI believes that the best way to derisk the technology is to put it out there and garner billions of use cases and put increasingly improved guardrails around it. Certainly, this seems to be the approach with Bing.1
Locked down
Of course, OpenAI’s services are delivered by API. This means that the firm can clamp down on any unfortunate uses of its models. The leak of LLaMa, the powerful LLM released by Meta is perhaps more pernicious. LLaMa’s model is available on torrents to download. Anyone could access it and fine-tune it how they like. A 4chan bot anyone? Or a bot fine-tuned for phishing attacks?
LLaMa’s breakthrough was that it, in many cases, performed as well, or better, than DeepMind’s and OpenAI’s best models. (The table below shows LLaMa’s model performance on various benchmarks. Its smallish 13b parameter model does very well.)
Meta achieved this with an optimised model with one-tenth of the parameters of GPT-3. It could run on locally a single GPU, meaning that compute is less of a blocker to playing with and personalising the model. Further optimisations could see LLMs with this kind of performance running on desktop computers. And then the technology will proliferate pretty wildly.
Anyway, that’s all I wanted to add. Sorry for the typo.
The full issue (with the typo corrected) is available here.
best
A
Although this seems like a Sisyphean task.