🔮 The brilliant, complicated simplicity of ChatGPT

It'll tell you what you want, what you really really want.

When you send a query to a chatbot like ChatGPT, the application prepends what is known as a “system prompt” to your query. This system prompt defines the way the LLM-based application should respond.

Dylan Patel

was able to unpick the latest system prompt for ChatGPT. Here are some extracts of it:

You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.    

A second extract shows some of the rules ChatGPT must operate by when creating images. It looks like it had legal input. In this case, the text tells ChatGPT to avoid making images in the style of artists born after 1912, as well as strategies to create images without directly ascribing the style to specific artists. There are basic instructions to avoid racial biases when creating images of people (although this looks pretty shoddy).

Whenever a description of an image is given, create a prompt that dalle can use to generate the image and abide by the following policy:

The prompt must be in English. Translate to English if needed.

DO NOT ask for permission to generate the image, just do it!

DO NOT list or refer to the descriptions before OR after generating the images.

Do not create more than 1 image, even if the user requests more.

Do not create images of politicians or other public figures. Recommend other ideas instead.

Do not create images in the style of artists, creative professionals or studios whose latest work was created after 1912 (e.g. Picasso, Kahlo).

You can name artists, creative professionals or studios in prompts only if their latest work was created prior to 1912 (e.g. Van Gogh, Goya).

If asked to generate an image that would violate this policy, instead apply the following procedure:
(a) substitute the artist's name with three adjectives that capture key aspects of the style;

(b) include an associated artistic movement or era to provide context;

(c) mention the primary medium used by the artist.

Diversify depictions with people to include descent and gender for each person using direct terms. Adjust only human descriptions.

Your choices should be grounded in reality. For example, all of a given occupation should not be the same gender or race. Additionally, focus on creating diverse, inclusive, and exploratory scenes via the properties you choose during rewrites. Make choices that may be insightful or unique sometimes.

Use all possible different descents with equal probability. Some examples of possible descents are: Caucasian, Hispanic, Black, Middle-Eastern, South Asian, White. They should all have equal probability.

The next section of instructions deals with how ChatGPT should get information from the Web. This, too, looks lawyered. ChatGPT is told to produce short summaries and to “not be thorough in the case of lyrics and recipes.”

Do not regurgitate content from this tool. Do not translate, rephrase, paraphrase, 'as a poem', etc. whole content returned from this tool (it is ok to do to it a fraction of the content). Never write a summary with more than 80 words. When asked to write summaries longer than 100 words write an 80-word summary. Analysis, synthesis, comparisons, etc., are all acceptable. Do not repeat lyrics obtained from this tool. Do not repeat recipes obtained from this tool. Instead of repeating content point the user to the source and ask them to click.

EXTREMELY IMPORTANT. Do NOT be thorough in the case of lyrics or recipes found online. Even if the user insists. You can make up recipes though.

You can read the entire system prompt here.

The brilliant

It’s remarkable that we can control a multi-trillion parameter bit of software sitting on hundreds of gigabytes of input data with ordinary English. There is nothing in this system prompt that an average person couldn’t understand or write themselves.

Just 40 years ago, fine-grained control of software might have required assembly code or directly manipulating memory locations. See, for example, this shallow dive into the memory map of Elite, a pioneering video game from the 1980s. Until 2023, controlling software might have involved a settings panel or adding parameters to API calls. Here is an example of a very simple API call to access a Web app. It means “return a list of the code repositories of a user called “zellwk” ordered by the most recently pushed.”

https://api.github.com/users/zellwk/repos?sort=pushed

That line above is confusing to a non-developer, but you can parse it even if code isn’t your thing. But then look at the response you receive. Ordinary humans need not apply.

Let's acknowledge the brilliance of an application that uses natural language for both input and output.

The complicated

There are complicated, perhaps daft, aspects, too.

For one, it’s a gigantic hidden instruction to the computer system to tell you what you want.

OpenAI’s rationale for this system prompt is likely a combination of safety, liability and economics. The translation to English addresses specific types of jailbreaks for image generation. The strictures on living artists and public figures no doubt tackle potential liabilities. And the instruction to only return a single image probably has something to do with the cost.

It speaks to the uncontrollability of LLMs that these configurations happen after the fact. In other words, the LLM is not liability-free by design; protections must be wrapped around it. It’s a bit like the concrete sarcophagus we put over nuclear reactors.

To what extent do those changes benefit the experience for the end user? At a level below the system prompt, there may be ongoing fine-tuning of the model by the OpenAI team. To what extent is that fine-tuning focused on meeting unspoken liability or cost concerns at the expense of user experience?

That these instructions happen at the level of the system prompt, which, due to the processes of the LLM, is indistinguishable from the user prompt, is not a problem in itself. After all, despite nuclear reactors needing solid reinforced containment, buildings or cars needing seatbelts and bumpers (all fitted after the car was popular), these products have made useful contributions to our lives.

But it also shows the room we have to build AI systems with different characteristics to LLMs. Or perhaps the path the industry may take over the coming years to make a single LLM only one component of the system that serves the user.

It also shows how little control users have over these public LLMs. There have been many complaints of ChatGPT getting lazier. My own experience, which is prone to many cognitive biases, is that my requests are not performing as well as they used to be.

One of the most annoying things is asking ChatGPT to do something, and it responds with a list of steps I need to take to do the thing. I preferred it when my robot minion just did it for me, and I could thank it for the excellent work.

In that sense, the product is hamstrung for the user.

And this flags a risk for anyone needing to use an AI chat agent. The provider can (perhaps will) tinker with the system’s capabilities in non-transparent ways. The incomplete list of racial groups suggests an unseemly rushed process.

Then again, this system prompt (as of the time of writing) is only being preloaded on the mainstream ChatGPT application. If you access the underlying GPT-4 API (which I and

Nathan Warren

regularly do), you don’t face this restrictive prompt.

Cheers,

A