When we communicate with each other, we are unconsciously looking back into the past: Our brains references past interactions with human language, and as our mouth begins to form words, our brain is a step ahead, steering us towards intelligible words and sentences.
ChatGPT and other “Large Language Models” do the same thing, they look back and observe all the words in a sentence and then find related words and create a coherent and consistent next sentence.
If you’ve ever read sheet music as a musician, you know what this process of moving back and forwards feels like. If you want to get good at reading music, you are always reading slightly ahead of what you are playing or singing. So while we read the music our brains look back, steer our body, and then move forward again at a blinding speed to the notes we havent played yet.
But what is the language model actually doing? Is it just grabbing an arbitrary word out of a big pile of words and autocompleting? Not really, I’ll use music as a metaphor to explain what is going on.
Lets visualise two musicians, sitting in a room looking at the same sheet music for the first time. One of the musicians is an AI and the other is a human. What do they see?
The human musician looks at the clef, the time signature, the key, tempo and the expressive information. Then we have the notes themselves; each note has a harmonic value, represented by where it is written on the stave and a rhythmic value, represented by how it is written. We humans read these notes from left to right, line by line, from the top of the page to the bottom, and as we go our brain steers our fingers, mouths, lips or vocal chords in the direction indicated on the sheet music.
The AI doesn’t read like this at all. It’s completely Alien in how it consumes all this information. It reads all the contextual information and all the notes on the page, all at once!
Then it breaks everything on the page up into smaller segments and groups everything that is statistically similar together. From here the AI starts looking for patterns in all these small statistical blocks of music. It sees the rhythmic and harmonic value of each note as a number. (side note: jazz musicians do this all the time, they look at relative harmony instead of literal harmony) Then the AI starts looking for context and importance of the notes by statistically analysing how close certain notes are + the frequency of those relationships and the distances and grouping. This allows the AI to determine which notes deserve more attention and have the most impact on the composition.
(Remember its doing this all at once with everything on the page, not note for note).
Then the AI continues to shed its humanity as it puts all these correlations together in tightly packed strings of numbers that it then relates back to the individual notes. A spectrum of context and statistical relations squished together into something unthinkable for a human, called an embedding.
Because the AI consumed everything all at once and because it has attached these embeddings to all the blocks of information on the page, if we change a note or ask it to write some new ones the AI can very quickly reference the entire piece of music + all those numbers and embeddings and define coherent next steps or make a change over the entirety of the composition without slipping up.
Thanks to the language model, the music has become interlinked and interwoven in a very dense manner.
So that’s the literal side of things, for both the human and the AI musician, information that is not explicitly written on the page of music is also present:
When we humans read music, we have a back catalogue of practiced scales and chords, a grid of tonal information that the music either fits into — in the form of consonance or dissonantly- moves away from. Then we also have cultural context, we just know why Mozart sound European, or why Antonio Carlos Jobim sound Brazilian. There is a library of (not directly related) contextual information our brains reference while we read and perform this peice of music that allow us to have a deeper understanding of what is written on the page.
The AI does the same, but it has a back catalogue of billions of pages of music and it turns out that the whole process I described, of finding out which notes it should pay attention to, just gets better and better the bigger that back catalogue is. So when the AI musician can reference all music ever committed to paper, suddenly the information and context its able to bring to the table goes well beyond human abilities.
The most amazing part of this evolutionary technology is that the AI succesfully navigates a labyrinth of data, it can find the right information or word amongst hunderds of billions of densly packed pathways and connections and it does it at a speed and scale that is staggering and amazing.
Except for two crucial aspects:
Efficiency and Judgement
Efficiency: The amount of energy the AI requires to do something as trivial as, lets say, read and perform Twinkle Twinkle Little Star, is just silly.
And Judgement: Well good news for team humanity. We humans might not be able to rewrite Breaking Bad scripts in 10 seconds flat, while making Walt speak like Homer Simpson, but we can do something much cooler.
We can say “no, that’s a pointless waste of time, I’m going to go do something useful instead”(like playing music)….and we know exactly why.
Thanks for reading,
Kai