The Token

In 1866 a copper cable reached across the floor of the Atlantic, and for the first time a merchant in London could speak to New York in minutes instead of weeks. The price was brutal: as much as ten dollars a word in the first years, around two hundred dollars in today’s money, and a hundred dollars for a twenty-word message, near two thousand dollars today.

So the merchants stopped writing English.

They bought code books. In 1878 a San Francisco shipping firm printed A. Chesebrough’s private telegraphic code, and in it one word carried a whole sailing instruction. The word was:

“Abdication”

which translates to:

“To Cork for orders to discharge in the U.K., with option of charterers to order her to Liverpool direct before sailing, at 2s. 6d. less; and 5s. extra, if ordered to the continent to discharge between Havre and Hamburg, both ports inclusive.”

Forty-two words of instruction for the price of one. The meaning never travelled down the wire at all. It sat in identical books at both ends, and the word was a key into them. The most famous of these books, the ABC Telegraphic Code, pressed thousands of sentences into single words for financiers, merchants and shipowners, and a clerk in a telegraph office would receive a string of nonsense, open the book, and copy out the meaning line by line.

The clerk did not need to understand the message. The clerk needed the codebook, and the discipline to look things up.

Keep that clerk in mind. You are about to meet a modern one.

When you type a question into an AI assistant, the natural picture is of something reading it. Your words go in, the machine takes them the way you would, letter by letter, word by word, and understanding follows. It has read the whole internet, after all. Reading is the one thing it must be good at.

That is not what happens.

The model never sees your words. It never sees a single letter. The most widely read entity in human history cannot read.

Before your sentence reaches the model, a small program chops it into pieces called tokens and replaces each piece with a number from a fixed codebook. The codebook for OpenAI’s current models holds around two hundred thousand entries. A common word is usually one token. A rarer word is broken into several. The model receives the numbers, does arithmetic on them, and produces numbers in reply, which the codebook turns back into text for you.

The pieces are not words, and they were not chosen by a linguist. The codebook is built by scanning mountains of text and keeping whichever chunks appear most often. Chunks that earn their keep get a number. Everything else is assembled from smaller pieces, the way the Victorian clerk assembled a rare sentence from ordinary words when the book had no code for it.

Watch it happen. Run “Abdication” itself through one openly published codebook of 49,408 entries and it comes out as three numbers, because the word is too rare to have earned an entry of its own:

The word is rare. Its parts are everywhere, which is why all three numbers are low, and low numbers are the codebook’s oldest, commonest entries. Now feed in the full forty-two-word instruction that “Abdication” once carried, and it costs fifty-nine tokens. The word to is one token and appears six times, billed at full price each time. Liverpool and Hamburg appear often enough in modern text to hold whole entries. Havre, a port the modern text has mostly forgotten, shatters in two, and the old shilling-and-pence price of 2s. 6d. costs six tokens for eight characters. What the 1878 book compressed hardest, the modern codebook handles worst, because those phrases died out before the training text was written.

This one fact explains a family of oddities that otherwise look like stupidity.

It explains the famous strawberry problem. For a long time, models asked how many times the letter r appears in strawberry would confidently answer two. The model was not being careless. It had never seen the letters: it saw something like the chunks straw and berry, two numbers, and letters are invisible inside a number. Newer models mostly pass the test now, and the workaround is the giveaway: they spell the word out one letter at a time in their working, turning each letter into its own token, exactly as you would read a document to a colleague over a bad phone line.

It explains why your bill is counted in tokens. Every AI provider prices its service the way the cable company did, per chunk carried. You are not paying for thoughts. You are paying for traffic.

It explains why the machine is cheaper in English than in most other languages. The codebook keeps whatever appears most often in its training text, and its training text is mostly English, so English packs tightly and other languages shatter into fragments. Researchers at Oxford measured the gap and found that the same sentence can cost up to fifteen times more tokens in one language than another. The gap has narrowed as vendors rebuild their codebooks, and a newer OpenAI codebook cut token counts sharply for several Indian languages, but the tilt remains. The wire never charged everyone equally either.

And it explains the machine’s ghost words. In early 2023 two researchers, Jessica Rumbelow and Matthew Watkins, found tokens that broke the models that carried them. Strings like SolidGoldMagikarp, mostly old Reddit usernames, had earned a number in the codebook but almost never appeared in the text the model learned from. Asked to repeat them, models dodged, insulted the user, or answered about something else entirely. An entry in the book with no experience behind it is a word the clerk has never once looked up.

Two habits follow from the codebook, and both help straight away.

When the letters in something matter, spell them out. Ask the machine to write a name, a serial number or a reference code out character by character before it works with the thing. Each letter then becomes its own token, and the machine can finally see what you see.

When the bill or the speed surprises you, count chunks instead of words. A long pasted document costs thousands of tokens even when the machine needs only a few of them, and the same meaning in another language can cost several times what it costs in English. Paste the part you need. Part eight comes back to this when we look at the window it all has to fit through.

The machine is the best clerk the codebook has ever had. It has processed more text than any person who ever lived, and every character of it arrived as a number from the book. Like the clerk in the telegraph office, it never had to learn English. It learned the codebook instead.

Next week, the thought. You will learn what the machine is doing when it says it is thinking, why you are paying for writing you will never read, and what a Berlin horse has to do with it.