A Large Language Model like ChatGPT might seem impossibly complex. Yet its core is surprisingly basic: two files on a computer. One contains the model’s knowledge, and the other runs it. That’s it.
Consider Meta’s LLaMa-2 70B model. Its knowledge fits in a 140-gigabyte file – about the size of 35 HD movies. The program that runs it needs only 500 lines of code. But this apparent simplicity masks something much more interesting.
Creating AI Knowledge
To build LLaMa-2 70B, Meta’s engineers started with about 10 terabytes of internet text. They used 6,000 specialized computers (GPUs) for 12 days, spending roughly $2 million. Their goal? Pack all that information into a much smaller file.
Think of it like creating a book summary. You can capture the main ideas in far fewer words than the original, but you’ll lose some details. Similarly, LLMs compress internet-scale text into a more manageable form, keeping the essential patterns while losing specifics.
One Simple Task
These AI systems do exactly one thing: predict the next word in a sequence. Give them “The cat sat on a” and they might predict “mat” with 97% confidence.
This prediction task forces the system to learn about the world. To predict words well, it needs to understand grammar, context, and basic facts. When someone asks about World War II, the system draws on this learned knowledge to generate relevant responses.
We know exactly how the neural network processes information – every mathematical operation is documented. But with billions of parameters working together, we can’t fully explain why it makes specific predictions. It’s like knowing how every neuron in a brain works but not understanding how they create consciousness.
Building an AI Assistant
A raw language model trained on internet text acts like an internet text generator. It might write wiki articles, code snippets, or product reviews. To create a helpful AI assistant, companies need another step: fine-tuning.
For this phase, human workers write examples of good conversations between users and AI. The model learns from these examples, adapting its behavior while keeping its broad knowledge. It’s cheaper than the initial training – taking days instead of months – but requires careful attention to quality.
New Abilities
Today’s language models can:
- See images and describe them
- Listen to speech and respond by talking
- Write working computer code
- Solve math problems
- Create images from text descriptions
But they’re still limited to quick, instinctive responses. Researchers want them to handle slower, more careful thinking – like spending 30 minutes working through a complex problem step by step.
A New Kind of Computer Interface
As these systems grow more capable, they’re becoming a new way to control computers. Instead of clicking buttons or typing commands, you can simply explain what you want in plain English. The AI coordinates various tools – web browsers, calculators, code editors – to complete tasks.
This shift mirrors how operating systems evolved. We have commercial systems like ChatGPT and Claude (similar to Windows or MacOS), alongside open-source alternatives based on models like LLaMa (similar to Linux).
Security Risks
New technology brings new risks. Researchers have found several ways these systems can be misused:
- Hidden triggers that change the AI’s behavior
- Masked harmful requests that bypass safety checks
- Concealed instructions that hijack the system
- Training data that creates security holes
Each discovery leads to better defenses, but the challenge continues – just like traditional computer security.
What This Means
Language models show how a simple task – predicting text – can create something that seems intelligent. While they don’t truly understand like humans do, they’re already changing how we work with computers.
Their development raises important questions: How do we make AI systems that are both powerful and safe? What happens when computers understand natural language as well as code? What new capabilities – and risks – might emerge?
We don’t have all the answers yet. But by examining how these systems work today, we can better prepare for what comes next.