How Meaning Becomes Geometry
At the heart of every modern artificial intelligence system lies a deceptively simple idea: meaning can be represented as numbers. More specifically, words, sentences, images, and even abstract concepts can be converted into lists of numbers called vectors, and these vectors can be placed in a geometric space where distance and direction carry semantic significance. This process is called embedding, and it is one of the most foundational concepts in machine learning, large language models, and AI agents alike.
If you have ever wondered how a language model knows that “king” and “queen” are related, or how a search engine retrieves documents that match the meaning of your query rather than just the exact words, the answer lies in embeddings. They are the bridge between the messy, ambiguous world of human language and the precise, mathematical world that algorithms require.
This article will explain what embeddings are, how they work, why they matter, and walk you through concrete examples so you can build a deep conceptual understanding of this critical idea.
The Problem: Machines Don’t Understand Words
Computers operate on numbers. They can add, multiply, compare, and sort numbers with extraordinary speed, but they have no native ability to understand that “dog” and “puppy” are closely related while “dog” and “democracy” are not. To a computer, these are just different sequences of characters with no inherent relationship.
Early approaches to this problem used simple techniques like one-hot encoding. In this method, every word in a vocabulary gets assigned a unique position, and is represented as a vector of all zeros except for a single one at its position.
Example: Imagine a tiny vocabulary of just four words: “cat”, “dog”, “fish”, “car”. With one-hot encoding, each word becomes a four-dimensional vector:
“cat” = [1, 0, 0, 0]
“dog” = [0, 1, 0, 0]
“fish” = [0, 0, 1, 0]
“car” = [0, 0, 0, 1]
In this representation, every word is equally distant from every other word. The distance between “cat” and “dog” is the same as between “cat” and “car”. This is a fundamental failure because it encodes no semantic knowledge whatsoever. A model using one-hot vectors has no starting point for understanding that animals are more related to each other than to vehicles. Every relationship must be learned entirely from scratch, and the vectors themselves carry no meaning.
This limitation motivated the development of embeddings: dense, lower-dimensional vectors where the numbers themselves encode meaning.
What Exactly Is an Embedding?
An embedding is a learned mapping that converts a discrete item, such as a word, a token, or an entity, into a continuous vector of real numbers. Unlike one-hot vectors, embedding vectors are dense, meaning most or all of their values are non-zero, and they exist in a relatively low-dimensional space, typically ranging from 50 to several thousand dimensions depending on the model.
The key insight is that these vectors are not assigned randomly. They are learned through training, meaning the model adjusts the numbers over millions of examples until words that appear in similar contexts end up with similar vectors. This is grounded in a powerful linguistic hypothesis: words that occur in similar contexts tend to have similar meanings.
Example: After training, a model might produce these simplified three-dimensional embeddings:
“cat” = [0.82, 0.41, -0.15]
“dog” = [0.79, 0.45, -0.12]
“fish” = [0.65, 0.30, 0.55]
“car” = [-0.70, 0.85, 0.10]
Notice how “cat” and “dog” have very similar numbers across all three dimensions. Their first values are close (0.82 vs 0.79), their second values are close (0.41 vs 0.45), and their third values are close (-0.15 vs -0.12). This numerical closeness reflects their semantic closeness as both are common household pets that appear in similar sentences in training data. Meanwhile, “car” has very different numbers, particularly a negative first dimension (-0.70), placing it far away in the geometric space. The model has learned, purely from patterns in text, that cats and dogs belong together conceptually while cars do not.
How Embeddings Are Learned: The Training Process
Embeddings are not hand-crafted by humans. Instead, they emerge from training a neural network on a specific task, and the embedding vectors are adjusted through gradient descent to minimize the model’s errors.
The most intuitive way to understand this is through a prediction task. Imagine a model that is shown a sentence with one word removed and must predict the missing word. Consider the sentence: “The ___ chased the ball across the yard.” To predict the missing word, the model needs to understand what types of entities chase balls in yards. Through exposure to millions of such examples, the model learns that “dog”, “puppy”, “retriever”, and “cat” are all reasonable answers, while “table” or “democracy” are not.
As training progresses, words that serve as valid answers to similar fill-in-the-blank problems gradually have their vectors moved closer together in the embedding space, while words that never co-occur in similar contexts are pushed further apart. The result is a geometric space where proximity equals semantic similarity.
Example: Suppose during training the model encounters these contexts:
“The doctor prescribed ___” (medicine, antibiotics, rest)
“The nurse prescribed ___” (painkillers, treatment, medication)
“The engineer designed ___” (bridge, system, circuit)
Because “doctor” and “nurse” both appear before “prescribed” and are followed by medical terms, their embedding vectors will converge toward similar coordinates. The model does not know what a doctor or nurse is in real life, but it has learned that these words are functionally interchangeable in many sentence structures. “Engineer” appears in different contexts with different completion words, so its vector will drift to a different region of the space. This is how the geometry of the embedding space comes to mirror the structure of human concepts.
The Magic of Vector Arithmetic: When Math Captures Relationships
One of the most remarkable discoveries about embeddings is that they support meaningful arithmetic. You can add and subtract embedding vectors and get semantically valid results. This was one of the first strong pieces of evidence that embeddings capture deep structural relationships in language, not just surface-level word similarities.
Example: The most famous demonstration of vector arithmetic in embeddings is the analogy test:
vector(“king”) — vector(“man”) + vector(“woman”) = ?
The resulting vector lands closest to the vector for “queen” in the embedding space.
What is happening here mathematically is fascinating. When you subtract “man” from “king”, you are removing the component of the vector that represents masculinity. What remains is a vector that encodes the concept of royalty. When you then add “woman”, you reintroduce a gender component, but now it is the feminine one. The resulting vector therefore encodes royalty plus femininity, which corresponds to “queen”. This works because the embedding space has organized the gender relationship as a consistent directional offset. The vector from “man” to “woman” points in roughly the same direction as the vector from “king” to “queen”. The model has discovered an abstract relational pattern and encoded it as geometry.
This property extends to many other relationships. For instance, vector(“Paris”) — vector(“France”) + vector(“Germany”) yields a result close to vector(“Berlin”). The spatial offset between a country and its capital is approximately the same direction and magnitude across different examples. The embedding space has learned geography without ever seeing a map.
Measuring Similarity: Distance and Cosine Similarity
Once words are represented as vectors, we need a way to quantify how similar two vectors are. The two most common methods are Euclidean distance and cosine similarity, and understanding the difference between them is essential.
Euclidean distance measures the straight-line distance between two points in space, just like measuring the distance between two cities on a map. Cosine similarity, on the other hand, measures the angle between two vectors rather than the distance between their endpoints. Two vectors pointing in nearly the same direction have a cosine similarity close to 1, even if one is much longer than the other. Two vectors pointing in opposite directions have a cosine similarity close to -1.
Example: Consider three simplified two-dimensional embeddings:
Vector A (“happy”) = [2, 3]
Vector B (“joyful”) = [4, 6]
Vector C (“sad”) = [-2, -3]
The Euclidean distance between A and B is about 3.6 (they are far apart as points in space). But their cosine similarity is 1.0, because they point in exactly the same direction; B is simply a scaled-up version of A. This tells us they share the same semantic orientation. Meanwhile, C points in the exact opposite direction from both A and B, giving it a cosine similarity of -1.0 with either. In practice, cosine similarity is the more commonly used measure for embeddings because it captures semantic orientation regardless of vector magnitude. A word that appears frequently might have a longer vector due to training dynamics, but it still points in the same meaningful direction as a rarer synonym.
Embeddings in Large Language Models
In modern large language models, embeddings play an even more sophisticated role than simple word vectors. When text enters an LLM, the first thing that happens is tokenization, which breaks the input into sub-word units called tokens. Each token is then looked up in an embedding table, which is essentially a large matrix where each row corresponds to a token and contains its learned vector representation.
However, LLMs go far beyond static word embeddings. Through the transformer architecture’s attention mechanism, the model creates contextual embeddings. This means the same word can have different vector representations depending on the sentence it appears in.
Example: Consider the word “bank” in two different sentences:
Sentence 1: “I deposited money at the bank.”
Sentence 2: “I sat on the bank of the river.”
In a static embedding model, “bank” would have a single vector regardless of context, which is a significant limitation since the word has multiple meanings. In an LLM with contextual embeddings, the model produces different internal representations for “bank” in each sentence. In sentence 1, the surrounding words “deposited”, “money” cause the attention mechanism to pull the representation of “bank” toward the financial region of the embedding space, close to words like “account”, “savings”, and “finance”. In sentence 2, the words “sat”, “river” shift the representation toward the geographical meaning, close to “shore”, “riverbed”, and “waterside”. The model dynamically adjusts the embedding based on context, which is why LLMs can handle ambiguity, polysemy, and nuance far better than earlier approaches.
Embeddings as the Foundation for AI Agents
AI agents, systems that can perceive their environment, reason about it, and take actions to achieve goals, rely heavily on embeddings in ways that go beyond text understanding. Agents need to store and retrieve memories, compare their current situation against past experiences, and find relevant information from large knowledge bases. Embeddings make all of this possible.
When an AI agent needs to recall something relevant to its current task, it converts its current context into an embedding vector and then searches through its memory, which is also stored as embedding vectors, to find the entries that are closest in the geometric space. This is called semantic retrieval, and it is fundamentally different from keyword search because it works on meaning rather than exact word matches.
Example: An AI agent is helping a user plan a trip. The user asks: “What about that beach place we discussed last week?” The agent has hundreds of past conversation entries stored as embeddings. It converts the current query into a vector and searches for the most similar stored vectors.
Stored memory entries and their similarity scores to the query:
“User mentioned wanting to visit Cancun for a seaside vacation” (similarity: 0.89)
“User asked about flight prices to Bali” (similarity: 0.82)
“User discussed hotel budget for the trip” (similarity: 0.75)
“User asked about Python programming tips” (similarity: 0.12)
The embedding of the query “beach place we discussed” is semantically closest to the memory about Cancun and a seaside vacation even though the query never mentions “Cancun” or “seaside” by name. The embedding captures that “beach place” and “seaside vacation” are conceptually the same thing. The entry about Python programming has a very low similarity score because it exists in a completely different region of the embedding space. The agent uses these similarity scores to retrieve the most relevant memories, giving it the ability to maintain coherent, context-aware conversations across time without needing to search for exact keywords.
Beyond Words: Embeddings for Everything
While we have focused primarily on word and text embeddings, the concept extends to virtually any type of data. Images can be embedded into vectors that capture visual content, enabling systems to find visually similar images or match text descriptions to photographs. Audio signals can be embedded to represent speaker identity or musical genre. User behavior on a website, such as browsing patterns and purchase history, can be embedded to power recommendation systems. Even entire documents, code repositories, or molecular structures can be represented as embedding vectors.
The unifying principle is always the same: convert complex, high-dimensional, or unstructured data into a dense vector where geometric relationships are meaningful. Once data lives in an embedding space, you can apply the same mathematical toolkit, including distance calculations, nearest-neighbor searches, clustering, and arithmetic, regardless of whether the original data was text, images, or something else entirely.
Example: In a multimodal embedding space, the text “a golden retriever playing on the beach” and a photograph of exactly that scene would be mapped to nearby vectors, even though one is text and the other is an image. This is how modern search systems can accept a text query and return relevant images, or accept an image and return descriptive text.
The multimodal model has learned to project both text and images into the same geometric space such that semantically matching pairs end up close together. The text about a golden retriever on a beach and the corresponding image both encode similar high-level concepts: a specific dog breed, a playful activity, and a beach setting. Despite being fundamentally different data types, their embeddings share a neighborhood in the vector space. This is what makes cross-modal search possible and is a powerful illustration of how embeddings serve as a universal language that different types of data can be translated into.
Limitations and Challenges of Embeddings
Despite their power, embeddings have important limitations that are worth understanding.
First, embeddings inherit the biases present in their training data. If the text used to train an embedding model contains societal biases, such as associating certain professions more strongly with one gender, those biases will be encoded in the geometry of the embedding space. The vector arithmetic that works for king/queen analogies also works for biased associations, which means the bias becomes systematic and mathematical rather than anecdotal.
Second, embeddings are not easily interpretable. Unlike a feature that you might label as “size” or “color”, the individual dimensions in an embedding vector do not have clear human-readable meanings. Dimension 47 of a 300-dimensional embedding might contribute to many different semantic properties simultaneously, making it difficult to explain exactly why two words ended up close together or far apart.
Third, the quality of embeddings depends heavily on the quality and diversity of training data. Embeddings trained on medical literature will capture medical terminology well but may poorly represent casual conversational language, and vice versa. This means there is no single universal embedding that works perfectly for all tasks.
Embeddings represent one of the most elegant ideas in artificial intelligence: the insight that meaning can be captured as geometry. By converting words, tokens, and concepts into dense numerical vectors, machine learning systems gain the ability to reason about similarity, analogy, and context in a mathematically rigorous way.
We have seen how one-hot encoding fails to capture any semantic relationships, and how learned embeddings solve this by placing related items close together in a continuous space. We explored how training on prediction tasks causes embedding vectors to organize themselves in meaningful ways, and how this organization supports remarkable capabilities like vector arithmetic. We examined how cosine similarity provides a robust way to measure semantic relatedness, how LLMs extend embeddings from static to context-dependent representations, and how AI agents use embedding-based memory retrieval to maintain coherent behavior over time.
Understanding embeddings is not just an academic exercise. It is the conceptual key that unlocks a deeper understanding of virtually every modern AI system. Whether you are studying how LLMs generate text, how recommendation engines suggest content, or how AI agents remember and reason, you will find embeddings at the foundation. They are the translation layer between human meaning and machine computation, and mastering this concept will fundamentally change how you think about what AI systems are actually doing under the hood.
Understanding Embeddings was originally published in AI Evergreen on Medium, where people are continuing the conversation by highlighting and responding to this story.