What Are Embeddings: The Foundation of Large Language Models?

September 25, 2024

Raido Linde

In the last 2 years, there has been a massive growth in generative AI with market size projected to grow from USD 20.9 billion in 2024 to USD 136.7 billion by 2030 at a compound annual growth rate of 36.7% according to Marketsandmarkets Research. This is the biggest disruption in the digital age, enabling organizations to make sense of the data to lower operational costs and enhance decision-making.

One small secret to this massive success is embedding models that power large language models (LLMs). It enabling these machine learning models to understand human language. While LLM and embedding models are related concepts, they perform very different tasks in machine learning.

Therefore, in this blog, we will cover what are embedding models, how these models work and why they are one of the essential building blocks of LLMs.

What Are Embeddings?

Embedding models transform data such as words, sentences, or images into numbers so that LLM models can understand them. It breaks down the data into numerical representations and embeds them in vector space based on their relationship. Closely related data will be near each other, while unrelated data will be far apart. This allows generative AI systems to perform mathematical calculations to find similarities and create new content from user input (aka a prompt).

What is the difference between LLMs and embedding models?

The reality is that LLMs can not understand human language; they can understand the numerical relationships between words. The embeddings are there to bridge that gap. They enable LLMs to understand human language without actually understanding language as crazy as it may sound.

However, unlike LLMs, embedding models cannot produce content; they can only encode the data into numbers and place them in the vector database for comparisons and analysis. As a result, most LLMs include the feature of the embedding model in their systems.

Essential Components of Large Language Models

‍

1. Collecting Data

When you are building an LLM model you need a large amount of data through various sources such as the web, websites, social media, books, and blogs. The more data you can provide the more patterns these models can learn and usually have higher quality output.

2. Preprocessing and Tokenization

Now the content needs to be broken down into smaller computer-understandable tokens such as words or subworlds. These tokens are the foundation to all LLM models.

3. Creating Embeddings

Embeddings are the next crucial step in this process. Embedding enables transforming these tokens into a vector, which is a numerical representation of words or images so that models can under their meanings and relationships. For example, tokens with similar meanings such as “cat” and “kitten” will have similar vectors. This enables models to understand they are closely related to each other.

4. Model Architecture

LLM models use a special neural network architecture, which is based on the transformer models introduced in the paper "Attention is All You Need" by Google researcher Vaswani et al. in 2017. Essentially this system takes the embeddings as tokens and processes these through an attention mechanism to understand how they are related in sequential data. As a result, it can transform input text, and data given to the model and create an output sequence of the text when the user requests it.

5. Training the Model

Now the model can be trained by feeding it the data that was collected in step 1. It learns to predict words and their relationships, adjusting its internal structure based on how accurate it is. This process requires powerful computer hardware and can take a long time. After that, it is ready to be used by users who can simply ask natural questions and get output back in the form of text, images, or audio.

6. Fine-Tuning and Deployment

Each LLM model can be further fine-tuned for specific tasks or use cases. Once it is tested and its accuracy is confirmed for a certain level, the model is deployed for all kinds of applications such as chatbots, virtual assistants, or others.

Understanding embeddings

Embedding models transform content into numbers so machine learning models can understand the meaning and relationship of its context. Explore simplified version of embedding model:

In the early days embedding used a one-hot encoding approach. Data was encoded to a list of vector numbers as seen in steps 1 (Token Representation) and 2 (Transformation to Vectors) and these were then placed orthogonal to each other. Hence didn't provide any meaningful relationship information and could have led to inefficiency in models.

Over the years the technology has improved and enables semantic embedding approaches as seen in step 3 (Vector Space). This enables us to create space with similar content next to each other that models can quickly retrieve, analyze, and use to create similar content. More about this later, but let's continue for now.

Types of Embedding, Use Cases, and Tools

Type of Embedding	Description	Use-Case	Example of a Tool
Word Embeddings	Dense vector representations of words that capture semantic relationships.	enhances search engines by improving keyword relevance.	Word2Vec
Sentence Embeddings	Vectors that represent entire sentences, capturing their meaning in context.	Employed in document similarity detection.	Sentence Transformers
Image Embeddings	Vector representations of images that capture visual features for comparison.	Used in image retrieval systems to find similar images based on visual content.	TensorFlow Image Embedding API
Audio Embeddings	Representations of audio signals that capture features for sound classification.	Used in speech recognition systems to transcribe audio.	OpenAI's Whisper
Contextual Embeddings	Dynamic embeddings that consider context, producing different vectors for the same word in different sentences.	Used in language translation to understand context.	BERT (Bidirectional Encoder Representations from Transformers)
Graph Embeddings	Representations of nodes or entire graphs that capture relationships and properties in a lower-dimensional space.	Enhances fraud detection by analyzing transaction patterns.	Node2Vec
Multimodal Embeddings	Embeddings that integrate information from multiple modalities (e.g., text, images, audio) to provide a comprehensive representation.	Enhances interactive AI systems by integrating various inputs.	CLIP (Contrastive Language-Image Pretraining)

Classical vs. Semantic Approaches in Embeddings

As natural language processing evolves, the shift from classical methods to more advanced semantic approaches has transformed how machines understand and generate language.

Overview

Aspect	Classical Approaches	Semantic Approaches
Main Techniques	- One-hot Encoding - Count-based - TF-IDF - N-grams	- Word2Vec - GloVe - ELMo
Context Consideration	Limited or no consideration of context	Considers word context and semantics
Dimensionality	High-dimensional (vocabulary size) for one-hot encoding	Lower-dimensional, dense representations
Semantic Capture	Limited semantic capture	Strong semantic capture
Long-distance Dependencies	Poor at capturing long-distance dependencies	Better at capturing long-distance dependencies, especially ELMo
Training Method	Statistical methods, frequency counts	Neural network-based, predictive models
Scalability	Generally more scalable (especially TF-IDF, Count-based)	Less scalable due to complex training
Training Speed	Faster to train	Slower to train, especially ELMo
Accuracy	Less accurate for complex NLP tasks	More accurate, especially for context-dependent tasks
Flexibility	Less flexible, fixed representations	More flexible, can generate context-dependent embeddings
Handling Unseen Words	Poor handling of unseen words	Better handling of unseen or rare words
Applications	Basic NLP tasks, information retrieval	Advanced NLP tasks, sentiment analysis, machine translation

1. Classical Approach

The classical method for language representation primarily relies on statistical techniques and symbolic representations. Key characteristics include:

Bag of Words (BoW): Each document is represented by the frequency of words without considering the context in which the words appear. This leads to sparse vectors that can be computationally expensive.

TF-IDF: A refinement of BoW, Term Frequency-Inverse Document Frequency captures word relevance by down-weighting common terms and up-weighting rare but significant words.

Word Co-occurrence Matrices: This involves tracking the proximity of words in a corpus, but like BoW, it lacks a deeper understanding of word meanings.

While efficient for smaller datasets, these methods struggle with scalability and semantic understanding. They fail to capture the contextual meaning of words and lead to poor generalization in real-world applications.

2. Semantic Approach (Embedding + LLM)

The semantic approach represents a leap forward, focusing on contextual understanding and rich language representations. Some key features are:

Word Embeddings (e.g., Word2Vec, GloVe): These capture the semantic relationships between words by representing them as dense vectors in a lower-dimensional space. Words with similar meanings tend to have closer vector representations.

Contextual Embeddings (e.g., BERT, GPT): Large Language Models (LLMs) further extend embeddings by incorporating context, meaning a word like “bank” can have different vector representations depending on its surrounding text.

Fine-tuning and Transfer Learning: LLMs can be fine-tuned for specific tasks, making them versatile. Pre-trained models like GPT or BERT excel at various downstream tasks like question answering, summarization, or natural language understanding.

The semantic approach captures nuances in meaning, context, and syntax, allowing for more accurate, scalable, and generalizable results across diverse NLP tasks.

Conclusion

Embeddings are the foundational building blocks of large language models, giving them the power to understand human language or any data, which it can use to recreate new content such as text, images, or audio. Without embedding we would not have ChatGPT, Claude, or any other generative AI applications. With ConfidentialMind you get open-source models that have these features built into. But that’s not all, we quantize each of our models, making them smaller and more cost-efficient. So, that our clients can harness the power of AI today, not tomorrow.