- Explain the computational and semantic weaknesses of one-hot word vectors.
- Describe how distance and direction can represent meaning in an embedding space.
- Distinguish standalone embeddings from contextual embeddings.
- Explain how GloVe learns vectors from word co-occurrence counts.
- Outline a Keras text-classification pipeline that uses an embedding layer.
- Choose among frozen, fine-tuned, and task-specific embeddings based on the data and domain.
- Basic familiarity with one-hot encoding and vocabulary construction.
- An understanding of vectors, Euclidean distance, and dot products.
- Introductory knowledge of regression, squared error, and gradient descent.
- Basic familiarity with neural-network layers and Keras.
Why One-Hot Word Vectors Fall Short
One-hot representations are expensive for large vocabularies and cannot express relationships between word meanings.
Suppose a vocabulary contains 100,000 words. One-hot encoding assigns each word a 100,000-element vector containing one 1 and zeros in every other position. This representation creates two major difficulties.
The first problem is computational cost. Every token receives a vector whose length equals the vocabulary size, even for frequent words such astheanda. With a 500,000-word vocabulary, each vector has 500,000 positions. This also increases the number of model weights and therefore the demand for training data and computation.
The deeper problem is that one-hot vectors do not encode meaning. Consider the synonymsmovieandfilm. Their vectors disagree at two positions: the location of the 1 for each word. The two squared differences add to 2, so their Euclidean distance is the square root of 2. The opposite wordsgoodandbadhave exactly the same distance.
Representing Meaning Through Geometry
Useful word vectors place related words near one another and encode relationships through consistent directions.
A better representation should connect geometry with meaning. Vectors for synonyms such asmovieandfilmshould be nearby, while words with very different meanings should be separated. Direction should carry information too, not merely distance.
Imagine viewing two dimensions of such vectors.factory,home, andbuildingcould form one group;bicycle,truck, andcaranother; with separate groups for fruits such asappleandbanana, and sports terms such asbaseballandfootball. The vector forapplebelongs with the fruit words because its usage patterns place it there.
Directions can express analogies. If the displacement frompuppytodogrepresents a shift from a young animal to its adult form, applying a similar displacement tocalfshould lead towardcow. Learned vectors also support relationships such as the sibling analogy below, without a hand-written dictionary or rule set.
Given vectors forking,man, andwoman, what vector calculation should produce a point nearqueen?
Useking − man + woman ≈ queen.
From One Vector per Word to Context-Sensitive Vectors
Standalone embeddings capture general usage but merge a word's different senses into a single representation.
The wordbankmay refer to a financial institution, a river's edge, an airplane turning, or relying on something. A standalone embedding supplies only one vector for the word, so it tends to blend these senses into an average that may fit none of the individual contexts especially well.
For example,bankshould receive different representations in a sentence about depositing money and one about sitting beside a river. Transformers create these contextual representations by considering other words in the sentence. Before reaching that stage, however, it is useful to understand how standalone embeddings are learned.
Learning Standalone Vectors with GloVe
GloVe extracts compact word vectors from sentence-level co-occurrence patterns.
You shall know a word by the company it keeps.John Firth
This observation supplies the central intuition: words used in similar surroundings are likely to be related. In “The acting in the ___ was superb,” words such asmovie,film,play, ormusicalmake sense, whereasbananaortensordo not.
Building the Co-Occurrence Matrix
Use a large collection of text, divide it into sentences, and count how often every pair of distinct words appears within the same sentence. The words do not need to be adjacent. These counts form a square word-word matrixX.
With vocabulary sizeV, the matrix hasVrows andVcolumns. Many cells are zero because numerous word pairs never share a sentence. The diagonal can normally be ignored because the useful relationships lie in the off-diagonal entries.
Related words need not occur together to be recognized as related. Instead, their rows can follow similar patterns across the vocabulary. For instance, the rows forappleandbananamay rise and fall similarly because both occur with terms such asfruit,eat,ripe, andpeel. These row patterns are how context enters the data.
Modeling Co-Occurrence with GloVe
For each word pair, the model approximates the logarithm of its observed count with three components:
- A bias
bᵢrepresenting the inherent frequency of wordi. - A corresponding bias
bⱼfor wordj. - The dot product of the two embedding vectors, which represents their connection.
The logarithm compresses counts that range from zero to very large values, making the regression easier to manage. Because the logarithm of zero is undefined, 1 can be added to the counts when carrying out the calculation.
The biases absorb how frequently the individual words occur, while the dot product captures how strongly the pair is related. A small dot product indicates little connection; a larger one represents a stronger relationship.
Estimating the Embeddings
The co-occurrence values are the training data; the biases and embedding coordinates are unknown model weights. Training minimizes the total squared difference between observed log counts and model predictions.
Start the unknown parameters at random values, then use gradient descent or stochastic gradient descent to reduce this loss. The model is a regression rather than a neural network, but gradient descent still applies because the required gradients can be calculated.
After optimization, each vocabulary word has a learned vectorwᵢ. The biases served to account for frequency during fitting and can be discarded once the embeddings have been obtained.
Characteristics of the Learned Vectors
- Dense: coordinates are learned real values instead of isolated zeros and ones.
- Low-dimensional: the vector is much shorter than the vocabulary-sized one-hot representation.
- Semantically structured: distances and directions can correspond to meaningful word relationships.
Embedding length is a hyperparameter. More dimensions provide additional capacity for complex patterns, but too many parameters can fit noise in the co-occurrence data. Begin with a smaller vector and increase its size until task performance stops improving or overfitting begins.
Inspecting and Reusing Embeddings
Projection reveals the structure of learned vectors, and pretrained GloVe vectors can transfer general language information to another task.
Techniques such as t-SNE project high-dimensional embeddings into two dimensions for display. The resulting plots can show nearby family terms such asbrother,sister,uncle, andaunt, as well as a region containing royalty words such asking,queen, andemperor. Such views also expose consistent vector directions.
Pretrained GloVe packages provide vectors learned from a large corpus such as Wikipedia. Available packages use different dimensionalities, including 100, 200, and 300, so a model can select and reuse a suitable set instead of learning general language patterns entirely from its own data.
For sentiment classification of movie reviews, would you reuse pretrained GloVe vectors or learn embeddings from the task data? Which considerations should guide the decision?
Consider how much labeled data is available, how closely the pretrained corpus matches the review domain, and the available computation. Frozen pretrained vectors are attractive with little data; fine-tuning or learning task-specific vectors becomes more plausible with sufficient domain data.
Applying Embeddings in Keras
A Keras classification model maps text to token IDs, looks up embeddings, pools the resulting sequence, and predicts a class.
Text classification requires a fixed-length representation even though raw inputs contain different numbers of words. A typical Keras workflow uses the following sequence.
- Configure
TextVectorizationwithoutput_mode='int'. It standardizes text, splits it into tokens, and replaces tokens with integer IDs; it does not create one-hot vectors. - Choose a maximum sequence length. Pad shorter inputs with zero-valued pad-token IDs, and keep the initial tokens while removing material beyond the maximum from longer inputs.
- Pass the IDs to an
Embeddinglayer. Itsinput_dimis the vocabulary size, including padding and unknown tokens, whileoutput_dimis the embedding length. Its weights may be pretrained or learned from the task. - Apply
GlobalAveragePooling1Dto average each embedding coordinate across token positions, producing one sentence vector. This simple operation discards positional detail. - Feed the pooled representation through one or more dense layers, such as a ReLU layer, and then through a softmax output layer for classification.
The result replaces a large sparse one-hot representation with a compact dense vector for the input sentence.
Freeze, Fine-Tune, or Learn from the Beginning
There are three ways to handle an embedding layer in a downstream model.
- Frozen pretrained vectors: initialize the layer with GloVe and set
trainable=False. This is fast, preserves general language knowledge, and can reduce overfitting with a small dataset. In the song-genre example, this approach achieved about 63% accuracy. - Fine-tuned vectors: begin with GloVe but leave
trainable=Trueso backpropagation can adapt the vectors to the domain. Accuracy in the same example rose to roughly 69%. - Vectors learned from scratch: randomly initialize the embedding layer and train it for the target objective. This needs more data but can capture domain-specific patterns. A 64-dimensional layer trained on the song data reached about 72%.
The preferred strategy depends on the amount and character of the data. With only a few hundred examples, frozen pretrained vectors are the safer starting point. With many examples, learning task-specific vectors may work well. A domain mismatch also matters: embeddings trained on Wikipedia may not fully represent specialized language such as legal or medical jargon, so adaptation can help.
Putting the Ideas Together
Word embeddings use patterns of context to create compact semantic representations, while contextual models extend them to individual word occurrences.
One-hot encoding is both vocabulary-sized and semantically empty: every distinct pair of words has the same Euclidean distance. Dense embeddings address these limits by using far fewer learned coordinates whose geometry reflects word relationships.
GloVe learns those coordinates by fitting word biases and embedding dot products to logarithmic sentence-level co-occurrence counts. Gradient descent minimizes squared reconstruction error, and the final vectors can exhibit both clusters and analogical directions.
In a neural network, an embedding layer converts integer token IDs into dense vectors. Pretrained weights can remain fixed, be adapted through training, or be replaced by vectors learned for the specific task.
A standalone vector still merges all senses of a polysemous word. Contextual embeddings produced with transformers address that remaining limitation by constructing a representation from the surrounding sentence.
Common misconceptions
Watch out: A shorter semantic relationship should appear as a shorter distance between one-hot vectors.
Instead: All distinct one-hot vectors are separated by the same Euclidean distance, √2, regardless of whether their words are synonyms, opposites, or unrelated.
Watch out: A single standalone vector can preserve every meaning of a word such as bank.
Instead: One vector per word blends the different senses into an average representation. Distinguishing each occurrence requires a contextual embedding based on surrounding words.
Watch out: Global average pooling preserves the position and order of every token embedding.
Instead: Averaging produces a convenient sentence vector but removes positional information and some nuance.
Lesson recap
- One-hot vectors grow with the vocabulary and assign the same distance to every distinct word pair.
- Dense word embeddings use distance and direction to represent semantic connections.
- GloVe learns standalone vectors by reconstructing logarithmic word co-occurrence counts with biases and dot products.
- Squared reconstruction error is minimized with gradient descent, after which the learned word vectors are retained.
- A Keras pipeline converts text to integer IDs, looks up embeddings, pools the token vectors, and applies classification layers.
- Embeddings may be frozen, fine-tuned, or learned from scratch according to data size and domain fit.
- Standalone embeddings cannot resolve polysemy; contextual transformer representations incorporate surrounding words.