Skip-Gram Word Embeddings

Article BigQuant

Summary

This tutorial walks through a small Python implementation of the skip-gram model, a method for learning word vectors by predicting surrounding tokens from a center token. It represents input and output embeddings as separate matrices and uses a toy sentence and vocabulary to illustrate the process.

For each context token, the example computes softmax probabilities and cross-entropy loss, derives gradients for the center vector and output vectors, then sums those values across the context window. A gradient step updates the center token’s input vector and the output matrix. The tutorial includes sample costs, gradients, and updated weights as evidence of how the calculation behaves. Its example is deliberately tiny and uses randomly initialized vectors; it explains the mechanics rather than evaluating model quality or showing an application to market data. The code also contains inconsistent variable names in places, so readers should check the implementation before relying on it.

Key ideas

Skip-gram learns token representations by predicting context tokens from a center token.
The example calculates each prediction with softmax and cross-entropy loss.
Gradients from all context tokens are accumulated before updating the embedding matrices.
Only the center token’s input vector receives an input-matrix gradient in this example.
The toy output illustrates the algorithm’s mechanics but does not assess embedding quality.

Implementing Skip-Gram Word Embeddings with Softmax Gradients

Summary

Key ideas

Tags