Transformer Core: Self-Attention Mechanism Principle Analysis

Introduction: From Word to Vector

Computers cannot directly understand text; they can only process numbers. Therefore, we need to convert each word into a numerical vector (Embedding). For example, in the sentence "The cat sat," each of the three words is represented as a 4-dimensional vector, forming the input matrix $X \in \mathbb{R}^{3 \times 4}$ (3 words, each 4-dimensional).

1. Core Magic: Self-Attention Mechanism

In traditional sequence models (such as RNN/LSTM), text is processed one word at a time in order. The disruptive innovation of the Transformer is that it breaks the temporal order, allowing all words in a sentence to examine each other simultaneously — this is the self-attention mechanism.

1.1 Core Roles: Q, K, V

The core of the self-attention mechanism is to multiply the input matrix $X$ by three learnable weight matrices respectively, obtaining three new matrices:

• Q (Query): What features does the current word want to find? (For example, "bank" acts as the subject querying other words to determine whether it means "financial institution" or "riverbank.")
• K (Key): What feature labels does each word in the sentence carry? Used to match with the Query.
• V (Value): If a match is found, what specific information content can this word actually provide.

1.2 Mathematical Derivation Formula

1.3 Step-by-Step Formula Breakdown

1. Step 1: Compute relevance scores $QK^T$
   Use the Query matrix $Q$ to dot product with the transpose of the Key matrix $K$ of all words. The larger the dot product, the better the features of the two words match, indicating higher relevance. The result is a $3 \times 3$ scoring matrix, where $\text{Scores}_{ij}$ represents the degree of attention the $i$-th word pays to the $j$-th word.
2. Step 2: Scale
   Divide by $\sqrt{d_k}$ (in this demo $d_k = 3$, so divide by $\sqrt{3} \approx 1.732$). This prevents the dot product results from becoming too large in higher dimensions, which would cause the gradient of the subsequent Softmax function to vanish.
3. Step 3: Softmax Normalization
   Apply the Softmax operation to each row of the scaled scoring matrix, converting it into a probability distribution (each row sums to 1). This represents the attention weights the current word assigns to all words in the sentence.
4. Step 4: Weighted Sum
   Multiply the attention weight matrix just computed by the value matrix $V$, performing a weighted sum. The final output $Z$ is a brand-new word vector representation that integrates the contextual information of the entire sentence.

2. Multi-Head Attention

In an actual Transformer, multiple attention heads (Multi-Head) are used. Each head uses different $W^Q, W^K, W^V$ weight matrices to understand the sentence from different subspaces (perspectives):

• Some heads may focus on grammatical structure (subject-verb-object relationships)
• Some heads may focus on semantic similarity (synonyms, near-synonyms)
• Some heads may focus on tense and modality

3. Positional Encoding

The self-attention mechanism itself is orderless (similar to a bag-of-words model); it cannot distinguish "cat bites dog" from "dog bites cat." Therefore, the Transformer must inject positional information through positional encoding. The common method uses sine and cosine functions:

The positional encoding is added to the word embedding vector to form the final input representation.

4. Feed-Forward Network & Residual Connection

After the self-attention layer, the Transformer also passes through a Feed-Forward Network:

Meanwhile, to prevent vanishing gradients and accelerate training, the Transformer uses residual connections and layer normalization:

5. Summary

The core innovation of the Transformer lies in the self-attention mechanism, which enables the model to process the entire sequence in parallel and dynamically attend to different parts of the sequence. Through the ingenious design of the Q, K, V matrices, the Transformer achieves efficient context modeling, becoming the cornerstone of modern NLP (such as GPT, BERT) and multimodal models (such as CLIP, Stable Diffusion).

Self-Attention Mechanism: Why It Is Effective

The self-attention mechanism is powerful because it fundamentally solves two core shortcomings of traditional neural networks (such as RNN, LSTM) when processing sequence data: the long-distance dependency challenge and the parallel computation bottleneck. The following breaks down the roots of its effectiveness from five dimensions.

---

1. Solving the "Forgetting" Problem: Long-Distance Dependencies

In RNNs or LSTMs, information is passed sequentially; the longer the sentence, the more easily information from the beginning attenuates (vanishing gradient), causing the model to "forget the beginning."

• Self-Attention Advantage: Each word can directly interact and compute with all words in the sequence (including the very first words).
• Physical Meaning: No matter how far apart two words are, self-attention can establish a "shortcut connection" to directly measure their relevance.
  For example, in the sentence: "The person who was walking the dog in the park wore a red coat." Self-attention allows "person" and "red coat" to be directly associated, unaffected by the long intervening phrase.

2. Parallel Computation and the "God's-Eye View"

RNNs must compute sequentially by time step (processing the first word before the second), limiting GPU parallel capability.

• Self-Attention Advantage: It can receive the entire sentence at once, simultaneously computing correlations between all words through matrix operations.
• Physical Meaning: It is equivalent to having a "God's-eye view" at the very first step, seeing the full picture of the sentence rather than crawling step by step. This fully leverages the parallel capabilities of modern hardware, significantly boosting training speed.

3. Dynamic Weight Allocation: Focusing on What Matters

The convolutional kernel parameters of Convolutional Neural Networks (CNNs) are static; regardless of how the input content changes, the way features are extracted is fixed.

• Self-Attention Advantage: Attention weights are dynamically generated, entirely dependent on the current input content.
• Physical Meaning: The model can adjust its focus in real-time based on context. For example, when processing the pronoun "it," self-attention assigns higher weights to preceding nouns (such as "apple" or "car"), aiding disambiguation. This dynamic nature gives the model tremendous flexibility.

4. Multiple Representations: Building a Rich Semantic Map

The Transformer uses Multi-Head Attention, meaning multiple sets of self-attention computations are executed in parallel.

• Self-Attention Advantage: Different "heads" can learn different levels of dependency relationships.
• Physical Meaning:
  • Some heads focus on grammatical relationships (such as subject and predicate);
  • Some heads focus on coreference relationships (such as "Xiao Ming" and "he");
  • Some heads focus on adjacent word modification (such as adjectives and nouns).
  Multi-perspective representation makes text understanding more three-dimensional and richer.

5. Mathematical Essence: Weighted Sum of Feature Vectors

Core self-attention formula:

Attention(Q, K, V) = softmax( QKT / √dk ) V

• QK^T: Compute a similarity matrix, measuring how well each element matches every other element.
• Softmax: Convert matching scores into a probability distribution (weights), making all weights sum to 1.
• Multiply by V: Perform a weighted sum of all Value vectors using the computed weights — the larger the weight, the greater the contribution of that position to the final output.

---

Summary

The self-attention mechanism breaks through the bottlenecks of traditional models in long-sequence modeling and semantic capture through global perception, dynamic weighting, and parallel computation. It not only allows the model to "see" each word but also deeply understands the interactions between words — this is the fundamental reason why it is effective.