Theoretical Positioning of GD-Attention in Prior Work: Why It Cannot Be Reduced to a Variant of Softmax

1. Why This Map is Necessary

The true nature of GD-Attention cannot be fully appreciated if it is merely understood as "another version of Softmax attention (e.g., sparser mixture, faster computation)."

In the standard context of attention research, attention has primarily been understood as "weighted blending." From there, it has branched out into sparser blending, efficient routing, more discrete selection, energy-based interpretations, and decision-making involving abstention or rejection.

However, the core of GD-Attention cannot be neatly categorized into any single one of these lineages. At its center, GD-Attention establishes a semantic energy landscape among meaning candidates, identifies an energy-minimizing consistency point, and employs a nonlinear semantic selection mechanism that selects the key corresponding to that point. Furthermore, it incorporates a structure that excludes candidates whose consistency does not meet a threshold.

Therefore, to position GD-Attention correctly, it is necessary to clarify not just that it is "a variant of attention," but where it approaches and where it fundamentally diverges within the existing research map of attention, routing, energy-based modeling, and abstention.

▶GD-Attention Github Demo

2. Topography of Existing Research: Six Main Lineages

The following classification is not strictly mutually exclusive but a conceptual arrangement to clarify the comparison axes for GD-Attention. Existing LLM research and technologies around attention can be roughly categorized into the following six groups:

1. Dense Blending [1, 2]

Standard attention represented by the Transformer, which mixes multiple candidates with weights. Smooth optimization and high expressiveness are its strengths, but it is not designed as a mechanism to explicitly fix a final winner, making it less suitable for scenarios that demand explicit discrete selection among semantic candidates. GD-Attention first distances itself from designs centered on this "weighted average" and foregrounds discrete selection under explicit conditions.

2. Sparse Blending [3, 4]

The trend of making the dense mixture of Softmax sparse. It sets some weights to zero to concentrate attention. However, this remains within "mixture control," and unique selection or explicit consistency filtering is not the main role. The core of GD-Attention lies not in sparse blending itself, but in discrete key selection based on semantic energy.

3. Conditional Routing [14-16]

Routing systems like MoE (Mixture of Experts) switch computational paths depending on the input. Using "only what is necessary" is a departure from dense blending, but what is selected here are "computational resources or structures (experts/paths)," not fixing a winner among semantic candidates. GD-Attention targets "semantic key selection," and both differ in their selection targets and basis. Therefore, while routing systems can be objects of comparison in terms of selection, they should not be placed at the forefront as the closest precedents to GD-Attention.

4. Hard Selection / Hard Alignment [5-8]

The lineage of Pointer Networks and hard/monotonic attention. This direction treats attention not as a "blend" but as a "select" (to point), which is quite close to GD-Attention. What is important here is not to claim the idea of reading attention as selection itself as a novelty. Even in recent theoretical research, there is a clear lineage of reading softmax attention as a token selection mechanism [8]. Where GD-Attention steps forward from this is that it provides a single framework encompassing a semantic energy landscape, a unique energy-minimizing consistency point, threshold exclusion of consistency breakdown, and residual-based attention refinement, rather than just simple hard selection.

5. Energy-based Attention [9, 10]

The lineage that understands attention from the perspective of energy landscapes and associative memory, reading it as a process of finding stable points or local minima (e.g., Modern Hopfield, Energy Transformer). GD-Attention shares the concept of establishing a semantic energy landscape among candidates. However, while general energy-based attention understands the overall behavior of attention from an energy perspective, GD-Attention strongly connects the energy-based interpretation to the selection mechanism of attention in that it determines an energy-minimizing consistency point through semantic energy evaluation between queries and keys, and performs selection based on that point.

6. Abstention / Reject Option [11-13]

Research that integrates structures into decision-making such as "not forcing an answer and rejecting if conditions are insufficient," like selective prediction. GD-Attention has a structure that excludes candidates not meeting conditions via threshold judgment, sharing a connection with reject option research in this regard. However, its core is not general selective prediction, but filtering inside attention based on semantic consistency.

3. Points of Comparison: What is Close, What to Avoid

3.1 Primary Close Precedents

The main close precedents are in five directions: Pointer Networks, hard / monotonic attention, max-margin token selection, energy-based attention, and selective prediction.

• Perspective of selection: Pointer Networks and hard attention are the closest, but GD-Attention differs in that it presupposes a semantic energy landscape and energy-minimizing consistency point calculation, and further excludes consistency breakdowns by a threshold.

• Theoretical closeness: In terms of theoretical closeness, the lineage of max-margin token selection [8] is particularly important, and top-k attention [7] is reasonably positioned as an example of a broader hard/sparse selection family.

• Perspective of energy: While sharing the perspective, GD-Attention foregrounds semantic energy evaluation between queries and keys and the calculation of a unique energy-minimizing consistency point.

• Perspective of abstention: Selective prediction research is important, but GD-Attention is positioned as threshold filtering based on semantic consistency "inside the attention operator."

3.2 Comparisons to De-emphasize

In contrast, MoE / routing are auxiliary comparisons, and retrieval systems are reasonably kept as even more peripheral conceptual comparisons.

• MoE / routing share superficial similarities in "selection," but differ from GD-Attention in that the selection targets are experts/paths. Therefore, they are useful as auxiliary comparisons, but placing them at the forefront as main close precedents can easily mislead the positioning.

• Retrieval systems are conceptually close in the broad sense of candidate selection, but because their main subject is external memory search, they belong to a different comparison hierarchy than GD-Attention, which handles semantic selection inside the attention operator.

4. The Correct Positioning of GD-Attention and Redefining Its Novelty

Based on the above mapping, the positioning of GD-Attention can be defined as follows:

GD-Attention can be positioned as an energy-based discrete semantic selection mechanism with consistency-based filtering.

Precedents exist for selection itself, energy interpretation, and reject options individually. Therefore, the true uniqueness of GD-Attention does not lie in "reading attention as selection" itself.

The core novelty of GD-Attention is: Rather than presenting "an alternative to Softmax," it reformulates semantic selection in attention as a nonlinear selection mechanism equipped with a semantic energy landscape, a unique energy-minimizing consistency point, a jump direction, threshold exclusion for consistency breakdown, and residual-based attention refinement.

Visual Map (Two-Axis Model) Proposal

Rather than a chronological timeline, the following two-axis mapping is the most effective:

• X-axis: "Blending" -> "Discrete Selection"

• Y-axis: Weakness -> Strength of "Selection Explicitness / Consistency Control"

In this conceptual map, while standard attention (bottom-left), sparse attention (bottom-middle), hard selection (bottom-right), and energy-based (center) are placed, it is natural to position GD-Attention in the upper-right region where both "discrete selection" and "strength of consistency control" are strong.

Only when this positioning is made explicit does GD-Attention appear not as "an alternative to Softmax," but as research that redescribes semantic selection not merely as weighting, but as nonlinear selection based on semantic energy. In this sense, the value of GD-Attention lies not in improving attention weights, but in redescribing semantic selection within a framework of energy minimization and consistency control.

5. Example Paragraph for Related Work and References

Below is an example of an English paragraph to describe the above positioning in a Related Work section of a paper.

Standard attention is typically framed as a soft alignment or weighted blending mechanism [1, 2], from early neural alignment models to the Transformer. Later work introduced sparse variants such as sparsemax and entmax [3, 4], which reduce the support of the attention distribution but still remain within the mixture paradigm. A different line of work treats attention more discretely, as in Pointer Networks, monotonic attention, and recent max-margin analyses of token selection [5–8]. In parallel, energy-based interpretations of attention emerged through Modern Hopfield networks and the Energy Transformer [9, 10], while selective prediction introduced explicit abstention or reject options in neural decision-making [11–13]. Recent work on selective attention in Transformers also aims to suppress unnecessary context, but this line still differs from GD-Attention in that it primarily improves attention usage rather than formulating semantic selection through energy minimization and consistency-based filtering [17]. GD-Attention is best understood at the intersection of these lines: not merely as sparse attention, nor merely as routing [14–16], but as an energy-based discrete semantic selection mechanism equipped with consistency-based filtering and nonlinear alignment by semantic energy minimization.

Reference List

In this manuscript, we emphasize [5–13] as the main close precedents of GD-Attention, and use [14–17] as auxiliary comparisons.

A. The Origin of Attention (Dense Blending)

• [1] Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. Presented at ICLR 2015.

• [2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.

B. Sparse Blending

• [3] Martins, A., & Astudillo, R. (2016). From softmax to sparsemax: A sparse model of attention and multi-label classification. Proceedings of the 33rd International Conference on Machine Learning.

• [4] Correia, G. M., Niculae, V., & Martins, A. F. (2019). Adaptively sparse transformers. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

C. Selection / Hard Selection

• [5] Vinyals, O., Fortunato, M., & Jaitly, N. (2015). Pointer networks. Advances in Neural Information Processing Systems, 28.

• [6] Raffel, C., Luong, M. T., Liu, P. J., Weiss, R. J., & Eck, D. (2017). Online and linear-time attention by enforcing monotonic alignments. Proceedings of the 34th International Conference on Machine Learning, PMLR 70.

• [7] Gupta, A., et al. (2021). Memory-efficient transformers via top-k attention. Proceedings of SustaiNLP 2021.

• [8] Tarzanagh, D. A., Li, Y., Zhang, Y., & Oymak, S. (2023). Max-margin token selection in attention mechanism. Advances in Neural Information Processing Systems, 36.

D. Energy-based / Associative Memory

• [9] Ramsauer, H., et al. (2021). Hopfield networks is all you need. Presented at ICLR 2021.

• [10] Hoover, B., et al. (2023). Energy transformer. Advances in Neural Information Processing Systems, 36.

E. Abstention / Reject Option

• [11] Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. Advances in Neural Information Processing Systems, 30.

• [12] Geifman, Y., & El-Yaniv, R. (2019). SelectiveNet: A deep neural network with an integrated reject option. Proceedings of the 36th International Conference on Machine Learning, PMLR 97.

• [13] Hendrickx, K., Perini, L., Van der Plas, D., Meert, W., & Davis, J. (2024). Machine learning with a reject option: A survey. Machine Learning, 113(5), 3073-3110.

F. Routing / Conditional Computation

• [14] Shazeer, N., et al. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. Presented at ICLR 2017.

• [15] Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(1), 5232-5270.

• [16] Roy, A., et al. (2021). Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9, 53-68.

G. Recent Direct Comparisons / Preventing Misinterpretation

• [17] Leviathan, Y., Kalman, M., & Matias, Y. (2025). Selective attention improves transformer. Presented at ICLR 2025 (OpenReview).

Optional Additions (If addressing hard attention directly from a theoretical standpoint)

• [18] Yang, Y., et al. (2024). Masked hard-attention transformers recognize exactly the star-free languages. Advances in Neural Information Processing Systems, 37.

• [19] Jerad, S., et al. (2025). Unique hard attention: A tale of two sides. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Short Papers).