Unverified50% confidenceFactExact time
A bidirectional encoder uses context from both the left and right sides when encoding each token, unlike GPT-style autoregressive models that only look at context from left to right.
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedGPT-2采用纯解码器的Transformer堆叠结构，去掉了编码器-解码器的交叉注意力部分，专注于单向自回归语言建模75% similarUnverifiedGPT系列采用单向（因果）注意力掩码，使模型在预训练阶段每个token只能关注其之前的上下文，与BERT等双向模型形成根本区别70% similarUnverified不同模型（如Claude与GPT系列）在训练数据和对齐策略上的差异，使得它们在代码审查时能够互补各自的知识盲点69% similarUnverifiedLearnable positional encodings were adopted by models such as BERT and GPT-2, with the drawback that they cannot handle sequences longer than the training length.69% similarUnverifiedWeight tying sets the input embedding matrix and the output projection matrix as the same set of parameters; early models such as GPT-2 have already adopted this approach, which reduces parameters and helps training stability.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1032339API
curl https://kongchang.com/api/v1/knowledge/claims/1032339MCP
get_claim(id=1032339)