Nagi-ovo
Follow
Breezing homepage: [nagi.fun](nagi.fun)
159
Followers
19
Followings
Patron
主页
Archives
Portfolios
日寄
BPE
Latest
Hottest
Most Commented
LLM 演进史(六):揭开 Tokenizer 的神秘面纱
Tokenizer 是 LLM 中很重要但又没那么 fancy 的组件,在本系列之前的语言模型建模中,tokenizer 的实现方式是字符级的,将所有可能出现的 65 种字符制作嵌入表,然后用 embedding layer 对训练集进行编码向量化。而实践中…
LLM
15 min
10 months ago
Ownership of this blog data is guaranteed by blockchain and smart contracts to the creator alone.
Blockchain ID
#61009
Owner
0x6380302480224d53ec4c2c318d1c7be2c55a7582
Transaction Hash
Creation 0xe99aa0c3...eadd5b56dd
Last Update 0x183e93de...ca7c86d2ef
IPFS Address
ipfs://bafkreid6e67datddz2iqadcmnro7m7dkhq3ocwp24mfkybtupiznpzoy7y