tokenize
English
Etymology
Pronunciation
- Lua error in Module:parameters at line 291: Parameter 1 should be a valid language or etymology language code; the value "GA" is not valid. See WT:LOL and WT:LOL/E. IPA(key): /ˈtoʊ.kən.aɪz/
Verb
Lua error in Module:en-headword at line 1142: Legacy parameter 1=STEM no longer supported, just use 'en-verb' without params
- (transitive, computing) To reduce to a set of tokens by lexical analysis.
- (transitive, computing) To substitute sensitive data with meaningless placeholders.
- (transitive) To treat as a token minority.