tokenize
Jump to navigation
Jump to search
English[edit]
Etymology[edit]
Pronunciation[edit]
- (General American) IPA(key): /ˈtoʊ.kən.aɪz/
Verb[edit]
tokenize (third-person singular simple present tokenizes, present participle tokenizing, simple past and past participle tokenized)
- (transitive, computing) To reduce to a set of tokens by lexical analysis.
- (transitive, computing) To substitute sensitive data with meaningless placeholders.
- (transitive) To treat as a token minority.