tokenize

From Wiktionary, the free dictionary
Archived revision by WingerBot (talk | contribs) as of 05:31, 15 October 2019.
Jump to navigation Jump to search

English

English Wikipedia has an article on:
Wikipedia

Etymology

token +‎ -ize

Pronunciation

  • Lua error in Module:parameters at line 291: Parameter 1 should be a valid language or etymology language code; the value "GA" is not valid. See WT:LOL and WT:LOL/E. IPA(key): /ˈtoʊ.kən.aɪz/

Verb

Lua error in Module:en-headword at line 1142: Legacy parameter 1=STEM no longer supported, just use 'en-verb' without params

  1. (transitive, computing) To reduce to a set of tokens by lexical analysis.
  2. (transitive, computing) To substitute sensitive data with meaningless placeholders.
  3. (transitive) To treat as a token minority.

Derived terms

Retrieved from "https://en.wiktionary.org/w/index.php?title=tokenize&oldid=55537087"