tokenizer

From Wiktionary, the free dictionary

Archived revision by NadandoBot (talk | contribs) as of 22:47, 20 May 2017.

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Jump to navigation Jump to search

English

Etymology

tokenize +‎ -er

Noun

tokenizer (plural tokenizers)

(computing) A system that parses an input stream into its component tokens.

Retrieved from "https://en.wiktionary.org/w/index.php?title=tokenizer&oldid=43036057"