tokenizer

tokenizer

English

Etymology

Etymology tree
English token
Proto-Indo-European *-id-
Proto-Indo-European *-yéti
Proto-Indo-European *-idyéti
Proto-Hellenic *-íďďō
Ancient Greek -ῐ́ζω (-ĭ́zō)bor.
Late Latin -izōder.
Middle French -iserbor.
Middle English -isen
English -ize
English tokenize
Proto-Indo-European *-yósder.
Proto-Italic *-āsjos
Latin -āriusnom.
Latin -āriusbor.
Proto-Germanic *-ārijaz
Proto-West Germanic *-ārī
Old English -ere
Middle English -ere
English -er
English tokenizer

    From tokenize + -er.

    Noun

    tokenizer (plural tokenizers)

    1. (computing) A system that parses an input stream into its component tokens.
    This article is issued from Wiktionary. The text is available under Creative Commons Attribution-Share Alike 4.0 unless otherwise noted. Additional terms may apply for the media files.