mistokenize
Word Rarity
Very rare
Frequency: Zipf frequency 0.00 in modern English
26
Scrabble Score
11
Letters
4
Syllables
9
Unique Chars
Meaning
verb
To tokenize incorrectly.
Example Sentences
- "C++ considers ::*, .* and ->* each to be a single token and a single operator. Some pre-Release 2.0 implementations mistokenize expressions involving pointer-to-pointer-to-member."
- "These were mostly proper names, such as Ronny Johnsen, or foreign language items such as ambre solaire (French) and fairie queene (Middle English), as well as a few misspelt or mistokenized items."
- "The sentences were tokenized into words using the regex tokenizer which avoided the problems of mistokenizing while using the default NLTK tokenizer."
- "Similarly, all other two-character atomic representations in SMILES are being mistokenized."