Tokenization, at its heart , is the process of separating a larger piece of data into discrete units called tokens . Think of it like segmenting a sentence into items . These items can then be processed https://rebeccatwzq507886.prublogger.com/40644273/tokenization-explained-a-simple-guide