Hey Onur, just got back from a trip around Japan. You’ve probably
already worked out the answer to this question but here is how I test
tokenizers;
require 'ferret'
$stdin.each do |line|
stk = Ferret::Analysis::StandardTokenizer.new(line)
while tk = stk.next()
puts " <#{tk.text}> from #{tk.start_offset} to