# Error decoding input string **URL:** \ **Category:** Ferret\ **Created:** [May 19, 2008, 6:01pm UTC](https://www.ruby-forum.com/t/error-decoding-input-string/138034 "2008-05-19T18:01:29Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![soderic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/8baadc/32.png) [@soderic](https://www.ruby-forum.com/u/soderic)\ **Post date:** [May 19, 2008, 6:01pm UTC](https://www.ruby-forum.com/t/error-decoding-input-string/138034/1 "2008-05-19T18:01:29Z")
Hi, I am trying to index a number of Spanish language text files, but a large fraction of the files are generating errors like the following… Error: exception 2 not handled: Error decoding input string. Check that you have the locale set correctly however it looks to me like my locale matches the file type. Running the file command on the files returns $ file /media/…/raw/abc/20Jan2007\_abc\_001041\_67.es /media/…/raw/abc/20Jan2007\_abc\_001041\_67.es: UTF-8 Unicode text and my locale is $ locale LANG=en\_US.UTF-8 LC\_CTYPE=“en\_US.UTF-8” LC\_NUMERIC=“en\_US.UTF-8” LC\_TIME=“en\_US.UTF-8” LC\_COLLATE=“en\_US.UTF-8” LC\_MONETARY=“en\_US.UTF-8” LC\_MESSAGES=“en\_US.UTF-8” LC\_PAPER=“en\_US.UTF-8” LC\_NAME=“en\_US.UTF-8” LC\_ADDRESS=“en\_US.UTF-8” LC\_TELEPHONE=“en\_US.UTF-8” LC\_MEASUREMENT=“en\_US.UTF-8” LC\_IDENTIFICATION=“en\_US.UTF-8” LC\_ALL= after enough of these errors are generated, I begin to get errors for having too many open files, and the indexing fails. Error: exception 2 not handled: Too many open files Any suggestions would be greatly appreciated. Thanks, Eric ---
**Author:** ![soderic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/8baadc/32.png) [@soderic](https://www.ruby-forum.com/u/soderic)\ **Post date:** [May 19, 2008, 11:16pm UTC](https://www.ruby-forum.com/t/error-decoding-input-string/138034/2 "2008-05-19T23:16:21Z")
Hi! Are you _sure_ this is all valid UTF8? I dont know how the file command determines this, and if it always is right. Maybe try to play around with iconv to ensure whatever you send to Ferret really is UTF8. Cheers, Jens On 19.05.2008, at 18:00, Eric S. wrote: > the file command on the files returns > > $ file /media/…/raw/abc/20Jan2007\_abc\_001041\_67.es > /media/…/raw/abc/20Jan2007\_abc\_001041\_67.es: UTF-8 Unicode text > LC\_MONETARY=“en\_US.UTF-8” > after enough of these errors are generated, I begin to get errors for > removed\_email\_address@domain.invalid > [http://rubyforge.org/mailman/listinfo/ferret-talk](http://rubyforge.org/mailman/listinfo/ferret-talk) – Jens Krämer Finkenlust 14, 06449 Aschersleben, Germany VAT Id DE251962952 [http://www.jkraemer.net/](http://www.jkraemer.net/) - Blog [http://www.omdb.org/](http://www.omdb.org/) - The new free film database ---
**Author:** ![soderic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/8baadc/32.png) [@soderic](https://www.ruby-forum.com/u/soderic)\ **Post date:** [May 20, 2008, 8:00pm UTC](https://www.ruby-forum.com/t/error-decoding-input-string/138034/3 "2008-05-20T20:00:36Z")
Hi Jens, Thanks for the reply! I used iconv (thanks for the pointer, I had no idea this tool existed) and was able to convert all of the articles to and from utf8 without any errors being generated, so I am pretty sure that the input sources are valid utf8. I should mention that I am using an old version of ferret. v.0.9.6 which is the last version to have a pure-ruby implementation. I’m using this version because I have added in some changes which allow me to specify the scoring algorithm used on a per-search basis. I haven’t however made any changes to the indexing portion of the application. I current have an iconv script creating transliterated ASCII copies of all my articles, so I am going to try to index over these. Also, I am thinking of trying to index using Lucene since there is a chance that the older version of ferret is compatible with lucene indexes. If you have any other suggestions I’d love to hear them, but I understand that I can’t expect much help with such an old version. Do you know of a way to specify custom scoring algorithms in the current versions of ferret? Best, Eric On Monday, May 19, at 23:15, Jens K. wrote: > Hi! > > Are you _sure_ this is all valid UTF8? I dont know how the file > command determines this, and if it always is right. > Maybe try to play around with iconv to ensure whatever you send to > Ferret really is UTF8. > > Cheers, > Jens > > On 19.05.2008, at 18:00, Eric S. wrote: > > > Hi, > > > > I am trying to index a number of Spanish language text files, but a > > large fraction of the files are generating errors like the > > following… > > > > Error: exception 2 not handled: Error decoding input string. Check > > that you have the locale set correctly > > > > however it looks to me like my locale matches the file type. > > Running > > the file command on the files returns > > > > $ file /media/…/raw/abc/20Jan2007\_abc\_001041\_67.es > > /media/…/raw/abc/20Jan2007\_abc\_001041\_67.es: UTF-8 Unicode text > > > and my locale is > > > > $ locale > > LANG=en\_US.UTF-8 > > LC\_CTYPE=“en\_US.UTF-8” > > LC\_NUMERIC=“en\_US.UTF-8” > > LC\_TIME=“en\_US.UTF-8” > > LC\_COLLATE=“en\_US.UTF-8” > > LC\_MONETARY=“en\_US.UTF-8” > > LC\_MESSAGES=“en\_US.UTF-8” > > LC\_PAPER=“en\_US.UTF-8” > > LC\_NAME=“en\_US.UTF-8” > > LC\_ADDRESS=“en\_US.UTF-8” > > LC\_TELEPHONE=“en\_US.UTF-8” > > LC\_MEASUREMENT=“en\_US.UTF-8” > > LC\_IDENTIFICATION=“en\_US.UTF-8” > > LC\_ALL= > > > > after enough of these errors are generated, I begin to get errors > > for > > having too many open files, and the indexing fails. > > > > Error: exception 2 not handled: Too many open files > > > > Any suggestions would be greatly appreciated. > > > > Thanks, > > Eric > > > > * * * > > > > Ferret-talk mailing list > > removed\_email\_address@domain.invalid > > [http://rubyforge.org/mailman/listinfo/ferret-talk](http://rubyforge.org/mailman/listinfo/ferret-talk) > > – > Jens Krämer > Finkenlust 14, 06449 Aschersleben, Germany > VAT Id DE251962952 > [http://www.jkraemer.net/](http://www.jkraemer.net/) - Blog > [http://www.omdb.org/](http://www.omdb.org/) - The new free film database ---
**Author:** ![soderic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/8baadc/32.png) [@soderic](https://www.ruby-forum.com/u/soderic)\ **Post date:** [May 30, 2008, 11:46am UTC](https://www.ruby-forum.com/t/error-decoding-input-string/138034/4 "2008-05-30T11:46:09Z")
Hi, So I’ve tried switching to the latest version of Ferret (0.11.06), but I am still getting the following errors. ,---- | Error: exception 2 not handled: Error decoding input string. Check that you have the locale set correctly | from spanish\_indexer.rb:45 | from spanish\_indexer.rb:38:in `each' | from spanish_indexer.rb:38`---- The articles are recognized as valid utf8 using iconv, and I believe my locale is set properly ,---- | LANG=en\_US.UTF-8 | LC\_CTYPE=“en\_US.UTF-8” | LC\_NUMERIC=“en\_US.UTF-8” | LC\_TIME=“en\_US.UTF-8” | LC\_COLLATE=“en\_US.UTF-8” | LC\_MONETARY=“en\_US.UTF-8” | LC\_MESSAGES=“en\_US.UTF-8” | LC\_PAPER=“en\_US.UTF-8” | LC\_NAME=“en\_US.UTF-8” | LC\_ADDRESS=“en\_US.UTF-8” | LC\_TELEPHONE=“en\_US.UTF-8” | LC\_MEASUREMENT=“en\_US.UTF-8” | LC\_IDENTIFICATION=“en\_US.UTF-8” | LC\_ALL= `---- what’s weird here is that the errors don’t always happen on the same articles, if I try to run indexing three times, printing out the articles that throw this error, I get a different list of articles each time. In fact I just changed my indexing script so that it keeps trying to index failed articles ,---- | # ind is my index | # | # add\_arts is a method which takes a list of articles, tries to | # index them, and returns a list of the articles that | # threw errors during indexing | # | puts art\_paths.size.to\_s + “articles” | missed = add\_arts(art\_paths, ind) | while missed.size \> 0 | missed = add\_arts(missed, ind) | puts missed.size | end `---- and I was able to index all of the articles with the following output ,---- | 5843 articles | 34 | 16 | 10 | 9 | 7 | 7 | 6 | 1 | 0 `---- any ideas what could be causing this non-deterministic behavior? Thanks, Eric