# Encoding issue

**URL:** <https://www.ruby-forum.com/t/encoding-issue/169429>\
**Category:** Rails\
**Created:** [June 3, 2009, 5:11pm UTC](https://www.ruby-forum.com/t/encoding-issue/169429 "2009-06-03T17:11:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mk27](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/5e9695/32.png) [@mk27](https://www.ruby-forum.com/u/mk27)\
**Post date:** [June 3, 2009, 5:11pm UTC](https://www.ruby-forum.com/t/encoding-issue/169429/1 "2009-06-03T17:11:38Z")

</div>

I’m having a problem with output gleaned from searching ISO-8859-9 web  
pages.

I thought I had rectified this by opening the pages “r:ISO…”, and by  
using .force\_encoding and .encode! with the lines. This seems to work,  
as according to logger.info output, execution makes it all the way  
through the controller AND the view code…meaning I would have thought  
there is nothing left for me to do.

After the controller code has run I get:  
Rendering docsearch/search  
Then the view code runs and I get:  
Completed in 1179ms (View: 11, DB: 1) | 200 OK  
[[http://localhost/docdir/search](http://localhost/docdir/search)]

THEN,  
Processing ApplicationController#search  
which seems strange, since  
Processing DocdirController#search  
already happened. I had not noticed this before; what is the difference  
between these two, and what is the significance of “Processing  
ApplicationController#search” AFTER the view is already complete?

Anyway, that’s when I get  
ArgumentError (invalid byte sequence in UTF-8):  
internal:prelude:8:in `synchronize' /usr/local/lib/ruby/1.9.1/webrick/httpserver.rb:111:in `service’  
/usr/local/lib/ruby/1.9.1/webrick/httpserver.rb:70:in `run' /usr/local/lib/ruby/1.9.1/webrick/server.rb:183:in `block in  
start\_thread’

So I have tried setting both internal and external encodings:  
Encoding.default\_external = @dir.encoding  
Encoding.default\_internal = @dir.encoding  
In the hopes that this would mean everything – parsing, IO, etc. will  
be done in ISO-8859-1, but obviously this is not the case. I presume I  
cannot set the encoding for the scripts that aren’t mine, ie, the  
webrick scripts that are throwing this error.

What can I do? AFAICT, all my code has executed without error, every  
line dealt with is output via the logger, in the controller AND in the  
view, A-ok, and then this happens? Why can I not just set _everything_  
to one encoding for a duration? Is it webrick? I also tried using  
Encoding::Converter on everything read in, to no avail…Help!

–MK

---

<div class="post-metadata">

**Author:** ![mk27](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/5e9695/32.png) [@mk27](https://www.ruby-forum.com/u/mk27)\
**Post date:** [June 3, 2009, 10:41pm UTC](https://www.ruby-forum.com/t/encoding-issue/169429/2 "2009-06-03T22:41:35Z")

</div>

Well, I solved it using Encoding::Convert. The difference between  
“each” (which does not allow in place changes) and “each\_with\_index” had  
me hung up. Ruby newbie!

I would still LOVE to know why/how iconv self-destructed tho…

---

<div class="post-metadata">

**Author:** ![mk27](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/5e9695/32.png) [@mk27](https://www.ruby-forum.com/u/mk27)\
**Post date:** [June 3, 2009, 8:37pm UTC](https://www.ruby-forum.com/t/encoding-issue/169429/3 "2009-06-03T20:37:25Z")

</div>

This works in a non-rails script:

def enconvert(text)  
conv=Encoding::Converter.new(“ISO-8859-1”,“UTF-8”,  
:undef =\> :replace, :invalid =\> :replace)  
text.each { |ln| ln=conv.convert(ln) }  
end

but in rails I still get an error, after “converting” the same page.

SURELY SOMEONE has dealt with encoding conversion before??

ps. iconv mysteriously self-destructed on my system this morning. It  
worked briefly, but now

require “iconv”

produces an error (so iconv is now unusable on my system until I rebuild  
ruby). Does anyone know if Encoding::Converter uses iconv under the  
hood?