# Setting encoding of pages in Capybara

**URL:** <https://www.ruby-forum.com/t/setting-encoding-of-pages-in-capybara/194575>\
**Category:** Ruby\
**Created:** [September 7, 2010, 12:35pm UTC](https://www.ruby-forum.com/t/setting-encoding-of-pages-in-capybara/194575 "2010-09-07T12:35:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![james-calivar](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/2bfe46/32.png) [@james-calivar](https://www.ruby-forum.com/u/james-calivar)\
**Post date:** [September 7, 2010, 12:35pm UTC](https://www.ruby-forum.com/t/setting-encoding-of-pages-in-capybara/194575/1 "2010-09-07T12:35:59Z")

</div>

Hi all,

Quick encoding question: say I’m trying to grab data from a Japanese  
page  
using Capybara and Rack::Test, and I get badly encoded text in the  
response.  
e.g. running this script:

require ‘rubygems’  
require ‘capybara’  
require ‘rack/test’  
require ‘rack/proxy’

Capybara.default\_selector = :css

class Japan \< Rack::Proxy  
def rewrite\_env(env)  
env[‘HTTP\_HOST’] = ‘[l-tike.com](http://l-tike.com)’  
env  
end  
end

session = Capybara::Session.new(:rack\_test, Japan.new)  
session.visit ‘/pickup/concert\_more.html’  
puts session.body

You’ll see weird characters in the output, and I can’t find nodes that  
should be there with css/xpath. How do I set the encoding so that  
Nokogiri  
parses the page properly?

---

<div class="post-metadata">

**Author:** ![james-calivar](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/2bfe46/32.png) [@james-calivar](https://www.ruby-forum.com/u/james-calivar)\
**Post date:** [September 7, 2010, 1:30pm UTC](https://www.ruby-forum.com/t/setting-encoding-of-pages-in-capybara/194575/2 "2010-09-07T13:30:27Z")

</div>

Hi,

On Tue, Sep 7, 2010 at 6:27 AM, James C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Hi all,
> 
> Quick encoding question: say I’m trying to grab data from a Japanese page  
> using Capybara and Rack::Test, and I get badly encoded text in the  
> response.  
> e.g. running this script:

First, a quick note, that this question is probably more appropriate for  
the  
capybara or nokogiri mailing lists. You’re likely to get a quicker  
response  
from those groups.

> env[‘HTTP\_HOST’] = ‘[l-tike.com](http://l-tike.com)’  
> env  
> end  
> end
> 
> session = Capybara::Session.new(:rack\_test, Japan.new)  
> session.visit ‘/pickup/concert\_more.html’  
> puts session.body

It looks like this page claims (in its header) to be encoding in  
SHIFT\_JIS,  
but the page is encoded in UTF-8. LibXML’s guesses at encoding are not  
perfect, and in this case the misleading information causes it to trust  
the  
header and use the wrong encoding.

If this page is edited to contain

instead of

then all is well.

Perhaps someone with more experience than me using non-western character  
sets will have a deeper insight into libxml’s behavior here?