# Nokogiri not getting html body sometimes

**URL:** <https://www.ruby-forum.com/t/nokogiri-not-getting-html-body-sometimes/168432>\
**Category:** Ruby\
**Created:** [May 20, 2009, 7:44pm UTC](https://www.ruby-forum.com/t/nokogiri-not-getting-html-body-sometimes/168432 "2009-05-20T19:44:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![juuser](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/5f8ce5/32.png) [@juuser](https://www.ruby-forum.com/u/juuser)\
**Post date:** [May 20, 2009, 7:44pm UTC](https://www.ruby-forum.com/t/nokogiri-not-getting-html-body-sometimes/168432/1 "2009-05-20T19:44:51Z")

</div>

I’m using Mechanize to get imdb page and then Nokogiri Node#search  
method to get some info from the page, but I’ve stumbled onto one  
special case where #search doesn’t work properly, e.g. all other pages  
I’ve tried so far work as expected.

It seems that some special characters are causing the trouble for  
Nokogiri, because when I tried to print document itself it outputted  
only half of tag and no body tags at all!

Anyway here is the code snippet which I’d expect to output “false” 4  
times. Instead, it outputs false, false, true, false. Try with some  
other imdb url and it’s ok.

require ‘mechanize’

mech = WWW::Mechanize.new {|agent| agent.user\_agent\_alias = ‘Windows  
Mozilla’}  
mech.get(“[Meu Nome Não é Johnny (2008) - IMDb](http://www.imdb.com/title/tt1092016/)”) do |page|  
puts page.search(“/html”).empty?  
puts page.search(“/html/head”).empty?  
puts page.search(“/html/body”).empty?  
puts page.body.empty?  
end

What could be the problem?

I’m using ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

---

<div class="post-metadata">

**Author:** ![juuser](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/5f8ce5/32.png) [@juuser](https://www.ruby-forum.com/u/juuser)\
**Post date:** [May 21, 2009, 4:28pm UTC](https://www.ruby-forum.com/t/nokogiri-not-getting-html-body-sometimes/168432/2 "2009-05-21T16:28:20Z")

</div>

i think you’d better set the encoding first.

mech.get(“[Meu Nome Não é Johnny (2008) - IMDb](http://www.imdb.com/title/tt1092016/)”) do |page|  
page.encoding = ‘ISO-8859-1’  
#… the rest of ur code  
end

---

<div class="post-metadata">

**Author:** ![juuser](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/5f8ce5/32.png) [@juuser](https://www.ruby-forum.com/u/juuser)\
**Post date:** [May 21, 2009, 6:32pm UTC](https://www.ruby-forum.com/t/nokogiri-not-getting-html-body-sometimes/168432/3 "2009-05-21T18:32:22Z")

</div>

Thank you! It did the trick.

Best regards,  
Jarmo

Lui Core wrote:

> i think you’d better set the encoding first.
> 
> mech.get(“[Meu Nome Não é Johnny (2008) - IMDb](http://www.imdb.com/title/tt1092016/)”) do |page|  
> page.encoding = ‘ISO-8859-1’  
> #… the rest of ur code  
> end