# Mechanize: How to scrape accented chars that weren't entered as entities?

**URL:** <https://www.ruby-forum.com/t/mechanize-how-to-scrape-accented-chars-that-werent-entered-as-entities/171680>\
**Category:** Ruby\
**Created:** [July 7, 2009, 3:19pm UTC](https://www.ruby-forum.com/t/mechanize-how-to-scrape-accented-chars-that-werent-entered-as-entities/171680 "2009-07-07T15:19:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Patrick\_Lajeunesse](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/9d8465/32.png) [@Patrick\_Lajeunesse](https://www.ruby-forum.com/u/Patrick_Lajeunesse)\
**Post date:** [July 7, 2009, 3:19pm UTC](https://www.ruby-forum.com/t/mechanize-how-to-scrape-accented-chars-that-werent-entered-as-entities/171680/1 "2009-07-07T15:19:24Z")

</div>

Hi,  
I’m trying to scrape links using Mechanize. Sometimes accented  
characters  
(on French pages) are corrupt once Ruby gets them. To see what I mean,  
check  
this:

require ‘mechanize’  
a = WWW::Mechanize.new  
page =  
a.get('[http://www.agr.gc.ca/cb/index\_f.php?s1=n&s2=index&page=2009\_07](http://www.agr.gc.ca/cb/index_f.php?s1=n&s2=index&page=2009_07)  
')  
page.links.each do |a\_link|  
puts a\_link  
end

Of course, it’s only the accents that are entered in plain text (i.e.,  
without entities) that have this problem. But in an imperfect world, I  
can’t  
always count on accents being entered properly.

Is there anything I can do about this? I’ve tried using Iconv to convert  
the  
strings to UTF-8, but that just resulted in a different (but still  
wrong)  
character in place of the broken ones.

Thanks for any help,

Patrick

---

<div class="post-metadata">

**Author:** ![Patrick\_Lajeunesse](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/9d8465/32.png) [@Patrick\_Lajeunesse](https://www.ruby-forum.com/u/Patrick_Lajeunesse)\
**Post date:** [July 7, 2009, 6:15pm UTC](https://www.ruby-forum.com/t/mechanize-how-to-scrape-accented-chars-that-werent-entered-as-entities/171680/2 "2009-07-07T18:15:26Z")

</div>

On Tue, Jul 07, 2009 at 10:18:50PM +0900, Patrick Lajeunesse wrote:

> puts a\_link  
> end
> 
> Of course, it’s only the accents that are entered in plain text (i.e.,  
> without entities) that have this problem. But in an imperfect world, I can’t  
> always count on accents being entered properly.
> 
> Is there anything I can do about this? I’ve tried using Iconv to convert the  
> strings to UTF-8, but that just resulted in a different (but still wrong)  
> character in place of the broken ones.

What version of nokogiri / mechanize do you have installed? I ran your  
code and was able to see the accents:

[Skitch | Evernote](http://skitch.com/aaron.patterson/bs4qt/terminal-bash-80x24)

Most of the time, these encoding issues are due to the server  
incorrectly identifying the encoding of the content. Is this content  
supposed to be ISO-8859-1?

---

<div class="post-metadata">

**Author:** ![Patrick\_Lajeunesse](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/9d8465/32.png) [@Patrick\_Lajeunesse](https://www.ruby-forum.com/u/Patrick_Lajeunesse)\
**Post date:** [July 8, 2009, 4:57am UTC](https://www.ruby-forum.com/t/mechanize-how-to-scrape-accented-chars-that-werent-entered-as-entities/171680/3 "2009-07-08T04:57:29Z")

</div>

Thanks Aaron - I thought I was up-to-date, but I guess I not. I did a  
gem  
update and got 0.9.3 - and then it worked fine.  
Thanks again,

Patrick

On Tue, Jul 7, 2009 at 12:13 PM, Aaron P.  
\<removed\_email\_address@domain.invalid