# Youtube...urgent, please help **URL:** \ **Category:** Ruby\ **Created:** [March 17, 2009, 5:45am UTC](https://www.ruby-forum.com/t/youtube-urgent-please-help/163072 "2009-03-17T05:45:08Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [March 17, 2009, 5:45am UTC](https://www.ruby-forum.com/t/youtube-urgent-please-help/163072/1 "2009-03-17T05:45:08Z")
Hi, I’m new to ruby and my co. has given me an assignment in ruby. It is regarding html extraction. It works fine except for some sites like [http://www.youtube.com](http://www.youtube.com), [http://www.gmail.com](http://www.gmail.com) where i’ll get errors like ‘400 Bad Request’ and ‘getaddrinfo: Name or service not known (SocketError)’ respectively for each of the 2 sites. I came to know that may be it is because the url is being redirected. But i’m not sure about it. My code for html extraction is : require ‘rubygems’ require ‘hpricot’ require ‘open-uri’ require ‘dbi’ puts “Enter domain name :” domain = gets #concatinating ‘[http://www](http://www).’ with the url to open the page url = “[http://www](http://www).”+domain document = open(url) #getting the original url of the site url2 = document.base\_uri.to\_s Can anybody please help. It is urgent. I’ll be really greatful for those who reply Regards, Arun K. ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [March 17, 2009, 6:45am UTC](https://www.ruby-forum.com/t/youtube-urgent-please-help/163072/2 "2009-03-17T06:45:11Z")
Arun K. wrote: > Hi, > > I’m new to ruby and my co. has given me an assignment in ruby. It is > regarding html extraction. You probably want Mechanize. > domain = gets > #concatinating ‘[http://www](http://www).’ with the url to open the page > url = “[http://www](http://www).”+domain Take a look at that URL – I’d say you don’t need ‘www’ in that. But I’m guessing what’s hurting is the newline at the end of it. Quick fix: domain = gets.chomp url = “http://#{domain}” ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [March 17, 2009, 7:00am UTC](https://www.ruby-forum.com/t/youtube-urgent-please-help/163072/3 "2009-03-17T07:00:30Z")
David M. wrote: > Arun K. wrote: > > > Hi, > > > > I’m new to ruby and my co. has given me an assignment in ruby. It is > > regarding html extraction. > > You probably want Mechanize. > > > domain = gets > > #concatinating ‘[http://www](http://www).’ with the url to open the page > > url = “[http://www](http://www).”+domain > > Take a look at that URL – I’d say you don’t need ‘www’ in that. > > But I’m guessing what’s hurting is the newline at the end of it. > > Quick fix: > > domain = gets.chomp > url = “http://#{domain}” > Sorry to say David, I tried that but the same error is producing. Is it > because i’ve not set the user agent. Can u please tell me how to set the > user\_agent for mozilla. > Thanks for ur immediate reply ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [March 17, 2009, 7:12am UTC](https://www.ruby-forum.com/t/youtube-urgent-please-help/163072/4 "2009-03-17T07:12:09Z")
On Tue, Mar 17, 2009 at 11:28 AM, Arun K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Sorry to say David, I tried that but the same error is producing. Is it > because i’ve not set the user agent. Can u please tell me how to set the > user\_agent for mozilla. [http://mechanize.rubyforge.org/mechanize/EXAMPLES\_rdoc.html](http://mechanize.rubyforge.org/mechanize/EXAMPLES_rdoc.html) has some examples setting the user agent. Google around and see what the mozilla user agent should be - [List of User-Agents (Spiders, Robots, Browser)](http://www.user-agents.org/index.shtml?moz) has an extensive list, for instance. > Thanks for ur immediate reply Don’t do that, it’s annoying. martin ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [March 17, 2009, 7:38am UTC](https://www.ruby-forum.com/t/youtube-urgent-please-help/163072/5 "2009-03-17T07:38:30Z")
2009/3/17 Arun K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > Can i use user-agents in hpricot? or if it can be used only for > mechanize. I’ve found a user-agent for mozilla : > Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR > 1.1.4322; .NET CLR 2.0.50727) > But still it is showing the same error. I found this: [http://schf.uc.org/articles/2007/02/14/scraping-gmail-with-mechanize-and-hpricot](http://schf.uc.org/articles/2007/02/14/scraping-gmail-with-mechanize-and-hpricot) It scraps gmail. If my memory doesn’t fail, it is one that gives you some problems. Cheers, Serabe ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [March 17, 2009, 7:28am UTC](https://www.ruby-forum.com/t/youtube-urgent-please-help/163072/6 "2009-03-17T07:28:04Z")
Martin DeMello wrote: > On Tue, Mar 17, 2009 at 11:28 AM, Arun K. > [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > Sorry to say David, I tried that but the same error is producing. Is it > > because i’ve not set the user agent. Can u please tell me how to set the > > user\_agent for mozilla. > > [http://mechanize.rubyforge.org/mechanize/EXAMPLES\_rdoc.html](http://mechanize.rubyforge.org/mechanize/EXAMPLES_rdoc.html) has some > examples setting the user agent. Google around and see what the > mozilla user agent should be - > [List of User-Agents (Spiders, Robots, Browser)](http://www.user-agents.org/index.shtml?moz) has an extensive list, for > instance. > > > Thanks for ur immediate reply > > Don’t do that, it’s annoying. > > martin Can i use user-agents in hpricot? or if it can be used only for mechanize. I’ve found a user-agent for mozilla : Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) But still it is showing the same error. ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [March 17, 2009, 8:26pm UTC](https://www.ruby-forum.com/t/youtube-urgent-please-help/163072/7 "2009-03-17T20:26:16Z")
Martin DeMello wrote: > On Tue, Mar 17, 2009 at 11:55 AM, Arun K. > [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > Can i use user-agents in hpricot? or if it can be used only for > > mechanize. > > Hpricot is an html parser, I don’t think it concerns itself with > actually fetching the page. Use mechanize for that. What’s more, mechanize doesn’t even use hpricot anymore – it uses nokogiri. ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [March 17, 2009, 7:56am UTC](https://www.ruby-forum.com/t/youtube-urgent-please-help/163072/8 "2009-03-17T07:56:44Z")
On Tue, Mar 17, 2009 at 11:55 AM, Arun K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Can i use user-agents in hpricot? or if it can be used only for > mechanize. Hpricot is an html parser, I don’t think it concerns itself with actually fetching the page. Use mechanize for that. martin