# HTTP headers and source

**URL:** <https://www.ruby-forum.com/t/http-headers-and-source/112132>\
**Category:** Ruby\
**Created:** [August 15, 2007, 6:59pm UTC](https://www.ruby-forum.com/t/http-headers-and-source/112132 "2007-08-15T18:59:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![h4z3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/54ee81/32.png) [@h4z3](https://www.ruby-forum.com/u/h4z3)\
**Post date:** [August 15, 2007, 6:59pm UTC](https://www.ruby-forum.com/t/http-headers-and-source/112132/1 "2007-08-15T18:59:39Z")

</div>

Ok guys, lets say i wanted to grab the source for [google.com](http://google.com) or  
something… it wont allow if unless i send the correct headers to spoof  
the program… Can anyone give me a working example of how to send  
headers and download a webpage source?

I tried looking through all of the docs and coming up with something but  
i failed…

Thanks for any reply’s

---

<div class="post-metadata">

**Author:** ![h4z3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/54ee81/32.png) [@h4z3](https://www.ruby-forum.com/u/h4z3)\
**Post date:** [August 15, 2007, 7:18pm UTC](https://www.ruby-forum.com/t/http-headers-and-source/112132/2 "2007-08-15T19:18:15Z")

</div>

> example of how to send headers and download a webpage source?
> 
> I tried looking through all of the docs and coming up with  
> something but i failed…
> 
> ## Thanks for any reply’s
> 
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

With open-uri[0] you can open URIs just like local files. That would be  
entirely sufficient to get the content of the index page of [google.com](http://google.com),  
for  
example. Instead of a simple URL you can also pass the open call a  
URI[1]  
object, for which you can explicitly call headers if you need to.

You could then also also use Hpricot[2] to do all sorts of nifty HTML  
parsing

[0] - [http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/](http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/)  
[1] - [http://www.ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html](http://www.ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html)  
[2] - [http://code.whytheluckystiff.net/hpricot/](http://code.whytheluckystiff.net/hpricot/)

Felix

---

<div class="post-metadata">

**Author:** ![h4z3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/54ee81/32.png) [@h4z3](https://www.ruby-forum.com/u/h4z3)\
**Post date:** [August 15, 2007, 9:26pm UTC](https://www.ruby-forum.com/t/http-headers-and-source/112132/3 "2007-08-15T21:26:29Z")

</div>

Here’s my suggestion:

Firefox + LiveHTTPHeaders -  
[http://livehttpheaders.mozdev.org/installation.html](http://livehttpheaders.mozdev.org/installation.html)

LHH shows all HTTP chatter, so there’s nothing that a server can see  
that you can’t. From there it’s just a matter of imitating the headers  
with Net::HTTP.

Remember, though, that you have some vague sort of obligation to  
maintain netiquette. If a server rejects automated requests, they may  
have a good reason to, and you’re going against their wishes to mimic  
a real browser. I doubt the Feds are going to come kicking your door  
in over it, but it’s still worth trying to be respectful.

Google, for example, has an API that they encourage for automated  
usage. Here are some details:

> **[Google for Developers - Software Development Guides, Tools & More](https://developers.google.com/?csw=1)**
>
> Explore Google's developer tools for AI, Cloud, app, web, and software development to help you build smarter and ship faster.

-rking