# Screen scraping programaticprogrammer.com?

**URL:** <https://www.ruby-forum.com/t/screen-scraping-programaticprogrammer-com/114502>\
**Category:** Ruby\
**Created:** [September 13, 2007, 2:39pm UTC](https://www.ruby-forum.com/t/screen-scraping-programaticprogrammer-com/114502 "2007-09-13T14:39:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![7stud](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/7/57b2e6/32.png) [@7stud](https://www.ruby-forum.com/u/7stud)\
**Post date:** [September 13, 2007, 2:39pm UTC](https://www.ruby-forum.com/t/screen-scraping-programaticprogrammer-com/114502/1 "2007-09-13T14:39:39Z")

</div>

The following is from “Programming Ruby 2nd” p.133:

* * *

require “net/http”

h = Net::HTTP.new(“[www.programaticprogrammer.com](http://www.programaticprogrammer.com)”, 80)  
response = h.get(“/index.html”)

## if response.message == “OK” puts response.body.scan(/\<img src=“(.\*?)”/m).uniq end

It doesn’t work: nothing is printed. So, I modified it a little:

* * *

require “net/http”

h = Net::HTTP.new(“[www.programaticprogrammer.com](http://www.programaticprogrammer.com)”, 80)  
response = h.get(“/index.html”)

puts response.message  
puts response.code

## if response.message == “OK” puts “_" puts response.body.scan(/\<img src="(._?)”/m).uniq end

and the output was:

Found  
302

I clicked a link on their home page and tried to access the page that  
was displayed, but I got the same result. What am I doing wrong?

---

<div class="post-metadata">

**Author:** ![7stud](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/7/57b2e6/32.png) [@7stud](https://www.ruby-forum.com/u/7stud)\
**Post date:** [September 13, 2007, 2:48pm UTC](https://www.ruby-forum.com/t/screen-scraping-programaticprogrammer-com/114502/2 "2007-09-13T14:48:19Z")

</div>

> * * *
> 
> and the output was:
> 
> Found  
> 302
> 
> I clicked a link on their home page and tried to access the page that  
> was displayed, but I got the same result. What am I doing wrong?

Wrong URL. How about using [www.pragmaticprogrammer.com](http://www.pragmaticprogrammer.com) instead?

I think the prOGRamatic programmers are slowly dying out anyway in  
favour of the pragmatic programmers… 😉

Ronald

---

<div class="post-metadata">

**Author:** ![7stud](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/7/57b2e6/32.png) [@7stud](https://www.ruby-forum.com/u/7stud)\
**Post date:** [September 13, 2007, 2:54pm UTC](https://www.ruby-forum.com/t/screen-scraping-programaticprogrammer-com/114502/3 "2007-09-13T14:54:20Z")

</div>

Ronald F. wrote:

> Wrong URL. How about using [www.pragmaticprogrammer.com](http://www.pragmaticprogrammer.com) instead?

Whoops. Thanks.

---

<div class="post-metadata">

**Author:** ![7stud](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/7/57b2e6/32.png) [@7stud](https://www.ruby-forum.com/u/7stud)\
**Post date:** [September 13, 2007, 2:58pm UTC](https://www.ruby-forum.com/t/screen-scraping-programaticprogrammer-com/114502/4 "2007-09-13T14:58:20Z")

</div>

I’ve just checked it, and the code on [pragmaticprogrammer.com](http://pragmaticprogrammer.com) (you made  
a  
typo, probably) returns a 200 OK, and the program works as expected. It  
could be the typo, but programmaticprogrammer does not exist for me, and  
you’re getting a 302 (it’s a redirect to another page) so I don’t know  
what’s happening exactly. Take a look on HTTP response codes if the  
problem  
wasn’t the URL.

Regards.

Diego.

---

<div class="post-metadata">

**Author:** ![7stud](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/7/57b2e6/32.png) [@7stud](https://www.ruby-forum.com/u/7stud)\
**Post date:** [September 17, 2007, 1:25pm UTC](https://www.ruby-forum.com/t/screen-scraping-programaticprogrammer-com/114502/5 "2007-09-17T13:25:41Z")

</div>

Just remember that with screen scraping, you are anticipating a file  
served by a file server, on top of that you are generally  
anticipating a very particular structure in that document. Web sites  
change frequently and without notice and even the smallest changes  
can blow out your scraper. So be very careful to inspect the various  
pages of sites you plan to scrape, and then try to write your scraper  
to check for things and not fail if it isn’t found.

With some clever programming and a little knowledge of the site, you  
can make a simple but smart scraper. However, it will still be pretty  
fragile. html/xhtml is just too loose and human-language like, full  
of ambiguity and implicit meaning that humans would get, but machines  
would work hard to fail at.