# Open URI and web scraping

**URL:** <https://www.ruby-forum.com/t/open-uri-and-web-scraping/120002>\
**Category:** Ruby\
**Created:** [November 12, 2007, 8:42pm UTC](https://www.ruby-forum.com/t/open-uri-and-web-scraping/120002 "2007-11-12T20:42:27Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeannibee](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/85e7bf/32.png) [@jeannibee](https://www.ruby-forum.com/u/jeannibee)\
**Post date:** [November 12, 2007, 8:42pm UTC](https://www.ruby-forum.com/t/open-uri-and-web-scraping/120002/1 "2007-11-12T20:42:27Z")

</div>

Hi

I apologize if this is off topic but I couldn’t figure out how to reach  
the right audience based on the lists on this forum.

Here goes; I am writing a web scraper in an attempt to mimic a VERY  
light version of ‘wget’. I have to call a reports page, get the html,  
write the html (and any associated images for graphs) to the filesystem  
and this end up being used in a pdf converter to create a nice pdf file  
for a client.

There are other more easy solutions such as printer drivers and such,  
but at my office this is not an option. The current ‘flow’ I’m using is  
the only one I am allowed to use right or wrong. (and understand a lot  
of the right ways to do it are NOT distributable under their licenses  
agreements so I have to create it from scratch).

That being said, here’s my problem…

The images in the document are NOT referenced as static files on the  
file system they are actually HTTP urls to the webservers’ servlet to be  
generated ‘on the fly’.

Example would be : ![](http://myserver:8080/Someservlet?name=blah&param=value&etc=etc)

When I try and get these images (and I write the content from the http  
request to the file system) the filesizes are sized 0. No errors happen  
on the http request.

Also if I copy/paste thsi URL in a browser windows I’m told an error of:

XML Parsing Error: no element found  
Location: [http://myserver:8080/Someservlet?name=blah&param=value&etc=etc](http://myserver:8080/Someservlet?name=blah&param=value&etc=etc)  
Line Number 1, Column 1:

BUT, this image is displayed PERFECTLY in the report.

How can I get this image to download? (I suspect it’s the mime type  
being set on the server side but I am not 100% sure)

PSEUDO CODE:  
img\_path =  
“[http://myserver:8080/Someservlet?name=blah&param=value&etc=etc](http://myserver:8080/Someservlet?name=blah&param=value&etc=etc)”  
open( img\_path,  
“User-Agent” =\> “Ruby/#{RUBY\_VERSION}”,  
“From” =\> “me@mecom”, “Referer” =\> “[http://localhost:8080/](http://localhost:8080/)”) { |i|  
dump\_uri\_info(i)  
puts “IMAGE INFO!!! → #{i.read}”  
write\_to\_binary( rnd\_filename, i.read )  
}

def dump\_uri\_info(f)  
puts “[[URI information…]]”  
puts “Fetched document: #{f.base\_uri}”  
puts “Content Type: #{f.content\_type}”  
puts “Charset: #{f.charset}”  
puts “Content-Encoding: #{f.content\_encoding}”  
puts “Last Modified: #{f.last\_modified}”  
end

* * *

OUTPUT

* * *

[[URI information…]]  
Fetched document:  
[http://myserver:8080/Someservlet?name=blah&param=value&etc=etc](http://myserver:8080/Someservlet?name=blah&param=value&etc=etc)  
Content Type: application/voicexml+xml  
Charset:  
Content-Encoding:  
Last Modified:  
IMAGE INFO!!! →   
Writing to file ::  
D:\sandbox\auto\_attendant\archive\_reports\trunk\dumps\1194882652\_854.gif

last comment… if I ‘fetch’ an actual image referenced on the file  
system ([http://myserver:8080/images/log.gif](http://myserver:8080/images/log.gif)) It will be created properly  
and if I look at it it’s perfect. Its’ only the dynamically generated  
images.

Thanks for your help.