# How come this code doesnt work as designed?

**URL:** <https://www.ruby-forum.com/t/how-come-this-code-doesnt-work-as-designed/129493>\
**Category:** Ruby\
**Created:** [February 19, 2008, 2:12am UTC](https://www.ruby-forum.com/t/how-come-this-code-doesnt-work-as-designed/129493 "2008-02-19T02:12:12Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![gmoniey](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/6a8cbe/32.png) [@gmoniey](https://www.ruby-forum.com/u/gmoniey)\
**Post date:** [February 19, 2008, 2:12am UTC](https://www.ruby-forum.com/t/how-come-this-code-doesnt-work-as-designed/129493/1 "2008-02-19T02:12:12Z")

</div>

Hi,

I found this web crawler code online using mechanize,

require ‘mechanize’

agent = WWW::Mechanize.new  
page = agent.get(‘[http://example.com/](http://example.com/)’)

stack = page.links  
counter = 0;

out = File.open(“out.txt”, “w”)  
while l = stack.pop  
begin  
next unless l.uri.host == agent.history.first.uri.host  
if not agent.visited? l.href  
counter += 1  
out.puts l.href  
stack.push(\*(agent.click(l).links))  
end  
rescue  
#puts “Error encountered”  
end  
end

puts "Total unique links: " + counter.to\_s

So I gave it a try, and although it seemed to be working, I noticed that  
the stack size quickly rocketed, and after examining the output, I  
noticed that there are several duplicates (For example, one output file  
had over 50k urls, but when I removed the duplicates, there was only a  
bit over 9k urls). So I modified the code using a Hash to avoid  
duplicates (although this design means that I am storing multiple copies  
of all the urls), but the same thing happened, so I was wondering if  
anyone could figure out what I am doing wrong. Here is the modified  
code:

require ‘mechanize’

agent = WWW::Mechanize.new  
page = agent.get(‘[http://example.com/](http://example.com/)’)

stack = page.links  
hash = Hash.new  
counter = 0;

out = File.open(“out.txt”, “w”)  
while l = stack.pop  
begin  
next unless l.uri.host == agent.history.first.uri.host  
if not agent.visited? l.href  
counter += 1  
out.puts "url:1 " + l.href  
agent.click(l).links.each do |link|  
if(hash[link] == nil)  
hash.store(link,link)  
stack.push(link)  
end  
end  
#stack.push(\*(agent.click(l).links))  
end  
rescue  
#puts “Error encountered”  
end  
end

puts "Total unique links: " + counter.to\_s

Note: I am aware that crawling sites at random is not accepted, and this  
script is not intended for that, I am crawling personal sites