# Looping through & downloading web page - WATIR codes

**URL:** <https://www.ruby-forum.com/t/looping-through-downloading-web-page-watir-codes/78349>\
**Category:** Ruby\
**Created:** [December 5, 2006, 7:56am UTC](https://www.ruby-forum.com/t/looping-through-downloading-web-page-watir-codes/78349 "2006-12-05T07:56:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![curious](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/6bbea6/32.png) [@curious](https://www.ruby-forum.com/u/curious)\
**Post date:** [December 5, 2006, 7:56am UTC](https://www.ruby-forum.com/t/looping-through-downloading-web-page-watir-codes/78349/1 "2006-12-05T07:56:23Z")

</div>

What I am trying to realize through code below is to read a line by  
line in items.txt, and download each ensuing pages by having WATIR  
click ‘Next’ link.

Now what I wish this code to do is when WATIR reaches the last page,  
and there is no more “Next” link on the page, then go to the second  
line in items.txt, searching again, and download ensuing pages again.

But unfortunately, the WATIR code close down when there is no more  
“Next” link on the first item search… and it does not go to searching  
with the second line in items.txt.

Could anybody help me how I should adjust the code below so that when  
there is no more “Next” link on the page, then the second line in  
items.txt goes into searching, download each page by clicking “Next”  
link… on and on??

any suggestion will be deeply appreciated.

Thanks.

=====================================================================  
require ‘watir’

ie = Watir::IE.start(“[http://www.test.com](http://www.test.com)”)

file = File.open(“c:/items.txt”, “r”)  
lines = file.readlines()

PageNo = 1

0.upto(lines.length - 1) {|j|

$item = lines[j]

ie.text\_field(:name, “item”).value = $item

ie.button(:name, “Search”).click

a = ie.contains\_text(“Next”)

while a

File.open(“#{PageNo}.html”, “wb”) { |f|

f \<\< ‘’  
f \<\< ie.html  
f \<\< ‘’

PageNo = PageNo + 1

ie.link(:text, “Next”).click

}

end

}

file.close()  
lines.clear()

---

<div class="post-metadata">

**Author:** ![curious](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/6bbea6/32.png) [@curious](https://www.ruby-forum.com/u/curious)\
**Post date:** [December 5, 2006, 9:56am UTC](https://www.ruby-forum.com/t/looping-through-downloading-web-page-watir-codes/78349/2 "2006-12-05T09:56:35Z")

</div>

curious wrote:

> with the second line in items.txt.
> 
> Could anybody help me how I should adjust the code below so that when  
> there is no more “Next” link on the page, then the second line in  
> items.txt goes into searching, download each page by clicking “Next”  
> link… on and on??
> 
> any suggestion will be deeply appreciated.

Rather than simply rewriting your code for you, I offer these  
suggestions:

1. Always close the opened file handle when the write is completed. You  
are  
not closing your file handles until the last file is read, which in the  
worst case leaves many file handles open and abandoned.

2. Instead of doing this:

0.upto(lines.length - 1) {|j|  
$item = lines[j]

Do it this way:

lines.each do |item|

# … your code here

end

1. Avoid use of global variables when possible.

I would offer more suggestions but I don’t actually understand what your  
program is meant to accomplish.

---

<div class="post-metadata">

**Author:** ![curious](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/6bbea6/32.png) [@curious](https://www.ruby-forum.com/u/curious)\
**Post date:** [December 5, 2006, 10:00am UTC](https://www.ruby-forum.com/t/looping-through-downloading-web-page-watir-codes/78349/3 "2006-12-05T10:00:36Z")

</div>

I found out what the problem is…

it is the last part…

ie.link(:text, “Next”).click… in the last web page, although “Next”  
text is contained, link is not present since it is the last page…

Is there anyway, I can control “ie.link(:text, “Next”).click” by  
checking the presence of link beforehand?? so if link is present, then  
proceed, otherwise… halt… I think my code will do as I wish once  
this gets done…

thanks.

---

<div class="post-metadata">

**Author:** ![curious](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/6bbea6/32.png) [@curious](https://www.ruby-forum.com/u/curious)\
**Post date:** [December 5, 2006, 10:20am UTC](https://www.ruby-forum.com/t/looping-through-downloading-web-page-watir-codes/78349/4 "2006-12-05T10:20:41Z")

</div>

curious wrote:

> I found out what the problem is…
> 
> it is the last part…
> 
> ie.link(:text, “Next”).click… in the last web page, although “Next”  
> text is contained, link is not present since it is the last page…
> 
> Is there anyway, I can control “ie.link(:text, “Next”).click” by  
> checking the presence of link beforehand??

Sorry, I don’t know enough about Watir to answer this question.

> so if link is present, then  
> proceed, otherwise… halt… I think my code will do as I wish once  
> this gets done…

No, it won’t. It won’t because you are not closing your file handles. If  
you  
don’t close your file handles, the writes will not be completed and this  
could lead to file corruption.

At the end of the process, after writing any number files, you close the  
last of the file handles, but you must close all of them.

I’m sorry I can’t answer more of your questions.