Looping through & downloading web page - WATIR codes

Ruby
1

What I am trying to realize through code below is to read a line by
line in items.txt, and download each ensuing pages by having WATIR
click ‘Next’ link.

Now what I wish this code to do is when WATIR reaches the last page,
and there is no more “Next” link on the page, then go to the second
line in items.txt, searching again, and download ensuing pages again.

But unfortunately, the WATIR code close down when there is no more
“Next” link on the first item search… and it does not go to searching
with the second line in items.txt.

Could anybody help me how I should adjust the code below so that when
there is no more “Next” link on the page, then the second line in
items.txt goes into searching, download each page by clicking “Next”
link… on and on??

any suggestion will be deeply appreciated.

Thanks.

=====================================================================
require ‘watir’

ie = Watir::IE.start(“http://www.test.com”)

file = File.open(“c:/items.txt”, “r”)
lines = file.readlines()

PageNo = 1

0.upto(lines.length - 1) {|j|

$item = lines[j]

ie.text_field(:name, “item”).value = $item

ie.button(:name, “Search”).click

a = ie.contains_text(“Next”)

while a

File.open(“#{PageNo}.html”, “wb”) { |f|

f << ‘’
f << ie.html
f << ‘’

PageNo = PageNo + 1

ie.link(:text, “Next”).click

}

end

}

file.close()
lines.clear()

2

curious wrote:

with the second line in items.txt.

Could anybody help me how I should adjust the code below so that when
there is no more “Next” link on the page, then the second line in
items.txt goes into searching, download each page by clicking “Next”
link… on and on??

any suggestion will be deeply appreciated.

Rather than simply rewriting your code for you, I offer these
suggestions:

  1. Always close the opened file handle when the write is completed. You
    are
    not closing your file handles until the last file is read, which in the
    worst case leaves many file handles open and abandoned.

  2. Instead of doing this:

0.upto(lines.length - 1) {|j|
$item = lines[j]

Do it this way:

lines.each do |item|

… your code here

end

  1. Avoid use of global variables when possible.

I would offer more suggestions but I don’t actually understand what your
program is meant to accomplish.

3

I found out what the problem is…

it is the last part…

ie.link(:text, “Next”).click… in the last web page, although “Next”
text is contained, link is not present since it is the last page…

Is there anyway, I can control “ie.link(:text, “Next”).click” by
checking the presence of link beforehand?? so if link is present, then
proceed, otherwise… halt… I think my code will do as I wish once
this gets done…

thanks.

4

curious wrote:

I found out what the problem is…

it is the last part…

ie.link(:text, “Next”).click… in the last web page, although “Next”
text is contained, link is not present since it is the last page…

Is there anyway, I can control “ie.link(:text, “Next”).click” by
checking the presence of link beforehand??

Sorry, I don’t know enough about Watir to answer this question.

so if link is present, then
proceed, otherwise… halt… I think my code will do as I wish once
this gets done…

No, it won’t. It won’t because you are not closing your file handles. If
you
don’t close your file handles, the writes will not be completed and this
could lead to file corruption.

At the end of the process, after writing any number files, you close the
last of the file handles, but you must close all of them.

I’m sorry I can’t answer more of your questions.