Downloading a CSV using URI - mechanize

Ruby
1

Hi All,

Got an issue. I am downloading a csv in a script written in Mechanize. I
am setting some parameters on the webpage and then submit my form. This
directs me to another page where i have the following href link (got it
through firebug):

CSV

In my script to simulate click on this CSV href link, i take out the URI
and download the file. The code for this is below:

#Submitting the page with button Chnange Columns
page=pageAgent.submit(checkboxForm,checkboxForm.buttons.first)
#Retrieving the csv link href
csv_href=pageAgent.page.link_with(:text => ‘CSV’).uri
pageAgent.download("#{csv_href}",‘c:/magma/1.csv’)}

Issue: The downloaded CSV file is missing some data which should be
there. When i download the file directly from webpage i get correct
results in csv file. So i was wondering, could it be an issue that i
download the file using URI? As i noticed that the URI is lot bigger as
compared to the href link generated on the page after the search
parameters are set. Also if someone can tell me how can i download a
file using the href itself using mechanize.

Help much appreciated.

I have also attached a txt file that has the URI and the href both for
reference. Didnt wanted this post to be long to read unnecessarily. :slight_smile:

2

If you want fresh data each time you need to extract the URI from the
page every time.

Odds are using an older URI is giving you an older report.

3

On Mon, May 6, 2013 at 1:17 PM, Rochit S. [email protected] wrote:

Got an issue. I am downloading a csv in a script written in Mechanize. I
am setting some parameters on the webpage and then submit my form. This
directs me to another page where i have the following href link (got it
through firebug):

Show us the code. The two items you show in the attachment don’t quite
match.

4

tamouse mailing lists wrote in post #1108020:

On Mon, May 6, 2013 at 1:17 PM, Rochit S. [email protected] wrote:

Got an issue. I am downloading a csv in a script written in Mechanize. I
am setting some parameters on the webpage and then submit my form. This
directs me to another page where i have the following href link (got it
through firebug):

Show us the code. The two items you show in the attachment don’t quite
match.

Hi Tamouse and Joel

I have attached the full script for your reference.

Thanks

5

Joel P. wrote in post #1108049:

Obviously I can’t test your code under the same conditions, but from the
look of it you should get the same file if you do this manually.

The main thing which looks like it could go wrong is the checkbox
section.

allChkBoxes = checkboxForm.checkboxes
chkNames = [] # <= This is overwritten as a String each time
allChkBoxes.each do |chkBx|
chkNames = chkBx.name # Why not just do “chkBx.check” ?
checkboxForm.checkbox_with(:name => “#{chkNames}”).check
end

I think you’re doing something differently with the options when running
this manually. Unless there’s more than one link called CSV on the page?

Hi Joel, i am setting all the checkboxes to true, so when the csv is
downloaded i get all the columns. Setting the checkbox as true adds that
specific column in the csv. It should not affect the csv output.

What do you think? is my approach correct where i am getting the URI of
the href ‘csv’ and dowloading the file. Also i think there is just one
csv hyperlink on the page, though i will double check again.

thanks.

6

As far as I can see, you are getting the correct CSV link, although I
have to read between the lines as I can’t see the actual page or
results.

I think what you need to do is isolate exactly what the difference is
between the report you want and the one you’re getting. That should give
you a bit more of a clue as to where the difference is coming from.

7

Obviously I can’t test your code under the same conditions, but from the
look of it you should get the same file if you do this manually.

The main thing which looks like it could go wrong is the checkbox
section.

allChkBoxes = checkboxForm.checkboxes
chkNames = [] # <= This is overwritten as a String each time
allChkBoxes.each do |chkBx|
chkNames = chkBx.name # Why not just do “chkBx.check” ?
checkboxForm.checkbox_with(:name => “#{chkNames}”).check
end

I think you’re doing something differently with the options when running
this manually. Unless there’s more than one link called CSV on the page?

8

Indeed. If there’s any AJAX going on; you’ll do better with
watir-webdriver.

9

Joel P. wrote in post #1108083:

Indeed. If there’s any AJAX going on; you’ll do better with
watir-webdriver.

Hi Joel, all.
I think there is some issue when i set the multiselect
listbox.select_all.
Somehow the options in the select list that have a space in them. Eg,
‘Dev Task’ will not be selected. If option was ‘DevTask’ then it works
fine.
Could this be an issue with mechanize? Is there a way i can use regex
while
Selecting the options with a space. ?
Thanks

10

On Mon, May 6, 2013 at 11:56 PM, Rochit S. [email protected]
wrote:

Hi Tamouse and Joel

I have attached the full script for your reference.

Thanks

Nothing pops out at me from a short inspection. What I would suggest
is loading up the ruby debugger and stepping through it, inspecting
your page and fragment values to see if you have indeed got what you
think you should get. If the page you’re hitting on does any
javascript to put in page content that is necessary to the execution
of the function you want, mechanize won’t handle that.

11

Joel P. wrote in post #1108049:

chkNames = chkBx.name # Why not just do “chkBx.check” ?
checkboxForm.checkbox_with(:name => “#{chkNames}”).check

12

Joel P. wrote in post #1108143:

Joel P. wrote in post #1108049:

chkNames = chkBx.name # Why not just do “chkBx.check” ?
checkboxForm.checkbox_with(:name => “#{chkNames}”).check

Hi Joel,
The above you mentioned is for the checkboxes. The issue is with this:

searchForm.field_with(:name => ‘bug_severity’).select_all

When i do select_all for this MultiSelect called ‘bug_severity’ it does
not Select the options that have a space in them - eg, ‘Dev Task’ will
not be selected.

Any ideas why and how to rectify?

Thanks

13

I’ve never used a multiselect in mechanize, so I don’t know about that.

I think you can usually select either on value or on content, perhaps
the values don’t have spaces and you could use them.

Usually the best option is to work with the objects, so if you can find
a way to iterate through each option and select it without having to
access it by name, that might work.

Have a look at your options in IRB, something like:
p searchForm.field_with(:name => ‘bug_severity’).methods.sort