Using Mechanize To Submit Forms

Ruby
1

Hi guys,

I’m trying to use Mechanize with Ruby to scrape data from this site
(http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do). The first part of my
task is trying to automate an initial search form, I can do this fine
using the following code.

require ‘rubygems’
require ‘mechanize’

agent = Mechanize.new
page = agent.get(‘http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do’)

search_form = page.form(‘HPLCDS0101Form’)
search_form.callJorEFlg = 1
search_form[‘method’] = ‘search’
search_form.exeKind = ‘HPLCDS0101E’
search_form.searchListCount = 2500
search_form.checkbox_with(:value => ‘001’).check
search_form.checkbox_with(:value => ‘002’).check
search_form.checkbox_with(:value => ‘004’).check

page = agent.submit(search_form)
stockform = page.form(‘HPLCDS0301Form’)

stock = stockform.button_with(:value => ‘Display of stock price’)

pp stock

When I submit the form, the data I get returned contains over 2300 links
of the type below. The onclick=“chart(‘1378’)” with the number is the
stock ticker symbol that determines which stock is going to be opened
when you click that particular link.

Basically, I want to be able to click all the links of the above type,
and save the returned data in a file.

Can anyone help nudge me along here? Really appreciate it, thanks!

2

Mechanize can return a Nokogiri::HTML document when you do:

doc = page.parser

Using Nokogiri you can do something like:

doc.css(‘input[@class=“negativeButton”]’)

to retrieve an array of the input tags with a class of
“negativeButton”. You can fine-tune the returned values as needed
using select() or reject() or by switching from .css() to .xpath().

You’ll need to look at the contents of the chart() Javascript function
which is being called by the onclick() handler to figure out what the
add the parameter to, and that will give you the full URL to retrieve.

3

Oh, BTW, these will give you some more info…

4

On Mar 28, 2010, at 15:29 , Robert Molloy wrote:

Basically, I want to be able to click all the links of the above type,
and save the returned data in a file.

You’re prolly looking for something like:

search_form.buttons_with(:onclick => /^chart/).each do |button|
button.click
end

this is untested, but a good portion was glommed from mechanize’s tests.
Do this:

gem unpack mechanize
cd mechanize*
grep -r buttons_with .

and such. I find lots of stuff that way.