I’m trying to use Mechanize with Ruby to scrape data from this site
(http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do). The first part of my
task is trying to automate an initial search form, I can do this fine
using the following code.
agent = Mechanize.new
page = agent.get(‘http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do’)
search_form = page.form(‘HPLCDS0101Form’)
search_form.callJorEFlg = 1
search_form[‘method’] = ‘search’
search_form.exeKind = ‘HPLCDS0101E’
search_form.searchListCount = 2500
search_form.checkbox_with(:value => ‘001’).check
search_form.checkbox_with(:value => ‘002’).check
search_form.checkbox_with(:value => ‘004’).check
page = agent.submit(search_form)
stockform = page.form(‘HPLCDS0301Form’)
stock = stockform.button_with(:value => ‘Display of stock price’)
When I submit the form, the data I get returned contains over 2300 links
of the type below. The onclick=“chart(‘1378’)” with the number is the
stock ticker symbol that determines which stock is going to be opened
when you click that particular link.
Basically, I want to be able to click all the links of the above type,
and save the returned data in a file.
Can anyone help nudge me along here? Really appreciate it, thanks!