Watir - Chrome - How to retrieve all “li” entries that are not all visible?

Ruby
1

I’m using Watir on test automation of a Chrome based chatting/IM
platform.

One of the case is to get all contacts info from a frame. Contacts are
placed in groups, and groups can be expanded or collapsed (by
toggle-clicking) to show or hide the contacts.

The HTML source is like the following (where group “Default” is
expanded):

  • ... /li>
  • ... /li>
  • Default (0/118) ... /li>
  • (ABN) James
  • (ABN) Justin
  • (ABN) Matt
    • .......

My Watir script of the related function is like the following:

def getContactCount(browser, outputfile)
iframe_chat = browser.iframe(:src =>
“/tribe/connectionlist/index.html#appId/3”)
ul_conns = iframe_chat.div(:id => “connectionList”).div(:id =>
“connectionListItems”).ul(:class => “connections”)
numGroups = ul_conns.lis(:class => /group/).length
outputfile.puts "Contact count – total groups: " + numGroups.to_s
grpName = Array.new
ul_conns.lis(:class => /group/).each { |li| grpName.push li.data_name
}
for igrp in 1…numGroups
list_grp = ul_conns.li(:data_name => grpName[igrp-1])
outputfile.puts "Group " + list_grp.text
list_grp.click #-- expand group
outputfile.puts "Number of contacts found: " + ul_conns.lis(:class
=> “connection”).length.to_s
ul_conns.lis(:class => “connection”).each { |li| outputfile.puts "
" + li.span(:class => “name”).text }
list_grp.click #-- collapse group
end
end

And this is the output:

Contact count – total groups: 3
Group SUGAR (0/11)
Number of contacts found: 11
(GFI) Joachim
(ICAP) Kissel
(JB Drax) James
…
Group Desk Connections (1/88)
Number of contacts found: 55
ICM_am
(ABN) Alex
(ABN) James
(ABN) Justin
…
Group Default (0/118)
Number of contacts found: 54
(ABN) James
(ABN) Justin
(ABN) Matt
…

The problem is, for group “Desk Connections” and “Default”, only part of
the contacts were retrieved by the script (55 of 88, 54 of 118,
respectively).

Actually, when I was checking the HTML source in the Chrome browser,
only part of them were listed as well if the contacts within a group are
too many to be displayed in all. But when I scrolled the frame down
(there is a vertical scroll bar for the frame), more were shown (but the
ones past above frame top became invisible).

Could anyone help? How do I get all the "li"s (class=“connection”)?

Let me know if any more info is needed.

Thanks a lot.
Xuemin

2

Once you’re on the page, you could just read all the HTML into a parser
like Nokogiri, and then iterate through the nodes with some CSS or
XPATH.

3

Thanks for the advice.

Here’s what I did:

I installed Nokogiri gem.
In my script I added the parsing.
But I found that I still have to click each group name in order to get
the contact listed under that group, otherwise Nokogiri/Xpath returns
empty result.

Then I put the Xpath method into the loop like the following:

for igrp in 1…numGroups
list_grp = ul_conns.li(:data_name => grpName[igrp-1])
outputfile.puts "Group " + list_grp.text
list_grp.click #-- expand group
outputfile.puts "Number of contacts found: " + ul_conns.lis(:class =>
“connection”).length.to_s

page_html = Nokogiri::HTML.parse(browser.html)
outputfile.puts
page_html.xpath("//div[@id=‘connectionListItems’]/ul/li[@class=‘connection’]").count
outputfile.puts
page_html.xpath("//div[@id=‘connectionListItems’]/ul/li[@class=‘connection’]").text

list_grp.click #-- collapse group
end

But the result is the same as before, i.e. only part of the contact list
was retrieved by Nokogiri+Xpath.

Any idea why? Did I miss anything?

Thanks.

4

I was thinking of dropping all the HTML of the page into Nokogiri and
letting it sort out the results without clicking on anything, however
there’s one very important thing to check first.

Are the results already there, hidden, before you expand the groups?

If not then it’s probably populated by JavaScript when you click, in
which case there is no alternative but to click or somehow trigger the
JavaScript or underlying information request in order to retrieve this
information.

If the information is there, hidden, then all you need is the right
selector for Nokogiri to pick it all up in one go.