For some hobby project of mine I am trying to come up with a way to let
a Rails application interact with some external website.
I am trying to achieve the following. Lets for example take a banner on
a website. Which is the easiest to explain.
Banners usually are on the same spot on websites, and when you click
them, you get bounced to a couple of URLs before you end up at the
company’s website whos ad you clicked.
I want to automate this process!
Imagine the following scenario:
- I tell my Rails site to click a certain link/banner on a certain
(external) website. Maybe through a DOMID, maybe through XPATH, im not
certain on this yet. - The server will need to retrieve / load the external page through a
GET and then do another GET on the link I wanted to click. - The server needs then to follow and gather all the links that it is
bounced to, before ending up at the final site.
How would I achieve this using Ruby on Rails?
Then there is an added problem:
If we take the same example using banners again, some banners are
rendered through JavaScript, such as Google Ads. The website HTML which
we GET does not contain the banners, but the JS on that page tells the
browser to fetch banners from Google directly.
For this to work in my project I need some kind of server-side
JavaScript processing scheme to make the banners show up on the server
side, before we can click any links and follow their hops.
Any advice is greatly appreciated!