Forum: Ruby Download csv file from secure web site (https

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Alice G. (Guest)
on 2009-05-27 03:19
I am a newbie to Ruby and I created a script to process a file which I
am getting from my local directory. However, in order to obtain this
file, I do a manual login to a secure http site (https) which presents
me with a logon screen where I enter the username and password. After
successful authentication, I am presented with a download page which has
as extension a php page which has a download button to get the file.
After I download on my local Windows Vista machine, I use FileZilla to
place it on the linux server.
How can I automate this in Ruby with open-uri or other extension for
that matter that would enable me to authenticate and get the file
directly on my linux server?
Any insight/code-example will be deeply appreciated.
Thanks so much, -- Alice
Reid T. (Guest)
on 2009-05-27 03:25
(Received via mailing list)
Alice Gheorghiu wrote:
> directly on my linux server?
> Any insight/code-example will be deeply appreciated.
> Thanks so much, -- Alice

can you ssh/scp directly to the secure http site (https) from a command
line on
the linux host?

If so, you should be able to setup a passwordless login for scp and get
the file
via a cron'd job using scp (google  ssh passwordless login)

$ scp removed_email_address@domain.invalid:/path/to/file
/path/to/download/to/filename
Alice G. (Guest)
on 2009-05-27 04:12
Reid T. wrote:
> Alice Gheorghiu wrote:
>> directly on my linux server?
>> Any insight/code-example will be deeply appreciated.
>> Thanks so much, -- Alice
>
> can you ssh/scp directly to the secure http site (https) from a command
> line on
> the linux host?
$ scp removed_email_address@domain.invalid:/path/to/file
/path/to/download/to/filename
Tried ssh and I am getting error as follows in the ssh command:
$ ssh username@https://www.desired_website.com/pageWLinkToReport.php
$ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
service > not known
> If so, you should be able to setup a passwordless login for scp and get
> the file
> via a cron'd job using scp (google  ssh passwordless login) - not possible yet
but maybe - I can ask the data provider to set our account up for
automatic
download?
Alice G. (Guest)
on 2009-05-27 20:04
Alice Gheorghiu wrote:
> Reid T. wrote:
>> Alice Gheorghiu wrote:
>>> directly on my linux server?
>>> Any insight/code-example will be deeply appreciated.
>>> Thanks so much, -- Alice
>>
>> can you ssh/scp directly to the secure http site (https) from a command
>> line on
>> the linux host?
> $ scp removed_email_address@domain.invalid:/path/to/file
> /path/to/download/to/filename
Tried ssh and I am getting error as follows in the ssh command:
$ ssh username@https://www.desired_website.com/pageWLinkToReport.php
$ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
service not known
>> If so, you should be able to setup a passwordless login for scp and get
>> the file
>> via a cron'd job using scp (google  ssh passwordless login)
- not possible yet but maybe - I can ask the data provider to set our
account up for automaticdownload?
I spoke with both data providers and they cannot help much. Any
suggestions of Ruby articles or sample code that may treat a similar
problem? Thanks again so very much.
Joel VanderWerf (Guest)
on 2009-05-27 20:20
(Received via mailing list)
Alice Gheorghiu wrote:
> Tried ssh and I am getting error as follows in the ssh command:
> $ ssh username@https://www.desired_website.com/pageWLinkToReport.php
> $ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
> service not known

If it works at all, it would be something like:

   ssh username@desired_website.com

but the fact that there is a https interface does not imply that there
is an ssh interface. They are handled by different server software, and
unless the provider explicitly set up an ssh server, it won't be there.
And for that matter, having an account on the https server doesn't imply
having an account that you can access via ssh, even if the ssh server is
running.

It sounds like a task for something like mechanize[1], but I don't know
how it is with https. Some others on this list will know, I hope... or
try googling for "ruby mechanize https".

[1] http://mechanize.rubyforge.org/mechanize/
Alice G. (Guest)
on 2009-05-27 21:09
Joel VanderWerf wrote:
> Alice Gheorghiu wrote:
Tried ssh and I am getting error as follows in the ssh command:
$ ssh username@https://www.desired_website.com/pageWLinkToReport.php
$ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
service not known

> If it works at all, it would be something like:
>    ssh username@desired_website.com

> but the fact that there is a https interface does not imply that there
> is an ssh interface. They are handled by different server software, and
> unless the provider explicitly set up an ssh server, it won't be there.
> And for that matter, having an account on the https server doesn't imply
> having an account that you can access via ssh, even if the ssh server is
> running.
>
> It sounds like a task for something like mechanize[1], but I don't know
> how it is with https. Some others on this list will know, I hope... or
> try googling for "ruby mechanize https".
> [1] http://mechanize.rubyforge.org/mechanize/
You are right, whereby their reluctance to assist. The site is a good
starting point. Will browse and will be back with the code if I can make
it work. Thanks a million.
Alice G. (Guest)
on 2009-06-17 23:39
Alice Gheorghiu wrote:
> Joel VanderWerf wrote:
>> Alice Gheorghiu wrote:
> Tried ssh and I am getting error as follows in the ssh command:
> $ ssh username@https://www.desired_website.com/pageWLinkToReport.php
> $ ssh: https://www.desired_website.com/pageWLinkToReport.php: Name or
> service not known
>
>> If it works at all, it would be something like:
>>    ssh username@desired_website.com
>
>> but the fact that there is a https interface does not imply that there
>> is an ssh interface. They are handled by different server software, and
>> unless the provider explicitly set up an ssh server, it won't be there.
>> And for that matter, having an account on the https server doesn't imply
>> having an account that you can access via ssh, even if the ssh server is
>> running.
>>
>> It sounds like a task for something like mechanize[1], but I don't know
>> how it is with https. Some others on this list will know, I hope... or
>> try googling for "ruby mechanize https".
>> [1] http://mechanize.rubyforge.org/mechanize/
> You are right, whereby their reluctance to assist. The site is a good
> starting point. Will browse and will be back with the code if I can make
> it work. Thanks a million.
I was able to work through the code: here it is:
<code>
!#/usr/bin/rb
####################################################
#This program handles files from Vitelity web-site #
#using nokogiri mechanize package                  #
#Created on 06/09/09                               #
#By Alice G.                                       #
####################################################

require 'rubygems'
require 'mechanize'
require 'date'

unless ARGV.size == 3
puts "Usage: Enter username, password and e-mail address"
exit
end

#Define the cdr start and end date
d = DateTime.now

#Create a new object from mechanize class to parse the web pages
a = WWW::Mechanize.new
  a.get('http://portal.vitelity.net/login.php') do |page|
    # Click the login link
    #login_page = a.click(page.links.text(/Log In/))
    #login_page = a.click(page.links.find { |link| /Log In/.match
link.text })

    # Submit the login form
      my_page = page.form_with(:action => 'signin.php') do |f|
       f.username  = ARGV[0]
       f.pass  = ARGV[1]
      end.click_button

    # Click the Call Detail Report link on the Vitelity index page
      #cdr_page = a.click(my_page.links.find { |link| /Call
Records/.match link.text})
      cdr_page = a.get('http://portal.vitelity.net/index.php?p=cdr')
      cdr_report=cdr_page.form_with(:action => 'index.php') do |f|
      f.toemail = ARGV[2]
      f.frommonth=d.month
      f.fromday = d.day - 1
      f.fromyear = d.year
      f.tomonth=d.month
      f.today=d.day
      f.toyear=d.year
      end.click_button
 end
</code>
The script accepts 3 parameters, username, password to the site and the
e-mail where the file should be sent. To get info on form field names
you need to source the web page and refer to them in your source code.
This topic is locked and can not be replied to.