Ruby web crawler for https

Actually there is code in ruby which collects all the URLs from your
site and stores them in a file.
I took him out
I understand that code is relevant for http connection, and uses GET and
HTTP requests.
I’ll scan the address type:
How to edit this code?

How to edit this code?

vi crawler.rb


change this line

    if %r{http://([^/]+)/([^/]+)}i =~ $_

to this

    if %r{https?://([^/]+)/([^/]+)}i =~ $_

– Sergey Avseyev

Simple ETL (Extract Transform Load) Tool for web

Here’s the complete sample how to fully grab & parse the