Using an HTTP proxy to record web traffic


#1

As part of debugging WWW::Mechanize, I realized that the best way to
compare what my browser is doing to what mechanize is doing would be to
compare their HTTP transactions incoming and outgoing. What I want is
an HTTP proxy that can record the transactions and also supports SSL.

The Ruby project doesn’t have any code
http://rubyforge.org/projects/rpp/

This free program doesn’t support SSL, and costs $600.

This doesn’t support SSL but is supposed to soon.
http://kevinlangdon.com/serviceCapture/

Any other suggestions?


#2

Dan K. wrote:

As part of debugging WWW::Mechanize, I realized that the best way to
compare what my browser is doing to what mechanize is doing would be to
compare their HTTP transactions incoming and outgoing. What I want is
an HTTP proxy that can record the transactions and also supports SSL.

MouseHole is a scriptable proxy. http://mousehole.rubyforge.org/

A script to record traffic would a cinch. Basically,

  1. Download MouseHole from the above link. Instructions for running it
    are there as well.

  2. Add a script named `proxyrecord.user.rb’ to your
    ~/.mouseHole/userScripts/ directory.
    Contains:

    MouseHole.script do
        name "Proxy Record"
        namespace "http://yourdomain.com/"
        description "Records all traffic, very snoopy."
        include_match %r!http://.*!
        version "1.0"
    
        rewrite do |req, res|
            # store contents of `req' and `res' in the database or a
            # log file or something
        end
    end
    
  3. In your browser, setup MouseHole as your proxy
    (http://127.0.0.1:37004/)

  4. Visit http://127.0.0.1:37004/ and activate the Proxy Record script by
    checking its box.

As for SSL, I don’t think proxies can actually intercept unencrypted
SSL. It uses the CONNECT method.

_why


#3

Very helpful, why, thanks. I presume I would need to use something
like WATIR to see the unencrypted contents of an SSL connection by
getting it directly from IE.