Using an HTTP proxy to record web traffic

Ruby
1

As part of debugging WWW::Mechanize, I realized that the best way to
compare what my browser is doing to what mechanize is doing would be to
compare their HTTP transactions incoming and outgoing. What I want is
an HTTP proxy that can record the transactions and also supports SSL.

The Ruby project doesn’t have any code
http://rubyforge.org/projects/rpp/

This free program doesn’t support SSL, and costs $600.
http://www.proxy-sniffer.com/download.html

This doesn’t support SSL but is supposed to soon.
http://kevinlangdon.com/serviceCapture/

Any other suggestions?

2

Dan K. wrote:

As part of debugging WWW::Mechanize, I realized that the best way to
compare what my browser is doing to what mechanize is doing would be to
compare their HTTP transactions incoming and outgoing. What I want is
an HTTP proxy that can record the transactions and also supports SSL.

MouseHole is a scriptable proxy. http://mousehole.rubyforge.org/

A script to record traffic would a cinch. Basically,

  1. Download MouseHole from the above link. Instructions for running it
    are there as well.

  2. Add a script named `proxyrecord.user.rb’ to your
    ~/.mouseHole/userScripts/ directory.
    Contains:

    MouseHole.script do
    name "Proxy Record"
    namespace "http://yourdomain.com/"
    description "Records all traffic, very snoopy."
    include_match %r!http://.*!
    version "1.0"

    rewrite do |req, res|
        # store contents of `req' and `res' in the database or a
        # log file or something
    end
end

  3. In your browser, setup MouseHole as your proxy
    (http://127.0.0.1:37004/)

  4. Visit http://127.0.0.1:37004/ and activate the Proxy Record script by
    checking its box.

As for SSL, I don’t think proxies can actually intercept unencrypted
SSL. It uses the CONNECT method.

_why

3

Very helpful, why, thanks. I presume I would need to use something
like WATIR to see the unencrypted contents of an SSL connection by
getting it directly from IE.