I am trying to right at application which will simply record the headers of all get requests made from my browser. It would be ideal to just listen in on port 80, but something tells me that wont work because of security risks. So I was wondering if there is a way to do this through a proxy. I am relatively new to ruby, and I havent been able to find anything in the rdoc. Any help is greatly appreciated. Thanks!
on 2008-02-06 02:39
on 2008-02-06 03:35
Well, this will work to record the headers. You'll need to install the mongrel gem. # gem install mongrel # headerlog.rb require 'rubygems' require 'mongrel' require 'logger' class HeaderHandler < Mongrel::HttpHandler @@logger = Logger.new('headers.log') def process(request,response) response.start(200) do |head, out| head["Content-Type"] = "text/plain" @@logger.info request.params end end end server = Mongrel::HttpServer.new("127.0.0.1", "2222") server.register("/", HeaderHandler.new) server.register("/favicon.ico", Mongrel::Error404Handler.new("")) server.run.join --- Then run it # ruby headerlog.rb Its set to listen on port 2222 and will record all the headers to a file named headers.log. You could change that port to whatever you want. I see no security implications of listening on port 80. The only problem would be if you have something else listening on port 80. Are you trying to put something in front of an existing application to capture the headers, then have it proxy the request to something else? In that case, what kind of application is it? You might be able to do that in the app itself. Or, perhaps turn on some sort of debug logging in apache or whatever server you are using. I might have some other ideas if this is more complex than the simple example I listed above. Hope that is helpful.
on 2008-02-06 06:41
thanks for your help dusty. I haven't gotten a chance to run it yet (I wont till tomorrow morning). My approach thus far is to create a proxy server, and have IE or Firefox route its requests through there. With that in mind I have 3 questions: 1) Is there another (doesn't necessarily have to be better) approach that you would suggest? 2) If I set IE7 to proxy its requests to port 2222, does the ruby script you provided return the requested data (i.e. does the web page still load in the browser) 3) If I set the port to 80 in the script above, will it behave as a "listener" and not interfere with the request from the browser (my guess is no on this one). Thanks for your help, I appreciate it very much.
on 2008-02-06 21:47
I had a feeling it was more than just capturing the headers. Do you happen to be running apache on the webserver? You can log all of it and not worry about a proxy. Here are two ways. 1. mod_dumpio # Put this in your apache config DumpIOInput On LogLevel debug 2. mod_log_forensic # Put this in your apache config ForensicLog /some/path/to/a/logfile.log This would simply capture the traffic on the web server itself.
on 2008-02-06 21:50
BTW - if you are just looking to do this on your computer so you can see what is going on while you hit remote sites, then I'd just use ethereal (or wireshark, I believe its called now). You can see the whole packet, probably the easiest way. Although, I'm sure someone on this list has written a ruby proxy server before, I haven't. Sounds interesting though, I might take a stab at it if I can find some spare time.
on 2008-02-06 21:58
On Feb 6, 2008 3:49 PM, dusty <firstname.lastname@example.org> wrote: > > There was some discussion of HTTP proxies on the Ruby/Eventmachine mailing list a few weeks back, so you might search through that archive. It sounds like the OP wants to write a reverse proxy, which isn't hard to do at all using EventMachine. The thing that made the subject more interesting, however, was the requirement that the proxy be transparent, so that the proxied traffic appears to be coming from the original peer's source IP address. That takes kernel support and is not available on all platforms.
on 2008-02-06 22:05
Thanks for the heads up, I'll check it out. Been playing with event machine a little bit recently, its nice!
on 2008-02-06 22:31
thanks for the info guys. I have looked into wireshark, and I have been back and forth on whether to do a packet level implementation (i.e. run wireshark command line, and then parse the data), or do a http level implementation (i.e. the proxy). The thing I dont like about packet level is the sheer number of packets that I would have to sift through.
on 2008-02-07 02:42
http://www.ruby-forum.com/topic/139859 may help you. Gotoken
on 2008-02-07 04:50
on 2008-02-07 08:39
GOTO Kentaro wrote: > http://www.ruby-forum.com/topic/139859 may help you. > > Gotoken Thanks, that looks very helpful, I will give it a shot tomorrow.
on 2008-02-07 20:42
Hi Gotoken, At first I thought it was the answer to my problems, but it seems that not all pages will load with it. i think pages which are heavily ajax dependent are the ones that dont load. I will try and tinker some more with it. GOTO Kentaro wrote: > http://www.ruby-forum.com/topic/139859 may help you. > > Gotoken
on 2008-02-09 14:32
Hi, WEBrick::HTTPProxyServer is not fast and maybe too slow in your situation. Could you report again when you find a simple way to reproduce the problem? I'll try to help you trouble shooting then. Thanks, Gotoken