I got bored so I’m writing a little web spider in ruby. I’ve got several
planned like content type limits etc, but I’m stuck with the size limit.
If I check against content_length from the headers of the response
already downloaded the body. How can I check and exclude based on the
before the whole body is downloaded?