# Buffer problem

**URL:** <https://www.ruby-forum.com/t/buffer-problem/83324>\
**Category:** Rails\
**Created:** [January 25, 2007, 6:11am UTC](https://www.ruby-forum.com/t/buffer-problem/83324 "2007-01-25T06:11:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [January 25, 2007, 6:11am UTC](https://www.ruby-forum.com/t/buffer-problem/83324/1 "2007-01-25T06:11:39Z")

</div>

I am trying to spider a site using Hpricot, but I keep getting out of  
buffer error. It will only let me do about two sites at a time, is  
there a way to clear the buffer after I process each page so I won’t  
blow the buffer?

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [January 25, 2007, 8:47am UTC](https://www.ruby-forum.com/t/buffer-problem/83324/2 "2007-01-25T08:47:48Z")

</div>

Can you post the code you are using?

On 1/24/07, removed\_email\_address@domain.invalid [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> I am trying to spider a site using Hpricot, but I keep getting out of  
> buffer error. It will only let me do about two sites at a time, is  
> there a way to clear the buffer after I process each page so I won’t  
> blow the buffer?
> 
> >

–  
Thanks,  
-Steve  
[http://www.stevelongdo.com](http://www.stevelongdo.com)

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [January 26, 2007, 12:55pm UTC](https://www.ruby-forum.com/t/buffer-problem/83324/3 "2007-01-26T12:55:00Z")

</div>

Anyone someone has to know something about the buffer?

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [January 25, 2007, 1:45pm UTC](https://www.ruby-forum.com/t/buffer-problem/83324/4 "2007-01-25T13:45:44Z")

</div>

require ‘rubygems’  
require ‘hpricot’  
require ‘open-uri’  
require ‘active\_record’

ActiveRecord::Base.establish\_connection(  
#connection info  
)

class Major \< ActiveRecord::Base  
has\_many :courses  
end

class Course \< ActiveRecord::Base  
belongs\_to :major  
end

def scrape(url)  
doc = Hpricot(open(url))  
tables =(doc/“table”)  
(tables[6]/“tr”).each do |major|  
createMajor major  
end  
end  
def createMajor(data)  
newMajor = Major.new  
newMajor.title = data.search(“td”).first.inner\_html  
newMajor.abbrev =data.search(“acronym”).inner\_html  
newMajor.link\_to = data.search(“a”).to\_s.split(‘"’)[1]  
puts newMajor.save  
end  
def courses(url)  
puts url  
doc = Hpricot(open(“[http://courses.tamu.edu/](http://courses.tamu.edu/)”+url.to\_s))  
courses = (doc/“//td[@class=‘sectionheading’]”)  
courses.each do |course|  
createCourse course  
end  
end  
def createCourse(data)  
course = data.inner\_html.strip.split(’ ‘)  
major = course[0]  
course\_no = course[1]  
puts major,course\_no  
course.pop  
course\_name = course.slice!(3,course.length).join(’ ')  
puts course\_name  
end

AllMajors = Major.find(:all, :limit=\>3,:offset=\>0)  
AllMajors.each do |course|  
courses(course.link\_to,course.id)  
end  
#scrape(url goes here)

This what I was last test with. I had it where scrape would call  
courses, but that broke the buffer before I even got output, this  
outputs the data from two pages and then breaks.