Dealing with very large XML output

Rails
1

I’ve got an app that has the potential to serve up a very large RSS
feed. I’m doing some looping the xml template to simulate return data
for thousands of records. Perhaps this needs to be redesigned to grab
chucks but I’ll ask the question anyway (because I am curious).

The response doesn’t seem to start going out until the template is done
looping through all the data. Is there anyway to tell it to start
sending the data to the client after x number of bytes is ready to go
(like a flush or something)?

Regards

2

Homer S. wrote:

I’ve got an app that has the potential to serve up a very large RSS
feed. I’m doing some looping the xml template to simulate return data
for thousands of records. Perhaps this needs to be redesigned to grab
chucks but I’ll ask the question anyway (because I am curious).

The response doesn’t seem to start going out until the template is done
looping through all the data. Is there anyway to tell it to start
sending the data to the client after x number of bytes is ready to go
(like a flush or something)?

Regards

As far as I know, you cant send a page before it’s done rendering. Now
without doing some sort of store in the session and update via ajax
thing, which would only work in a browser, and not in any xml client.

Although I’m only 75% sure of that. But given the way rails is designed
it seems like it would be a hard thing to do with it’s support for
after_filters and whatnot.

3

Thank for the reply.

It turns out the biggest problem with the speed is the XML creation is
so slow.

I’m doing something like below to create the RSS feed. If I just output
the data in plain text (html with no xml creation) I can render 235,000
records in 45 seconds (I can live with that).

If I try a couple hundred records with the code below it takes 20
seconds. If I get into the 100K record with the code below ruby.exe
heads into la-la land.

Any faster ways to generate the rss xml?

xml.instruct! :xml, :version=>“1.0”
xml.rss(:version=>“2.0”){
xml.channel{
xml.title(“Your Blog’s Title”)
xml.link(“http://www.yoursite.tld/”)
xml.description(“What your site is all about.”)
xml.language(‘en-us’)
for post in @posts
xml.item do
xml.title(post.title)
xml.description(post.html_content)
xml.author(“Your N. Here”)
xml.pubDate(post.created_on.strftime(“%a, %d %b %Y %H:%M:%S
%z”))
xml.link(post.link)
xml.guid(post.link)
end
end
}
}

4

How about this… move the rendering to a background process with say,
BackgroundDrb.
Your action redirects the client to the location of the static file and
you
let someone like apache stream the file.

Could get quite I/O intensive, but atleast you can stream. Might be ok
if
you only handle a small number of requests.