Hi all, I am working on a project that requires rss parsing for pre-fetched web pages. I don't need caching or anything fancy, just the rss parsing itself. I am currently using sporkmonger's feedtools, but I am wondering if anything out there is better in terms of performance. Hopefully someone has tested a few parsers. If not, I'll run some simple tests and post back to the list. Some parsers under consideration: http://raa.ruby-lang.org/project/rss/ http://simple-rss.rubyforge.org/ http://sporkmonger.com/projects/feedtools/ Thanks Ray
on 19.08.2006 09:16
on 19.08.2006 09:33
One more. http://syndication.rubyforge.org/doc/
on 20.08.2006 04:16
> One more. > http://syndication.rubyforge.org/doc/ Right now, I'm recommending that people who care about performance use the UFP instead. Cheers, Bob Aman
on 28.08.2006 20:04
Just wanted to give everyone who's curious an update on this situation. I ended up comparing just Feedtools and Syndication, since the others were not as feature-complete as I would have liked. In our system tests, Syndication performs a lot better than Feedtools, but I won't publicize the results here since those are specific to our system. In just the parsing portion of parsing http://www.digg.com/rss/chandrasonic/index2.xml (grabbing 40 entries, 40 titles, 40 dates, etc etc) Syndication took 0.062467 seconds of total time and Feedtools took 4.227067 seconds of total time. This level of performance difference is highly reproduceable and I am using this specific feed just to show some numbers. Syndication does have some downsides however. The programmer must specifically whether a feed is RSS or Atom since Syndication has no built-in distinguishing mechanism. Syndication also seems to have incomplete time-stamp support. Slashdot time stamps, for example, doesn't work for me. Disclaimer: I'm not an expert on Ruby, RSS feeds or benchmarking, but just wanted to share my results. Thanks Bob for all your help along the way.
on 05.09.2006 00:34
> http://www.digg.com/rss/chandrasonic/index2.xml (grabbing 40 entries, 40 > > Disclaimer: I'm not an expert on Ruby, RSS feeds or benchmarking, but > just wanted to share my results. Thanks Bob for all your help along the > way. The results aren't even remotely suprising to me, but yeah, they really underscore several points: 1) REXML is a performance stinker (and by extension, FeedTools 10x more so) 2) If performance is an issue, a parse-at-all-costs parser is going to be a problem unless it's written in C or something similarly fast 3) As always, use the right tool for the job -- FeedTools isn't going to scale, so if you need scalability, don't use it 4) If you're dealing with ~500 feeds or less, you want to always get the right answers back, you don't mind the size of the library, and you want a Ruby-only solution, FeedTools is what you want In case it's not obvious, point #4 has a lot of conditions. At some point in the distant future, I intend to write a pure C library that -will- scale, but that's a long way off, because for the forseeable future I'm going to be working on GentleCMS. Cheers, Bob Aman
on 31.01.2008 09:20
Hi Bob, I am working on a RSS parser script. Here I have to parser thousands and thousands of RSS feeds every hour. I am looking for a optimized parser which can take parse all these feeds. Please suggest the RSS parser you have come across. Thanks in advance.
on 14.02.2009 14:25
Sunil Khedar wrote: > Hi Bob, > > I am working on a RSS parser script. Here I have to parser thousands and > thousands of RSS feeds every hour. > > I am looking for a optimized parser which can take parse all these > feeds. Please suggest the RSS parser you have come across. > > Thanks in advance. Hi, I am looking for an high-performance parser too. Have you come across any solution? Thanks!
on 14.02.2009 16:06
On Feb 14, 8:24 am, Marco Colli <collimarc...@gmail.com> wrote: > > Hi, I am looking for an high-performance parser too. > Have you come across any solution? If speed is the #1 issue, I would try libxml-ruby. It's lower lever then you might want, but that gives you a choice on how to parse, and it's not be hard to build an rss specific layer on top of that if you really want it. T.
on 15.02.2009 12:35
On Sat, Feb 14, 2009 at 10:24 PM, Marco Colli <collimarco91@gmail.com> wrote: > > Hi, I am looking for an high-performance parser too. > Have you come across any solution? http://www.rubyinside.com/feedzirra-a-new-ruby-feed-library-built-for-speed-1485.html ^ manveru
on 16.02.2009 04:50
have you tried vtd-xml (http://vtd-xml.sf.net) ----- Original Message ----- From: "Michael Fellinger" <m.fellinger@gmail.com> To: "ruby-talk ML" <ruby-talk@ruby-lang.org> Sent: Sunday, February 15, 2009 3:33:01 AM GMT -08:00 US/Canada Pacific Subject: Re: best-performing Rss parser On Sat, Feb 14, 2009 at 10:24 PM, Marco Colli <collimarco91@gmail.com> wrote: > > Hi, I am looking for an high-performance parser too. > Have you come across any solution? http://www.rubyinside.com/feedzirra-a-new-ruby-feed-library-built-for-speed-1485.html ^ manveru
on 17.02.2009 03:59
On Thu, Jan 31, 2008 at 1:20 AM, Sunil Khedar <sunil@truesparrow.com> wrote: > I am working on a RSS parser script. Here I have to parser thousands and > thousands of RSS feeds every hour. > > I am looking for a optimized parser which can take parse all these > feeds. Please suggest the RSS parser you have come across. > Sounds like a case of premature optimization to me. If you intend to do anything like stick the data parsed from the feeds into a database or search index, I think you'll quickly find that will become the bottleneck, rather than the feed processing itself. My company went through something similar, with a performance obsessed former C++ programmer looking for the fastest feed parsing solution available. He settled on building his own, highly procedural feed processor around libxml-ruby after benchmarking several of the solutions available. However, soon after he discovered that updating the database and search index was a far bigger bottleneck, one he spent the next several months addressing. Feed parsing speed went completely by the wayside. If you intend to do any sort of indexing of the feeds at all, you should really focus on building a maintainable feed reader, as opposed to a fast one. The database and/or search index are going to be your bottleneck anyway, so don't let the desire for speed trump things like correctness and code clarity. Feed processing is something that scales horizontally using a queue and multiple feed reader processes, as opposed to databases and search indexes which generally don't scale quite as well. Given that, I would suggest looking at existing solutions like feedtools and feedzirra before trying to write your own, and if you do, go with Nokogiri. It has a nice, clear, easy-to-use API and is relatively fast.
on 17.02.2009 08:10
Tony Arcieri wrote: > On Thu, Jan 31, 2008 at 1:20 AM, Sunil Khedar <sunil@truesparrow.com> wrote: > >> I am working on a RSS parser script. Here I have to parser thousands and >> thousands of RSS feeds every hour. >> >> I am looking for a optimized parser which can take parse all these >> feeds. Please suggest the RSS parser you have come across. >> Check out Paul Dix's Feedzirra. It's an order of magnitude faster than the alternatives.