Fetching RSS feeds asynchronously in background

Lennart_K · January 30, 2009, 1:10pm

Hello everyone,

I want to fetch some (5-10) RSS feeds from another server. I plan to
fetch the RSS feeds asynchronously and store them on the hard disk
every 30 minutes so I can read them from there on every request. (To
treat the remote server with care and to avoid longer page generation
time).

What is the best way to do that?

I think about starting a fetcher-thread on the start of my Rails
application. I want to avoid cron jobs to keep everything in the Rails
application.

Thank you!

So long
Lennart K.

Lennart_K · January 31, 2009, 3:12pm

Hi Lennart,

You may have interests on this:

github.com

janx/newsmailer/blob/a4329bab234d8cc0252d9d7a72e97036b8a278df/app/models/crawler.rb

class Crawler

  def start
    num = REFRESH_THREADS ? REFRESH_THREADS : 1
    num = 1 if num < 1

    sliced = slice Feed.all, num
    threads = []
    for feeds in sliced
      threads << Thread.new(feeds) { |fs| fetch fs }
    end

    threads.each {|t| t.join}
  end

  def deliver
    User.all.each do |user|
      user.news.each do |article|
        article.send_to user
      end

This file has been truncated. show original

Best,
Jan

Lennart K. [email protected] [2009-01-30
13:09:19 +0100]: