Forum: Ruby on Rails Caching from screen scraping

Announcement (2017-05-07): is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see and for other Rails- und Ruby-related community platforms.
D2b2017098da3e2fd1cdcfbb72ecdbfb?d=identicon&s=25 Jason Edgecombe (Guest)
on 2006-01-27 18:12
(Received via mailing list)
Hi all,

I need to do some screen scraping  from my rails app. Given an ethernet
(MAC) adress, I scrape results from an internal web page that returns
location and hostname. How can I cache the result from that screen
scraping as to be polite to the scrapee? I would like to expire the
results daily. In perl, I would use Cache::File. Can I use rails caching
for this? What's the best way? The screen scrape is used internally in
my app and not viewed directly.

Jason Edgecombe
3f900b38ec3b2c45427c354722fa4ce3?d=identicon&s=25 Tom Fakes (tomfakes)
on 2006-01-27 19:49
(Received via mailing list)
Here are two ways you can do this:

1.  If you don't mind the entire action being cached for the 24 hours,
can use the Rails Action cache, with the new Action Cache plugin that
the ability to expire the cache on a timer.

In your action you can do this:

	caches_action :my_action

	def my_action
		response.time_to_live =
		# Do some screen scraping

2. If caching the entire action doesn't work for you, you can use Rails
fragment caching.  Check out the code for the Action Cache plugin to see
what you can do.  Here's a summary:

	a. Build a small object that keeps the expiry time and the data to
	b. Use YAML to serialize this (YAML.dump and YAML.load)
	c. Add this to the fragment cache with a unique key (read_fragment
and write_fragment)

If you build this yourself, it should be about 3 lines of code to save
the cache, and about 4 lines to read and check if the cache has expired.
This topic is locked and can not be replied to.