Hi all, I need to do some screen scraping from my rails app. Given an ethernet (MAC) adress, I scrape results from an internal web page that returns location and hostname. How can I cache the result from that screen scraping as to be polite to the scrapee? I would like to expire the results daily. In perl, I would use Cache::File. Can I use rails caching for this? What's the best way? The screen scrape is used internally in my app and not viewed directly. Sincerely, Jason E.
on 2006-01-27 19:12
on 2006-01-27 20:49
Here are two ways you can do this: 1. If you don't mind the entire action being cached for the 24 hours, you can use the Rails Action cache, with the new Action Cache plugin that adds the ability to expire the cache on a timer. In your action you can do this: caches_action :my_action def my_action response.time_to_live = 1.day # Do some screen scraping end 2. If caching the entire action doesn't work for you, you can use Rails fragment caching. Check out the code for the Action Cache plugin to see what you can do. Here's a summary: a. Build a small object that keeps the expiry time and the data to cache b. Use YAML to serialize this (YAML.dump and YAML.load) c. Add this to the fragment cache with a unique key (read_fragment and write_fragment) If you build this yourself, it should be about 3 lines of code to save to the cache, and about 4 lines to read and check if the cache has expired.