# Optimization anyone **URL:** \ **Category:** Ruby\ **Created:** [November 22, 2005, 11:40am UTC](https://www.ruby-forum.com/t/optimization-anyone/49388 "2005-11-22T11:40:02Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![hsanson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/f0a364/32.png) [@hsanson](https://www.ruby-forum.com/u/hsanson)\ **Post date:** [November 22, 2005, 11:40am UTC](https://www.ruby-forum.com/t/optimization-anyone/49388/1 "2005-11-22T11:40:02Z")
I have this little script that takes a list of keyword sets, each set has only two keywords and for each one of them the script creates a regular expression like this: Regexp.new("#{key1}._#{key2}|#{key2}._#{key1}") then I match it to a string that contains a long text fetched from a web page. a more complete pseudo-code ######################################### long\_text = get\_web\_page(url) keyword\_hash = load\_keyword\_array\_from\_database keyword\_hash.each\_pair { |id, value| ``` key1 = value[0] key2 = value[1] r = Regexp.new("#{key1}\.*#{key2}|#{key2}\.*#{key1}") return id if long_text =~ r ``` } return -1 ########################################### Now this code works perfect, the problem is that the keyword\_hash has more than 300 elements and running this code can take between 50 to 120 seconds. Since I am processing more than 1000 pages with this code it takes forever. I solved this problem by replacing the regular expression match to r1 = Regexp.new("#{key1}._#{key2}") r2 = Regexp.new("#{key2}._#{key1}") return id if long\_text =~ r1 or long\_text =~ r2 I simply put the or statement outside the regular expresion and the speedup was from 50~120sec to 0.40 secs per page. using the Benchmark class and running some test I got normal: 0 0 27.688000 0.015000 27.703000 ( 27.765000 ) fast: 0.469000 0.000000 0.484000 (0.954000) the speed difference is totally diferent. Is this expected when using regular expressions?? regards, Horacio ---
**Author:** ![hsanson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/f0a364/32.png) [@hsanson](https://www.ruby-forum.com/u/hsanson)\ **Post date:** [November 22, 2005, 11:52am UTC](https://www.ruby-forum.com/t/optimization-anyone/49388/2 "2005-11-22T11:52:06Z")
> > > > > “H” == Horacio S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) writes: H\> Regexp.new(“#{key1}._#{key2}|#{key2}._#{key1}”) vs H\> r1 = Regexp.new(“#{key1}._#{key2}“) H\> r2 = Regexp.new(”#{key2}._#{key1}”) H\> Is this expected when using regular expressions?? yes, ruby has some optimizations. For example with the regexp /abc.\*def/ svg% ruby -rjj -e ‘/abc.\*def/.dump’ Regexp /abc.\*def/ 0 exactn “abc” (3) 1 anychar\_repeat 2 exactn “def” (3) 3 end must : abc optimize : exactn svg% It call the regexp engine (which is slow) only when it has found the substring “abc” in the string Now if you use /abc.\*def|def.\*abc/ you break this optimization svg% ruby -rjj -e ‘/abc.\*def|def.\*abc/.dump’ Regexp /abc.\*def|def.\*abc/ 0 on\_failure\_jump ==\> 5 1 exactn “abc” (3) 2 anychar\_repeat 3 exactn “def” (3) 4 jump ==\> 8 5 exactn “def” (3) 6 anychar\_repeat 7 exactn “abc” (3) 8 end svg% it must call the stupid (:-)) regexp engine for each line Guy Decoux ---
**Author:** ![hsanson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/f0a364/32.png) [@hsanson](https://www.ruby-forum.com/u/hsanson)\ **Post date:** [November 22, 2005, 12:20pm UTC](https://www.ruby-forum.com/t/optimization-anyone/49388/3 "2005-11-22T12:20:19Z")
Horacio S. wrote: > ``` > r = Regexp.new("#{key1}\.*#{key2}|#{key2}\.*#{key1}") > > ``` > > code it takes forever. > I simply put the or statement outside the regular expresion and the > > the speed difference is totally diferent. > > Is this expected when using regular expressions?? On obvious optimization is to create all regexps during load\_keyword\_array\_from\_database() and not during iteration of the hash. That way you just have to do it once and can reuse those regexps with multiple pages you check. Another possible optimization is to take your approach of splitting the regexps a bit further and create two regexps - one for each keyword - and return the id if both match. This works only correctly if (i) keywords don’t overlap or (ii) you can use \b to ensure matching on word boundaries. Kind regards ``` robert ``` ---
**Author:** ![hsanson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/f0a364/32.png) [@hsanson](https://www.ruby-forum.com/u/hsanson)\ **Post date:** [November 22, 2005, 5:34pm UTC](https://www.ruby-forum.com/t/optimization-anyone/49388/4 "2005-11-22T17:34:30Z")
ts [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) writes: > yes, ruby has some optimizations. For example with the regexp /abc.\*def/ Do you know how oniguruma does that, per chance? ---
**Author:** ![hsanson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/f0a364/32.png) [@hsanson](https://www.ruby-forum.com/u/hsanson)\ **Post date:** [November 22, 2005, 5:42pm UTC](https://www.ruby-forum.com/t/optimization-anyone/49388/5 "2005-11-22T17:42:33Z")
> > > > > “C” == Christian N. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) writes: C\> Do you know how oniguruma does that, per chance? You can compile oniguruma with debugging options moulon% ./ruby -e ‘/abc.\*def/’ list:814d418 string:814d3e8abc qualifier:814d478{0,-1} anychar:814d448 string:814d4d8def optimize: EXACT\_BM anchor: [] sub anchor: [] exact: [abc]: length: 3 code length: 11 [exact3:abc] [anychar\*-peek-next:d] [exact3:def] [end] moulon% Guy Decoux ---
**Author:** ![hsanson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/f0a364/32.png) [@hsanson](https://www.ruby-forum.com/u/hsanson)\ **Post date:** [November 22, 2005, 3:17pm UTC](https://www.ruby-forum.com/t/optimization-anyone/49388/6 "2005-11-22T15:17:32Z")
There’s a whole section in Mastering Regular Expressions that goes into the differences between regexp engines. Summary: it makes a big difference how you setup your patterns! On 11/22/05, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > key2 = value[1] > > more than 300 elements and running this code can take between 50 to > > > > fast: > > multiple pages you check. ## – ‘There was an owl lived in an oak. The more he heard, the less he spoke. The less he spoke, the more he heard.’ Christian L. removed\_email\_address@domain.invalid