# Odd Regexp Issue **URL:** \ **Category:** Ruby\ **Created:** [May 23, 2007, 10:23pm UTC](https://www.ruby-forum.com/t/odd-regexp-issue/104927 "2007-05-23T22:23:32Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\ **Post date:** [May 23, 2007, 10:23pm UTC](https://www.ruby-forum.com/t/odd-regexp-issue/104927/1 "2007-05-23T22:23:32Z")
I’m writing a web crawler, and in that crawler I want to remove all scripts in the pages I crawl. I should be able to do a simple gsub!(//,"") right? Well, I do that and unfortunately it doesn’t remove some scripts. Take google for instance. It removes the first script, but not the second. I’m really confused. Since google has two scripts, so it’s not like the full regexp should ever fail to be triggered. Any insight on the issue would be GREAT?! 😃 Thanks, Kyle H. ---
**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\ **Post date:** [May 23, 2007, 10:36pm UTC](https://www.ruby-forum.com/t/odd-regexp-issue/104927/2 "2007-05-23T22:36:20Z")
Kyle H. wrote: > Thanks, > Kyle H. Try multiline mode: gsub!(//m,"") ---
**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\ **Post date:** [May 23, 2007, 10:45pm UTC](https://www.ruby-forum.com/t/odd-regexp-issue/104927/3 "2007-05-23T22:45:21Z")
Joel VanderWerf wrote: … > Try multiline mode: gsub!(//m,"") Luis is right, it needs to be non-greedy, as well: gsub!(//m,"") ---
**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\ **Post date:** [May 23, 2007, 11:10pm UTC](https://www.ruby-forum.com/t/odd-regexp-issue/104927/4 "2007-05-23T23:10:48Z")
Well, that seemed to do the trick 😃 Thanks a lot, I didn’t know that regexp only applied to one line by default, HRM! Thanks, Kyle H. ---
**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\ **Post date:** [May 23, 2007, 10:33pm UTC](https://www.ruby-forum.com/t/odd-regexp-issue/104927/5 "2007-05-23T22:33:20Z")
On 5/23/07, Kyle H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I should be able to do a simple gsub!(//,“”) right? Well, I do > that and unfortunately it doesn’t remove some scripts. Take google for > instance. It removes the first script, but not the second. I’m really > confused. Since google has two scripts, > so it’s not like the full regexp should ever fail to be triggered. gsub(//m,“”) If there are new lines inside the string you need to use the m modifier to make the dot (.) include new lines as well. And the ? is to make the match non-greedy. Without it it would match the start of the first script and the end of the last script. ---
**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\ **Post date:** [May 23, 2007, 11:24pm UTC](https://www.ruby-forum.com/t/odd-regexp-issue/104927/6 "2007-05-23T23:24:25Z")
On May 23, 2007, at 4:32 PM, Luis P. wrote: > [http://ktulu.com.ar/blog/](http://ktulu.com.ar/blog/) > Of course, you do realize that you’re saying “scripts” but you’re > removing “comments” with this regexp. I can have: in a page with not a in sight! -Rob Rob B. [http://agileconsultingllc.com](http://agileconsultingllc.com) removed\_email\_address@domain.invalid ---
**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\ **Post date:** [May 23, 2007, 11:32pm UTC](https://www.ruby-forum.com/t/odd-regexp-issue/104927/7 "2007-05-23T23:32:26Z")
On May 23, 2007, at 5:10 PM, Kyle H. wrote: > Well, that seemed to do the trick 😃 > > Thanks a lot, I didn’t know that regexp only applied to one line by > default, HRM! > > Thanks, > Kyle H. Actually, it is the . expression that doesn’t match a newline without the ‘m’ option. That option just changes ‘.’ from matching “any character except a newline” to matching “any character”. The Regular Expression section of chapter 22 in the pickaxe covers all this (p. 324-328) -Rob Rob B. [http://agileconsultingllc.com](http://agileconsultingllc.com) removed\_email\_address@domain.invalid ---
**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\ **Post date:** [May 25, 2007, 3:35pm UTC](https://www.ruby-forum.com/t/odd-regexp-issue/104927/8 "2007-05-25T15:35:19Z")
Kyle H. wrote: > I’m writing a web crawler, and in that crawler I want to remove all > scripts in the pages I crawl. > > I should be able to do a simple gsub!(//,“”) right? Well, I do > that and unfortunately it doesn’t remove some scripts. Take google for > instance. It removes the first script, but not the second. I’m really > confused. Since google has two scripts, > so it’s not like the full regexp should ever fail to be triggered. > > Any insight on the issue would be GREAT?! 😃 > > Thanks, > Kyle H. I’m not sure what are you after actually, but apart from the tags Rob mentioned, you might need to remove the onClick, onMouseOver and other handlers. And since the handlers can be within almost any tag it would be very hard to find and remove them correctly with just a few regexps. You should use a real HTML parser (the preffered Ruby one seems to be called hpricot … I guess the author wanted to be funny). If this is meant to make the display of the pages secure you should also rather “keep only the tags and attributes that are safe” than “remove stuff that’s not safe”. You might easily overlook something. If you happened to use the-language-that-musn’t-be-named, you’d just use HTML::TagFilter ([HTML::TagFilter - A fine-grained html-filter, xss-blocker and mailto-obfuscator - metacpan.org](http://search.cpan.org/~wross/HTML-TagFilter-1.03/TagFilter.pm)). Good luck. Jenda