# Code duplication **URL:** \ **Category:** Ruby\ **Created:** [April 6, 2009, 3:51pm UTC](https://www.ruby-forum.com/t/code-duplication/164845 "2009-04-06T15:51:18Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [April 6, 2009, 3:51pm UTC](https://www.ruby-forum.com/t/code-duplication/164845/1 "2009-04-06T15:51:18Z")
Hi all, The following is the code for extracting the html contents of a website. I have included the code in case url redirect and BadRequest error. #getting the HTTP response from ‘uri’ response = Net::HTTP.get\_response(uri) case response # if the url is redirecting then fetch the contents of the redirected url when Net::HTTPRedirection then uri = URI.parse(response[‘Location’]) response = Net::HTTP.get\_response(uri) # in case of a bad request error when Net::HTTPBadRequest then http = Net::HTTP.start(uri.host, uri.port) #getting the html data by setting the path as ‘/’ and using a user agent response = http.get("/", “User-Agent”=\>“Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)”) end data = response.body My tutor is saying that there is a duplication in the above code. ie. code for html reading is specified twice without any purpose and it should be removed. I’ve no idea where there is a mistake. I’m a newbee to ruby and i don’t understand the problem correctly or where things went wrong. Can anyone please help me to find the mistake. Thanks in advance. Regards Arun ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [April 6, 2009, 4:28pm UTC](https://www.ruby-forum.com/t/code-duplication/164845/2 "2009-04-06T16:28:15Z")
Arun K. wrote: > Hi all, > The following is the code for extracting the html contents of a > website. I have included the code in case url redirect and BadRequest > error. > > #getting the HTTP response from ‘uri’ > response = Net::HTTP.get\_response(uri) > case response > # if the url is redirecting then fetch the contents of the > redirected url > when Net::HTTPRedirection then uri = URI.parse(response[‘Location’]) > response = > Net::HTTP.get\_response(uri) > > # in case of a bad request error > > when Net::HTTPBadRequest then http = Net::HTTP.start(uri.host, > uri.port) > #getting the html data by setting the path as ‘/’ and using a user > agent > response = http.get("/", “User-Agent”=\>“Mozilla/4.0 (compatible; MSIE > 5.5; Windows NT 5.0)”) > end > > data = response.body > > My tutor is saying that there is a duplication in the above code. ie. > code for html reading is specified twice without any purpose and it > should be removed. I’ve no idea where there is a mistake. I’m a newbee > to ruby and i don’t understand the problem correctly or where things > went wrong. Can anyone please help me to find the mistake. > > Thanks in advance. > > Regards > Arun What is the use of this below statement ? response = http.get("/", “User-Agent”=\>“Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)”) Since you had already got the response object using get\_response, then why it is needed? ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [April 6, 2009, 4:39pm UTC](https://www.ruby-forum.com/t/code-duplication/164845/3 "2009-04-06T16:39:58Z")
Loga G. wrote: > Arun K. wrote: > > > Hi all, > > The following is the code for extracting the html contents of a > > website. I have included the code in case url redirect and BadRequest > > error. > > > > #getting the HTTP response from ‘uri’ > > response = Net::HTTP.get\_response(uri) > > case response > > # if the url is redirecting then fetch the contents of the > > redirected url > > when Net::HTTPRedirection then uri = URI.parse(response[‘Location’]) > > response = > > Net::HTTP.get\_response(uri) > > > > # in case of a bad request error > > > > when Net::HTTPBadRequest then http = Net::HTTP.start(uri.host, > > uri.port) > > #getting the html data by setting the path as ‘/’ and using a user > > agent > > response = http.get(“/”, “User-Agent”=\>“Mozilla/4.0 (compatible; MSIE > > 5.5; Windows NT 5.0)”) > > end > > > > data = response.body > > > > My tutor is saying that there is a duplication in the above code. ie. > > code for html reading is specified twice without any purpose and it > > should be removed. I’ve no idea where there is a mistake. I’m a newbee > > to ruby and i don’t understand the problem correctly or where things > > went wrong. Can anyone please help me to find the mistake. > > > > Thanks in advance. > > > > Regards > > Arun > > What is the use of this below statement ? > response = http.get(“/”, “User-Agent”=\>“Mozilla/4.0 > (compatible; MSIE > 5.5; Windows NT 5.0)”) > > Since you had already got the response object using get\_response, then > why it is needed? Hi, Thanks for the reply. If it is a bad request error, then I have to communicate to the port and host and then I’ve to fetch the data. For eg. if i try to fetch html contents from [youtube.com](http://youtube.com), i get a bad request error. So I used the Net::HTTP.start() and then I used the path and user agent to retreive the contents and stored it in response. I dont think that there is any other way. If I remove that part, I’m not able to read the html. Thanks Arun ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [April 6, 2009, 5:21pm UTC](https://www.ruby-forum.com/t/code-duplication/164845/4 "2009-04-06T17:21:02Z")
On 6 Apr 2009, at 14:50, Arun K. wrote: > My tutor is saying that there is a duplication in the above code. ie. > code for html reading is specified twice without any purpose and it > should be removed. I’ve no idea where there is a mistake. I’m a newbee > to ruby and i don’t understand the problem correctly or where things > went wrong. Can anyone please help me to find the mistake. There are probably better solutions, but the following illustrates the point your tutor is making: MOZILLA\_HEADER = { “User-Agent”=\>“Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)” } def get\_http\_response uri, max\_redirects = 0 Net::HTTP.start(uri) do |connection| response = connection.get(uri.path, MOZILLA\_HEADER) response &&= case response when Net::HTTPRedirection if max\_redirects \> 0 then get\_http\_response URI.parse(response[‘Location’]), (max\_redirects - 1) else raise “Too many redirects” end when Net::HTTPRedirection get\_http\_response URI.parse(“http://#{uri.host}:#{uri.port}/”), max\_redirects end end end data = get\_http\_response(my\_uri, 3).body See how get\_http\_response is recursive in the case of an erroneous response? This minimises the actual HTTP interaction code as well as elegantly handling redirects. Whilst this could result in many more http connections being used, it also makes them clear up after themselves which is always good. Ellie ## Eleanor McHugh Games With Brains [http://slides.games-with-brains.net](http://slides.games-with-brains.net) raise ArgumentError unless @reality.responds\_to? :reason ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [April 7, 2009, 6:32am UTC](https://www.ruby-forum.com/t/code-duplication/164845/5 "2009-04-07T06:32:31Z")
> There are probably better solutions, but the following illustrates the > point your tutor is making: > > MOZILLA\_HEADER = { “User-Agent”=\>“Mozilla/4.0 (compatible; MSIE 5.5; > Windows NT 5.0)” } > > def get\_http\_response uri, max\_redirects = 0 > Net::HTTP.start(uri) do |connection| > response = connection.get(uri.path, MOZILLA\_HEADER) > response &&= case response > when Net::HTTPRedirection > if max\_redirects \> 0 then > get\_http\_response URI.parse(response[‘Location’]), > (max\_redirects - 1) > else > raise “Too many redirects” > end > when Net::HTTPRedirection > get\_http\_response URI.parse(“http://#{uri.host}:#{uri.port}/”), > max\_redirects > end > end > end > > data = get\_http\_response(my\_uri, 3).body Thanks Ellie,You gave me a clue of not only solving the code duplication but also about handling the redirects. Thanks a lot. Regards Arun ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [April 7, 2009, 1:19pm UTC](https://www.ruby-forum.com/t/code-duplication/164845/6 "2009-04-07T13:19:36Z")
Eleanor McHugh wrote: > On 7 Apr 2009, at 05:31, Arun K. wrote: > > > Thanks Ellie,You gave me a clue of not only solving the code > > duplication but also about handling the redirects. Thanks a lot. > > My pleasure 🙂 > > Ellie > > ## Eleanor McHugh Games With Brains [http://slides.games-with-brains.net](http://slides.games-with-brains.net) > > raise ArgumentError unless @reality.responds\_to? :reason Hi Ellie, I once again thank you for your reply. It helped me a lot. Now I want to share some doubt with you. 1. How can i specify the redirect limit without declaring it inside a method. Is it possible? 2. By including a redirect limit, will I be able to make the code for url redirection the most effective one or should i include some aditions to the code to handle redirection effectively? Thanks Arun ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [April 7, 2009, 2:32pm UTC](https://www.ruby-forum.com/t/code-duplication/164845/7 "2009-04-07T14:32:19Z")
On 7 Apr 2009, at 12:18, Arun K. wrote: > Hi Ellie, > I once again thank you for your reply. It helped me a lot. Now I > want to > share some doubt with you. > > 1. How can i specify the redirect limit without declaring it inside a > method. Is it possible? The redirect limit isn’t declared inside the method but as one of the parameters of the method, which is why it allows recursive execution as each redirect is received. You’ll note that I provided an initial value as part of the initial functional call: data = get\_http\_response(my\_uri, 3).body but in a real-world program you either specify a constant and use that: MAXIMUM\_REDIRECTS = 3 data = get\_http\_response(MAXIMUM\_REDIRECTS, 3).body or else wrap everything together into an object where this value would be either an instance or class variable depending on your intent. > 1. By including a redirect limit, will I be able to make the code for > url redirection the most effective one or should i include some > aditions > to the code to handle redirection effectively? I can’t really answer that question without knowing more about the real-world problem you’re trying to solve. However in general I’d say that whenever you have a recursive problem like this it’s sensible to ensure that it’s throttled to prevent resource exhaustion. For a very graphic example of why this is important - especially with network applications - read up on the Morris Worm 🙂 Ellie ## Eleanor McHugh Games With Brains [http://slides.games-with-brains.net](http://slides.games-with-brains.net) raise ArgumentError unless @reality.responds\_to? :reason ---
**Author:** ![arunvoip](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e8c25b/32.png) [@arunvoip](https://www.ruby-forum.com/u/arunvoip)\ **Post date:** [April 7, 2009, 1:07pm UTC](https://www.ruby-forum.com/t/code-duplication/164845/8 "2009-04-07T13:07:58Z")
On 7 Apr 2009, at 05:31, Arun K. wrote: > Thanks Ellie,You gave me a clue of not only solving the code > duplication but also about handling the redirects. Thanks a lot. My pleasure 🙂 Ellie ## Eleanor McHugh Games With Brains [http://slides.games-with-brains.net](http://slides.games-with-brains.net) raise ArgumentError unless @reality.responds\_to? :reason