# Error handling with exceptions in multi-thread download

**URL:** <https://www.ruby-forum.com/t/error-handling-with-exceptions-in-multi-thread-download/193974>\
**Category:** Ruby\
**Created:** [August 27, 2010, 11:58am UTC](https://www.ruby-forum.com/t/error-handling-with-exceptions-in-multi-thread-download/193974 "2010-08-27T11:58:25Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![luofeiyu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/9de0a6/32.png) [@luofeiyu](https://www.ruby-forum.com/u/luofeiyu)\
**Post date:** [August 27, 2010, 11:58am UTC](https://www.ruby-forum.com/t/error-handling-with-exceptions-in-multi-thread-download/193974/1 "2010-08-27T11:58:25Z")

</div>

```
there are three program fragments,all of them have same problems,
all of them can run ,but when i downloaded 30% data,an error

```

ocurr  
undefined local variable or method `web’ for Data:Class (NameError)  
cthreaddown is an array which contains many web.  
would you kind to tell me how to fix it?  
p1  
`
     for page_to_fetch in cthreaddown
     threads<<Thread.new(page_to_fetch) do |web|
         datafile="/home/pt/stock/"
         open(datafile,'a+'){|refile|
         open(web){|webfile|
              refile.puts webfile.read
              puts "#{web} is over"
         }}
        end
      end
     threads.each {|thr|
        begin
         thr.join
         rescue Timeout::Error => e
           puts "#{web} failed"
         rescue NameError => e
           puts "#{web} failed"
         rescue => e
           puts "#{web} failed"
        end
       }
      `  
p2  
`
     for page_to_fetch in cthreaddown
     threads<<Thread.new(page_to_fetch) do |web|
         datafile="/home/pt/stock/"
         open(datafile,'a+'){|refile|
         begin
        open(web){|webfile|
              refile.puts webfile.read
              puts "#{web} is over"
         }}
        rescue NameError => e
           puts "#{web} failed"
        rescue => e
           puts "#{web} wrong"
         end
       end
      end
      threads.each {|thr| thr.join }
     `

```
 p3
 [code]
 for page_to_fetch in cthreaddown
 threads<<Thread.new(page_to_fetch) do |web|
     datafile="/home/pt/stock/"
     open(datafile,'a+'){|refile|
     begin
    open(web){|webfile|
          refile.puts webfile.read
          puts "#{web} is over"
     }}
    rescue NameError => e
       puts "#{web} failed"
    rescue => e
       puts "#{web} wrong"
     end
   end
  end

```

threads.each {|thr|  
begin  
thr.join  
rescue Timeout::Error =\> e  
puts “#{web} failed”  
rescue NameError =\> e  
puts “#{web} failed”  
rescue =\> e  
puts “#{web} failed”  
end  
}

---

<div class="post-metadata">

**Author:** ![luofeiyu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/9de0a6/32.png) [@luofeiyu](https://www.ruby-forum.com/u/luofeiyu)\
**Post date:** [August 27, 2010, 12:16pm UTC](https://www.ruby-forum.com/t/error-handling-with-exceptions-in-multi-thread-download/193974/2 "2010-08-27T12:16:52Z")

</div>

You probably have a TimeOut error. And the errors actually comes from:

```
    rescue Timeout::Error => e
      puts "#{web} failed"

```

because web is not defined there.

B

---

<div class="post-metadata">

**Author:** ![luofeiyu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/9de0a6/32.png) [@luofeiyu](https://www.ruby-forum.com/u/luofeiyu)\
**Post date:** [August 27, 2010, 2:24pm UTC](https://www.ruby-forum.com/t/error-handling-with-exceptions-in-multi-thread-download/193974/3 "2010-08-27T14:24:14Z")

</div>

i have omitted many lines of code,no TimeOut error,  
for page\_to\_fetch in cthreaddown  
threads\<\<Thread.new(page\_to\_fetch) do |web|  
i have got an array— cthreaddown before  
it can run , can get 30% data ,and then stop,

undefined local variable or method `web’ for Data:Class (NameError)

---

<div class="post-metadata">

**Author:** ![luofeiyu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/9de0a6/32.png) [@luofeiyu](https://www.ruby-forum.com/u/luofeiyu)\
**Post date:** [August 27, 2010, 3:07pm UTC](https://www.ruby-forum.com/t/error-handling-with-exceptions-in-multi-thread-download/193974/4 "2010-08-27T15:07:48Z")

</div>

2010/8/27 Pen T. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> there are three program fragments,all of them have same problems,  
> all of them can run ,but when i downloaded 30% data,an error  
> ocurr  
> undefined local variable or method `web’ for Data:Class (NameError)

You need to look into class methods of class Data. Somewhere there  
you will find usage of “web” which is undefined. Unfortunately you  
did not post that class so we cannot tell you where your error is.

> cthreaddown is an array which contains many web.  
> would you kind to tell me how to fix it?  
> p1  
> [code]  
> for page\_to\_fetch in cthreaddown  
> threads\<\<Thread.new(page\_to\_fetch) do |web|  
> datafile=“/home/pt/stock/”  
> open(datafile,‘a+’){|refile|  
> open(web){|webfile|  
> refile.puts webfile.read

You should use #write with #read and not #puts, because #puts does not  
write out arguments unmodified. Also note that if webfile is large  
this approach will break. You should better use blocked reading.

buffer = “”  
while webfile.read(1024, buffer)  
refile.write buffer  
end

> ```
> puts "#{web} failed"
> rescue => e
> puts "#{web} failed"
> end
> }
> [/code]
> 
> ```

Cheers

robert

---

<div class="post-metadata">

**Author:** ![luofeiyu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/9de0a6/32.png) [@luofeiyu](https://www.ruby-forum.com/u/luofeiyu)\
**Post date:** [August 27, 2010, 4:23pm UTC](https://www.ruby-forum.com/t/error-handling-with-exceptions-in-multi-thread-download/193974/5 "2010-08-27T16:23:58Z")

</div>

sorry for that: def self.getdaystock(fromdate)

---

<div class="post-metadata">

**Author:** ![luofeiyu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/9de0a6/32.png) [@luofeiyu](https://www.ruby-forum.com/u/luofeiyu)\
**Post date:** [August 27, 2010, 4:22pm UTC](https://www.ruby-forum.com/t/error-handling-with-exceptions-in-multi-thread-download/193974/6 "2010-08-27T16:22:23Z")

</div>

require ‘rubygems’  
require ‘open-uri’  
require ‘nokogiri’  
require ‘mysql’  
require ‘date’

class Data  
def self.getdaystock()  
time0=Time.now  
i=0  
cdown=[]  
threads=[]  
cthreaddown=[]  
open(“/home/pt/usa/stocklist”,“r”){|item|  
while line=item.gets  
cdown\<\<line  
end}  
a=(fromdate.split(“-”)[1].to\_i-1).to\_s  
b=fromdate.split(“-”)[2].to\_s  
c=fromdate.split(“-”)[0].to\_s  
d=Time.now.mon.to\_s  
e=Time.now.day.to\_s  
f=Time.now.year.to\_s  
cdown=cdown.map {|item|  
item=‘[http://ichart.finance.yahoo.com/table.csv?s=‘+item.to\_s+’&amp;a=‘+a+’&amp;b=‘+b+’&amp;c='+c+\](http://ichart.finance.yahoo.com/table.csv?s='+item.to_s+'&amp;a='+a+'&amp;b='+b+'&amp;c='+c+%5C)  
‘&d=’+d+’&e=‘+e+’&f=‘+f+’&g=d&ignore=.csv’}  
cdown.each.with\_index {|item,idx|  
cthreaddown\<\< item  
i=i+1  
if i==100 or idx==cdown.size-1 then  
cthreaddown=cthreaddown.uniq  
for page\_to\_fetch in cthreaddown  
threads\<\<Thread.new(page\_to\_fetch) do |web|  
datafile=“/home/pt/usa/stock/”+web.split(“=”)[1].to\_s.gsub(“&a”,“”)  
begin  
open(datafile,‘a+’){|refile|  
open(web){|webfile|  
refile.puts webfile.read  
puts “#{web} over” }}  
rescue Timeout::Error =\> e  
puts “#{web} failed”  
rescue NameError =\> e  
puts “#{web} failed”  
rescue =\> e  
puts “#{web} wrong”  
end  
end  
end  
threads.each {|thr|  
begin  
thr.join  
rescue Timeout::Error =\> e  
puts “#{web} failed”  
rescue NameError =\> e  
puts “#{web} failed”  
rescue =\> e  
puts “#{web} failed”  
end }  
i=0  
cthreaddown=[]  
threads = []  
sleep 0.01  
else  
end  
}  
end  
end

i have opened 100 threads. there are 6000 companies in  
/home/pt/usa/stocklist .

---

<div class="post-metadata">

**Author:** ![luofeiyu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/9de0a6/32.png) [@luofeiyu](https://www.ruby-forum.com/u/luofeiyu)\
**Post date:** [August 27, 2010, 5:41pm UTC](https://www.ruby-forum.com/t/error-handling-with-exceptions-in-multi-thread-download/193974/7 "2010-08-27T17:41:14Z")

</div>

On 27.08.2010 16:22, Pen T. wrote:

> cdown=[]  
> e=Time.now.day.to\_s  
> threads\<\<Thread.new(page\_to\_fetch) do |web|  
> rescue =\> e  
> puts “#{web} failed”  
> end  
> end
> 
> i have opened 100 threads. there are 6000 companies in  
> /home/pt/usa/stocklist .

Well, there you have it. The error is in your exception handling code.

Btw, the whole code could use a bit of commenting and also be simplified  
in areas. For example: you store Time.now initially and then call it  
over and over again to fetch individual fields. This is not only  
inefficient but also error prone because you will not get all values  
from the same point in time.

Also, it’s a bad die to hardcode the URL in the middle of the code.  
It’s much better to pass this as parameter or at least store it in some  
constant for easier reference and documentation purposes.

Cheers

robert