Synchronize between download and save in multi-threads?

i want to download all usa listed company historial data from yahoo
finance,
at first i download all compamy name list and save it in
/home/pt/usastock/totalname(amex nyse nasdaq,6000 companies),here is my
program to dowload their historial data from yahoo finance with
multi-threads:
require ‘rubygems’
require ‘open-uri’
threads = []
puts time0=Time.now
i=0
j=0
str=[]
open(’/home/pt/usastock/totalname’,‘r’){|item|
while line=item.gets
line=line.chomp
i=i+1
str<<“http://ichart.finance.yahoo.com/table.csv?s=”+line
if i==10 or item.eof == true then
begin
for page_to_fetch in str
threads<<Thread.new(page_to_fetch) do |url|
open(url){|urlfile|
datafile="/home/pt/test/"+url.split("=")[1].to_s
open(datafile,‘w’){|file|
file.puts urlfile.read
j=j+1
puts “#{url} #{j} is over”
}}
end
end
threads.each {|thr| thr.join}
i=0
str=[]
threads = []
rescue Timeout::Error => e
rescue => e
end
else
end
end
}
puts Time.now-time0
it can run seccessfully,when i change
if i==10 or item.eof == true then (which means i open 10 threads)
into
if i==100 or item.eof == true then (which means i open 100
threads)
there is a problem,some data will not saved in my designated position,
(datafile="/home/pt/axel/"+url.split("=")[1].to_s)
some of them are saved such as “/home/pt/table.csv?s=YHOO”
according to my program,it should be saved as “/home/pt/test/YHOO”
i think that maybe it is a problem of synchronizimg between download
and save
in my program.
would you mind to fix it?
thinks.

i think that maybe it is a problem of synchronizimg between download
and save
in my program.
would you mind to fix it?

The “Mutex” class might be useful to you.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs