# Trouble with threads

**URL:** <https://www.ruby-forum.com/t/trouble-with-threads/129724>\
**Category:** Ruby\
**Created:** [February 21, 2008, 2:06am UTC](https://www.ruby-forum.com/t/trouble-with-threads/129724 "2008-02-21T02:06:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![gmoniey](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/6a8cbe/32.png) [@gmoniey](https://www.ruby-forum.com/u/gmoniey)\
**Post date:** [February 21, 2008, 2:06am UTC](https://www.ruby-forum.com/t/trouble-with-threads/129724/1 "2008-02-21T02:06:14Z")

</div>

I am trying to expand my web crawler to use multiple threads (with  
mechanize), and I ma having some trouble. It seems that each thread is  
not creating a local variable, but rather they are sharing the “index”  
variable below:

threads = []  
mutex = Mutex.new

10.times do |i|  
threads[i] = Thread.new(i) { |index|  
while index \< @will\_visit.size  
current\_link = @will\_visit[index]  
begin  
index += 10  
puts current\_link  
page = @agent.get(current\_link)  
if(page.kind\_of? WWW::Mechanize::Page)  
page.links.each do |link|  
mutex.synchronize do  
if(validLink?(link))  
@will\_visit.push(link.href)  
end  
end  
end  
end

```
    puts "Currently visiting page #{index} of #{@will_visit.size}"
  rescue Exception => msg
    puts "Error with " + current_link

```

puts msg  
puts msg.backtrace  
end  
end  
}  
end  
threads.each {|t| t.join }

From what I have read from google, the ‘index’ variable should be  
independent between threads, but it seems that it is shared. The problem  
may also be with the face that @agent is shared, but I am not sure

---

<div class="post-metadata">

**Author:** ![gmoniey](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/6a8cbe/32.png) [@gmoniey](https://www.ruby-forum.com/u/gmoniey)\
**Post date:** [February 21, 2008, 2:25am UTC](https://www.ruby-forum.com/t/trouble-with-threads/129724/2 "2008-02-21T02:25:11Z")

</div>

On Thu, 21 Feb 2008 10:06:14 +0900, gm gm [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> From what I have read from google, the ‘index’ variable should be  
> independent between threads, but it seems that it is shared. The problem  
> may also be with the face that @agent is shared, but I am not sure

‘index’ is indeed independent here; @agent being shared, however, is  
very likely to cause problems. As far as I know, WWW::Mechanize agents  
are not safe for use by multiple threads. Each thread will need its  
own agent.

-mental

---

<div class="post-metadata">

**Author:** ![gmoniey](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/6a8cbe/32.png) [@gmoniey](https://www.ruby-forum.com/u/gmoniey)\
**Post date:** [February 21, 2008, 3:03am UTC](https://www.ruby-forum.com/t/trouble-with-threads/129724/3 "2008-02-21T03:03:16Z")

</div>

On Wed, Feb 20, 2008 at 5:24 PM, MenTaLguY [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> ‘index’ is indeed independent here; @agent being shared, however, is  
> very likely to cause problems. As far as I know, WWW::Mechanize agents  
> are not safe for use by multiple threads. Each thread will need its  
> own agent.

That agrees completely with my direct experience. WWW::Mechanize is,  
sadly, not threadsafe - it reuses a buffer for each request, and if  
you start a new request before another completes, the new request  
will clobber the input buffer. I gladly share my painfully won  
experience with you.

Judson

---

<div class="post-metadata">

**Author:** ![gmoniey](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/6a8cbe/32.png) [@gmoniey](https://www.ruby-forum.com/u/gmoniey)\
**Post date:** [February 21, 2008, 3:28am UTC](https://www.ruby-forum.com/t/trouble-with-threads/129724/4 "2008-02-21T03:28:53Z")

</div>

Judson L. wrote:

> On Wed, Feb 20, 2008 at 5:24 PM, MenTaLguY [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > ‘index’ is indeed independent here; @agent being shared, however, is  
> > very likely to cause problems. As far as I know, WWW::Mechanize agents  
> > are not safe for use by multiple threads. Each thread will need its  
> > own agent.
> 
> That agrees completely with my direct experience. WWW::Mechanize is,  
> sadly, not threadsafe - it reuses a buffer for each request, and if  
> you start a new request before another completes, the new request  
> will clobber the input buffer. I gladly share my painfully won  
> experience with you.
> 
> Judson

that would explain the crazy output. I will try and modify it a bit, and  
see how it works. Thanks!

---

<div class="post-metadata">

**Author:** ![gmoniey](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/6a8cbe/32.png) [@gmoniey](https://www.ruby-forum.com/u/gmoniey)\
**Post date:** [February 21, 2008, 3:35am UTC](https://www.ruby-forum.com/t/trouble-with-threads/129724/5 "2008-02-21T03:35:09Z")

</div>

Judson L. wrote:

> On Wed, Feb 20, 2008 at 5:24 PM, MenTaLguY [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > ‘index’ is indeed independent here; @agent being shared, however, is  
> > very likely to cause problems. As far as I know, WWW::Mechanize agents  
> > are not safe for use by multiple threads. Each thread will need its  
> > own agent.
> 
> That agrees completely with my direct experience. WWW::Mechanize is,  
> sadly, not threadsafe - it reuses a buffer for each request, and if  
> you start a new request before another completes, the new request  
> will clobber the input buffer. I gladly share my painfully won  
> experience with you.
> 
> Judson

that seemed to make a big difference…also, do you think I need to put  
a mutex around the ‘out.puts’ call (ie file output?)

---

<div class="post-metadata">

**Author:** ![gmoniey](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/6a8cbe/32.png) [@gmoniey](https://www.ruby-forum.com/u/gmoniey)\
**Post date:** [February 21, 2008, 3:55am UTC](https://www.ruby-forum.com/t/trouble-with-threads/129724/6 "2008-02-21T03:55:39Z")

</div>

gm gm wrote:

> I am trying to expand my web crawler to use multiple threads (with  
> mechanize), and I ma having some trouble. It seems that each thread is  
> not creating a local variable, but rather they are sharing the “index”  
> variable

threads = []

10.times do |i|  
threads[i] = Thread.new(i) do |index|  
if index == 0  
index += 100  
end

```
puts index

```

end  
end

threads.each do |t|  
t.join  
end

–output:–  
100  
1  
2  
3  
4  
5  
6  
7  
8  
9

If the threads shared the index variable, then each line of the output  
would be 100. Instead, i gets assigned to index for each thread, and  
nothing a thread does to its index variable has any effect on another  
thread’s index variable.