Forum: Ruby multithreaded file access

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
C7d5bc5b054035d95f287797c2595694?d=identicon&s=25 Matias Surdi (Guest)
on 2005-12-23 17:41
(Received via mailing list)
Hi...

I've a class which is run by many threads at the same time.... this
class has to append a line to a text file eventually.

I do this with:

File.new('filename','a').puts("this is the string")


Is this already thread safe??? how can I make it so?????


Thanks...
882053229c4ca225e43cd9cd7fc21625?d=identicon&s=25 Jellen (Guest)
on 2005-12-23 18:14
(Received via mailing list)
Well, I think it's OK to do that.
Seeing is believing:
# first one
a = Thread.new do
  5.times do
    f = File.new("qq.txt", "a").puts "I am a..."
    f.close if f
  end
end
b = Thread.new do
  5.times do
    f = File.new("qq.txt", "a").puts "I am b..."
    f.close if f
  end
end
a.join
b.join

# and this one
f = File.new("test.txt", "a")
a = Thread.new do
  5.times do
    f.puts "I am a..."
    sleep 1
  end
end
b = Thread.new do
  5.times do
    f.puts "I am b..."
    sleep 1
  end
end
a.join
b.join
f.close


Both program are ok. (But I am not sure myself:)
Ace7fa5337acbdf5897a6fc035897580?d=identicon&s=25 J. Ryan Sobol (Guest)
on 2005-12-23 18:56
(Received via mailing list)
On Dec 23, 2005, at 12:12 PM, Jellen wrote:

>   5.times do
>   5.times do
> a.join
> b.join
> f.close
>
>
> Both program are ok. (But I am not sure myself:)

Interesting examples, Jellen, but I don't think it answers Matias'
question, which was

> File.new('filename','a').puts("this is the string")
>
>
> Is this already thread safe??? how can I make it so?????

Correct me if I'm wrong, but your examples only prove that the thread
on the CPU will be able to append the file.  I *think* Matias wants
to know if the statement ( File.new('filename','a').puts("this is the
string") ) is atomic.  Or in other words, do you need to enforce
mutual exclusive access to the file with a mutex?  Unfortunately, I
don't have an answer to that question.
E20e89d58211a3631842daecc1245de2?d=identicon&s=25 Ilmari Heikkinen (Guest)
on 2005-12-23 22:16
(Received via mailing list)
On 12/23/05, J. Ryan Sobol <ryansobol@gmail.com> wrote:
> Correct me if I'm wrong, but your examples only prove that the thread
> on the CPU will be able to append the file.  I *think* Matias wants
> to know if the statement ( File.new('filename','a').puts("this is the
> string") ) is atomic.  Or in other words, do you need to enforce
> mutual exclusive access to the file with a mutex?  Unfortunately, I
> don't have an answer to that question.

[kig@jugend:~] cat fw_test.rb
def writer(i, fn, ok)
  Thread.new{
    t_str = "#{i}" * 65536
    while ok.first
      File.open(fn, 'a'){|f|
        f.puts t_str
      }
    end
  }
end

ok = [true]
fn = 'fw_test.dat'
ts = (1..3).map{|i| writer i, fn, ok}

sleep 10

ok[0] = false

if File.readlines(fn).uniq.size == ts.size
  puts "puts in different threads seems to be atomic"
else
  puts "puts in different threads isn't atomic"
end

File.unlink fn

[kig@jugend:~] ruby fw_test.rb
puts in different threads seems to be atomic
Ace7fa5337acbdf5897a6fc035897580?d=identicon&s=25 J. Ryan Sobol (Guest)
on 2005-12-23 23:01
(Received via mailing list)
On Dec 23, 2005, at 4:13 PM, Ilmari Heikkinen wrote:

> end
>   puts "puts in different threads seems to be atomic"
> else
>   puts "puts in different threads isn't atomic"
> end
>
> File.unlink fn
>
> [kig@jugend:~] ruby fw_test.rb
> puts in different threads seems to be atomic

Crafty test program.  Coincidentally, my results differ from yours.

$ ruby fw_test.rb
puts in different threads isn't atomic
$ ruby -v
ruby 1.8.2 (2004-12-25) [powerpc-darwin8.3.0]

I'm getting 8 unique lines in File.readlines(fn).uniq, as opposed to
the 3 thread objects in ts.  Assuming I understand the program, that
means threads are not waiting their turn like they should.  FYI, I've
compiled my own ruby executable from DarwinPorts instead of using the
'broken' one shipped in OS X Tiger.

However, if I change the multiplication factor in line 3 to 655, then

$ ruby fw_test.rb
puts in different threads seems to be atomic

~ ryan ~

PS - I love the 205+ MB of text this app generates! :-D
E20e89d58211a3631842daecc1245de2?d=identicon&s=25 Ilmari Heikkinen (Guest)
on 2005-12-23 23:25
(Received via mailing list)
On 12/23/05, J. Ryan Sobol <ryansobol@gmail.com> wrote:
> On Dec 23, 2005, at 4:13 PM, Ilmari Heikkinen wrote:
> > [kig@jugend:~] ruby fw_test.rb
> > puts in different threads seems to be atomic
>
> Crafty test program.  Coincidentally, my results differ from yours.
>
> $ ruby fw_test.rb
> puts in different threads isn't atomic
> $ ruby -v
> ruby 1.8.2 (2004-12-25) [powerpc-darwin8.3.0]

Good! I thought it was odd that they seemed to be atomic..

And now we know to mutex disk writes (or use flock.)

> I'm getting 8 unique lines in File.readlines(fn).uniq, as opposed to
> the 3 thread objects in ts.  Assuming I understand the program, that
> means threads are not waiting their turn like they should.  FYI, I've
> compiled my own ruby executable from DarwinPorts instead of using the
> 'broken' one shipped in OS X Tiger.
>
> However, if I change the multiplication factor in line 3 to 655, then
>
> $ ruby fw_test.rb
> puts in different threads seems to be atomic

I don't know the reason (can guess, but too unsure)
Explanation anyone?

> ~ ryan ~
>
> PS - I love the 205+ MB of text this app generates! :-D

Ilmari
Ace7fa5337acbdf5897a6fc035897580?d=identicon&s=25 J. Ryan Sobol (Guest)
on 2005-12-23 23:46
(Received via mailing list)
On Dec 23, 2005, at 5:23 PM, Ilmari Heikkinen wrote:

>> ruby 1.8.2 (2004-12-25) [powerpc-darwin8.3.0]
>>
>> PS - I love the 205+ MB of text this app generates! :-D
>
> Ilmari

Regardless, the important thing is to know that it seems file IO via
puts is not thread safe, which I kind of assumed from the beginning.
This begs the question: which methods, especially concerning file IO,
are thread safe? (if any)

~ ryan ~
C7d5bc5b054035d95f287797c2595694?d=identicon&s=25 Matias Surdi (Guest)
on 2005-12-24 00:43
(Received via mailing list)
J. Ryan Sobol
escribió:>>> Crafty test program.  Coincidentally, my results differ from yours.
>>
>>
>
>
> Regardless, the important thing is to know that it seems file IO via
> puts is not thread safe, which I kind of assumed from the beginning.
> This begs the question: which methods, especially concerning file IO,
> are thread safe? (if any)
>
> ~ ryan ~
>
>


Hey!!.... I'm now more confused than before posting :-D
Fe9b2d0628c0943af374b2fe5b320a82?d=identicon&s=25 Eero Saynatkari (rue)
on 2005-12-24 01:11
Matias Surdi wrote:
> J. Ryan Sobol
> escribi󺦧t;>> Crafty test program.  Coincidentally, my results differ from yours.
>>>
>>>
>>
>>
>> Regardless, the important thing is to know that it seems file IO via
>> puts is not thread safe, which I kind of assumed from the beginning.
>> This begs the question: which methods, especially concerning file IO,
>> are thread safe? (if any)
>>
>> ~ ryan ~
>>
>>
>
>
> Hey!!.... I'm now more confused than before posting :-D

You should guard your critical section :) The simplest way
is using Mutex from the core. Consult your ri or make a visit
to http://www.ruby-doc.org.


E
C7d5bc5b054035d95f287797c2595694?d=identicon&s=25 Matias Surdi (Guest)
on 2005-12-24 09:11
(Received via mailing list)
Eero Saynatkari escribió:
>>>are thread safe? (if any)
> is using Mutex from the core. Consult your ri or make a visit
> to http://www.ruby-doc.org.
>
>
> E
>

Thanks everyone for replying...
Ace7fa5337acbdf5897a6fc035897580?d=identicon&s=25 J. Ryan Sobol (Guest)
on 2005-12-24 16:30
(Received via mailing list)
On Dec 23, 2005, at 6:42 PM, Matias Surdi wrote:
> Hey!!.... I'm now more confused than before posting :-D

http://www.rubycentral.com/book/tut_threads.html

Pay special attention to the section on mutual exclusion.

~ ryan ~
3a0d61d465c296f7f291b5bd09c90a4a?d=identicon&s=25 tony summerfelt (Guest)
on 2005-12-24 17:24
(Received via mailing list)
J. Ryan Sobol wrote on 12/23/2005 4:59 PM:

> Crafty test program.  Coincidentally, my results differ from yours.

i was in a similar situation.

i was under the impression from page 118 of my well thumbed copy of
'programming ruby' that any shared resource (eg. a file that will be
access by more than one thread) should be under the control of a
mutex...

the program i was working is long running (and been in use for almost
a year) and seems to be holding up ok...
E7559e558ececa67c40f452483b9ac8c?d=identicon&s=25 unknown (Guest)
on 2005-12-28 04:50
(Received via mailing list)
On Dec 23, 2005, at 11:38 AM, Matias Surdi wrote:
> I've a class which is run by many threads at the same time.... this
> class has to append a line to a text file eventually.
>
> I do this with:
>
> File.new('filename','a').puts("this is the string")

On Posix file systems, writes to a file in append mode are
atomic but that is only true when you make a direct system
call.  In Ruby that would be a call to IO#syswrite, which
bypasses all the standard buffering of puts and company.
You can't mix and match these types of IO calls.  It is one
or the other.

Caveats:  I think there is a limit on the size of the write
that will guaranteed to be atomic.  The Unix system calls
pathconf and fpathconf give you access to the PIBE_BUF limit
that specifies this limit for *pipes*.  I'm not sure if there
is a similar limit for files.  I just couldn't locate anything
specifically in my quick research.

Related Question: Does Ruby have its own buffering methodology
or does it use the C stdio library buffering for file I/O?
I just don't know enough about the Ruby internals to answer
this question.

Gary Wright
B000982a23d5c6a34292902caf225dd7?d=identicon&s=25 Yohanes Santoso (Guest)
on 2005-12-28 06:41
(Received via mailing list)
Matias Surdi <matiassurdi@gmail.com> writes:

> Hi...
>
> I've a class which is run by many threads at the same time.... this
> class has to append a line to a text file eventually.
>
> I do this with:
>
> File.new('filename','a').puts("this is the string")

Don't do this. It's not an atomic operation, either with buffered or
unbuffered IO.

> Is this already thread safe???

No.

> how can I make it so?????

If thread A outputs two line, and thread B outputs one line, would
this be acceptable?

A-1
B-1
A-2

Or should it be:

A-1
A-2
B-1

How granular do you want it? Per line? Per thread output?

Please investigate the mutex library.

file_mutex.synchronize {
  file.puts(".....")
  file.puts(".....")
}

YS.
5befe95e6648daec3dd5728cd36602d0?d=identicon&s=25 Robert Klemme (Guest)
on 2005-12-28 14:55
(Received via mailing list)
gwtmp01@mac.com wrote:
> On Dec 23, 2005, at 11:38 AM, Matias Surdi wrote:
>> I've a class which is run by many threads at the same time.... this
>> class has to append a line to a text file eventually.
>>
>> I do this with:
>>
>> File.new('filename','a').puts("this is the string")

Apart from MT issues this code has serious different issues: you do not
close the file (and you cannot because the IO is not returned from
puts).
You rather want

File.open('filename','a') {|io| io.puts("this is the string")}

Otherwise you risk that the text is not written to the file in the
proper
order - or you get even problems opening the file multiple times.

There are two possible solutions: synchronize every access to a file or
use
a Queue and a separate writing thread (might be better performance wise,
because you don't have to reopen the file all the time).

Kind regards

    robert
This topic is locked and can not be replied to.