Several processes writing to same file without flock-ing: Is controlled desaster possible?

Ruby
1

Problem: I have several processes running on the same host, all are Ruby
programs. Each process writes to the SAME file, using something like
fh.puts(“some data”). I want to ensure, that no data line gets lost, and
only complete data lines are written to the file. I don’t care about the
relative sequence of the data line. The solution should work on Cygwin
and on Linux.

The usual (?) way to implement this, would be to monitor writing to the
file, for example using flock. However, I wonder whether - under the
special restrictions outlined above - I could do without controlling the
concurrency, by using the following:

  • The file is opened for appending
  • The buffer is flushed after writing each line

I did an experiment with this approach (on Cygwin, with Ruby 2.3.3),
running 20 processes in parallel, and the resulting file looked fine,
but maybe this was just sheer luck. Do you think my approach is
guaranteed to work, or do we have a possible race condition?

For completeness, here is the code I have used:

Program wtf.rb:

#!/usr/bin/ruby
File.open(“garble#{ARGV[1]||’’}.txt”,
File::WRONLY|File::CREAT|File::APPEND, 0644) do |fh|
(ARGV[0]||1000).to_i.times do |n|
fh.puts("#{Time.now} : Process #{$$} in iteration #{n}")
fh.flush
end
end

Driver programm (execute it just as ‘driver.sh’, without parameters):

#!/bin/zsh
rm garble.txt
for n in $(seq ${2:-20})
do
wtf$3.rb ${1:-1000} >process$n.txt 2>&1 &
done
echo waiting for the child processes to finish …
wait
less garble.txt

2

Just a hypothesis, that you are writing such a small amount of data to
that file that the OS can swallow it almost instantly and Ruby is not
the speediest language around.
Try increasing the data you write to the a line (1k,2k, 10k) and I
suspect you will eventually reach a threshold where you will experience
problems (opening an already open file, etc.).

3

Of course you do close those files :slight_smile:

If you use File.open with a block, the file will be automatically closed
at the end of the block.

4

Földes László wrote in post #1179090:

Just a hypothesis, that you are writing such a small amount of data to
that file that the OS can swallow it almost instantly and Ruby is not
the speediest language around.
Try increasing the data you write to the a line (1k,2k, 10k) and I
suspect you will eventually reach a threshold where you will experience
problems (opening an already open file, etc.).

I agree that it makes sense to write a larger amount of data, but at
least the problem of “opening an open file” should occur with my test
case too, if it occurs at all, because I don’t close the files in
between anyway.

I tried my test again, writing 1000 times as much information on each
write (which would be around 50K), and here too did not encounter any
problems. Of course this doesn’t prove that problems can’t happen.

5

Földes László wrote in post #1179117:

Of course you do close those files :slight_smile:

If you use File.open with a block, the file will be automatically closed
at the end of the block.

Even without using a block, the files would be closed afterwards,
because the program terminates immediately after the end of the block,
as we can see from the listing, and this would implicitly close all open
files. I don’t see how closing a file at the end of the program is
related to my question.