I am having a very annoying issue. This is the situation:
I am reading a file on one machine that is continuosly being written to
and writing out to a socket. On another machine I am reading from the
socket and writing it to a file. The problem is that when I tail the
original file and the file I am creating, the new file has a delay of
around 1 minute until I see the new data. It seems that the OS (Linux
Redhat 5) is waiting until it gets a block of data before it actually
writes to the file. I want it to write immediately.
Here is a code snippet of how I am reading from the socket and writing
to the file.
file = File.new(fname, “w”)
file.sync = true
buff = “”
loop{
begin
ios = IO.select([server.socket], [file],[server.socket, file],
0.0001)
if ios[0].length > 0
buff = buff + ios[0].first.readpartial(8192)
end
if ios[1].length > 0
file.write buff
buff = “”
end
rescue Errno::ECONNRESET, SocketError
puts “ERROR: Lost connection to server”
exit 1
rescue EOFError
end
sleep 0.01
}
}
I first used just a regular readline from the socket and then did a
file.puts line, but I was having the same issue sot that is why I tried
to go to select. I am wondering if this is a Linux issue, since I
turned the file sync to true. However, I am not sure, so I am hoping
that other code heads out there could help me out.
writes to the file. I want it to write immediately.
Here is a code snippet of how I am reading from the socket and writing
to the file.
file = File.new(fname, “w”)
file.sync = true
Hi Matt,
“file.sync = true” looks good already.
Any chance you’re writing to a file on NFS or some other remote
filesystem?
Since you’re on Linux, try stracing both the Ruby process and your tail
process to see what it’s doing.
buff = “”
loop{
begin
ios = IO.select([server.socket], [file],[server.socket, file],
0.0001)
There’s no point in ever calling IO.select on a regular file,
they’re always ready for reading/writing.
Since you’re down to one socket, I wouldn’t bother with select at all:
file.write(server.socket.readpartial(8192))
I first used just a regular readline from the socket and then did a
file.puts line, but I was having the same issue sot that is why I tried
to go to select. I am wondering if this is a Linux issue, since I
turned the file sync to true. However, I am not sure, so I am hoping
that other code heads out there could help me out.
Whatever it is, there’s a high likelyhood that strace-ing one of the
processes involved will tell you what the problem is :>
Redhat 5) is waiting until it gets a block of data before it actually
“file.sync = true” looks good already.
You could try calling fsync as well (it’s not a writer, just call
“file.fsync”):
Implementation from IO
ios.fsync -> 0 or nil
Immediately writes all buffered data in ios to disk. Note that
fsync differs from using IO#sync=. The latter ensures that data
is flushed from Ruby’s buffers, but doesn’t not guarantee that the
underlying
operating system actually writes it to disk.
NotImplementedError is raised if the underlying operating system does
not support fsync(2).
I am writing to NFS, so I am thinking that this may be the issue. I’m
starting to think that there is no way for me to get what I want, except
to rewrite this in C or Assembly
NFS is your problem, not Ruby. See the nfs(5) manpage and see if you
(or your friendly systems administrator) can tune the NFS mount
options to invalidate the cache more frequently.
Ruby IO methods are already thin wrappers around corresponding C
functions (and thinner if you use the sys{write,read,…} variants).
You normally don’t need to go lower-level than that. Heck, with enough
time and effort, you even could write your own NFS client implementation
in Ruby (not a serious suggestion
I am writing to NFS, so I am thinking that this may be the issue. I’m
starting to think that there is no way for me to get what I want, except
to rewrite this in C or Assembly
NFS is your problem, not Ruby. See the nfs(5) manpage and see if you
(or your friendly systems administrator) can tune the NFS mount
options to invalidate the cache more frequently.
Ruby IO methods are already thin wrappers around corresponding C
functions (and thinner if you use the sys{write,read,…} variants).
You normally don’t need to go lower-level than that. Heck, with enough
time and effort, you even could write your own NFS client implementation
in Ruby (not a serious suggestion
Will do. Thanks again Eric for your help. I was pulling my hair out in
annoyance. Going to have to go to the SA’s dungeon to see if they can
accommodate me, hehe.
Redhat 5) is waiting until it gets a block of data before it actually
“file.sync = true” looks good already.
You could try calling fsync as well (it’s not a writer, just call
“file.fsync”):
Implementation from IO
ios.fsync -> 0 or nil
Immediately writes all buffered data in ios to disk. Note that
fsync differs from using IO#sync=. The latter ensures that data
is flushed from Ruby’s buffers, but doesn’t not guarantee that the
underlying
operating system actually writes it to disk.
NotImplementedError is raised if the underlying operating system does
not support fsync(2).
Hey fellas, thanks for the response.
I tried file.fsync, and checked to make sure that the return was not
nil, and it wasn’t. However, I am seeing the same activity. I took out
the select call and as Eric suggested and just used
“file.write(server.socket.readpartial(8k))” with and that did not seem
to help either.
I am writing to NFS, so I am thinking that this may be the issue. I’m
starting to think that there is no way for me to get what I want, except
to rewrite this in C or Assembly
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
