How can I trim top of file w/o disrupting other writers?


#1

As a way to learn ruby, I wrote a method to trim the first “n” lines
from a log file. I had planned to use it on a series of log files
that are constantly growing. However, some of the programs that are
appending to these logs don’t tolerate the somewhat brute-force method
I’m using, which is:

  1. Write the lines I want to keep to a temp file
  2. Rename the temp file to the original log file name

This works fine for some log files, but others stop receiving log data
after I run my program. I suspect it is because they have the file
open and don’t notice that the inode (or some other pointer to the log
file) has been pulled out from underneath them.

I realize this is more of a Linux/UNIX question, but I’d still like to
implement the solution as a ruby method if possible.

For what it’s worth, my current method is (please excuse the lack of
ruby-ness)

Trims a file down to last “n” number of lines.

If save_orig == TRUE then the trimmed lines are

appended to a file suffixed with _saved. Otherwise

the trimmed lines are discarded.

def trim_file (file_name, nbr_lines_to_keep = 5000, save_orig = FALSE)

The file to be trimmed must be writable by the process owner

if File.writable?(file_name)

# The file to be trimmed must either be owned by the process owner
# (i.e. same account that is running this program) or the process
# owner needs to be root (or running as root via sudo).
#
if (File.stat(file_name).owned?) || (Process.euid == 0)

   # get current uid and gid of the file (in case it
   # isn't same as default)
   #
   f_uid = File.stat(file_name).uid
   f_gid = File.stat(file_name).gid

   all_lines = IO.readlines(file_name)
   nbr_lines = all_lines.size
   if nbr_lines > nbr_lines_to_keep

     start_line  = nbr_lines - nbr_lines_to_keep
     tmpfilename = file_name + "_temptrimfile"

     tmpfile_nst = File.new(tmpfilename, "w")
     tmpfile_nst.puts(all_lines[start_line..nbr_lines])
     tmpfile_nst.close

     if save_orig == TRUE
         savefile_nst = File.new(file_name + "_saved", "a")
         savefile_nst.puts(all_lines[0..start_line-1])
         savefile_nst.close
         File.chown(f_uid, f_gid, file_name + "_saved")
     end

     File.rename(tmpfilename, file_name)
     # This script *could* be running as root or via sudo
     # in which case we need to preserve the original uid
     # and gid of the file.  Otherwise the trimmed and/or
     # saved file would be owned by root
     File.chown(f_uid, f_gid, file_name)
     print "#{file_name} => trimmed to #{nbr_lines_to_keep} lines.

"
print "First #{start_line} lines "
print “appended to #{file_name}_saved\n” if save_orig == TRUE
print “discarded\n” if save_orig ==
FALSE
else
print "File #{file_name} not trimmed. Nbr lines (#
{nbr_lines}) "
print “not greater than #{nbr_lines_to_keep}.\n”
end
else
puts “File #{file_name} not trimmed. You are not file owner or
root”
end
else
puts “File #{file_name} not trimmed. Not writable or doesn’t
exist.”
end
end # end method trim_file


#2

On 07/05/2009, at 9:35 AM, bradjpeek removed_email_address@domain.invalid wrote:

after I run my program. I suspect it is because they have the file
open and don’t notice that the inode (or some other pointer to the log
file) has been pulled out from underneath them.

Why can’t you simply open the file itself for read write then replace
the contents? That’s worked fine for me before

Blog: http://random8.zenunit.com/
Learn: http://sensei.zenunit.com/
Twitter: http://twitter.com/random8r


#3

On May 6, 9:15 pm, Julian L. removed_email_address@domain.invalid wrote:

This works fine for some log files, but others stop receiving log data
after I run my program. I suspect it is because they have the file
open and don’t notice that the inode (or some other pointer to the log
file) has been pulled out from underneath them.

Why can’t you simply open the file itself for read write then replace
the contents? That’s worked fine for me before

That worked. Thanks for the embarassingly obvious answer.

Actually, that occurred to me when I first wrote the script and I
can’t remember why I chose the rename approach.