Rsync functionality in Ruby?

Hello,

I wrote a ruby script to allow me to automatically update a repository
working copy from
an unversioned hierarchy. I use rsync to synchronise the directories
prior to svn
add/delete and commit. Worked great on my test system (linux). However,
because of
security issues, on the system where this script will be used there is
no rsync utility
available.

So, I was wondering, is there an existing rsync-like ruby solution? I
checked this
newsgroup, rubyforge, raa, etc but couldn’t find anything that
replicated rsync
functionality (lots of utilities that use/manage rsync though.)

Note that I don’t really need the “r” portion of rsync (the sync occurs
on the same
filesystem), just the ability to synchronise two hierarchies with
"–exclude "
capability.

I thought I’d ask before embarking on a wheel-reinvention project.

cheers,

paulv

p.s. I’m pushing my luck here, but guidance on an algorithm or recipe
for sync
functionality would also be appreciated. :o)

On Fri, 31 Aug 2007 18:50:05 +0200, Paul van Delst
[email protected] wrote:

Hello,

I wrote a ruby script to allow me to automatically update a repository
working copy from an unversioned hierarchy. I use rsync to synchronise
the directories prior to svn add/delete and commit. Worked great on my
test system (linux). However, because of security issues, on the system
where this script will be used there is no rsync utility available.

Can’t you compile it yourself as a user? If not, could you compile
“cpdup”?
I’m not sure it runs on Linux, but it’s perfect on *BSDs.

Just compile it with ./configure --prefix=$HOME/usr. That usually works
for any
program. I do it regularily to install Ruby as a non-root user.

I don’t think rsync installed as user is in any way more dangerous than
Ruby.

Hope this helps.

Regards,

Michael

On Aug 31, 12:49 pm, Paul van Delst [email protected] wrote:

functionality (lots of utilities that use/manage rsync though.)

p.s. I’m pushing my luck here, but guidance on an algorithm or recipe for sync
functionality would also be appreciated. :o)

I haven’t found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I’d be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :slight_smile:

Dusty Doris

dusty wrote:

newsgroup, rubyforge, raa, etc but couldn’t find anything that replicated rsync
functionality (lots of utilities that use/manage rsync though.)

I haven’t found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I’d be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :slight_smile:

Hi,

Thanks for the reply. But, if I understand your reply, I can’t use
anything that uses unix
rsync under the covers, because there is no rsync available on the
system in question.

cheers,

paulv

there is no rsync utility
available.

So, I was wondering, is there an existing rsync-like ruby
solution? I checked this
newsgroup, rubyforge, raa, etc but couldn’t find anything that
replicated rsync
functionality (lots of utilities that use/manage rsync though.)

You should look at FileUtils::cp_r with the :preserve option. It’s
not the same as rsync by any stretch, but it should be fairly similar
assuming you didn’t need --delete or --include/–exclude patterns. If
you do need those, then the source of FileUtils is probably a pretty
good place to start rolling your own since it properly does symlinks,
permissions, times, and other metadata.

Corey

On 01.09.2007 01:36, Corey J. wrote:

security issues, on the system where this script will be used there
the same as rsync by any stretch, but it should be fairly similar
assuming you didn’t need --delete or --include/–exclude patterns. If
you do need those, then the source of FileUtils is probably a pretty
good place to start rolling your own since it properly does symlinks,
permissions, times, and other metadata.

Other options for unidirectional update on system with GNU utils are:

  • use cp -au
  • maybe you can also facilitate tar

You can build synchronization out of these by invoking then twice, i.e.
once for each direction.

Other than that it should not be too difficult to implement something on
your own using methods in class File.

Kind regards

robert

On 01/09/07, Bob H. [email protected] wrote:

Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.

If you’re dicking around with expect you have the
password in plain text somewhere along the line, so you’d be lucky to
avoid frowns even then :slight_smile:

;Use RSA authentication - a blank passphrase on the key (for cronjobs),
or
use ssh-agent (for everything else - unit testing etc).

On 31-Aug-07, at 4:15 PM, dusty wrote:

I haven’t found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I’d be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :slight_smile:

Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.

Cheers,
Bob

Dusty Doris


Bob H. – tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/

On 2-Sep-07, at 3:44 AM, Dick D. wrote:

On 01/09/07, Bob H. [email protected] wrote:

Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.

If you’re dicking around with expect you have the
password in plain text somewhere along the line, so you’d be lucky to
avoid frowns even then :slight_smile:

I know, funny isn’t it :slight_smile:

;Use RSA authentication - a blank passphrase on the key (for
cronjobs), or
use ssh-agent (for everything else - unit testing etc).

I think I’m going to push a bit on the RSA authentication, see where
I get.

Cheers,
Bob


Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/


Bob H. – tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/

On Sep 1, 11:05 am, Bob H. [email protected] wrote:

the connection (I wound up going though disgusting conniptions using

Bob H. – tumblelog at http://www.recursive.ca/so/
Recursive Design Inc. – weblog athttp://www.recursive.ca/
hutchhttp://www.recursive.ca/ – works onhttp://www.raconteur.info/
cms-for-static-content/home/

Sorry I took so long to reply. Forgot about this thread. I am using
straight rsync. Running an rsync daemon on the servers I want. Then
I setup modules for the access. The reason I do this is to avoid
having ssh keys with blank passwords on my servers. Also, this is
only available on the internal network, so I’ll also use things like
firewall rules and access-list in rsync to make sure only the machines
I want to get rsync access can.

Here’s an example of a class that calls rsync to get a file listing in
a module.


require ‘rubygems’
require ‘open4’

class RsyncUtils

attr_reader :rsync

def initialize(path=’’)
paths = [’/bin’,
‘/usr/bin’,
‘/usr/local/bin’,
‘/opt/local/bin’
]
paths << path unless path.empty?
paths.each() do |path|
if(test(?x, “#{path}/rsync”))
@rsync = “#{path}/rsync”
end
end
raise “Cannot find rsync, please specify the path” unless @rsync
end

def list(hostname,path,recurs=false)
recursive = “–recursive” if recurs
r = run_command("#{@rsync} #{recursive} #{hostname}::#{path}")
if r.exitstatus == 0
r.results = RsyncFile.parse_list(r.stdout)
end
r
end

def list_recursive(hostname,path)
list(hostname,path,true)
end

private
def run_command(command)
pid, stdin, stdout, stderr = Open4.popen4(command)
ignored, status = Process::waitpid2 pid
RsyncResult.new(stdout.readlines,
stderr.readlines,
status.exitstatus)
end

end

Here is a class that hands back my results

class RsyncResult

attr_reader :stdout, :stderr, :exitstatus
attr_accessor :results

def initialize(stdout,stderr,exitstatus)
@stdout = stdout
@stderr = stderr
@exitstatus = exitstatus
end

end

Here is the class that parses the filelisting to tell me about the
files


class RsyncFile

attr_reader :name, :type, :time, :size

FILE_TYPE = 0
DIRECTORY_TYPE = 1
UNKNOWN_TYPE = 2

def initialize(name,type,time,size)
@name = name
@time = time
@size = size
@type = type
end

def is_file?
@type == FILE_TYPE
end

def is_dir?
@type == DIRECTORY_TYPE
end

def self.parse_list(files)
results = []
files.each do |file|
parts = file.split
case parts[0]
when /^[rwx-]{10}$/
type = FILE_TYPE
when /^d[rwx-]{9}$/
type = DIRECTORY_TYPE
else
type = UNKNOWN_TYPE
end
time = Time.local(*ParseDate::parsedate("#{parts[2]}
#{parts[3]}"))
size = parts[1].to_i
name = parts[4]
results << RsyncFile.new(name,type,time,size)
end
results
end
end

Here is how I’d use it in a script.

rsync = RsyncUtils.new
check = rsync.list(@options["–hostname"],@options["–
path"],@options["–recursive"])
if check.exitstatus > 0
raise “#{check.stderr}”
end

#This would list all the files - the rsync output
if @options["–verbose"]
puts check.stdout
exit(0)
end

Now assume I have another class that is do a comparison of the results
for me. That is where I’m calling crit.compare. That’s not as
important, as you can see I’m simply iterating through the results and
checking the Time on the current server vs. the time on the remove
file.

check.results.each do |r|
if r.type == RsyncFile::FILE_TYPE
diff = ((Time.now - r.time) - @options["–drift"].to_i).to_i
critical_files << r.name unless crit.compare(diff)
warning_files << r.name unless warn.compare(diff)
end
end

So, basically I would call the script

./rsync_file_age.rb --hostname somehost --path somedir/anotherdir/ –
warning 0:10 --critical 10:20

That would compile a command to RsyncUtils such as

rsync somehost::somedir/anotherdir/

It would then use open4 to run that command and would parse the
results.

Hope that is helpful.

BTW an rsyncd.conf file might look like this and would provide the
somedir module.

pid file = /var/run/rsyncd.pid
hosts allow = 10.0.0.10

uid = nobody
gid = nogroup
use chroot = no
max connections = 4
syslog facility = local5

[somedir]
path = /var/somedir
read only = true

Paul van Delst wrote:

Hello,

Hi.

I wrote a ruby script to allow me to automatically update a repository
working copy from an unversioned hierarchy.

Is there anyway to use SVN’s import command to get you 80% of
the way there? See, for example,

http://subversion.tigris.org/faq.html#in-place-import

Or, some combination of cp and svn diff? E.g.,

svn co working_copy
cp -r unversioned working_copy
svn diff > patch
[process the patch results]

or perhaps some sort of inverse, e.g.,

U_DIRS=find unversioned -type d
U_FILES=find unversioned -type f
svn co working_copy
cd working_copy
S_DIRS=find . \( -name .svn -prune \) -o -type d -print
S_FILES=find . \( -name .svn -prune \) -o -type f -print
svn rm --force [ S_DIRS - U_DIRS ]
svn rm --force [ S_FILES - U_FILES ]
cp -r unversioned/* .
svn ci -m’To sync with unversioned.’

Regards,

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs