Calling scp from ruby

rudy · September 7, 2006, 2:30am

I have a problem using “scp” from a ruby scrip. It copies my sql file to
the destination but it cuts the file the middle of line 290.
When i run scp manually i the copied file is normal.

I am using `` to call the unix command

Dir.chdir(sql_folder)
scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}/#{document_id}

Do you have any idea what i am doing wrong?

rudy · September 7, 2006, 5:09am

What I’m using when scping is: scp -r dirToCopy destination ( -r for
recursive copy folder)

but, I believe that when dirToCopy or destination is remote, you should
write:

USER@IP:/FOLDER/FILE

in your lines I can’t see the @ sign

Could it be like:
scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}/#{document_id}
scp -r #{sql_folder} #{remoteUSER}@#{remotesIP}:/#{document_id}

hope it helps,

Jean

rudy · September 7, 2006, 8:51am

On 9/7/06, Jean V. [email protected] wrote:

scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}/#{document_id}
scp -r #{sql_folder} #{remoteUSER}@#{remotesIP}:/#{document_id}

(The OP seems to know the proper syntax, as he is able to do it by
hand.)

(I have no clue what the cause is, these are just hint what to try)

try another file (to see whether it is dependent on some pattern in
the file, or whether it transfers the same size all the time)
try removing few line around that line
try using system instead of ``
look at the logs on the remote side

rudy · September 7, 2006, 9:05pm

Seid Rudy wrote:

I have a problem using “scp” from a ruby scrip. It copies my sql file to
the destination but it cuts the file the middle of line 290.
When i run scp manually i the copied file is normal.

I am using `` to call the unix command

Dir.chdir(sql_folder)
scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}/#{document_id}

Do you have any idea what i am doing wrong?

I want to emphasize this is just an off-the-wall guess. The backticked code is being forked, the parent process terminates prematurely, and
the
forked process is killed, also prematurely.

Have you tried “system” instead of back-ticks? “system” returns a
process
completion code, therefore it must wait until the process completes.

rudy · September 7, 2006, 11:14am

On 9/7/06, Jan S. [email protected] wrote:

Could it be like:
scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}/#{document_id}
scp -r #{sql_folder} #{remoteUSER}@#{remotesIP}:/#{document_id}

scp file #{remotemachine}:

is fine, it copies file to remotemachine to the home of the user with
the same name (or as preconfigured in .ssh/config - you can create
shortcuts for remotemachine and configure what the remote username
should be). You can add /remotefolder/remotefile after the colon to
change the remote location.

(The OP seems to know the proper syntax, as he is able to do it by hand.)

(I have no clue what the cause is, these are just hint what to try)

try another file (to see whether it is dependent on some pattern in
the file, or whether it transfers the same size all the time)

try removing few line around that line

try using system instead of ``

look at the logs on the remote side

Try the -v switch to scp. Perhaps it will break even more but perhaps
it would do the same and show useful information.

Thanks

Michal

rudy · September 7, 2006, 9:41pm

Quoting Paul L. [email protected]:

Why won’t you use Net::SFTP instead of calling the external scp?

rudy · September 7, 2006, 10:32pm

Pau Garcia i Quiles wrote:

Why won’t you use Net::SFTP instead of calling the external scp?

I can guess one reason. Net::SFTP doesn’t work with key-based
authentication, only passwords, and there is no password given in OP’s
code…

I hope that I am wrong and that this has changed recently.

rudy · September 7, 2006, 11:10pm

Pau Garcia i Quiles wrote:

Why won’t you use Net::SFTP instead of calling the external scp?

I have that question too. Do that do SSH format?

rudy · September 8, 2006, 8:33pm

Michal S. wrote:

Try the -v switch to scp. Perhaps it will break even more but perhaps
it would do the same and show useful information.

Thanks

Michal

hi i tried -v option and found out that scp is getting chopped @ byte
40960. Any clue how i can change the byte limit?

debug1: Sending command: scp -v -r -t
/var/local/rudyserv/wilcox/hr/_sql/
Entering directory: D0755 0 069999
Sending file modes: C0644 40960 069999.sql
Sending file modes: C0644 0 069999.err
Sending file modes: C0644 0 069999.tst
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.1 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 0

rudy@localhost $ ls -al …/sql/069999/069999.sql
-rw-r–r-- 1 seid users 41455 Sep 9 03:18 …/sql/069999/069999.sql

rudy · September 8, 2006, 8:46pm

On Sat, 9 Sep 2006, Seid Rudy wrote:

hi i tried -v option and found out that scp is getting chopped @ byte
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.1 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 0

rudy@localhost $ ls -al …/sql/069999/069999.sql
-rw-r–r-- 1 seid users 41455 Sep 9 03:18 …/sql/069999/069999.sql

i spawn scp all the time from ruby, with gigantic files of tens of gigs
and
have never seen this. i think this maybe either an os issue or your
disk is
failing (though didn’t you mention it worked by hand?).

one other possiblity is some pipe filling issue. by using backticks you
are
relying on ruby to read stdout as fast as scp produces it. i can’t
imagine
why there would be an issue there, but i’d try your command using
something
like

system “scp #{ src } #{ dst } >#{ log } 2>&1”

are you on windows by any chance?

-a

rudy · September 8, 2006, 9:59pm

unknown wrote:

On Sat, 9 Sep 2006, Seid Rudy wrote:

system “scp #{ src } #{ dst } >#{ log } 2>&1”

are you on windows by any chance?

no, unix

I did that and i got an empty log.

I also rerun the script without deleting the sql file so it appended to
the file like it should. When i looked at the destination file i noticed
the file stopped at the same spot. But when i look up further to the top
of the destinatio file i noticed it had the a complete content of the
sql file followed by an incomplete content of the sql file. Ofcourse the
source has content of two complete sql files appended in one.

rudy · September 9, 2006, 12:49am

Seid Rudy wrote:

unknown wrote:

On Sat, 9 Sep 2006, Seid Rudy wrote:

system “scp #{ src } #{ dst } >#{ log } 2>&1”

are you on windows by any chance?

no, unix

I did that and i got an empty log.

I also rerun the script without deleting the sql file so it appended to
the file like it should. When i looked at the destination file i noticed
the file stopped at the same spot. But when i look up further to the top
of the destinatio file i noticed it had the a complete content of the
sql file followed by an incomplete content of the sql file. Ofcourse the
source has content of two complete sql files appended in one.

I DON’T THINK IT’S scp ISSUE.
i get the same problem when i do this

system(“cat #{sqloutput}/#{date_folder}.sql
#{sqloutput}/#{date_folder}.jnk >#{sqloutput}/#{date_folder}.out”)

rudy · September 9, 2006, 1:17am

On Sat, 9 Sep 2006, Seid Rudy wrote:

I did that and i got an empty log.

system(“cat #{sqloutput}/#{date_folder}.sql
#{sqloutput}/#{date_folder}.jnk >#{sqloutput}/#{date_folder}.out”)

hmmm. are you sure you don’t have disk issues?

-a

rudy · September 11, 2006, 2:03am

Pau Garcia i Quiles wrote:

On Thursday 07 September 2006 22:29, Joel VanderWerf wrote:

Pau Garcia i Quiles wrote:

Why won’t you use Net::SFTP instead of calling the external scp?
I can guess one reason. Net::SFTP doesn’t work with key-based
authentication, only passwords, and there is no password given in OP’s
code…

I hope that I am wrong and that this has changed recently.

Actually, it works with key-based authentication:

Hey, it does work! Thanks.

rudy · September 10, 2006, 6:39pm

On Thursday 07 September 2006 22:29, Joel VanderWerf wrote:

Pau Garcia i Quiles wrote:

Why won’t you use Net::SFTP instead of calling the external scp?

I can guess one reason. Net::SFTP doesn’t work with key-based
authentication, only passwords, and there is no password given in OP’s
code…

I hope that I am wrong and that this has changed recently.

Actually, it works with key-based authentication:

require ‘net/ssh’
require ‘net/sftp’

Net::SSH.start( ‘192.168.1.37’,
:username=>‘pgquiles’
) do |session|

    session.sftp.connect do |sftp|
            sftp.put_file("backup.tar.bz2",

“/media/hda3/backup.tar.bz2”)
end
end

rudy · September 14, 2006, 6:55pm

Seid Rudy wrote:

Pau Garcia i Quiles wrote:

On Thursday 07 September 2006 22:29, Joel VanderWerf wrote:

Pau Garcia i Quiles wrote:

Why won’t you use Net::SFTP instead of calling the external scp?

I can guess one reason. Net::SFTP doesn’t work with key-based
authentication, only passwords, and there is no password given in OP’s
code…

I hope that I am wrong and that this has changed recently.

Actually, it works with key-based authentication:

---- code -------------
require ‘rubygems’

require_gem ‘net-ssh’
require_gem ‘net-sftp’

Net::SSH.start( ‘host’,‘user’,‘password’) do |session|
session.sftp.connect do |sftp|
sftp.put_file("#{log}", “#{remotesqlfolder}/#{date_folder}.log”)
end
end
---- end of code ------

— error -------------

Even though it gave me bunch of warnings it copied the file. BUT i still
have the same problem and that is it chops from the bottom of the
destination file (N) bytes.

i have tried
scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}/
system(“scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}/”)
system “scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}”
and the aboth
All have the same problem @ the same spot of the file.

Does anybody got any idea i can try?

rudy · September 13, 2006, 6:49pm

Pau Garcia i Quiles wrote:

On Thursday 07 September 2006 22:29, Joel VanderWerf wrote:

Pau Garcia i Quiles wrote:

Why won’t you use Net::SFTP instead of calling the external scp?

I can guess one reason. Net::SFTP doesn’t work with key-based
authentication, only passwords, and there is no password given in OP’s
code…

I hope that I am wrong and that this has changed recently.

Actually, it works with key-based authentication:

require ‘net/ssh’
require ‘net/sftp’

Net::SSH.start( ‘192.168.1.37’,
:username=>‘pgquiles’
) do |session|
    session.sftp.connect do |sftp|
            sftp.put_file("backup.tar.bz2", 
“/media/hda3/backup.tar.bz2”)
end
end

I get this error, not sure why?

---- code -------------
require ‘rubygems’

require_gem ‘net-ssh’
require_gem ‘net-sftp’

Net::SSH.start( ‘host’,‘user’,‘password’) do |session|
session.sftp.connect do |sftp|
sftp.put_file("#{log}", “#{remotesqlfolder}/#{date_folder}.log”)
end
end
---- end of code ------

— error -------------
/usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/userauth/services.rb:129:
warning: *' interpreted as argument prefix /usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/connection/driver.rb:74: warning:&’ interpreted as argument prefix
/usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/service/process/open.rb:130:
warning: &' interpreted as argument prefix /usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/service/process/open.rb:131: warning:&’ interpreted as argument prefix
/usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/service/process/open.rb:139:
warning: &' interpreted as argument prefix /usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/service/process/open.rb:140: warning:&’ interpreted as argument prefix
/usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/service/process/open.rb:141:
warning: &' interpreted as argument prefix /usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/service/process/open.rb:142: warning:&’ interpreted as argument prefix
/usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/service/process/popen3.rb:143:
warning: &' interpreted as argument prefix /usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/service/process/popen3.rb:162: warning:&’ interpreted as argument prefix
/usr/lib/ruby/gems/1.8/gems/net-sftp-1.1.0/lib/net/sftp/protocol/driver.rb:105:
warning: &' interpreted as argument prefix /usr/lib/ruby/gems/1.8/gems/net-sftp-1.1.0/lib/net/sftp/protocol/driver.rb:106: warning:&’ interpreted as argument prefix
/usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/connection/driver.rb:378:
warning: instance variable @processing_data_requests not initialized
/usr/lib/ruby/gems/1.8/gems/net-ssh-1.0.10/lib/net/ssh/connection/channel.rb:495:
warning: instance variable @on_window_adjust not initialized
/usr/lib/ruby/gems/1.8/gems/net-sftp-1.1.0/lib/net/sftp/operations/abstract.rb:64:
warning: `*’ interpreted as argument prefix
— end of error ------