Net::SSH AuthenticationFailed on some hosts but 'ssh' works

Rails Deployment
1

I have a custom capistrano recipe that runs commands on a couple of
remote hosts, using publickey ssh, and has been working for several
months. Recently (in the last week or so), something changed I have not
been able to identify, and the Net::SSH.start() method raise
AuthenticationFailed for only one of the remote hosts. The capistrano
recipe has not changed, the net-ssh gem is 2.0.8 and was updated long
ago. I can get the same error using either capistrano, or direct
Net::SSH.start within IRB.

I use ubuntu 8.04 on all computers involved, and follow regular security
updates, but have not been able to track which one might have caused
this, and why only one remote machine is affected.

Also, normal command-line ‘ssh’ works correctly every time for all
hosts.

To try and track this, I’ve done a series of tests connecting to two
remote machines, one working and one not, and comparing the differences,
both with command-line ssh and with Net::SSH with high debug levels set.
Most differences found were expected, like different host keys, but the
sequence of events was always the same, with one exception, the debug
output from Net::SSH showed a sequence of messages and responses, and
message 5 differed. Here follows a more explicit description of the
test:

Given that ‘ssh -l userx goodhost’ and ‘ssh -l userx badhost’ both work
correctly and identically with publickey authentication, we get the
following results in IRB (edited for clarity):

Net::SSH.start(‘goodhost’,‘userx’,{:verbose => Logger::DEBUG})
…
trying publickey
queueing packet nr 5 type 50 len 508
received packet nr 5 type 60 len 460
queueing packet nr 6 type 50 len 556
received packet nr 6 type 52 len 12
publickey succeeded

Net::SSH.start(‘badhost’,‘userx’,{:verbose => Logger::DEBUG})
…
trying publickey
queueing packet nr 5 type 50 len 508
received packet nr 5 type 51 len 44
all authorization methods failed

So, it seems packet 50 should receive 60 in response, but gets 51
instead. I have no idea what these numbers mean, and why different
responses are received by Net::SSH, when the command-line ssh works
fine.

As said before, this is a problem that has suddenly happened, with no
obvious change to the computer configurations, and all computers are
identically configured (with regards to ssh, ruby and Net::SSH).

Does anyone have any ideas, or further suggestions on where to look?

P.S. Things I’ve already tried that have not helped:

  • removing ssh gateway from my configs
  • downgrading net-ssh gem all the way back to 2.0.1
  • run command-line ssh with maximum verbocity to find differences
2

Craig Taverner wrote:

I have a custom capistrano recipe that runs commands on a couple of
remote hosts, using publickey ssh, and has been working for several
months. Recently (in the last week or so), something changed I have not
been able to identify, and the Net::SSH.start() method raise
AuthenticationFailed for only one of the remote hosts. The capistrano
recipe has not changed, the net-ssh gem is 2.0.8 and was updated long
ago. I can get the same error using either capistrano, or direct
Net::SSH.start within IRB.

I use ubuntu 8.04 on all computers involved, and follow regular security
updates, but have not been able to track which one might have caused
this, and why only one remote machine is affected.

Also, normal command-line ‘ssh’ works correctly every time for all
hosts.

To try and track this, I’ve done a series of tests connecting to two
remote machines, one working and one not, and comparing the differences,
both with command-line ssh and with Net::SSH with high debug levels set.
Most differences found were expected, like different host keys, but the
sequence of events was always the same, with one exception, the debug
output from Net::SSH showed a sequence of messages and responses, and
message 5 differed. Here follows a more explicit description of the
test:

Given that ‘ssh -l userx goodhost’ and ‘ssh -l userx badhost’ both work
correctly and identically with publickey authentication, we get the
following results in IRB (edited for clarity):

Net::SSH.start(‘goodhost’,‘userx’,{:verbose => Logger::DEBUG})
…
trying publickey
queueing packet nr 5 type 50 len 508
received packet nr 5 type 60 len 460
queueing packet nr 6 type 50 len 556
received packet nr 6 type 52 len 12
publickey succeeded

Net::SSH.start(‘badhost’,‘userx’,{:verbose => Logger::DEBUG})
…
trying publickey
queueing packet nr 5 type 50 len 508
received packet nr 5 type 51 len 44
all authorization methods failed

So, it seems packet 50 should receive 60 in response, but gets 51
instead. I have no idea what these numbers mean, and why different
responses are received by Net::SSH, when the command-line ssh works
fine.

As said before, this is a problem that has suddenly happened, with no
obvious change to the computer configurations, and all computers are
identically configured (with regards to ssh, ruby and Net::SSH).

Does anyone have any ideas, or further suggestions on where to look?

P.S. Things I’ve already tried that have not helped:

  • removing ssh gateway from my configs
  • downgrading net-ssh gem all the way back to 2.0.1
  • run command-line ssh with maximum verbocity to find differences

I’m having similar problems with Capistrano. I’ve setup a second server
on Slicehost using the exact same process (so I thought) as the first.
Cap deploy works fine for the first, but for the second prompts for my
password. My Capistrano recipe is the same for both, but only the new
setup is requiring a password, the first simply connects through my ssh
key. If I do enter the password, it fails every time with the response:

connection failed for: MY_SLICE_IP_ADDRESS
(Net::SSH::AuthenticationFailed: MY_SLICE_USERNAME)

SSH through the command line works just fine for both using ssh keys and
no password provided.

I’m not very familiar with Net::SSH but ran the two commands you
provided above and in both cases the public key was successfully
authorized.

So I guess I’m pretty stumped as well.

Thanks,
Ted

3

I was experiencing similar troubles. I was also able to ssh in fine
using openssh, but Net::SSH (2.0.15) was always returning auth failed.

Here’s a script that was failing:

require ‘rubygems’
require ‘net/ssh’
require ‘net/sftp’
Net::SSH.start(‘myserver’, ‘myuser’, { :verbose => Logger::DEBUG }) do
|ssh|
end

And it was always failing with this error:
E, [2009-10-22T22:23:42.401052 #5381] ERROR –
net.ssh.authentication.session[3f9e1f693000]: all authorization methods
failed (tried hostbased)

However, if I specify the allowed method explicitly, it works:

require ‘rubygems’
require ‘net/ssh’
require ‘net/sftp’
Net::SSH.start(‘myserver’, ‘myuser’, { :verbose => Logger::DEBUG,
:auth_methods => %w{ publickey } }) do |ssh|
end

D, [2009-10-22T22:24:35.399982 #5388] DEBUG –
net.ssh.authentication.methods.publickey[3fa5b5a21044]: publickey
succeeded (86:90:0d:ad:70:44:cd:29:61:24:66:4d:9a:d3:29:0a)

So for some reason, Net::SSH isn’t realizing that the public key is an
option.