On the total nondisclosure of the 8/9/06 security vulnerabil


#1

Dear Rails team,

The handling of the recent vulnerability in Rails has proven somewhat
problematic for us. We have recently adopted Rails as our web platform
of choice; previously, we used J2EE. We love Rails. We hate J2EE. We
don’t want to go back. It took a lot of effort and convincing to get the
management teams of our various projects to sign off on the use of
Rails. The nondisclosure policy in handling this vulnerability has
seriously jeopardized our (and many other people’s) ability to use Rails
in a commercial environment, so we would like to suggest that it be
changed.

The core issue is that releasing a patch to fix a critical security
vulnerability without telling anyone what the vulnerability is does very
little good as a knowledgeable cracker can just SVN diff the new version
with the old one and peruse that patch to have an exploit ready to go.
(The 1.1.4 to 1.1.5 patch is under 2,000 lines, not even that long.) It
only takes one person to do that and release it onto the net. Then,
thousands of script kiddies will have the exploit within minutes. I am
sure that, even as I type this, the 0-day exploit rooms on IRC are
buzzing with prepackaged copies of exploit code for the Rails bug. Not
disclosing the vulnerability does very little to keep the information
out of the hands of malicious hackers; it only harms legitimate users.

As a system administrator, how can I know whether my systems have
already been compromised when I don’t even know the general nature of
the vulnerability? I’m not suggesting that specific exploit code ought
to be posted, but we do need to know, at least generally, what is going
on in order to be able to effectively deal with it.

As a business, if I don’t have any clue what the problem is, I can’t
reassure my customers that their data has been safe, because I don’t
know. I have to spend resources to diff the new code with the old code
and find out exactly what the problem was, needlessly duplicating work
that dozens or hundreds of other people and companies (and crackers) are
simultaneously performing around the world, because of the decision to
withhold that information. Only then can I check to see whether the
exploit has been used on my system, potentially hours later.

Further, though the authors promise that the exploit is fixed in 1.1.5,
how can I be sure of that if I don’t even know what the exploit was?
Nobody is perfect. Everyone makes mistakes from time to time. If, by
some chance, there was a mistake in the fix in 1.1.5 (or perhaps the fix
opened up some other problem), it becomes that much harder for the
legitimate user community to catch and solve the problem. All bugs are
shallow to many eyes. But nobody will even discuss the original exploit
(except crackers amongst themselves), so how can we possibly audit the
fix? Even if, several days later, the details of the original exploit
are released, that’s several days that our public, Internet-facing
systems were sitting vulnerable to problems that the crackers surely
knew about; the problem could have been fixed in minutes if the whole
Rails developer community was looking at it.

Sun and the big J2EE app server vendors send out glossy brochures and
sales teams to wine and dine their customers. The glossy brochures and
the multibillion dollar brand name stamped onto the product give
managers a warm, fuzzy feeling. It makes them feel safe, because
(theoretically) they have someone to sue if things go wrong. Rails
doesn’t have that. Rails is produced by a decentralized team of people
on the Internet. While we, as technical people, recognize the technical
benefits of Rails over something like J2EE, it took a whole lot of
convincing to get the non-technical management people to forget about
the Sun cheerleaders and sign off on Rails for their expensive new
projects.

If our customers do not feel that their data is secure on our hosts,
they will not host with us. Since that is how we make our livings, this
becomes a big problem for us. The melodramatic “the sky is falling,
UPGRADE NOW NOW NOW” tone of the 1.1.5 announcement did not help,
either. If a customer reads this announcement, the customer will call us
up to ask what is going on. What are we supposed to tell him – “Uh… I
don’t know, the actual problem is secret, so we can’t really tell you
whether your data has been compromised, but some guys on the Internet
assure us that the issue is fixed now.”?? We would be back on J2EE
within the week if we do that, if we still had customers at all.

I therefore propose that full (or at least partial) disclosure be
adopted as standard policy for future Rails vulnerabilities. Doing so
will not give the cracking community any information that they wouldn’t
have in 30 minutes or an hour anyway – information that the legitimate
user community is denied for days, thus granting an advantage to the
malicious crackers. What it will do is allow Rails to exist as a
competitive, viable platform for commercial web development, and it will
allow system administrators to maintain the integrity of their systems
and their data.

I close with a quote I found at
http://en.wikipedia.org/wiki/Full_disclosure :

 A commercial, and in some respects a social doubt has been started

within the last year or two, whether it is right to discuss so openly
the security or insecurity of locks. Many well-meaning persons suppose
that the discussion respecting the means for baffling the supposed
safety of locks offers a premium for dishonesty, by showing others how
to be dishonest. This is a fallacy. Rogues are very keen in their
profession, and know already much more than we can teach them respecting
their several kinds of roguery.

 Rogues knew a good deal about lock-picking long before locksmiths

discussed it among themselves, as they have lately done. If a lock, let
it have been made in whatever country, or by whatever maker, is not so
inviolable as it has hitherto been deemed to be, surely it is to the
interest of honest persons to know this fact, because the dishonest are
tolerably certain to apply the knowledge practically; and the spread of
the knowledge is necessary to give fair play to those who might suffer
by ignorance.

     -- From A. C. Hobbs (Charles Tomlinson, ed.), Locks and Safes:

The Construction of Locks. Published by Virtue & Co., London, 1853
(revised 1868).


– Paul L., Senior Software Engineer –
— Networked Knowledge Systems —
---- P.O. Box 20772 Tampa, FL. 33622-0772 ----
----- (813)594-0064 Voice (813)594-0045 FAX -----
------ removed_email_address@domain.invalid ------


----- This email bound by the following: -----
---- http://www.nks.net/email_disclaimer.html ----


#2

Paul L. wrote:

Dear Rails team,

The handling of the recent vulnerability in Rails has proven somewhat
problematic for us. We have recently adopted Rails as our web platform
of choice; previously, we used J2EE. We love Rails. We hate J2EE. We
don’t want to go back. It took a lot of effort and convincing to get the

You make some good points and have valid concerns. However, the fact
remains that Rails is a very new framework and prone to security issues.
The developers aren’t perfect like any of us. Lets give credit though
on how fast they fixed the issue. I’m sure they dropped what they were
doing to get a patch out and I applaud them for that.


#3

We know absolutely nothing about how the vulnerability came to the
attention of the Rails team. Looking back at the recent history of
discovered vulnerabilities in closed-source ommercial products, roughly
the following has been the case:

“White hats” (whatever their motivations) often discover vulnerabilities
and bring them quietly to the attention of the affected vendors. At this
point several responses by the vendor are possible (and all of them have
been observed): 1) they can patch the problem, release the patch, and
then come fully clean (including exploit details) once the patch is
available; 2) they can quietly release a patch and hope no one catches
on that there was a problem, or 3) they can decide to do nothing and
hope (or pressure) the exploit discoverers to keep quiet while they
continue to “work on it.” In any case, closed-source processes are such
that it can take days or weeks to produce a patch.

The incredible but true fact is that companies like Microsoft, Sun,
Cisco, and others that often face this situation don’t face a lot of
real risk from not patching their problems. The opprobrium that
Microsoft right gets for their less-than-strict approach to security
vulnerabilities doesn’t keep enough people from buying their products
for them to apply more than Chairman and CEO-level lip service to the
problem.

What has happened several times is that the reporters of a problem felt
strongly enough about an unpatched vulnerability that they went public
with the information. This decidedly antisocial behavior does have the
effect of forcing the vendor’s hand, but it also makes everyone scared
as heck for the nontrivial amount of time it takes for a vendor to
release a patch.

But with open source software, we expect well-audited patches to new
vulnerabilities to be released within hours of discovery. So the pattern
of keeping quiet until the patch is available is not necessarily a bad
one because the time is so short. But it’s essential to give some
indication of the nature of the problem in the initial advisory!
Otherwise no one has any way to judge their vulnerability. Applying a
patch to a large number of servers is not something that can be done
instantaneously, especially since it completely invalidates all your
unit and regression testing. That means there is a significant-sized
window of time during which a sysadmin must be able to make an
intelligent decision whether to shut down or not.

We knew Rails is a relatively immature technology. What we have just
discovered is that the Rails team is a relatively immature team. They
will learn from this and get better, and/or they will partner with
people who know better. We know they tried to reduce the level of
problems by inhibiting the flow of information to the bad guys. But as
many have pointed out, that doesn’t slow the bad guys down at all,
especially when the source is open. If anything, the team should have
given a description of the vulnerability and postponed full technical
details for 24 hours to give people time to patch.


#4

Hm, there seem to be already proof-of-concept exploit
(found on ror2ru group:
http://groups.google.com/group/ror2ru/browse_thread/thread/e654a6ddedc29e7e/7b90204e50bd7974
)
Short summary of claims:
vulnerability is related to very-very automagic in routing code.

E.g. for Rails 1.1.4 url like http://127.0.0.1:3000/breakpoint_client
can easily take down your server to ever-waiting state (my app has gone
there, when I tried).

And for Rails 1.1.5 (you say, it is fixed?) http://127.0.0.1:3000/cgi
brings down routing totally.

Are these the only affected by “security patch” places, or we should
expect more unknown, so-called “fixed” bugs???

Damn!
I just don’t get for what reason DHH didn’t explain vulnerability (uhm,
it would be an interesting story on how it was found, analyzed and
resolved).

It was a question of several hours to find and publish exploit, so who
is really protected by that “opionated obscurity”?
Where is test code for that “patch”?
And there was already a ticket “#5408 Unhandled urls can cause loading
of arbitrary ruby files” on Rails TRAC from 06/16 about mentioned
issues…

And, maybe I’m wrong and that is not that security breach “patched” in
1.1.5? How can I know? David Heinemeier H., please, tell us,
obscurity is not a way to go.


#5

dseverin wrote:

And for Rails 1.1.5 (you say, it is fixed?) http://127.0.0.1:3000/cgi
brings down routing totally.

One more for 1.1.5:
Two subsequent calls:

http://127.0.0.1:3000/builder/blankslate
http://127.0.0.1:3000/active_support/dependencies

put server to errors “SystemStackError (stack level too deep)”
constantly for all further requests.

Nope, guys, the routing problems aren’t fully fixed, and one still can
require about 500 .rb files from standard Rails vendor/* directories
just typing some text as URL in browser.

I don’t know yet how to fight it… :frowning:
What configuration could anyone suggest to really secure my
applications?


#6

On winxp using webrick, things don’t look good… Still breakable…

  1. http://127.0.0.1:3000/builder/blankslate
    Routing Error
    Recognition failed for “/builder/blankslate”

  2. http://127.0.0.1:3000/active_support/dependencies
    SystemStackError
    stack level too deep
    RAILS_ROOT: ./script/…/config/…

Application Trace | Framework Trace | Full Trace
F:/Programs/ruby/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/vendor/builder/blankslate.rb:48:in
blank_slate_method_added' F:/Programs/ruby/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/vendor/builder/blankslate.rb:48:inblank_slate_method_added’
F:/Programs/ruby/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/vendor/builder/blankslate.rb:48:in
method_added' F:/Programs/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.4/lib/action_controller/routing.rb:688:indefine_hash_access_method’
F:/Programs/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.4/lib/action_controller/routing.rb:694:in
name_route' F:/Programs/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.4/lib/action_controller/routing.rb:650:innamed_route’
F:/Programs/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.4/lib/action_controller/routing.rb:655:in
`method_missing’
#{RAILS_ROOT}/config/routes.rb:16
#{RAILS_ROOT}/config/routes.rb:1
-e:4

  1. Any url:
    SystemStackError
    stack level too deep
    RAILS_ROOT: ./script/…/config/…

Application Trace | Framework Trace | Full Trace
F:/Programs/ruby/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/vendor/builder/blankslate.rb:48:in
blank_slate_method_added' F:/Programs/ruby/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/vendor/builder/blankslate.rb:48:inblank_slate_method_added’
F:/Programs/ruby/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/vendor/builder/blankslate.rb:48:in
method_added' F:/Programs/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.4/lib/action_controller/routing.rb:688:indefine_hash_access_method’
F:/Programs/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.4/lib/action_controller/routing.rb:694:in
name_route' F:/Programs/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.4/lib/action_controller/routing.rb:650:innamed_route’
F:/Programs/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.4/lib/action_controller/routing.rb:655:in
`method_missing’
#{RAILS_ROOT}/config/routes.rb:16
#{RAILS_ROOT}/config/routes.rb:1
-e:4


#7

The rails team is trying to contain damage. The fix works on everything
except for webrick. Give them some time. No one can be in production
with
webrick…

colin wrote:

On winxp using webrick, things don’t look good… Still breakable…


#8

dseverin wrote:

dseverin wrote:

And for Rails 1.1.5 (you say, it is fixed?) http://127.0.0.1:3000/cgi
brings down routing totally.

One more for 1.1.5:
Two subsequent calls:

http://127.0.0.1:3000/builder/blankslate
http://127.0.0.1:3000/active_support/dependencies

put server to errors “SystemStackError (stack level too deep)”
constantly for all further requests.

I’m running 1.1.5 and using those links I don’t get any crashes. I do
get routing errors, but no crashes.

Routing Error
Recognition failed for “/builder/blankslate”

Routing Error
Recognition failed for “/active_support/dependencies”


#9

John R. wrote:

dseverin wrote:

dseverin wrote:

And for Rails 1.1.5 (you say, it is fixed?) http://127.0.0.1:3000/cgi
brings down routing totally.

One more for 1.1.5:
Two subsequent calls:

http://127.0.0.1:3000/builder/blankslate
http://127.0.0.1:3000/active_support/dependencies

put server to errors “SystemStackError (stack level too deep)”
constantly for all further requests.

I’m running 1.1.5 and using those links I don’t get any crashes. I do
get routing errors, but no crashes.

Routing Error
Recognition failed for “/builder/blankslate”

Routing Error
Recognition failed for “/active_support/dependencies”

Same here. I’ve just tested a local webrick though, maybe a different
config in production mode will act differently…

Jeroen


#10

s1lence wrote:

The rails team is trying to contain damage. The fix works on everything
except for webrick. Give them some time. No one can be in production
with
webrick…

I have no crashes using these techniques under WebBrick on WinXP.


#11

Running with webrick in development mode both /cgi and those 2 requests
generate errors on all subsequent requests.

for the .cgi one:

…/ruby/lib/ruby/1.8/cgi.rb:773

action_controller/routing.rb:289:in `attempt_load’

routing.rb:
safe_load_paths.each do |load_path|
full_path = File.join(load_path, path)
file_path = full_path + ‘.rb’
if File.file?(file_path) # Found a .rb file? Load it up
require_dependency(file_path)

printing ‘safe_load_paths’ in attempt_load gives me all the dirs from my
app, rails, and some ruby library dirs(!). Also the path variable is
‘cgi’.
So it just traverses all lib directories and loads the first file names
cgi.rb, even if it’s in a ruby or rails library dir.

with safe_load_paths being defined as:

def safe_load_paths #:nodoc:
if defined?(RAILS_ROOT)
$LOAD_PATH.select do |base|
base = File.expand_path(base)
extended_root = File.expand_path(RAILS_ROOT)
base.match(/\A#{Regexp.escape(extended_root)}/*#{file_kinds(:lib)

  • ‘|’}/) || base =~ %r{rails-[\d.]+/builtin}
    end

where File.expand_path(RAILS_ROOT) is my application dir.

This:
base.match(/\A#{Regexp.escape(extended_root)}/*#{file_kinds(:lib) *
‘|’}/)
seems to match too much, some debugging output shows that
file_kinds(:lib) * ‘|’ returns ‘app|lib’

so this is base.match(/\A/path/to/your_app//*app|lib/)
with no parenthesis around app|lib !
So any dir matching ‘lib’ is included.

Fix:
actionpack-1.12.4\lib\action_controller\routing.rb: 276
CHANGE
base.match(/\A#{Regexp.escape(extended_root)}/#{file_kinds(:lib) *
‘|’}/) || base =~ %r{rails-[\d.]+/builtin}
TO
base.match(/\A#{Regexp.escape(extended_root)}/
(?:#{file_kinds(:lib) *
‘|’})/) || base =~ %r{rails-[\d.]+/builtin}

Vulnerability fixed :slight_smile:

…and I haven’t even finished my first rails app yet :wink:


#12

Paul L. wrote:

little good as a knowledgeable cracker can just SVN diff the new version
with the old one and peruse that patch to have an exploit ready to go.

Then, Paul L. wrote:

As a system administrator, how can I know whether my systems have
already been compromised when I don’t even know the general nature of
the vulnerability? I’m not suggesting that specific exploit code ought

Uhhh, am I missing something? You describe exactly how to figure out
what the problem and its solution are and then you say you have no way
to figure it out. All software has bugs, including security issues. Deal
with it.


#13

dseverin wrote:

E.g. for Rails 1.1.4 url like http://127.0.0.1:3000/breakpoint_client
can easily take down your server to ever-waiting state (my app has gone
there, when I tried).

This url takes down my server, running 1.1.5 and mongrel


#14

I don’t know yet how to fight it… :frowning:
What configuration could anyone suggest to really secure my
applications?

I already had a line of code at the end of my routes to catch the entire
URL of an umatched route and dump it into an array for further
processing. Whenever I try those links (running Lighttpd locally with
FastCGI) I get routing errors but no stack problems.

The line at the end of my routes.rb is:

map.with_options(:controller => ‘public/pages’) do |public|

public.connect ‘*url’, :action => ‘index’
end


#15

David M. wrote:

dseverin wrote:

E.g. for Rails 1.1.4 url like http://127.0.0.1:3000/breakpoint_client
can easily take down your server to ever-waiting state (my app has gone
there, when I tried).

This url takes down my server, running 1.1.5 and mongrel

Problem also exists with Webrick, and IIRC, the default config on my dev
machine is lighttpd. So this is a problem inside the Rails code.

Is that the same security flaw, or another one entirely?


#16

Daniel H. wrote:

Is that the same security flaw, or another one entirely?

Looks like a new one inadvertently introduced as part of 1.1.5. This
patch to routing.rb should fix (and explain why):

— routing.rb.orig 2006-08-10 12:20:12.830325000 -0500
+++ routing.rb 2006-08-10 12:20:26.043147000 -0500
@@ -273,7 +273,7 @@
$LOAD_PATH.select do |base|
base = File.expand_path(base)
extended_root = File.expand_path(RAILS_ROOT)

base.match(/\A#{Regexp.escape(extended_root)}/#{file_kinds(:lib) *
‘|’}/) || base =~ %r{rails-[\d.]+/builtin}
+
base.match(/\A#{Regexp.escape(extended_root)}/
(#{file_kinds(:lib) *
‘|’})/) || base =~ %r{rails-[\d.]+/builtin}
end
else
$LOAD_PATH


#17

Nick wrote:

Daniel H. wrote:

Is that the same security flaw, or another one entirely?

Looks like a new one inadvertently introduced as part of 1.1.5. This
patch to routing.rb should fix (and explain why):

Oy, that’s what you get with security through obscurity. Another bug.

I managed to get around the breakpoint_client hanging the entire machine
by adding it at the top of routes.rb, which is hackish - and I don’t
know what else is there that could be a vector of attack.

sigh :frowning:


#18

dseverin wrote:

I don’t know yet how to fight it… :frowning:
What configuration could anyone suggest to really secure my
applications?

You could always setup a catch-all as your last route:
map.connect ‘*url’, :controller => ‘my_controller’,
:action=>“unhandled”
and while you’re at it, implement a decent 404-handler that logs the
hits and
offers “smart” alternatives instead of a dummy 404 page :wink:

Daniel


#19

Mongrel on Windos using 1.1.5 still causes the error to occur for us as
well.

http://127.0.0.1:3000/active_support/dependencies


#20

Nick wrote:

I think we might want to modify that patch further. I don’t see why
there’s a /* separating the RAILS_ROOT from the (app|lib) portion. It
should be /+ right?

Modified patch below.

— routing.rb.orig 2006-08-10 12:20:12.830325000 -0500
+++ routing.rb 2006-08-10 12:20:26.043147000 -0500
@@ -273,7 +273,7 @@
$LOAD_PATH.select do |base|
base = File.expand_path(base)
extended_root = File.expand_path(RAILS_ROOT)

base.match(/\A#{Regexp.escape(extended_root)}/*#{file_kinds(:lib) *
‘|’}/) || base =~ %r{rails-[\d.]+/builtin}
+
base.match(/\A#{Regexp.escape(extended_root)}/+(#{file_kinds(:lib) *
‘|’})/) || base =~ %r{rails-[\d.]+/builtin}
end
else
$LOAD_PATH