Security question

I know there isn’t a definite answer to this but I thought I’d throw it
out
here anyway.

I’m running FreeBsd 7 with nginx. I have a total of 3 daemons. sshd ,
ftpd
and nginx. I’ve tried to for a day now to get nginx running in a FreeBsd
jail and
I’m about to throw the towel in. I was wondering how big a deal is it to
not
run it in a jail. Is chrooting it sufficient. What do people around here
normally
do ?

Thanks.

On Fri, Apr 11, 2008 at 05:36:26AM -0400, Amer Shah wrote:

I know there isn’t a definite answer to this but I thought I’d throw it out
here anyway.

I’m running FreeBsd 7 with nginx. I have a total of 3 daemons. sshd , ftpd
and nginx. I’ve tried to for a day now to get nginx running in a FreeBsd
jail and
I’m about to throw the towel in. I was wondering how big a deal is it to not
run it in a jail.

What problem with jail ?

Is chrooting it sufficient.

It should be enough.

What do people around here normally do ?

I run nginx in usual environment.
The workers run under nobody or other user.

On Fri, 2008-04-11 at 05:36 -0400, Amer Shah wrote:

I’m about to throw the towel in. I was wondering how big a deal is it
to not run it in a jail. Is chrooting it sufficient. What do people
around here normally
do ?

I usually run it as a normal process (as user nginx). It’s the
applications I worry about more than the web server itself. Since Nginx
(unlike a typical Apache configuration), doesn’t run applications within
its own process space (unlike Apache’s mod_php, mod_python, etc), it’s
fairly easy to run those applicatons under separate users and this
greatly alleviates many security risks.

My general feeling is that standard *nix permissions are adequate if
properly enforced.

Regards,
Cliff

Cliff W. wrote:

(unlike a typical Apache configuration), doesn’t run applications within
its own process space (unlike Apache’s mod_php, mod_python, etc), it’s
fairly easy to run those applicatons under separate users and this
greatly alleviates many security risks.

Have you played with any MAC schemes, eg grsecurity? Quite good for
locking a user into a defined set of directories and you can even limit
permissions to do stuff like incoming or outgoing net connections (why
would your PHP user need to create an outgoing network connection other
than when the user account is compromised…). This can greatly
decrease the damage an attacher can do

Ed W

Hello All.

Thanks for the feedback.

Igor, as for the jail, it’s nothing nginx specific, but as a relative
FreeBsd newbie, it’s a little overwhelming to setup the jail. I’ve
gotten
most of the way there but usually am at a loss at the last few steps.

I’ve moved my ftpd to listen only on the internal network now and so now
the
only internet facing daemons I have are sshd (running on a non-standard
port
with root logins disabled) and the nginxd which I will chroot now. I
think
this along with ipfw should be safe enough for now until I have a better
handle on freebsd.

As always, thanks for your helps guys. And as the original poster of the
‘config for static files’, I’ve now finalized the setup and configs and
ran
ab against it and am consistently clocking in 6000 RPS when running ab
against it from a remote machine. This really is quite amazing.

Hi

Actually that’s not true. Many web apps need to open an outgoing
network connection (for example, to fetch an RSS feed, process a credit
card, use OpenID, check a blog comment against akismet, etc).

True - lots don’t need to though! You just add the userid into a group
if they can see the network, otherwise they can’t. Quite simple.
Running servers and clients are two seperate permissions with grsec, so
for example you can allow them to check openid, but deny running an IRC
server…

But yes, I’ve investigated these things a bit and agree they can help
with security. Unfortunately they also tend to make fixing things
become a lot more obscure. Now when an app fails you must ask whether
it’s firewall rules, security framework rules, or simply an application
error.

Yep. True. Also if you are using VPS style containers then it’s also
easier to temporarily drop the firewall rules and drop the security to
debug (another tick for VPS containers!)

IMHO it’s much easier to setup a VPS (e.g. OpenVZ) than to fiddle with
most of the security frameworks (the most common question about SELinux
is how to disable it). You get adequate isolation at minimal cost, and
your app runs in a fairly standard environment.

Well actually you get no extra protection against your app being broken
into to, you just limit the damage caused.

I use linux vservers and I think they are excellent. Best feature I
like is that you can simply copy the container, boot it up and test an
upgrade of config change, then blow it away again. I move all the
shared files to a seperate mount (eg /var/www/ in a web app container)
and that way only the OS needs to be duplicated.

I have never got the hang of SELinux though - seems very complicated,
but potentially very brilliant. I don’t use all of grsec, but it’s a
better fit to my way of thinking. I like the look of the Suse stuff
(AppArmour, etc), but it’s not a supported option under gentoo.

Good luck

Ed W

On Fri, 2008-04-11 at 18:25 +0100, Ed W wrote:

Have you played with any MAC schemes, eg grsecurity? Quite good for
locking a user into a defined set of directories and you can even
limit permissions to do stuff like incoming or outgoing net
connections (why would your PHP user need to create an outgoing
network connection other than when the user account is
compromised…).

Actually that’s not true. Many web apps need to open an outgoing
network connection (for example, to fetch an RSS feed, process a credit
card, use OpenID, check a blog comment against akismet, etc).

But yes, I’ve investigated these things a bit and agree they can help
with security. Unfortunately they also tend to make fixing things
become a lot more obscure. Now when an app fails you must ask whether
it’s firewall rules, security framework rules, or simply an application
error.

IMHO it’s much easier to setup a VPS (e.g. OpenVZ) than to fiddle with
most of the security frameworks (the most common question about SELinux
is how to disable it). You get adequate isolation at minimal cost, and
your app runs in a fairly standard environment.

Regards,
Cliff

On Fri, 2008-04-11 at 21:26 +0100, Ed W wrote:

IMHO it’s much easier to setup a VPS (e.g. OpenVZ) than to fiddle with
most of the security frameworks (the most common question about SELinux
is how to disable it). You get adequate isolation at minimal cost, and
your app runs in a fairly standard environment.

Well actually you get no extra protection against your app being broken
into to, you just limit the damage caused.

But that’s pretty much the case no matter what you do. The security
frameworks simply prevent a broken/hacked application from being used to
further compromise the system. Using the example you gave earlier, to
prevent a hacked PHP application from opening a network connection. They
didn’t prevent the PHP app from being hacked in the first place (nor
could they).

Things like AppArmour can help prevent particular exploits in binary
applications (specifically buffer overruns leading to the execution of
arbitrary code or reading of protected areas), but in general the
purpose of security frameworks such as SELinux and GRSEC is to limit the
damage post-exploit. This is pretty much the same for a VPS. The scope
and method of containment is all that differs.

Regards,
Cliff

Cliff W. wrote:

into to, you just limit the damage caused.

But that’s pretty much the case no matter what you do. The security
frameworks simply prevent a broken/hacked application from being used to
further compromise the system. Using the example you gave earlier, to
prevent a hacked PHP application from opening a network connection. They
didn’t prevent the PHP app from being hacked in the first place (nor
could they).

Hmm, well I don’t want to start a battle here, but I somewhat disagree.

In my mind a vserver just gives you a completely normal server with no
extra frills, but the point is that you can pare it down to the min
software required

The hardening stuff reduces the capabilities at the process level.
So we can lock a particular process into only certain file systems and
reduce the ability to execute all executables (ok, filepermissions do
this also, but they are easy to misconfigure and hard to give proper
granularity compared with a MAC specification). The ability to limit
capabilities is very powerful though and can definitely be used to
reduce the possibility of an app being hacked at all.

Some of the other hardening features can reduce the susceptability of
applications to new exploits, eg stack overflows.

An overlooked part of grsec (and perhaps others) is mandatory logging of
events. For example segfaults are logged in syslog - this can be very
useful for detecting a hack attempt. You can even log execution of
certain binaries (filter out the known ones and you are left with an
“interesting” list which may allow you to detect a breakin)

but in general the
purpose of security frameworks such as SELinux and GRSEC is to limit the
damage post-exploit.

Well they certainly do that - but remember the ability to reduce
capabilities also. You can pare an application back much more tightly
than you can with only file permissions. The two frameworks you mention
above allow you to really lock down a given binary very very tightly and
so I think it’s fair to say that they dramatically reduce the chance of
an exploit as well as reducing the damage once one occurs?

A VPS in my mind really just gives you a much cleaner space to run each
app in and hence reduces the severity of a breach (perhaps reduces the
likely hood of a breach by having fewer services running, but that wasnt
the biggest attraction to me)

Anyway, both are useful to varying extents - I am certainly a big fan of
vservers and grsec to a lesser extent

Ed W

On Sat, 2008-04-12 at 01:21 +0100, Ed W wrote:

Hmm, well I don’t want to start a battle here, but I somewhat disagree.
capabilities is very powerful though and can definitely be used to
reduce the possibility of an app being hacked at all.

Yes and no. You can limit certain types of hacks, but for instance
there is no way you can prevent someone from tinkering with a WordPress
mysql instance to acquire user info or just defacing it.

This is my point about “scope”. The security framework limits the scope
of potential attacks at the process level, but requires attention to
detail. The VPS limits the scope to the VPS level. While this isn’t as
fine-grained as the security framework, it’s sufficient to prevent the
entire system (I mean host system) from being compromised. Worst case
scenario is reinstalling the VPS which is typically quite easy.

purpose of security frameworks such as SELinux and GRSEC is to limit the
damage post-exploit.

Well they certainly do that - but remember the ability to reduce
capabilities also. You can pare an application back much more tightly
than you can with only file permissions. The two frameworks you mention
above allow you to really lock down a given binary very very tightly and
so I think it’s fair to say that they dramatically reduce the chance of
an exploit as well as reducing the damage once one occurs?

Except that I’m unaware of any non-trivial application that is a single
application. Most web applications use at least a database, so that’s
another layer susceptible to attack and a layer that exceptions must be
made for in the framework. As the complexity of web apps increase
(think WordPress) I think it will get more difficult to properly secure
them with a security framework without impeding the functionality of the
application. Of course, for any particular app, this may or may not be
true, but in general I think the trend is that way. Also, many apps
support user-installed plugins which further complicates the issue.

A VPS in my mind really just gives you a much cleaner space to run each
app in and hence reduces the severity of a breach (perhaps reduces the
likely hood of a breach by having fewer services running, but that wasnt
the biggest attraction to me)

Well, both, but limiting the scope of a potential attack to an easily
rebuilt VPS is the most appealing factor, IMHO. Limiting services might
help in general, but most attacks on websites happen at the web
application layer. The way I see it, if a system’s primary purpose (VPS
or otherwise) is to provide a website, then if this service is
compromised then the attacker has won. Whether or not they also setup an
IRC bot isn’t too relevant (and this is something I usually deal with at
the gateway level anyway [I happen to like pfsense]). The point is
they’ve compromised the primary purpose of the system, so protecting the
rest of the system becomes rather moot, except as secondary damage. I
don’t think GRSEC (or any other security framework) is going to help
here, so the question becomes quick recovery (and hopefully an upgrade
of the application in question). This is where a VPS really helps.
Reinstalling a new OS on the bare metal can be time-consuming, whereas
rebuilding a VPS can take a relatively short time (especially if you
separate your services across VPS’, so you don’t need to rebuild things
like mail servers [ech!]).

Anyway, both are useful to varying extents - I am certainly a big fan of
vservers and grsec to a lesser extent

I’m inclined to think of virtualization as a primary defense and
security frameworks second, mostly for the recovery abilities I outlined
above, but also because the security frameworks require much more
thought and time to implement properly (and often you won’t realize
mistakes until it’s too late).

I should mention that I’m considering this from the standpoint of hosted
sites (which is what I do) rather than a specialized system (i.e.
intranet or dedicated server), so that affects my point of view.

Regards,
Cliff

If we’re talking PHP level, i find it best to run individual fastcgi
pools per user.

php-fpm makes this a breeze now.

if their scripts get exploited, the only files that can be modified
are their own. it doesn’t account for any PHP exploits though, and the
scripts can still be exploited and non-root activities can occur (i.e.
zombie bots or IRC bots can be launched, etc) - it won’t be a real
true ‘system compromise’ but it will cost you bandwidth, resources,
etc :slight_smile:

Hi

The way I see it, if a system’s primary purpose (VPS
or otherwise) is to provide a website, then if this service is
compromised then the attacker has won. Whether or not they also setup an
IRC bot isn’t too relevant (and this is something I usually deal with at
the gateway level anyway

We have different fears I think. You are (please excuse the
mis-summary) mainly interested in something like “defacing” websites, ie
breaking them enough to fiddle with the DB in some way, perhaps an SQL
injection attack or the like.

I’m more worried about them breaking in and getting a shell and NOT
being able to detect that for some time… They could be on that machine
and using it to attack other machines and information they learn on that
system might let them escalate access elsewhere.

As an aside. If you are largely worried about web “breakins”, there is
a tool for Apache called mod_security which is fantastic. It’s kind of
like pfsense for webservers. A few broadbrush rules on that make it
VERY much harder to break into a typical webapp. There is at least one
source of regularly updated rules to try and patch holes in common web
applications (although that’s tricky given you can install them with
varying URLs, but it’s a good crack at the problem)

Would be interesting to try and do something similar with nginx
perhaps…?

I’m inclined to think of virtualization as a primary defense and
security frameworks second, mostly for the recovery abilities I outlined
above, but also because the security frameworks require much more
thought and time to implement properly (and often you won’t realize
mistakes until it’s too late).

Agreed. Although I think defense is the wrong word because to my mind
the VPS provides only a little “defense” and more it provides easier
cleanup after the event

Ed W

Cliff W. wrote:

True and this is a real concern that I shouldn’t have dismissed. On the
plus side, it’s pretty difficult to get shell via a web application

If this were true then I am over worried, but it seems to me that a php
injection attack makes it trivial to get a shell as the PHP user. Any
PHP injection attack can run any PHP script, hence the phpshell util can
be uploaded, or a common attack seems to be to use php cURL to download
an exe to the temp folder and execute it.

Both of the above attacks are mitigated by using the security controls
of grsec or similar, and whilst a determined attacker will fine tune the
attack the key thing is that standard script kiddie attacks will be
mitigated (think wordpress exploit + 10 mins on google searching for
common wordpress keywords + automated attack script - if you kill the
key attack methods using grsec then you are right down at the bottom of
the “failed” list which can only be attacked by a more creative attacker
and likely your script kiddie is happy with his 1000+ easily cracked
systems and won’t bother with you)

This is the kind of thing which worries me more though. Shell on a host
which is behaving normally and the attack is not obvious until you find
a machine pumping millions of email messages…

As an aside I adjusted the examples on the wiki, but heads up all using
nginx. The default examples on the wiki leave you VULNERABLE to serious
php injection attacks. Most php apps are setup for apache and have
.htaccess rules as part of the deployment. Most users on nginx seem to
struggle just to setup fastcgi and I will give you even money they don’t
translate all the .htaccess script to nginx rules… Therefore you only
need to look for a typical php app which allows uploads into a web
accessible dir, upload a file xyz.php, then point your browser at the
uploaded file and you have just run run the script of your choice on the
host system. LOTS of php apps are vulnerable and I give you good odds
that most nginx systems are vulnerable because of the lack of debugged
standard configs.

I don’t want to list any widely deployed apps here to avoid giving too
many people a leg up, but it should be something that everyone here
RUSHES TO CHECK RIGHT NOW!!

Hence my previous point about using grsec to sandbox scripts, limit
network access, block temp dirs, etc will kill off most standard upload
script attacks. Better yet to fix the problem, but that’s harder…

Ed W

On Sat, 2008-04-12 at 11:48 +0100, Ed W wrote:

breaking them enough to fiddle with the DB in some way, perhaps an SQL
injection attack or the like.

I’m more worried about them breaking in and getting a shell and NOT
being able to detect that for some time… They could be on that machine
and using it to attack other machines and information they learn on that
system might let them escalate access elsewhere.

True and this is a real concern that I shouldn’t have dismissed. On the
plus side, it’s pretty difficult to get shell via a web application
(except for web applications such as hosting panels that specifically
open security holes… ahem, I mean “services”).

As an aside. If you are largely worried about web “breakins”, there is
a tool for Apache called mod_security which is fantastic. It’s kind of
like pfsense for webservers. A few broadbrush rules on that make it
VERY much harder to break into a typical webapp. There is at least one
source of regularly updated rules to try and patch holes in common web
applications (although that’s tricky given you can install them with
varying URLs, but it’s a good crack at the problem)

I’ll take a look at that. I don’t use Apache myself anymore (except for
a couple legacy mod_svn sites), but I know a few people who could use
it.

the VPS provides only a little “defense” and more it provides easier
cleanup after the event

I’ll concede that.

Cliff