RubyOnRails.com blocking based on referrer?


#1

It looks like the company hosting RubyOnRails.com (TextDrive, now
Joyent) is blocking all URLs that contain the term “site5.com
anywhere in the URL with a misleading error message (Precondition
Failed). Competition is healthy, and we encourage it, but if it is
their intention to do this sort of thing they should be open and
honest with their customers, and the community about it. This does
not appear to be limited to just the rails site either. One of our
contractors is a paying TextDrive customer (http://
www.mattmccray.com/) and any link containing Site5.com that goes to
his site results in a similar error message.

Case in point:
http://www.adamgreenfield.com/test.html
http://www.adamgreenfield.com/site5.com/test.html

Examples (don’t work):
http://forums.site5.com/showthread.php?t=6758
http://engineering.site5.com/
http://fs.site5.com/~agreenfield/test.html

Examples (work)
http://www.adamgreenfield.com/articles/2005/12/13/rails-hits-1-0

At any rate, if TextDrive doesn’t want to play fair with competing
hosting companies, then we would be happy to extend the following
offer to the rails project. We will offer a dedicated machine (with
the operating system of your choice) for the hosting needs of the
rails project:

2 x 2.8 GHz Intel Xeon EMT64 / 64 bit
4 GB DDR ECC Registered RAM
2 x 200 GB SATA 7200 RPM 8 MB cache disks
100 mbit/s uplink
1 TB data transfer (if you need more, just let us know and it
shouldn’t be a problem)

More hosting companies offering rails based solutions is a good thing
for the technology as a whole. Competition in this market will only
result in the best possible situation for the consumer. If TextDrive
wishes to take anti-competitive actions like this, it is my sincere
hope that Rails will take their hosting elsewhere, even if elsewhere
is not with us.


Adam C. Greenfield
Chief Technology Officer
Site5 Internet Solutions, Inc.
Phone: (888) 748-3526 x 906
E-Mail: removed_email_address@domain.invalid


#2

It looks like the company hosting RubyOnRails.com (TextDrive, now
Joyent) is blocking all URLs that contain the term “site5.com
anywhere in the URL with a misleading error message (Precondition
Failed).

Adam,

I had a quick look in our support system and I can’t see any mention
of this. Did you try contacting removed_email_address@domain.invalid or Dan the
rails sysadmin?

412 precondition failed indicates that those requests have fallen foul
of mod_security which we use to stop the millions of random referrer
spam / comment spam requests which get sent . I’ve contacted our
systems administrators and we’ve found the rule in question and
removed it from wrath. We’ll make sure it’s not

We have several automated processes which mine our logs for common
referrers which look ‘spammy’, occasionally these get overly agressive
pulling in legitimate referrers or post body patterns. I assume
that’s what has happened here.

We also like competition, and I can assure you that this blocking was
not intentional.


Cheers

Michael K.
removed_email_address@domain.invalid
removed_email_address@domain.invalid


#3

On Dec 13, 2005, at 10:49 PM, Michael K. wrote:

not intentional.

May I inquire as to specifically what the rule was? .site5.com. ?

Also, could one of your administrators do some research and let me
know how it got there? (Obviously, if this is an automated system, we
would not want to have the same thing occur in the future)


Adam C. Greenfield
Chief Technology Officer
Site5 Internet Solutions, Inc.
Phone: (888) 748-3526 x 906
E-Mail: removed_email_address@domain.invalid


#4

I would REALLY like to know more about this, because I have a lot of
friend active as linux administrators (working at a large hosting
company’). And they can’t even imagine having a automated proces that
could block hosts referrers? Are you crazy, hosting a application and
having a ‘automated proces’ that blocks clients because they originate
from a certain host? Whats next? blocking requests with no referers?


#5

John,

No, I live in the netherlands and they work for a dutch company doing
hosting for large Service Appliction Providers. They don’t do small
hosting things like textdrive/site5! If you don’t believe me checkout
whois of advany.com : )

I am just really chocked, I was going to buy a 500 dollars a month
package for one of my customers (they pay 1000 a month for there current
java package). But once you start trusting a company for hosting needs,
you are responsible, as far as your client is consirnd. I can’t imagin
such childish behaviour (if they can’t explain it). I can’t imagin
having a default blocking behaviour like this in place. This could cause
a lot of loss (of visitors).

Well, I am just shocked…


#6

Call me cynical, but your friends wouldn’t happen to work at site5? :wink:

(… and no, I have no affiliation with anyone)

  • John

On 14/12/2005, at 2:05 PM, Abdur-Rahman A. wrote:

that’s what has happened here.
system, we would not want to have the same thing occur in the


Rails mailing list
removed_email_address@domain.invalid
http://lists.rubyonrails.org/mailman/listinfo/rails


Rails mailing list
removed_email_address@domain.invalid
http://lists.rubyonrails.org/mailman/listinfo/rails


John M. removed_email_address@domain.invalid


#7

Frankly, it seems pretty obvious there wasn’t a malicious intent. Why
would Textdrive want to block visitors to a site that specifically
mentions Textdrive as the official hosting provider of RoR on the
homepage?

  • Derek

On 12/13/05, Abdur-Rahman A. removed_email_address@domain.invalid wrote:

you are responsible, as far as your client is consirnd. I can’t imagin

(… and no, I have no affiliation with anyone)

because they originate from a certain host? Whats next? blocking

May I inquire as to specifically what the rule was? .site5.com. ?
E-Mail: removed_email_address@domain.invalid
Rails mailing list
http://lists.rubyonrails.org/mailman/listinfo/rails


Rails mailing list
removed_email_address@domain.invalid
http://lists.rubyonrails.org/mailman/listinfo/rails


Derek H.
HighGroove Studios - http://www.highgroove.com
Atlanta, GA
Keeping it Simple.
404.593.4879


#8

It looks like the company hosting RubyOnRails.com (TextDrive, now
Joyent) is blocking all URLs that contain the term “site5.com

I’m disappointed that you didn’t check into the matter before blasting
TextDrive for foul play. Sure, if it was a deliberate act, it would
have been lame. But even if it was deliberate, it wouldn’t make any
sense?! Why would someone block referrers from a site5.com site?

If the intention was foul play, it should have been a text block. Like
not allowing site5.com to be entered in comments or on the wiki. But
referrers? That’s just making sure that potential customers who went
to a site5.com site first wouldn’t get to see what TextDrive had to
offer – which seems like bad business.

Maybe you can explain to me how this would have helped TextDrive. If
not, I hope, nay, expect, you’ll be retracting this broadside.

David Heinemeier H.
http://www.loudthinking.com – Broadcasting Brain
http://www.basecamphq.com – Online project management
http://www.backpackit.com – Personal information manager
http://www.rubyonrails.com – Web-application framework


#9

The people at Textdrive seem genuinely concerned and willing to
correct the error, so there’s no reason to believe this was done
intentionally. And as Derek mentioned, it doesn’t really make sense
for them to be blocking traffic from Site5 and affiliated sites. We
have many links to the Rails website throughout our various sites and
applications. Blocking all of that traffic would essentially waste a
large amount of free exposure, not to mention Page Rank.

Besides… this is a fairly trivial issue (technically speaking). The
Textdrive team seems to me to be pretty bright, and if they were going
to bother doing something like this out of malice, they could probably
come up with a much more devious solution. :wink:

But, like I said, I genuinely believe that this was not at all
intentional. No damage has been done, and the problem is being
addressed quickly. I’m not sure what more we could have asked for in
a response…

Many thanks to the Textdrive team for their assistance, and keep up
the good work–positive competition is advantageous for all parties
involved!

On 12/13/05, Abdur-Rahman A. removed_email_address@domain.invalid wrote:

But once you start trusting a company for hosting needs,
you are responsible, as far as your client is consirnd. I can’t imagin
such childish behaviour (if they can’t explain it). I can’t imagin
having a default blocking behaviour like this in place. This could cause
a lot of loss (of visitors).

Well, I am just shocked…


Best Regards,
Matt Lightner
CEO and Co-Founder
Site5.com


#10

On 12/13/05, David Heinemeier H. removed_email_address@domain.invalid
wrote:

Maybe you can explain to me how this would have helped TextDrive. If
not, I hope, nay, expect, you’ll be retracting this broadside.

David,

As you also noted, it doesn’t appear that TextDrive had anything to
gain by deliberately blocking traffic from Site5. That being said, we
don’t assume to know all of their conceivable motivations. I am glad
to hear that this was nothing but machine error, as anything else
would have been rather questionable, as I’m sure everyone would agree.

It seems that it would behoove TextDrive to investigate the
configuration of their mod_security installation to ensure that other
legitimate traffic sources don’t suffer this same fate. The symptoms
of the issue, at least at first, seems very suspicious. I’m not sure
if it would be antagonistic to the goals of the mod_security module to
include a more verbose 412 error page, but, if not, that would be a
relatively easy way to avoid future confusion on this kind of issue.

At any rate, it is clear that TextDrive has no ill-will against Site5,
and, in fact, has been quite helpful in addressing the issue. As
such, I can assure you that we certainly have no ill-will against
them. We look forward to a continued competitive (perhaps even more
open) relationship with the company. Clearly Site5 and TextDrive
share many common interests, and both companies would stand to gain
immensely from a more frequent dialogue (read: any dialogue at all
;-).

Again, I appreciate the attention on the matter–congrats on the 1.0
release of Rails!


Matt Lightner
CEO and Co-Founder
Site5.com


#11

You don’t get the point, why would textdrive have even a system for
blocking referrers? and why would it block everything that has the url
in the name? mhhh, I just can’t buy the supplied story, but they only
lost me as there client unless they come up with a better story. Many
other hosting companies out there but I liked textdrive as it was the
official sponsor of rails. I am just happy I didn’t make a big mistake
and thereby risking loosing a great client of mine…


#12

The people at Textdrive seem genuinely concerned and willing to
correct the error, so there’s no reason to believe this was done
intentionally. And as Derek mentioned, it doesn’t really make sense
for them to be blocking traffic from Site5 and affiliated sites. We
have many links to the Rails website throughout our various sites and
applications. Blocking all of that traffic would essentially waste a
large amount of free exposure, not to mention Page Rank.

I’m happy to hear that, Matt. I’d be equally happy if you guys could
bring this updated information to the various places that includes
this charge against TextDrive. As you can see from this thread, such
accusations can have consequences for a business like TextDrive that
relies on trust. Mr. Advany, for example, seemed ready to burn them at
the stake.

I hope that there’s an official apology in the making.

David Heinemeier H.
http://www.loudthinking.com – Broadcasting Brain
http://www.basecamphq.com – Online project management
http://www.backpackit.com – Personal information manager
http://www.rubyonrails.com – Web-application framework


#13

On 12/13/05, Abdur-Rahman A. removed_email_address@domain.invalid wrote:

You don’t get the point, why would textdrive have even a system for
blocking referrers? and why would it block everything that has the url
in the name? mhhh

http://www.modsecurity.org/


#14

Mhhh, ill just contact textdrive directly, still doesn’t feel good


#15

LOL, Im not ready to burn them at the stake, but its a realy big
accusation that I can’t really understand. But Ill just contact
textdrive directly, could find anything about this automaticly blocking
behavouir on there site. If something like this was said for my
business, It would affect me greatly. But its just really strange how
they configured there mod_security.

Well I am a bit sorry, for the somewhat overreacted replies, but I was a
bit shoked and still am! Hosting is a very basic consern for a software
engineer like me, I like it to be fully outsourced to people with
knowledge. But if you hear something, like blocking visitors bassed on
behavior (apart from site5) that a big deal for my client.

Vriendelijke groet,

Abdur-Rahman


#16

The issue was dealt with swiftly … I suspect direct communications
with Textdrive would have achieved the same result.

Raising an issue with a competitor (apparently without first
contacting) and then sneaking in a plug for one of your services is
shoddy.

David is absolutely correct.

  • John

On 14/12/2005, at 3:48 PM, David Heinemeier H. wrote:

this charge against TextDrive. As you can see from this thread, such
http://www.rubyonrails.com – Web-application framework


Rails mailing list
removed_email_address@domain.invalid
http://lists.rubyonrails.org/mailman/listinfo/rails


John M. removed_email_address@domain.invalid


#17

Heya :slight_smile:

services is shoddy.
I probably would have reacted similarly - but only because I couldn’t
really
imagine that a hosting company woudla ctually put in place an automated
system that could without their knowledge or intervention block
incomming
traffic in a manner that would effect all their clients.

I would have assumed that they must have done it deliberately, because
it is
completely baffling to me that they would let the system make such far
reaching decisions without human intervention or review.

The end result is not that I think TextDrive is malicious - but it is
another factor that goes into the decision process when I recommend
hosting
that if I use them I will also have to worry about their automated
security
system causing problems for my sites.

Soulhuntre

http://www.girl2.com - my girls
http://www.the-estate.com - my legacy
http://wiki.thegreybook.com - my project
http://weblog.soulhuntre.com - my thoughts


#18

On Dec 14, 2005, at 1:33 AM, John M. wrote:

Raising an issue with a competitor (apparently without first
contacting) and then sneaking in a plug for one of your services is
shoddy.

Site5 doesn’t sell dedicated server hosting

Sorry for the delayed reply, I didn’t catch that part of your message
until it was quoted later.


Adam C. Greenfield
Chief Technology Officer
Site5 Internet Solutions, Inc.
Phone: (888) 748-3526 x 906
E-Mail: removed_email_address@domain.invalid