Will Rails be able to do this -fast-?

Rails
1

Hi,

First of all I’d like to say I really love this Rails community.
Everyone is very helpful and I’ve learned so much that I simply
couldn’t have learned just by reading books. I was wondering if you
could help me with the following as well. My general question is: will
Rails be the way to go with this?

I need an application that does server monitoring. But not general
server monitoring (as in ping specific ports), it actually queries
servers for information and based on that information it will act.
There will be a list of servers and generally, when the end of the
list is reached, it will start all over again. The list will be
updated by the application, but also by user input. I need to be able
to set an interval between the requests, but the application also
needs to know about certain limitations. For example, there will be a
maximum of queries to a specific server in a single hour as some
database entries point to the same server, but with a different query.
At this point I ask you - is this all possible? The application needs
to run on itself for months, maybe even years in a row and be very low-
maintenance.

Well, if that’s possible with Rails, there’s another part of the
application that needs to act on specific results. For example, when
something’s not the way it should be, it needs to send an e-mail. The
thing is, this e-mail needs to be sent -fast- ! I’m talking about
milliseconds here. This is because the e-mail will be processed by
another server automatically. Every millisecond counts with this.
Again, I ask you: is Rails the way to go?

I’m quite sure I want to develop the front-end in Rails, but I’m
unsure as to whether the backend will be just as fast in Rails as in
any other programming language. Are there things to think about here?
For example, I don’t know how I would best send these e-mails. SMTP?
Is this fast in Rails? Any faster or slower than in any other
programming language? And why? I’m very inexperienced with the
properties of various available programming languages and wouldn’t
know where to start exploring this.

And how can I make this an ongoing process? I don’t want a cronjob-
powered script, for example. Later on, when there’s a lot to query, we
might move from sending a query once every five seconds to a hundred
times per second. How to do this? It could also be also time-specific.
Some database entries only need to be monitored from 8AM to 5PM for
example. But not all.

I hope you can help me with this, as I’m not sure where to start. If
you tell me to go and learn another programming language that’s
faster, I will. Just be honest.

Thank you very much.

2

www.spiceworks.com ?

3

In a prior life I wrote a control process for the FAA that managed a
distributed suite. The suite was spread over three sites, 45+ pieces
of hardware, 25 communicating apps, 165+ command line arguments and
provided near realtime advice to air traffic controllers. Features
included near real time sensing of failed hardware and software with
automated failover to redundant components. The system needed to be
site configurable to support (initially) four regions.

I started by developing a language specific to the problem at hand and
implemented this using lex and yacc. For infrastructure I used the
Parallel Virtual Machine (PVM) from Oak Ridge National Laboratory, all
coding was done in C with user application libraries available in C
and C++.

Ruby and Rails are fine applications but I would not consider them
suitable for any time sensitive application. You could build useful
support and analysis tools and you might even be able to build a
simulator, but… My guess is that in a distributed monitor and
control situation you’ld be looking at sense and respond times in the
multi tens of seconds.

On Sep 2, 5:52 pm, Frederick C. [email protected]

4

www.spiceworks.com?

I love Spiceworks, but it’s not what I’m looking for here. Sorry!

5

On Sep 2, 7:13 pm, jhaagmans [email protected] wrote:

faster, I will. Just be honest.

To be honest, if you’re sending emails, querying remote servers etc.
then you shouldn’t be worrying about milliseconds - either of those
two actions could take far longer than that. The smtp server you talk
too could easily sit on your message for seconds or even minutes -
email isn’t an instant delivery protocol. Your server-monitory thing
is going to be some sort of background process, you may find a single
instance of that gets more done if you’re using something like jruby
that has better threading than normal ruby. Stuff like eventmachine is
pretty cool for having a single process multiplex a lot of IO.

Fred

6

On Sep 3, 11:03 am, jhaagmans [email protected] wrote:

physically very close to the server sending out the e-mails. Just
believe me, any millisecond counts. That’s also the reason I will be
creating this application: I will need to be able to fine-tune it
later on so that it fits perfectly.

Just seems highly likely that any time you save will actually be lost
in the noise.

Is Rails, or at least Ruby, able to do server monitoring on pre-set
intervals? Forget about the speed issue for a moment, but would it be
able to query a server, wait for its response (we’re talking about
normal HTTP-requests), store this response and go on?

Don’t see why not. As you suspect this bit wouldn’t actually use rails
at all (maybe activerecord if you want to store the results in a
database)

many requests per second to one or two servers, also on a pre-set
interval, but it also needs to send out an e-mail when the response
differs from what is expected. I guess, especially for this last
application, Ruby won’t be able to help me. Am I right?

stuff like eventmachine does all the gnarly IO bits in C so can be
pretty effective for this sort of stuff if you’re worried about the
performance side

Fred

7

Thanks Fred :slight_smile:

Just seems highly likely that any time you save will actually be lost
in the noise.

If it does, it does. However, on average, an e-mail sent 1 millisecond
faster will also be received 1 ms faster. I know you think it doesn’t
matter, but for this application, it matters. Trust me on this. Even
server load isn’t as important as speed.

And thanks for pointing me towards the EventMachine, I knew it was
there, but I just forgot.

8

On 2 sep, 23:52, Frederick C. [email protected] wrote:

instance of that gets more done if you’re using something like jruby
that has better threading than normal ruby. Stuff like eventmachine is
pretty cool for having a single process multiplex a lot of IO.

Fred

Thanks Frederick,

The thing is, I can’t choose the tools I need to work with. I’m not
worrying about server response time from the servers that I will
query, that’s just a given and I cannot do anything about it. I also
cannot worry about e-mail delivery time -after it left my machine-
because that also is just a given. The e-mails will be sent to servers
physically very close to the server sending out the e-mails. Just
believe me, any millisecond counts. That’s also the reason I will be
creating this application: I will need to be able to fine-tune it
later on so that it fits perfectly.

To Rick:

Do you think C and/or C++ are more capable of doing this? I have no
prior experience with C, so I’ll have to look into the possibilities.

To all:

Is Rails, or at least Ruby, able to do server monitoring on pre-set
intervals? Forget about the speed issue for a moment, but would it be
able to query a server, wait for its response (we’re talking about
normal HTTP-requests), store this response and go on? There will be
three applications running on three servers. One is just querying -a
lot- of servers and it doesn’t matter whether it has to wait for the
previous request to finish or not (so I guess multi-threading will not
be needed, right?). It only needs to store the last response and the
time of that last response. The second is querying only a few servers
and there are limitations to how often it can query a specific server,
but it should do a request every second (meaning that when a request
takes over 1 second, it will need to start a new thread). “Normal”
Ruby will probably not be able to do this, right? The last one will do
many requests per second to one or two servers, also on a pre-set
interval, but it also needs to send out an e-mail when the response
differs from what is expected. I guess, especially for this last
application, Ruby won’t be able to help me. Am I right?

I’m going to buy some C books, at least for app 2 and 3, I guess :wink:

Thank you both!

9

On Sep 3, 12:00 pm, jhaagmans [email protected] wrote:

Thanks Fred :slight_smile:

Just seems highly likely that any time you save will actually be lost
in the noise.

If it does, it does. However, on average, an e-mail sent 1 millisecond
faster will also be received 1 ms faster. I know you think it doesn’t
matter, but for this application, it matters. Trust me on this. Even
server load isn’t as important as speed.

I have to agree with Fred - trying to optimize things to this level is
almost certainly a mistake. There are far too many other factors that
can randomly devour >10ms at a pass: the size of the OS timeslice
(typically around 10ms), swapping pages in from disk (50ms to ?),
network contention, etc.

On the other hand, if you’re really serious about being fast, SMTP is
doing it completely wrong. Write a real TCP client and you’ll clear
out a massive amount of overhead.

Unless this is for one of the high-frequency stock trading operations,
in which case I encourage you to spend as much time and money on it as
possible - tell them they need a custom OS or something… :wink:

–Matt J.

10

My point in bringing in C was that the tools you choose are those that
can do the job. If you have a real time requirement (something with
msecs response) you won’t be happy with interpreted code exchanging
messages via email. However, if what you’ve been given is more on the
order of “use Ruby on Rails and make this as fast as possible” then
you can certainly get the job done.

The real tradeoff here is that to achieve real speed you must
invariably give up flexibility. Even though you could write a hard
drive controller in Ruby I doubt you would be satisfied with the
result.

I would recommend that you spend some time defining what you are
trying to do and get requirements set for the areas of your
application in which performance is critical. You should have real
numbers here - or agreement with your user that “as fast as possible”
is adequate.

Take a look at SNMP as a mechanism for collecting data from remote
systems - you might look at MRTG for example. Quite often you can
partition the time sensitive portion out from the reporting portion.
You could, for example, build status logs to report out when all is
running in a pre-defined “good” way and make a nice web view that’s
driven from the logs. You obviously need to develop a strategy for
reporting and responding to the alert and alarm states that populate
the “not good” space. This is the place where time can be critical.

So go for it - sounds like fun.

11

On 3 sep, 18:08, Rick [email protected] wrote:

My point in bringing in C was that the tools you choose are those that
can do the job. If you have a real time requirement (something with
msecs response) you won’t be happy with interpreted code exchanging
messages via email. However, if what you’ve been given is more on the
order of “use Ruby on Rails and make this as fast as possible” then
you can certainly get the job done.

What I was given was:

  • I’d like you to develop an app in PHP
  • When x happens, an e-mail needs to be sent out as fast as possible
    and every millisecond counts.

At that moment I mentioned Rails because I had no clue on what this
app needed to do and I just love Rails and want to limit the time I
spend coding PHP scripts. However, now I know the details, I think it
might be important to look into other programming languages and/or
frameworks. Of course, cost is an issue and it’s an application for
which I could probably have a v0.1 ready in under 20 hours. If I need
to learn a new language, though, I might be taking alot longer. I
know, that’s his problem, but I want to offer a set of possibilities
and it’s for him to decide what’s most important.

is adequate.
The numbers aren’t disclosed. We will need to develop the application,
cross our fingers and hope it’s fast enough. It could easily be that
it’s not possible and our e-mail messages arrive too late every time.
That’s why I need to be sure that what we develop is the best we can
come up with within the budget. If C is the way to go, I will offer
that possibility. However, I’m quite sure I will need twice as long in
C and might have less time left for fine-tuning.

Take a look at SNMP as a mechanism for collecting data from remote
systems - you might look at MRTG for example. Quite often you can
partition the time sensitive portion out from the reporting portion.
You could, for example, build status logs to report out when all is
running in a pre-defined “good” way and make a nice web view that’s
driven from the logs. You obviously need to develop a strategy for
reporting and responding to the alert and alarm states that populate
the “not good” space. This is the place where time can be critical.

What you’re saying might be very important. I know the kind of
information I’m looking for, so if I can assign the highest priority
to the incoming data that contains this information, that could help.
The third time-critical application will actually be designed so that
when the time-critical stuff comes up, it drops everything it does and
sends out the e-mail.

So go for it - sounds like fun.

At least it’s very interesting! It will be fun when I figure it out, -
if- I figure it out :wink:

12

Well, just remember that Rails is a web framework, and Ruby is (as has
been mentioned) an interpreted language.
If speed matters, then you need to go as low in the language hierarchy
as possible. And usually when you talk about going low, you mean C… As
I was reminded here, though, bits of Ruby gems are written in C,
precisely for speed reasons :slight_smile:

And if every millisecond counts, then yes, you probably want to control
everything, and that means email is probably also the wrong way to go.
You’ll probably want to send your own messages through your own
interface. I did not test this and I can be wrong, but it may be faster
to just send one TCP packet with a code which gets translated to a real
message when it gets to the recipient. Again … Speed vs. Flexibility :slight_smile:

Good luck, and remember to share what you learned… If you can’t share
the code :wink:

13

Hi, Thanks Matt. Really appreciate your help :slight_smile:

On 3 sep, 18:29, Matt J. [email protected] wrote:

server load isn’t as important as speed.

I have to agree with Fred - trying to optimize things to this level is
almost certainly a mistake. There are far too many other factors that
can randomly devour >10ms at a pass: the size of the OS timeslice
(typically around 10ms), swapping pages in from disk (50ms to ?),
network contention, etc.

That doesn’t take away my first point. On average, every millisecond
that I can shave off, will result in the e-mail getting there faster.
And again, trust me, every millisecond counts. I don’t want to debate
on the need of shaving these milliseconds off; it’s the task I got. My
question is where to start shaving.

On the other hand, if you’re really serious about being fast, SMTP is
doing it completely wrong. Write a real TCP client and you’ll clear
out a massive amount of overhead.

Great, that’s useful. I’ll look into that.

Unless this is for one of the high-frequency stock trading operations,
in which case I encourage you to spend as much time and money on it as
possible - tell them they need a custom OS or something… :wink:

You’re actually on the right track, but also way off :wink:

The thing is, I -will- start work on this at some point and I -will-
need to make sure it’s my best effort. The budget also has some weight
of course, which is why I can’t get too fancy, developing a custom OS
for example :wink:

14

On 3 sep, 21:01, Aldric G. [email protected]
wrote:

Well, just remember that Rails is a web framework, and Ruby is (as has
been mentioned) an interpreted language.
If speed matters, then you need to go as low in the language hierarchy
as possible. And usually when you talk about going low, you mean C… As
I was reminded here, though, bits of Ruby gems are written in C,
precisely for speed reasons :slight_smile:

That helps :slight_smile:

And if every millisecond counts, then yes, you probably want to control
everything, and that means email is probably also the wrong way to go.
You’ll probably want to send your own messages through your own
interface. I did not test this and I can be wrong, but it may be faster
to just send one TCP packet with a code which gets translated to a real
message when it gets to the recipient. Again … Speed vs. Flexibility :slight_smile:

We don’t control the recipient and we have to send an e-mail. It might
change, but it won’t for the forseeable future.

Good luck, and remember to share what you learned… If you can’t share
the code :wink:

I will share what I can share, but if I don’t use Rails, I probably
won’t be sharing it here :wink:

15

You keep saying “every millisecond counts”, yet your requirement was
given basically as “I need an F1 racecar, here take this bicycle and
build me one” to which you responded “I’m more familiar with mopeds,
I’ll use that instead”.

16

jhaagmans wrote:

On 3 sep, 18:08, Rick [email protected] wrote:

My point in bringing in C was that the tools you choose are those that
can do the job. �If you have a real time requirement (something with
msecs response) you won’t be happy with interpreted code exchanging
messages via email. �However, if what you’ve been given is more on the
order of “use Ruby on Rails and make this as fast as possible” then
you can certainly get the job done.

What I was given was:

  • I’d like you to develop an app in PHP
  • When x happens, an e-mail needs to be sent out as fast as possible
    and every millisecond counts.

Why not just set up Nagios and have done with it? SNMP might be useful
too…

Anyway, if you’ve been told every millisecond counts, you need to know
why. Do you?

Best,

Marnen Laibow-Koser
http://www.marnen.org
[email protected]

17

Thanks Marnen :slight_smile:

On 3 sep, 21:22, Marnen Laibow-Koser <rails-mailing-l…@andreas-
s.net> wrote:

  • I’d like you to develop an app in PHP
  • When x happens, an e-mail needs to be sent out as fast as possible
    and every millisecond counts.

Why not just set up Nagios and have done with it? SNMP might be useful
too…

Because I will not be doing infrastructure monitoring. I’m querying
servers and applications running on servers (could even be PHP
scripts) for actual information which I will use in the applications.

Anyway, if you’ve been told every millisecond counts, you need to know
why. Do you?

I do. And he’s not exaggerating. But again, I can’t go into the
details. I’m absolutely 100% sure that anyone here would agree with me
that every millisecond counts. But I can’t say any more about it and
it’s useless to discuss whether it’s actually needed when I can’t
explain why. Just believe me.

Thanks again to all. By monday I will get back in touch with my client
and I will present him with options I think are usable. If anyone has
any comments on how to do this, please share them.

18

jhaagmans wrote:

I will share what I can share, but if I don’t use Rails, I probably
won’t be sharing it here :wink:

Well, you’ve got a lot of people curious now, I hope you leave us with a
way to follow up, at least!

“And in other news, a crazed developer was seen hanging from an F-16. He
had apparently painted the word ‘Jets on Rails’ on all the planes while
foaming at the mouth. Investigations continue as to how he circumvented
security.”
“Hey, I remember him!”

19

On 3 sep, 21:18, Aldric G. [email protected]
wrote:

jhaagmans wrote:

I will share what I can share, but if I don’t use Rails, I probably
won’t be sharing it here :wink:

Well, you’ve got a lot of people curious now, I hope you leave us with a
way to follow up, at least!

I’m working on a blog to share these kinds of things with the rest of
the world. I have a feeling everyone having a crazy idea starts
knocking on my door lately. I’ve never done the things I’m trying to
do now when I was using PHP. I learn so much so fast!

Lol @ news update xD

20

On 3 sep, 21:07, jemminger [email protected] wrote:

You keep saying “every millisecond counts”, yet your requirement was
given basically as “I need an F1 racecar, here take this bicycle and
build me one” to which you responded “I’m more familiar with mopeds,
I’ll use that instead”.

Well, I’m not more familiar with Rails. I’ve been using PHP for over
eight years now and I’m still learning Rails. However, when he asked
me whether I could do “something” in PHP, I stated that I preferred
using Rails. When it got clear what needed to be done, I wasn’t so
sure what I would need, which is why I started this topic. What’s
wrong about that?