Large-scale application


#1

Hi fellow railers!

We are discussing the possibility of writing a very large application in
Rails. By “very large” I mean a framework that would contain a few
hundred smaller applications that would need to talk to one big database
and have to share some common session data I’d like to get some
feedback on some possible ways of doing the following:

The application would basically run the university. It would handle
accounts payable, accounts receivable, financial aid, course
registration, student advising, payroll, grade checking, grade
submission, student housing, etc.

We’re working with Yale’s CAS system and we’ve got that working fine
with Rails, so single-sign-on isn’t a problem, but we would like to be
able to share session data across all of these applications.

So our thought is that we would just make one big Rails application.
However, the following questions come up:

  1. How would we separate those sub-programs (grade submission) into
    “modules” so we could organize the models, views and controllers?
    Components looks like one way, but does that make sense?

  2. Would we be able to deploy a new “module” (grade checking for
    example) without bringing down the whole system? (not a big deal, just a
    question)

We’ve currently got an idea of how to do this with
Spring+Hibernate+Sitemesh+some_other_Java_buzzwords and we’re not making
progress.

I would appreciate any comments on this. We’re just looking for
suggestions.

Thank you very much!

Brian H.
Web D.
Learning & Technology Services
Schofield 3-B
University of Wisconsin-Eau Claire
715 836 3585
removed_email_address@domain.invalid


#2

Hogan, Brian P. wrote:

registration, student advising, payroll, grade checking, grade
Components looks like one way, but does that make sense?
suggestions.

My suggestion is to hire it out. Big projects need a lot of project
management expertise. There are starting to be contract firms that will
bid Rails as well as Java projects – I can track down one or two on the
web if you’d like, although I suspect they read this list and will
probably contact you. And no, I don’t bid on projects this size or have
any financial interest in anyone that does. :slight_smile:


M. Edward (Ed) Borasky

http://linuxcapacityplanning.com


#3

When you say:

“We’ve currently got an idea of how to do this with
Spring+Hibernate+Sitemesh+some_other_Java_buzzwords and we’re not making
progress.”

What exactly are you not making any progress on “being able to deploy a
new
“module” without bringing down the whole system”, or just building the
application you are talking about in Java/Spring/Hibernate? The reason
I
ask is that if you development team doesn’t have any experience with
Ruby/Rails and you are struggling with developing the app in Java, I
don’t
think Ruby/Rails will be the magic cure. The applicaiton you are
talking
about could be developed in Java/Spring/etc or Ruby/Rails, the
programming
language/frameworks are probably not the problem.


#4

I would appreciate any comments on this. We’re just looking for suggestions.

Use ruby modules. Separate your controllers into folders like:

app/controllers/grade_checking/base_controller.rb

views in:

app/views/grade_check/*

class GradeChecking::IndexController < ApplicationController
end

One issue is you can’t have a /grade_checking_controller.rb. You’d
probably use routes to map ‘grade_checking/’ to its Index controller.
Justin F. has an article on this:
http://justinfrench.com/index.php?id=122

But, that would make one large rails app. Yes, bringing it down for
upgrades would bring down the whole system. Perhaps you should try
building it as several disconnected apps and skip the whole module
idea. You can easily share session data as long as they’re on the
same domain.


rick
http://techno-weenie.net


#5

I don’t think he’s looking for suggestions on outsourcing, from what
he wrote it seems like he’s doing the work in house with a small team
of developers and they are looking for suggestions/comments on how one
would go about such a large scale app. But I could have misread what
he is looking for. Plus outsourcing is no fun, who would ever let
somebody else write their Rails code, somebody else would have all the
fun. Haha.


#6

Hello Brian !

2006/1/3, Hogan, Brian P. removed_email_address@domain.invalid:

However, the following questions come up:

  1. How would we separate those sub-programs (grade submission) into
    “modules” so we could organize the models, views and controllers? Components
    looks like one way, but does that make sense?

You should take a look at Rails Engines:
https://opensvn.csie.org/traccgi/rails_engines/trac.cgi

  1. Would we be able to deploy a new “module” (grade checking for example)
    without bringing down the whole system? (not a big deal, just a question)

Hmmm, unless you were running in development mode, I don’t think this
would work, unfortunately. Besides, if you move most / all of your
code into engines, I’m not even sure if that would work. I’d have to
let James A.s comment on this.

Hope that helps !


#7

Hi Brian,

  1. I think the better way to organize the entire app is breaking in many
    small and specialized apps. I mean, one app for payroll, one app for
    grade checking, … You can use the common models (like Student) as
    plugin or as “lib”. If you’ll use one big database for all tables, I’d
    recommend you to use something like this, on each app:

ActiveRecord::Base.table_name_prefix = “payroll” ## or something like
this in application.rb

This way, all payroll related tables will be prefixed w/ “payroll”, to
avoid name conflicts w/ same-name-other-data table in some other app :slight_smile:

Of course, the common models (like student) can use some “common”
prefix, to indicate its a table used by many apps.

About the sessions: use ActiveRecord to persist the sessions. Read the
chapter about session storage in rails book, but basically (in your
case), it would be better to persist them at database :slight_smile:

  1. Yes and no. Using lighttpd, you can use dinamyc vhosting, to avoid
    restarting the http server. Using this, you are able to deploy a new
    module without bringing down the other apps. But depending on the usage
    (ie: nobody uses the app at, say, 01am), I’d prefer to edit the
    lightty’s conf file and restart the server. The restart takes no more
    than 1sec, and the sessions wouldn’t break, so, it doesn’t hurt :slight_smile:

Regards,
Juca


#8

On Tuesday 03 January 2006 10:03 am, M. Edward (Ed) Borasky wrote:

accounts payable, accounts receivable, financial aid, course
“modules” so we could organize the models, views and controllers?
I would appreciate any comments on this. We’re just looking for
suggestions.

My suggestion is to hire it out. Big projects need a lot of project
management expertise. There are starting to be contract firms that will
bid Rails as well as Java projects – I can track down one or two on the
web if you’d like, although I suspect they read this list and will
probably contact you. And no, I don’t bid on projects this size or have
any financial interest in anyone that does. :slight_smile:

I’ve never done anything with hundreds of sub programs either, but the
way
Brian has this organized, it seems to me he has a real shot at success,
due
to modularity.

If I read this correctly, each sub program has exactly three very well
defined
interactions with the others:

  1. The database
  2. Session/state data – could be placed in database and accessed with a
    number passed around
  3. A menu system to the access subprograms

I wonder if Brian and his crew can hack together a couple of the
subprograms
in Rails. Doing so would reveal dead ends, and/or build confidence, and
give
Management something to look at. It would also put him in a better
position
to evaluate the offerings of any contract firms bidding on the project.

If the trial subprograms don’t work out, the few days lost are a cheap
price
to pay for not going down the wrong track. If they do work out, it gets
them
off dead center.

I would think that, if doing a few subprograms at a time works out, they
could
shut down and add subprograms once a week (maybe 2am on Saturday night
:-).

Once again, the idea I just put forward is based on these subprograms
being
very modular with distinct, thin and well known interfaces. If they
start
becoming more entangled, all bets are off.

SteveT

Steve L.
Author:


#9

" As far as if you need to render a section from a different app
as i
often do, you can set up some proxy methods. Define a
fetch_remote_action defined in your application.rb or in a helper
that uses open-uri and localhost to grab pages and actions from other
apps and slurp the text in to be rendered inline with your other info
from the local app."

Sounds neat… but do you have an example of how to do that? I think
I’ve
seen a few examples of this but it does sound interesting.

Also, anything you could send me to kickstart the server setups for this
would be great… I’m pretty… not good when it comes to Apache /
Lightty
so I would appreciate any configuration guidance you can provide.

Thank you!

-bph


#10

On Jan 3, 2006, at 6:42 AM, Hogan, Brian P. wrote:

> We've currently got an idea of how to do this with Spring+Hibernate > +Sitemesh+some_other_Java_buzzwords and we're not making progress. > > I would appreciate any comments on this. We're just looking for > suggestions. > > Thank you very much!

Brain-

I think the easiest way to separate these apps out would be to use

subdomains. If you organized things like this:

grades.unisite.com
accounting.unisite.com
enrollment.unisite.com
and so on…

Then you could have your sessions stored in ActiveRecord and set the

sessions and cookies for the domain like this: *.unisite.com so
that your sessions and cookies could persist between subdomains. This
way you would be able to have a separate small rails app for each
module or subsection and you could take down one subdomain without
taking down the others. Each subdomain would have its own fcgi
listeners and either be running independently of the webserver so
each app could be started without the need to affect others or you
could have one instance of lighttpd running for each subdomain behind
a central proxy that sends requests to the appropriate rails app for
the section of the site.

One question I have about the different modules is this. Are they

all totally independent of each other as far as the need to be on the
same page? Like do you need two of the sub-apps to be on one page at
a time? Or are they separate enough that you could use a portal page
to select the app you want to use and get sent there?

I would stay away from rails components for an app of this size as

they for some reason are a big performance hit that you should avoid.

Cheers-
-Ezra Z.
Yakima Herald-Republic
WebMaster
http://yakimaherald.com
509-577-7732
removed_email_address@domain.invalid


#11

Brian,

From what I read on this thread you will need a bigger version of what I’m
doing for one client here at HXM. We have about 11 different apps
(websites)
sharing sessions (pstore but moving into activerecord) and code (models,
layouts, images, etc.). Basically we have all these different Rails apps
that share the code/resources through symlinks (Linux) and the sessions
through a uniform domain (.domain.com) and a single pstore dir in the
file
system. This allows us to have different apps with different
environments
(Dev/Prod) and to take down/up any site without affecting the others.

Hope this helps,

Adrian M.


#12

On Jan 3, 2006, at 12:42 PM, Brian H. wrote:

Also, anything you could send me to kickstart the server setups for
this would be great… I’m pretty… not good when it comes to
Apache / Lightty so I would appreciate any configuration guidance
you can provide.

Thank you!

-bph

Brain-

The proxy for other pages would look something like this:

require ‘net/http’
require ‘uri’

class Page < ActiveRecord::Base
def self.fetch(page, limit=5)
data = ‘’
begin
data = Net::HTTP.get_response(URI.parse(“http://192.168.0.2/#
{page}”))
rescue
limit -= 1
limit > 0 ? retry : raise
end
data.body
end
end

And you use it like any other model method:

@page = Page.fetch(“other/item”)

I use  a page model like this to grab the content from some php

pages that reside on a different server then my rails app for the
newspaper[1]. You can also put in a regex or two right above the
data.body part if you want to strip out some tags or rewrite url’s as
they come through the proxy. I have these lines right above data.body
in one of the methods I use like this:

     data.body.gsub!(/\/temporaryimages/, "http://img.yakima-

herald.com/temporaryimages")
data.body.gsub!(//wrappers/(\d+).news/i, ‘/page/’+section
+’/\1’)

So that I can use content from other server and alter the stream

before I display it to bend it to my needs. The limit and retyr stuff
I foiund necessary because I am requesting pages from different
physical boxes and want to retry up to 5 times before it gives up in
case the server times out the first couple of times. You can adjust
this to be however many times you want to retry. And if the other
pages you are requesting are on the same box then you can use
localhost instead of the ip address and it is still very fast since
the machines are on the same box.

I can help you out with the server setups for sure. I need to pull

together some stuff for you. Why don’t you send me an email off l;ist
and we can go over some custom lighty and apache config options for you.

[1] http://yakimaherald.com

Cheers-
-Ezra Z.
Yakima Herald-Republic
WebMaster
509-577-7732
removed_email_address@domain.invalid