How to chat

Rails
1

Railsers:

Seeing as how everyone who works out a Rails technique should blog about
it,
(and seeing as how I decline to research how to build a gem, plugin, or
blog), this post will sketch a system I developed to chat.

If anyone likes this, please put it on your blog, so other blogs can
point
to it and increase its Google pagerank, etc.

If anyone /doesn’t/ like it, speak up now, or forever … chat about it
later!

== Requirements ==

From the user’s viewpoint, a chat panel should work like this:

  • hit your page
  • all prior chat utterances appear in a scrolling DIV
  • user enters text into a long thin field
  • the text appears at the bottom of the DIV
  • the DIV scrolls down to display it
  • anyone else viewing that page sees it, too
  • you see their chats

That’s the core of chat. Here’s a naive implementation:

  • create a Chat database record and its model
  • put a DIV on your page
  • add a JavaScript DOM timer to your page
  • the timer sends AJAX “refresh” request
  • the server returns all the chat as HTML
  • the browser pushes the HTML into that DIV
  • put a form_remote_for :chat on your page
  • wire it to call an action, “utter” that adds
    a chat to the database
  • when the timer goes off, it “refreshes” this new chat

I hope to improve upon that implementation, but note that my
implementation
might be naive too. It will religiously observe the guideline “Premature
Optimization is the Root of all Hourly Billing Strategies”. I hope it is
at
least more accurate and responsive than some of the chat sample code I
downloaded. For example: It won’t top-post!

== Analysis ==

So let’s guess that sending a long chat history once per timer will be
slow.
This implies that each batch of chat should append to the end of the
history
DIV.

Let’s also guess we should not set our timer to run faster than every
couple
seconds. This implies each chat batch could contain more than one line.

We need a system to delimit a batch of chat, so the server can avoid
sending
the same batch of chat over and over again. We must store a “high water
mark” variable, and only send chat that exceeds this value. And we
shouldn’t
store it in a session variable, because they don’t scale. Session
variables
are essentially scaffolding; to use in a pinch, before we figure out a
truly
stateless solution.

So we add these lines to that algorithm:

  • add a hidden last_chat_id input field
  • the “refresh” timer sends the last_chat_id
  • the server fetches chat with id > last_chat_id
  • the “refresh” response contains the highest
    chat id
  • the browser pushes this into the hidden field

That system ass-u-me-s that our database will always increase and never
roll-over the chat.id variable. We could replace the variable with a
timestamp; the rest of the algorithm will work the same. Another
solution
would be to occassionally “accidentally” erase the entire chat table and
reset its index ID. (That solution has the added benefit of insulating
us
from compliance with court orders requesting proof that someone once
chatted
something naughty!)

To install a “high water mark”, we need one variable’s value,
last_chat_id,
to travel thru these layers:

  • a hidden form field
  • the ActiveForm helper that builds AJAX
  • the JavaScript inside that helper
  • the wire
  • the server’s param[] bundle of joy
  • the ActiveRecord parameters
  • the SQL statement
  • the SQL’s data store
  • the returned recordset (the maximum one in this batch)
  • the returned HTML inside the wire
  • the AJAX JavaScript handler
  • our browser’s page’s DOM elements
  • something that pushes last_chat_id back into its hidden field!

Gods bless N-tier architectures, huh?

But focus on “something”. If our chat action returned raw HTML (such as
a
TABLE full of rows, each containing a chat utterance), then our AJAX
handler
would push this into the DIV tag containing our chat history. Not into
the
FORM tag containing our hidden field.

This implies our HTML package should return a SCRIPT tag containing
JavaScript that updates that field to our hidden value.

That sounds like a plan, but here’s another problem.

Suppose our timer goes off, and our browser launches a “refresh”
message.
Then the server slows down under load, and the timer gets bored and
launches
another one. Each will have the same last_chat_id variable. Slowing down
our
timer is not an option, because if everyone starts chatting the server
could
slow down enough to encounter this problem.

A chat batch should come back to the browser containing enough
information
to censor duplicate lines. So this implies the chat batch is not HTML,
it’s
all JavaScript. It’s a series of addChat() calls, targetting a function
in
our application.js file with this signature:

addChat(panel_id, chat_id, fighter_name, utterance)

That will first check if our browser already contains a SPAN with an ID
set
to ‘chat_id’. If so, this is a duplicate chat, so just bounce out. (In a
stateless, high-latency system, sending a little more data than we need
is
always more efficient than risking sending a little less!)

If the chat utterance is not in the history DIV yet, push it in. (Code
below
will show how.)

That JavaScript function has one more variable, and explaining it will
finish our requirements.

== Productization ==

The ‘panel_id’ is a number indicating which chat panel we target. Our
application could have more than one chat - sometimes more than one on
each
page! So each DIV, FORM, and SPAN we have discussed will need an ID with
this number inside it, such as <div id=‘panel_<%= panel_id
%>_chat_history’…>, to make it unique. That’s so all our excessive
JavaScript can find the correct panel when it goes to harvest data out
or
plug data back in.

So henceforth, if I leave out the excessive <%= panel_id %> baggage, and
only write panel_1_, just imagine it’s there, because it’s easy to code.
Relatively.

== The System ==

So here’s the complete rig:

  • a chat model/table with

    • id
    • user_id → the usual user/login account system
    • panel_id → optional panel model/table
    • uttered_at datetime
    • utterance

  • the chat-endowed page has

    • ‘application.js’, containing
      addChat(panel_id, chat_id, fighter_name, utterance)
      scrollMe(panel_id) ← calls div.scrollTop = 0xffff;
      setLastChatId(panel_id, chat_id)
    • a DIV called ‘panel_1_chat_history’
    • a FORM to be called ‘panel_1_chat_form’
      with a big edit field and a submit button
      with a HIDDEN field called panel_1_last_chat_id
      and an ONSUBMIT with magic AJAXy stuff in it
      that targets :action => ‘utter’
      and updates this:
    • a SPAN called ‘panel_1_new_content_script’
    • a periodically_call_remote() that
      updates ‘panel_1_new_content_script’
      at :frequency => 5 seconds
      :with => ‘panel_1_last_chat_id’
      targets :action => ‘utter’

  • the chat-endowed page’s controller has (or imports*)

    • an action ‘utter’ which
      finds a chat in the params and saves it
      returns a SCRIPT containing
      JavaScript code that updates the ‘panel_1_chat_history’

(*Note there’s a slight gap between canonical MVC and Rails’s convenient
implementation. True MVC makes elements in each module pluggable, so
elements in Views can plug into elements in Controllers, independent of
each
other. Rails, by contrast, uses “Controller” to mean “thing that serves
logic for one coupled View”. [Unless I’m incredibly mistaken.] This is
still
a very good pattern - it enforces the most important separation; between
GUI
Layer concerns and Representation Layer concerns. Yet if this were true
MVC,
then my partial chat View could plug into a truly decoupled Controller.
Not
a Controller that couples with every page that contains chat panels. But
I
digress…)

Now notice that the same action, ‘utter’, serves both saving chat and
refreshing the history. This is because we have two AJAX conversations,
and
we might as well use both to reflect the latest chat back to the user.
That
allows the user to see their own chat, much sooner than our 5 second
timer
will allow. This preserves the illusion that other chats arrive
promptly,
too.

== Implementation ==

The following code snippets illustrate the stations on that cycle that
are
hard to code. The easy ones (like, uh, lots of RHTML, DDL, etc), are
left as
an exercise for the reader. The easy tests are left too (and, yes, I
have
them!).

Create a chat controller/view with one target action, ‘_index.rhtml’. Go
to
the your page that will host a chat panel, and add this line, wherever
you
want chat:

<%= render :partial => ‘chat/index’,
:locals => { :panel_id => 1 }
%>

Make the :panel_id unique, somehow. If you only have a few chats, simply
hard-code each ones ID. Always remember: Magic Numbers are a perfectly
safe
and valid coding practice if they are less than 10.

The first part of ‘_index.rhtml’ is the DIV that will contain our
scrolling
chat history:

That sets the font size a little smaller than normal (a trick I learned
from

cough< Google Chat), and it sets the overflow style to automatic
scroll
bars.

The

goes over the wire empty, and your browser will populate it
during
the split second before it paints.

Next, ‘_index.rhtml’ starts the FORM that will submit new text:

<%
@chat = Chat.new()
@chat.user_id = @user.id
@chat.panel_id = panel_id

form_remote_for( :chat,
:url=>{ :action=>:utter },
:html=>{ :id => ‘chat_panel_’+panel_id.to_s,
:style => ‘display:inline’ },
:update=>‘panel_’+panel_id.to_s+‘_new_content_script’,
:loading=>load,
:complete=>comp,
:after=>“chat_utterance.value=‘’;”
) do |f|
%>

The form contains TABLE markup to stretch things out, and it finishes
with
these important fields:

    <%= f.text_field :utterance, :style=>'width:100%' %>

…
<%= f.hidden_field :user_id %>
<%= f.hidden_field :panel_id %>

      <input type='hidden'
          id='panel_<%= panel_id %>_last_chat_id'
        name='last_chat_id'
        value='0' />

      <%= submit_tag 'Send' if @user %>
      <span id="loading"

style=“background-color:white;color:white;”>
sending…

…
<% end %>

I had to explain why we are doing these things, up in the Analysis
section,
so you will recognize the variables. Otherwise they would just look like
excessive markups. (Right?)

Notice we use the remote form’s :loading and :complete parameters to
flash
the text “sending…”. I will eventually upgrade to a system that
neither
twitches the page’s geometry, nor reveals the invisible word
“loading…” if
you drag-select under the Send button.

Another potential improvement: The controller will soon bounce any new
chat
with an empty utterance. (We should not reward the user for banging
Enter
after writing nothing, or spaces!) However, I suspect we could bounce at
/this/ layer, if we add more JavaScript into the mix.

When that FORM submits, it triggers the ‘utter’ action:

def utter
return unless request.xhr?

if params[:chat]
  if validate_utterance(params[:chat])
    params[:uttered_at] = Time.now
    chat = Chat.create!(params[:chat])
  else
    render(:nothing => true)  #  that's for wasting our time!
    return
  end
end

last_chat_id = params[:last_chat_id]

tex = Chat.get_chattage(
                  params[:panel_id] || params[:chat][:panel_id],
                  last_chat_id.to_i
                  )
render(:text => tex)

end

That (mildly overloaded) action possibly creates a new chat (if the user
really had something to say), and it then returns any pending chats,
possibly including the recently added chat record.

Without the render(:nothing=>true) line, we would reward any user who
banged
insistently. They would cause more server hits than others, and
would fetch chat more often, at the cost of tremendous overhead in their
browser and mild overhead in our server. So we don’t serve new chat to
anyone who did not write an utterance. They can wait for their timer.

(Contrarily, I should have set the limit on our input field. That will
also
prevent annoying error messages, flooding, etc.)

With only a few more code snippets to go, I will leave out the model
stuff
that anyone could write. Get your own model! We are nearing the end.

The faux-Controller method Chat.get_chattage() returns a snip of HTML
containing JavaScript, like this:

Notice I didn’t waste time adding //<![CDATA[ or whatever to mask that
script from naive browsers. If the user gets this far, they must have
JavaScript turned on! (I have also tested my code in Konqueror, Firefox,
and
other amateur browsers, and I suspect it’s generic…)

AJAX will push that SCRIPT into this SPAN:

<%= Chat.get_chattage(panel_id, 0) %>

The partial ‘_index.rhtml’ also calls get_chattage, so our page loads
with a
complete chat history. (Note that this call will take care of all the
pre-existing , with their correct IDs, in the history DIV, and
it
will take care of that sneaky ‘last_chat_id’ field! Nice, huh?

get_chattage() returns JavaScript that, one way or another, will call
these
functions, from ‘application.js’:

function scrollMe(panel_id){ // moves the history DIV to the bottom
var chat_history = ‘panel_’+panel_id+‘_chat_history’;
document.getElementById(chat_history).scrollTop = 0xffff;
}

var last_user_ids = {};
var last_toggles = {};

function addChat(panel_id, chat_id, fighter_name, utterance)
{
span_id = ‘panel_’+panel_id+‘chat’+chat_id;
if (document.getElementById(span_id)) return;
var chat_history = ‘panel_’+panel_id+‘_chat_history’;
div = document.getElementById(chat_history);
html = ‘<span id="’+span_id+'" width=“100%” ';
var toggle = last_toggles[panel_id];
toggle = !!toggle;

if (last_user_ids[panel_id] != fighter_name)
  {
  toggle = !toggle;
  last_toggles[panel_id] = toggle;
  }

var bgColor = toggle? 'white': '#eeeeee';
html +=

‘style="width:100%;color:black;background-color:’+bgColor+';"

';
html += fighter_name;
html += ': ';
html += utterance;
html += ‘
’;
div.innerHTML += html;
last_user_ids[panel_id] = fighter_name;
}

function setLastChatId(panel_id, id)
{
var last_chat_id =
document.getElementById(‘panel_’+panel_id+‘_last_chat_id’);
last_chat_id.value = id;
}

If you have been following along, you recognize things like
‘setLastChatId()’. But the ‘bgColor’ and ‘toggle’ stuff is new. Briefly,
it
assigns the same background color to contiguous utterances by the same
user,
and it alternates the color when alternate characters speak. So if you
send
2 chats, they both get Gray, for example. Then if your opponent chats,
they
get White. If someone else chats they get Grey, and if you chat again
you
now get White. So the system works to join utterances into apparent
paragraphs, and it always contrasts different speakers.

That elusive support function, Chat.get_chattage(), is straightforward
Rails
and ActiveRecord code, so I will just touch on its main concepts.
Firstly,
it needs to sanitize user input, and put it into a “string with quotes”.
It
does this without the help of h(), which I couldn’t figure out how to
import
(or how to Google for!):

def escapeHTML(str)
return(str.
gsub(/&/, “&”).
gsub(/"/, “"”).
gsub(/>/, “>”).
gsub(/</, “<”)
)
end

def enstring(str) # CONSIDER move this to a helper?
return ‘"’ + escapeHTML(str) + ‘"’
end

get_chattage also needs to fetch its relevant chat. All the above
architecture has finally pulled all these variables together, right
where
we need them:

def self.get_chattage(panel_id, last_chat_id)

where = ['panel_id = ? and id > ?',
               panel_id, last_chat_id]

chats = Chat.find(:all, :conditions => where, :order =>

‘uttered_at’)

if [] != chats
  tex = '<script>'

  chats.each do |chat|
    tex += chat.to_javascript
    last_chat_id = chat.id  if last_chat_id < chat.id
  end

  tex += 'setLastChatId('+panel_id.to_s+', ' + last_chat_id.to_s +

‘);’
tex += ‘scrollMe(’+panel_id.to_s+‘);’
return tex
end

return nil

end

If the system has new chats, we convert each one to a JavaScript
statement,
and stuff them all together with .to_javascript(). Finally, we detect
the
new “high water mark” - the new maximum chat_id, and we pass these to
the
browser. And we trigger the correct history DIV to scroll to the bottom.

Chat::to_javascript() looks a little bit like this:

def to_javascript
fighter_name = enstring(user.fighter_name)
utterance = enstring(self.utterance)
return “addChat(#{self.panel_id}, #{id}, #{fighter_name},
#{utterance});”
end

And that’s the bottom of the cycle. New chats follow this sequence:

  • a user enters them into the big edit field
  • AJAX sends them to the ‘utter’ action
  • ‘utter’ puts them into the database
  • the database sets their id to > last_chat_id
  • our find() detects them
  • we convert them into JavaScript
  • we AJAX them back to the browser
  • the browser evaluates their JavaScript
  • and sticks them into the history

== Testing ==

Every factoid in this post has a matching unit test. Really.

I will conclude the code snippets with an interesting test. Without
introduction, read it and try to see what it does:

require ‘rexml/document’
include REXML

class ChatTest < Test::Unit::TestCase
fixtures :chats, :users

def addChat(*foo)
assert_equal 2, foo[0]
@utterances << foo
end

def setLastChatId(panel_id, id)
assert_equal 2, panel_id
@last_chat_id = id
end

def scrollMe(panel_id)
assert_equal 2, panel_id
@scrollMe_was_called = true
end # TODO better name

def test_get_chattage
chattage = Chat.get_chattage(2, 0)

sauce = XPath.first(Document.new(chattage), '/script').text
@utterances = []
@last_chat_id = nil
@scrollMe_was_called = false

assert eval(sauce)  #  no jury would convict me

expect_utterances = [
  [2,  9, "userA", "chat utterance one"],
  [2, 10, "userB" , "chat utterance two"],
  [2, 11, "userA", "chat utterance three"] ]

assert_equal  expect_utterances, @utterances
assert_equal  11, @last_chat_id
assert @scrollMe_was_called

end
end

That test case uses a mock pattern called “self shunt”, where your own
test
case object works as your mock object.

We are mocking ‘application.js’. Look at our sample SCRIPT again:

The XPath stuff strips out the tags, leaving JavaScript that
looks
suspiciously similar to Ruby source. Rather than wait for all the web
browser venders in the world to get off their butts and add Ruby to
their
script engines, we simply write only Ruby-style JavaScript.

Then we eval() it. That calls all our mock functions, and they trivially
report they were called correctly.

== TODO ==

Right now, nothing retires old chats. If the user returns to a page left
on
an open server for a couple months, they will (efficiently) see a couple
months’ worth of chat. Something needs to clean all that up.

The current design is very flexible. (I even flexed it a little as I
wrote
this post!:wink: We could, for example, decorate each user’s name in the
list.
Decorate means to add user-specific links, icons, colors, etc. Because
we
have a working cycle, we can extend it by adding variables to the chat
records and the user records, then we insert them into addChat(), and
render
them in the DOM.

I suspect one could use “JSON” to render a record as JavaScript, but I
have
not researched it yet. It might make mocking our JavaScript hard!

I also suspect that there are libraries available to generate JavaScript
from Ruby generators. This implies one could test-first such JavaScript
by
querying those generators directly.

Lastly, I did not yet re-use this chat into more than one page. This
violates the advice I always give, that you should never add extra code
that
might work when you re-use. My excessive panel_id variable violated
that
rule! I might report back what happens when I add more chat panels, and
I
actually see what will happen when I stress that panel_id!

–
Phlip
http://www.greencheese.us/ZeekLand ← NOT a blog!!!

2

Slinky wrote:

was actually trying to do exactly this as part of my learning process.
The question how to copy data from the DOM thru JavaScript into Ajax is
a
FAQ, and should be in the FAQ that Faisal et al seem to be writing.

The answer is simple - :with => points to a Ruby string containing
JavaScript, and calls it each time it tries to do its thing. So put
inside
the JavaScript anything you want to happen, and if it returns a return
value, the Ajax stuff with append this to the end of the URL it creates.

You must use :method => ‘get’ with :with, so the Ajax stuff will then
actually use the parameters in the URL it sends out.

I could be wrong, and this important system seems to be
under-documented.

–
Phlip
Redirecting... ← NOT a blog!!!

3

This was an interesting email and addressed a problem I was having with
transporting values in a hidden field to the server for processing when
using periodically_call_remote

You seem to use the :with parameter with the function, though, and I
couldn’t seem to find evidence that this parameter is actually
supported by periodically_call_remote. What’s more, if it is
supported, does it function vaguely like the other functions that
include it, as in, running javascript contained within?

I’m having a very hard time getting local javascript browser values to
my ruby code. I’ve seen a lot of suggestions regarding using JSON, but
I can’t seem to find examples of how to actually package and send the
values via an argument in ruby. :frowning: Sorry to hijack your thread, but I
was actually trying to do exactly this as part of my learning process.

4

Phlip wrote:

actually use the parameters in the URL it sends out.

I could be wrong, and this important system seems to be under-documented.

–
Phlip
Redirecting... ← NOT a blog!!!

Thanks so much for your quick response- I’m definitely on the right
track to getting this functional now, but real quick, if I return a
value through :with, what key in the parameter structure will the
returned value be attached to? ‘with’?

5

Okay, never mind- managed to figure it out. The return value of the
:with parameter gets added direction into the link taht gets sent as
the remote request (which is why it has to use GET). Ruby parses
arguments in GET and POST the same, so you don’t have to worry about it
too much, but you do have to return a correctly-formatted string for
the keys and arguments to be parsed into param hash values correctly.

Basically like this:

key1=value1&key2=value2

You do that and you’ll get a bunch of nice key/value pairs added to
param[], with all the values being strings. From there you can do
whatever! In case you’re wondering, here’s what I did:

In the head of index.rhtml:

	<%= javascript_include_tag "prototype" %>
	<%= javascript_include_tag "scriptaculous" %>
	<%= javascript_tag "function stuff()
				{
					return 'stuff=234';
				}"
	%>

In the body:

	<%= periodically_call_remote( :url => { :action => 'add_item'},
		:newitem => false , :frequency => 5, :method => :get,
		:with => "stuff()")
	%>

I haven’t written the final, functional version, but this does
correctly read in the param values so that param[‘stuff’] = ‘234’.
Anything beyond that is just correctly building the string you need to
use so it has the appropriate values.

6

Can we add all this to the FAQ?

Slinky wrote:

return ‘stuff=234’;
I haven’t written the final, functional version, but this does
correctly read in the param values so that param[‘stuff’] = ‘234’.
Anything beyond that is just correctly building the string you need to
use so it has the appropriate values.

And figuring out how to work all their nested layers of escapes…

–
Phlip
Redirecting... ← NOT a blog!!!