Forum: Ruby Twitter Personalities (#208)

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

The three rules of Ruby Quiz:

1.  Please do not post any solutions or spoiler discussion for this
quiz until 48 hours have elapsed from the time this message was
sent.

2.  Support Ruby Quiz by submitting ideas and responses
as often as you can!
Visit: http://rubyquiz.strd6.com/suggestions

3.  Enjoy!

Suggestion:  A [QUIZ] in the subject of emails about the problem
helps everyone on Ruby Talk follow the discussion.  Please reply to
the original quiz message, if you can.

RSS Feed: http://rubyquiz.strd6.com/quizzes.rss

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

## Twitter Personalities (#208)

Merhaba Rubyists,

This week's quiz is to create a program that will generate messages
140 characters in length. There primary use will be to create a
Twitter "personality". At the end of the quiz period these
"personalities" will be unleashed on the internet and we'll see how
they do in the wild.

The programs will consist of two parts: a component for interacting
with Twitter, and a top secret "personality" module.

For the Twitter interface component there will be no no-spoiler
period. Please feel encouraged to discuss different libraries or
methods on the mailing list. Let's all work together to find the best
interface.

The "personality" component can take any inputs and will produce a 140
character message when called. The "personality" may remember state.
The no-spoiler period applies for the "personality" component; please
save them until everyone has had a chance to consider their own
implementations.


Have Fun!

This Quiz sounds fun!

I wish  I knew more about Ruby than I do now. I would participate in 
this
one, but I am a beginner rubyist :-(

Anyhow, I can not wait to see the results. Good luck to everyone
participating!


JC

On Jun 5, 2009, at 7:26 PM, Joshua Collins wrote:

> This Quiz sounds fun!
>
> I wish  I knew more about Ruby than I do now. I would participate in  
> this one, but I am a beginner rubyist :-(
>
> Anyhow, I can not wait to see the results. Good luck to everyone
> participating!

This is an almost perfect quiz for a beginner!  Don't give up!

Use it as a chance to stretch your skills.

Try this:

1.  Build a funny saying in a String
2.  Replace some words in that saying with variables that hold the
words you removed
3.  Adjust each variable so it gets two or three different words that
all could fit the saying, selected randomly

It's just like reverse Madlibs.  You may get more ideas to keep
improving it too.

It's not a spoiler to get help on the Twitter side, or you can just
skip that part completely.  Don't be afraid to change the task to suit
your skill.  If you learn something, you win.

You can even ask us questions on the personality side, just be clever
in avoiding spoiler material.  "Hey guys, I'm building a new
navigation system for NASA's next generation rocket and it needs to
randomly choose between three directions…"  We're here to help.  :)

"Never give up!  Never surrender!"

James Edward Gray II

From: "James Gray" <james@grayproductions.net>
>
> "Never give up!  Never surrender!"

http://grabthar.ytmnd.com/

;D


SCNR,

Bill

From: "Daniel Moore" <yahivin@gmail.com>
>
> The "personality" component can take any inputs and will produce a 140
> character message when called. The "personality" may remember state.
> The no-spoiler period applies for the "personality" component; please
> save them until everyone has had a chance to consider their own
> implementations.

So... to clarify, the personality component is not replying
to--or seeing--other messages as input, but is merely
producing a sequence of 140 character messages determined
solely according to its internal black box state?


Regards,

Bill

Bill Kelly wrote:
> to--or seeing--other messages as input, but is merely
> producing a sequence of 140 character messages determined
> solely according to its internal black box state?
>
>
> Regards,
>
> Bill

I don't think that is a necessary restriction, as it "can take any
inputs." I will interpret that to mean you can feed in whatever you
would like - a dictionary or Shakespeare, for example.

-Justin

Joshua,

James has some great suggestions and I couldn't have said it better.
One of the best ways to learn more about Ruby is to dive in. You may
find out that it is easier than you imagine. If you get stuck on a
particular issue you can search the web or ask on the mailing list;
there are many great resources and lots of people interested in
helping out.

Bill,

The personality component can see other messages as input and use
those to determine the next message to provide. Of course, in order
for the component to see the replies and other messages they would
need to be gathered and passed in as arguments in some form.

If you like you can share the techniques for gathering these messages
with the mailing list. The goal is to make it as easy as possible to
focus on the "personality" part without worrying about the
nitty-gritty of interfacing with Twitter. Additionally sharing
information on available libraries or general techniques will help all
the solutions to be better.

I will give the Quiz a shot. After all, I will surely learn from it :-)

Thanks for the encouragement guys!

Wow !! Absolutely great quiz ! :D

So the "personality" is just a random message, perhaps a Clint Eastwood
quote or something?

Jayanth

On Sun, Jun 7, 2009 at 8:42 PM, Sandro Paganotti 
<sandro.paganotti@gmail.com

Daniel Moore wrote:
> Visit: http://rubyquiz.strd6.com/suggestions
>
> The programs will consist of two parts: a component for interacting
> save them until everyone has had a chance to consider their own
> implementations.
>
>
> Have Fun!
>   

Posting to Twitter, I found, was pretty straightforward using the
Twitter4r gem, although it only worked for me under Ruby 1.9:

require 'twitter'
client = Twitter::Client.new :login => "myname", :password => 
"mypassword"
client.status :post, "Working on Ruby Quiz."


I hope that helps anyone who might have been hesitant because of the
Twitter interface part.

-Justin

On Jun 8, 2009, at 1:46 AM, Srijayanth Sridhar wrote:

> So the "personality" is just a random message, perhaps a Clint  
> Eastwood
> quote or something?

It is if you want it to be.  :)

If you would prefer to be trickier, you could try to build intelligent
replies to things people say.

James Edward Gray II

Ok, I've just finished mine... it was a really crazy experience... now 
I'm
going to create a twitter account for it and generate some random 
phrases
with different personalities :D

Sandro Paganotti wrote:
> [Note:  parts of this message were removed to make it a legal post.]
> 
> Ok, I've just finished mine... it was a really crazy experience... now I'm
> going to create a twitter account for it and generate some random phrases
> with different personalities :D

You mean like "The Policeman's Beard is Half-Constructed"?

Sort of :P (here's the bot account: https://twitter.com/twsapiens)

Sandro

Phlip wrote:
> You mean like "The Policeman's Beard is Half-Constructed"?

Racter ftw!

Justin Collins wrote:
>> as often as you can!
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>
>> The no-spoiler period applies for the "personality" component; please
> require 'twitter'
> client = Twitter::Client.new :login => "myname", :password => 
> "mypassword"
> client.status :post, "Working on Ruby Quiz."
>
>
> I hope that helps anyone who might have been hesitant because of the 
> Twitter interface part.
>
> -Justin
>


So, here is my full solution, which just pulls transcriptions of
Dijkstra's notes and tries to extract decent quotes. Also attempts to
find relavent replies. Works most of the time. You can see some results
at http://twitter.com/ewdbot and get a nicer-looking version from
http://gist.github.com/125781

#A little Twitter thing to post random quotes from Edsger Dijkstra

require 'yaml'
require 'open-uri'
require 'twitter'  #twitter4r gem
require 'hpricot'

class DijkstraQuote
    class << self

        #Match the website's naming scheme
        def random_ewd
            (rand(1289) + 30).to_s.rjust(4, "0")
        end

        #Get a random quote
        def get_quote
            quote = nil
            while quote.nil? do
                quote = fetch random_ewd
            end

            quote
        end

        #Get a random quote from the specified set of notes
        def fetch ewd

            #Check if we've already retrieved this set of notes
            if File.exists? "ewd#{ewd}"
                quotes = YAML.load_file "ewd#{ewd}"
                quotes[rand quotes.length]
            else
                $stderr.puts "Fetching EWD#{ewd}..." if $DEBUG

                #Fetch a transcript of Dijkstra's notes
                begin
                    file =
open("http://www.cs.utexas.edu/~EWD/transcriptions/EWD#{ewd[0,2]}xx/EWD#{ewd}.html")
{ |f| Hpricot(f) }
                rescue OpenURI::HTTPError
                    $stderr.puts "Not found" if $DEBUG
                    return nil
                end

                text = file.to_plain_text

                #Pick some sentences that seem good
                lines = text.split(/\.(?:\s+|\n+)/).map do |l|
                    l.gsub(/\t|\n|\r/, " ").squeeze(" ").strip
                end.select do |l|
                    l.length > 40 and l.length < 140 and l[0,1] ==
l[0,1].upcase and l[-3, 3] != "viz"
                end

                #Cache them for later
                File.open "ewd#{ewd}", "w" do |f|
                    YAML.dump lines, f
                end

                #Return a random sentence
                lines[rand lines.length]
            end
        end

        #(Very crudely) tries to find a quote that is related to the
message.
        #Note that this only searches the cache.
        def find_related message

            #Get the longer words
            words = message.gsub(/[^a-zA-Z ]/, "").split.select { |w|
w.length > 4 }

            quote = nil
            if words.length > 0
                word = /#{Regexp.union(words)}/i

                #Check local files for the search word
                Dir.glob("ewd*").find do |f|
                    text = File.read f
                    if text =~ word

                        #Get the list of quotes and grab one
                        quotes = YAML.load_file f
                        matches = quotes.grep word
                        quote = matches[rand(matches.length)]
                    else
                        false
                    end
                end
            else
                quote = get_quote
            end

            quote
        end
    end
end

#Post to Twitter
class DijkstraTwitter

    def initialize
        @twitter = Twitter::Client.new :login => "?", :password => "?"
    end

    #Post a random quote. If _ask_ is true, asks for approval first
    def post_random ask = false
        quote = DijkstraQuote.get_quote

        if ask and ask_permission("Would you like to post this:
\"#{quote}\"") or not ask
            @twitter.status :post, quote
        end
    end

    #Check unanswered @ewd messages and come up with responses
    def post_replies ask = false
        require 'set'

        #Keep track of what has been replied to already
        if File.exist? "ewdreplies"
            replied = YAML.load_file "ewdreplies"
        else
            replied = Set.new
        end

        replies = @twitter.status(:replies)

        replies.each do |status|
            status_id = status.id.to_s
            next if replied.include? status_id
            message = status.text
            user = status.user.screen_name
            if reply_to message, user, ask
                replied << status_id
            end
        end

        File.open "ewdreplies", "w" do |f|
            YAML.dump replied, f
        end
    end

    #Send a reply if we can find one
    def reply_to message, sender, ask = false
        puts "Responding to \"#{message}\""

        quote = DijkstraQuote.find_related message
        if not quote
            puts "Nothing related. Skipping."
            return false
        elsif ask and ask_permission("Would you like to post this:
\"#{quote}\"") or not ask
            response = "@#{sender} #{quote}"
            if response.length > 140
                response = response[0,140]
            end

            @twitter.status(:post, "@#{sender} #{quote}")
            true
        else
            if ask and ask_permission "Ignore this reply?"
                true
            else
                false
            end
        end
    end

    #Ask if the user would like to post the quote which was found
    def ask_permission message
        puts message

        response = nil
        until response =~ /^(y|n)/i
            print "(Y/N): "
            response = gets
        end

        $1.downcase == "y"
    end
end

#Try it out
puts DijkstraQuote.get_quote

#dt = DijkstraTwitter.new
#dt.post_random true
#dt.post_replies true


My favorite so far: "In this sense, Programming = Mathematics + Murphy's
Law"

-Justin

On Jun 5, 2:54 pm, Daniel Moore <yahi...@gmail.com> wrote:
> Visit:http://rubyquiz.strd6.com/suggestions
>
> The programs will consist of two parts: a component for interacting
> save them until everyone has had a chance to consider their own
> implementations.
>
> Have Fun!
> --
> -Danielhttp://rubyquiz.strd6.com

Here's the twitter "personality" I made: http://twitter.com/iloveweather

It posts about the weather conditions of a random US city.  The phrase
list needs some expanding, and there's a little code to clean up
before I post it.  It was pretty fun, and I'm really enjoying seeing
tweets like "Step outside, Bliss. It's nice." and "It's days like this
that make people happy to be in Beverly."

It doesn't do replies (yet).

More to come,
Chris

Hi,

here my implementation of a Markov chain based text generator for 
Twitter.

class MarkovText

  def initialize file
    @word_map = {}
    do_preprocessing file
  end

  def do_preprocessing file
    text = File.readlines(file).collect{|l| l.chomp}.join
    text.gsub! /([a-zA-Z0-9])([,:.!?;])/, '\1 \2'
    text.squeeze! ' '
    words = text.split ' '
    for i in (0..words.size-2)
      # if we encounter this word first initialize with empty Hash
      @word_map[words[i]] ||= {}
      @word_map[words[i]][words[i+1]] ||= 0.0
      @word_map[words[i]][words[i+1]]  += 1.0
    end

    # generate prob distribution for text
    @word_map.keys.each do |k|
      sum = @word_map[k].values.inject(&:+)
      @word_map[k].keys.each{|l| @word_map[k][l] /= sum}
    end
  end

  def generate start_word = nil
    text = [start_word || @word_map.keys[rand(@word_map.keys.size-1)] ]
    while !(/[.!?:;]/ === text.last) do
      w = choose_next(text.last)
      break if !w
      text << w
    end
    text
  end

  def choose_next word
    roll = rand
    cand = @word_map[word]
    return nil if !cand
    i = 0
    while roll > cand.values[i]
      roll -= cand.values[i]
      i += 1
    end
    cand.keys[i]
  end
end

And the inteface code which maintains a collection of text generators 
and
does interfacing with the output (stdout at the moment) and restriction 
to
140 chars etc:

# load Markov chain text generator
require 'markov_text'

class TwPerson

  def initialize min, max, *files
    @max  = max
    @min  = min

    @name = "Mark V. Shaney"
    @pwd  = "markovianSplitPersonality"

    @gens = []
    files.each do |f|
      @gens << MarkovText.new(f)
    end

    @pool = []
    generate_text
  end

  def run
    loop do
      make_post
      time = rand(@max-@min) + @min
      puts "> Made my post, sleeping for #{time}s"
      sleep(time)
    end
  end

  def make_post
    puts "> Trying to post"
    text = ""
    while  (text.length + @pool.last.length) < 139
      text += " " + get_next
    end
    text.squeeze! ' '
    text.gsub!(/( )([.,;:?!])/) do |m|
      m[1]
    end
    do_post text
  end

  def generate_text
    puts "> Generating new text pool"
    m = @gens.shift
    @pool += m.generate
    @gens << m
  end

  def do_post txt
    puts txt
  end

  def get_next
    generate_text if @pool.size < 2
    @pool.shift
  end

end

This solution is quick and dirty and does not claim to be either elegant
or free from errors ;)

Thorsten

Here's my solution: http://github.com/sandropaganotti/Twitter-Sapiensit 
uses
Wordnet to create a group of words semantically-related to a starting
sentence (its personality);
the uses this words to generate a new sentence.

Sandro

Uops. real link:
http://github.com/sandropaganotti/Twitter-Sapiens<http://github.com/sandropaganotti/Twitter-Sapiensit>

On Wed, Jun 10, 2009 at 8:13 AM, Sandro Paganotti <

There were two main components to this week's quiz, the interface to
Twitter and the personality that generates messages.

Let's start with the Twitter interfaces.

Justin Collins' @ewdbot uses the Twitter4r gem[1]. The usage is
straightforward. When given the account credentials it creates an
object that provides methods to post status updates and get
information from Twitter.

    def initialize
      @twitter = Twitter::Client.new :login => "?", :password => "?"
    end

    ...
    # Posting a quote
    @twitter.status :post, quote

    # Getting replies
    replies = @twitter.status(:replies)


Sandro Pagonatti's @twsapiens uses the Twitter gem[2].

    # Initialize Twitter credentials
    base = Twitter::Base.new(Twitter::HTTPAuth.new('twsapiens', <psw 
here>))

    # Get a random status text from the account's friends timeline
    base.friends_timeline.sort{|a,b| rand()<=>rand()}.first.text


One comment about randomizing arrays is that Ruby 1.9.1 provides a
`shuffle` method to `Array`, so now you no longer need to use
`sort_by{ rand }` or the like if you are on 1.9.1.

Both of the gems make connecting to Twitter via HTTP a breeze, though
Twitter4r seems a little bit simpler to use. No matter which one you
choose it shouldn't be more than a couple of lines to connect.

Now that we're all connected to Twitter let's examine some ways to
generate messages to send.

@ewdbot by Justin selects a random Edsger Dijkstra quotation from the
Edsger W. Dijkstra Archive[3]. The quotation is selected by
downloading a random page from the archive, selecting sentences that
meet certain size and regex requirements, then choosing a random
sentence from that list. The trimmed word lists are cached locally to
make future access easier. Justin's solution is well written and
definitely worth examining if you are interested in learning more.

Remember that Sandro was selecting a random status message from
@twsapiens friends timeline? Well that message is used to seed the
personality. Sandro uses the Linguistics gem[4] to create a collection
of words related to the selected message. Those words are then
arranged according to grammar rules from Ola Bini's port[5] of Peter
Norvig's _Paradigms of Artificial Intelligence Programming_. Some of
these sentences come out a little crudely constructed, but it is a
very difficult problem for a program to construct it's own sentences.
This sentence in particular is rather thought provoking:

    the character in table on a variation by he by volume with he on a
variation typecast a notebook computer

Thorsten Hater submitted a Markov chain text generator. I tested it
out on some of the EWD quotations and it had some successful results.
It would be interesting to combine the Markov chain text generation
with trending topics in an attempt to create a popularity bot. Let the
mailing list know when your bot hits 1,000,000 followers!

Twitter Personality Accounts:

* @iloveweather - Chris Shea
* @ewdbot - Justin Collins
* @twsapiens - Sandro Pagonatti

Thank you everyone for your great responses to this week's quiz!

[1]: http://twitter4r.rubyforge.org/
[2]: http://twitter.rubyforge.org/
[3]: http://www.cs.utexas.edu/~EWD/transcriptions/transcriptions.html
[4]: http://www.deveiate.org/projects/Linguistics/
[5]: 
http://github.com/olabini/paipr/tree/7334d50b3b62924bc7c02effd22be0201658945b/lib/ch02