Index only partially built

I have an application which I’m running using Mongrel and Apache as
described here Deployment Strategies for Rails on Windows servers | New Auburn Personal Computer Services LLC.

I have a model Person which I am attempting to use acts_as_ferret with.

When I first try to do a search the index begins to get built but it its
fails halfway through with the following error in the browser:

===
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /myapp/people/search.

Reason: Error reading from remote server

I’m guessing this is Apache giving up on receiving anything from Mongrel
as the index is taking so long to build. If I attempt to do the search
again then only half of the data seems to have be indexed.

How can I index all of the database entries?

Is there a way I can manually build the index before using the
application? From the console for example?

Matthew Planchant wrote:

Is there a way I can manually build the index before using the
application? From the console for example?

ruby script/console production
Person.rebuild_index

ruby script/console production
Person.rebuild_index

When I try this false is returned and some of my data still isn’t being
index. How can I find out what is going wrong?

Raj S. wrote:

Matthew Planchant wrote:

Is there a way I can manually build the index before using the
application? From the console for example?

ruby script/console production
Person.rebuild_index

Thanks. Thought it might have been something like that.

On Mon, Nov 20, 2006 at 01:41:55PM +0100, Matthew Planchant wrote:

Is there a way I can manually build the index before using the
application? From the console for example?

yeah, just do

Person.rebuild_index in your console.

Jens


webit! Gesellschaft für neue Medien mbH www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer [email protected]
Schnorrstraße 76 Tel +49 351 46766 0
D-01069 Dresden Fax +49 351 46766 66

On Mon, Nov 20, 2006 at 02:12:32PM +0100, Matthew Planchant wrote:

ruby script/console production
Person.rebuild_index

When I try this false is returned and some of my data still isn’t being
index. How can I find out what is going wrong?

the return value of rebuild_index has no special meaning, so this is ok.
how do you know some of your data isn’t indexed ?

However, AAF logs the fields and values it indexes when a record is
saved or created, so you might find some helpful info in the log file
(you might have to set the log level to debug when doing this in
production mode)

Jens


webit! Gesellschaft für neue Medien mbH www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer [email protected]
Schnorrstraße 76 Tel +49 351 46766 0
D-01069 Dresden Fax +49 351 46766 66

yeah, just do

Person.rebuild_index in your console.

This is returning false and some of the data is still no indexed?

I’m getting some output like this in the log:

Error retrieving value for field primary_organisation_name: undefined
method name' for nil:NilClass Error retrieving value for field preferred_address_address: undefined methodaddress’ for nil:NilClass
Error retrieving value for field primary_organisation_name: undefined
method `name’ for nil:NilClass

Could this be the cause of rebulid missing out some of the data?

Jens K. wrote:

(you might have to set the log level to debug when doing this in
production mode)

How do I do this?

the return value of rebuild_index has no special meaning, so this is ok.
how do you know some of your data isn’t indexed ?

When I do a search the data which I know is in the db isn’t found.

Here’s how I know it isn’t all being indexed. If I search for person X
then they are not found. If I do directly to X’s edit page and make and
an amendment X can now be found with a search.

However, AAF logs the fields and values it indexes when a record is
saved or created, so you might find some helpful info in the log file
(you might have to set the log level to debug when doing this in
production mode)

OK. I’ll take a look.

On Mon, Nov 20, 2006 at 03:17:15PM +0100, Matthew Planchant wrote:

method name' for nil:NilClass Error retrieving value for field preferred_address_address: undefined method address’ for nil:NilClass
Error retrieving value for field primary_organisation_name: undefined
method `name’ for nil:NilClass

Could this be the cause of rebulid missing out some of the data?

seems your primary_organisation and preferred_address are nil, indeed.

Jens


webit! Gesellschaft für neue Medien mbH www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer [email protected]
Schnorrstraße 76 Tel +49 351 46766 0
D-01069 Dresden Fax +49 351 46766 66

Matthew Planchant wrote:

Jens K. wrote:

(you might have to set the log level to debug when doing this in
production mode)

How do I do this?

OK. I added ‘config.log_level = :debug’ to production.rb.

The rebuild_index sees to be working OK but then terminates prematurely.
Why might this happen?

Jens K. wrote:

On Mon, Nov 20, 2006 at 03:17:15PM +0100, Matthew Planchant wrote:

method name' for nil:NilClass Error retrieving value for field preferred_address_address: undefined methodaddress’ for nil:NilClass
Error retrieving value for field primary_organisation_name: undefined
method `name’ for nil:NilClass

Could this be the cause of rebulid missing out some of the data?

seems your primary_organisation and preferred_address are nil, indeed.

Does this mean that contacts with NULL values in the database for
primary_organisation and preferred_address will not be included in the
index? If so how can I get around this?

Here’s a summary of what I have:

===
class Person < ActiveRecord::Base
acts_as_ferret :additional_fields => [:organisation_names,
:preferred_address_address, :primary_organisation_name]

has_many :person_organisations, :dependent => true
has_many :organisations, :through => :person_organisations

belongs_to :preferred_address, :foreign_key => ‘preferred_address_id’,
:class_name => ‘Address’

belongs_to :primary_organisation, :foreign_key =>
‘primary_organisation_id’, :class_name => ‘Organisation’

def primary_organisation_name
return primary_organisation.name
end

def preferred_address_address
return preferred_address.address
end

def organisation_names
organisations.collect { |organisation| organisation.name }.join ’ ’
end

def self.full_text_search(q, options = {})
return nil if q.nil? or q == “”
default_options = {:limit => 10, :page => 1}
options = default_options.merge options
options[:offset] = options[:limit] * (options.delete(:page).to_i-1)
results = Person.find_by_contents(q, options)
return [results.total_hits, results]
end
end

class Organisation < ActiveRecord::Base
acts_as_ferret :additional_fields => [:address_address]

belongs_to :address
has_many :documents

has_many :person_organisations, :dependent => true
has_many :persons, :through => :person_organisations

def address_address
return address.address
end

def self.full_text_search(q, options = {})
return nil if q.nil? or q==""
default_options = {:limit => 10, :page => 1}
options = default_options.merge options
options[:offset] = options[:limit] * (options.delete(:page).to_i-1)
results = Organisation.find_by_contents(q, options)
return [results.total_hits, results]
end
end

class Document < ActiveRecord::Base
acts_as_ferret :additional_fields => [:organisation_name,
:topic_titles]

has_many :document_topics, :dependent => true
has_many :topics, :through => :document_topics

belongs_to :organisation

def topic_titles
topics.collect { |topic| topic.title }.join ’ ’
end

def organisation_name
return organisation.name
end

def self.full_text_search(q, options = {})
return nil if q.nil? or q == “”
default_options = {:limit => 10, :page => 1}
options = default_options.merge options
options[:offset] = options[:limit] * (options.delete(:page).to_i-1)
results = Document.find_by_contents(q, options)
return [results.total_hits, results]
end
end

Jens K. wrote:

On Mon, Nov 20, 2006 at 03:17:15PM +0100, Matthew Planchant wrote:

method name' for nil:NilClass Error retrieving value for field preferred_address_address: undefined methodaddress’ for nil:NilClass
Error retrieving value for field primary_organisation_name: undefined
method `name’ for nil:NilClass

Could this be the cause of rebulid missing out some of the data?

seems your primary_organisation and preferred_address are nil, indeed.

So what does this mean? Is this likely to bring the rebuild process to
an end prematurely?

On Tue, Nov 21, 2006 at 05:44:31PM +0100, Matthew Planchant wrote:

seems your primary_organisation and preferred_address are nil, indeed.

So what does this mean? Is this likely to bring the rebuild process to
an end prematurely?

should not, but you should handle the case your relationship is nil,
i.e.:

def primary_organisation_name
primary_organisation.name rescue nil
end

Jens


webit! Gesellschaft für neue Medien mbH www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer [email protected]
Schnorrstraße 76 Tel +49 351 46766 0
D-01069 Dresden Fax +49 351 46766 66

Jens K. wrote:

On Tue, Nov 21, 2006 at 05:44:31PM +0100, Matthew Planchant wrote:

seems your primary_organisation and preferred_address are nil, indeed.

So what does this mean? Is this likely to bring the rebuild process to
an end prematurely?

should not, but you should handle the case your relationship is nil,
i.e.:

def primary_organisation_name
primary_organisation.name rescue nil
end

Ok. Thanks. What does this do?

Jens K. wrote:

On Tue, Nov 21, 2006 at 05:44:31PM +0100, Matthew Planchant wrote:

seems your primary_organisation and preferred_address are nil, indeed.

So what does this mean? Is this likely to bring the rebuild process to
an end prematurely?

should not, but you should handle the case your relationship is nil,
i.e.:

def primary_organisation_name
primary_organisation.name rescue nil
end

How does this work for relationships such as:

def organisation_names
organisations.collect { |organisation| organisation.name }.join ’ ’
end

?

Has anyone else had the rebuild ending prematurely?