Can we keep validation in memcached?

Rails
#1

Hi All ,

Can we validate request input data using memcache ? Actually I am facing
scenario in which user uploading CSV file which can have hundreds of
rows
and to validate each row using ActiveRecord taking long time.

I have below question:-

  1. So can we cache our validation rule in memcache ?
  2. If it is possible to store validation rules in memcache but can I
    able
    to perform
    validation check ?
#2

Memcached is only useful for storing the responses and unless the
validation rules are incredibly complicated using memcached is not going
to
speed anything up.

Simply not the right tool for the job, the real question is why are your
validation rules so slow.

#3

Thanks Peter to reply.
Actually my table consist more than 15 columns and each column have lots
of
validation like presence validation,format validation.
Major issue is user will upload CSV file which can consist hundred’s of
rows.Through active record validating each row consuming lots of time.

That’s why I am thinking to move validation rule to cache level. It will
fast process.

Is this can be possible to validate data on cache level ?

On Sun, Nov 3, 2013 at 2:38 AM, Peter H. <

#4

Thanks Peter to reply.
Actually my table consist more than 15 columns and each column have lots
of
validation like presence validation,format validation.
Major issue is user will upload CSV file which can consist hundred’s of
rows.Through active record validating each row consuming lots of time.

That’s why I am thinking to move validation rule to cache level. It will
fast process.

Is this can be possible to validate data on cache level ?

#5

Thanks for reply.
@josh - So you are saying that we can validate request on memcache level
also. Please correct me If I am wrong ?

#6

@Fahim

In my opinion memcached is not a suitable tool for the purpose you’ve
mentioned. In fact it’d be waste of resources in such a case.
I would go for processing that thing in background using any of the
available gem for processing the background job.
Showing an interim screen to the user mentioning that your uploaded file
is
being processed and process it in background with the help of
ActiveRecord

  • DelayedJob.
#7

On Nov 4, 2013, at 2:42 AM, Sur M. [email protected] wrote:

@Fahim

In my opinion memcached is not a suitable tool for the purpose you’ve mentioned.
In fact it’d be waste of resources in such a case.
I would go for processing that thing in background using any of the available
gem for processing the background job.
Showing an interim screen to the user mentioning that your uploaded file is
being processed and process it in background with the help of ActiveRecord +
DelayedJob.

I tend to agree. ActiveRecord does not do well for mass insert/updates,
and better to offload this task and not keep the Rails stack busy with
it.

I also wouldnt hold up the user with a wait screen. Id put a little
indicator in an area that shows the processing is being done but does
not restrict further use of the site, except for that new data. When the
background processing is finished, the indicator can be removed.

#8

Why do you think memcached is only useful for storing response data?
That
is simply not true. Memcache has been around longer than Rails, and is
useful as a general purpose, fast, volatile data store. You can use it
for
anything you like, and it applies well in any case where there is
deterministic output given a certain set of inputs.

#9

On Nov 4, 2013, at 4:32 AM, Fahim P. [email protected] wrote:

@sur and @tam thanks for reply…

@sur - Currently I am using rescue …I am using worker’s but still I am
thinking to use memcache for fast performance…What you say can I achieve
validation on cache level.

@tam - I even showing processing image till process done…

Friends …can i achieve fast performance through putting validation on cache
level …or I have to assume that this is max performance code.

Heres what I dont get: what could you possibly be caching here? If whats
happening is a user is uploading a CSV file that is going to inserted
into table, what is going to be cached?

#10

@sur and @tam thanks for reply…

@sur - Currently I am using rescue …I am using worker’s but still I
am
thinking to use memcache for fast performance…What you say can I
achieve
validation on cache level.

@tam - I even showing processing image till process done…

Friends …can i achieve fast performance through putting validation on
cache level …or I have to assume that this is max performance
code…
On Nov 4, 2013, at 2:42 AM, Sur M. [email protected] wrote:

@Fahim

In my opinion memcached is not a suitable tool for the purpose you’ve
mentioned. In fact it’d be waste of resources in such a case.
I would go for processing that thing in background using any of the
available gem for processing the background job.
Showing an interim screen to the user mentioning that your uploaded file
is being processed and process it in background with the help of
ActiveRecord + DelayedJob.

I tend to agree. ActiveRecord does not do well for mass insert/updates,
and
better to offload this task and not keep the Rails stack busy with it.

I also wouldnt hold up the user with a wait screen. Id put a little
indicator in an area that shows the processing is being done but does
not
restrict further use of the site, except for that new data. When the
background processing is finished, the indicator can be removed.

On Monday, 4 November 2013 00:26:08 UTC+5:30, Fahim P. wrote:
Thanks for reply.
@josh - So you are saying that we can validate request on memcache level
also. Please correct me If I am wrong ?

On Sunday, November 3, 2013 1:46:20 AM UTC+5:30, Fahim P. wrote:
Hi All ,

Can we validate request input data using memcache ? Actually I am facing
scenario in which user uploading CSV file which can have hundreds of
rows
and to validate each row using ActiveRecord taking long time.

I have below question:-

  1. So can we cache our validation rule in memcache ?
  2. If it is possible to store validation rules in memcache but can I able
    to perform
    validation check ?

–
You received this message because you are subscribed to the Google G.
“Ruby on Rails: Talk” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [email protected]
To post to this group, send email to [email protected]n.invalid.
To view this discussion on the web visit
https://groups.google.com/d/msgid/rubyonrails-talk/70f5120b-b8f2-4726-8467-586edaf2873f%40googlegroups.com
.

#11

Thanks @tamouse for reply.
Actually my table consist more than 15 columns and each column have lots
of validation like presence validation,format validation.
Major issue is user will upload CSV file which can consist hundred’s of
rows.Through active record validating each row consuming lots of time.

That’s why I am thinking to move validation rule to cache level. It will
fast process.

If I understood you correctly,

Validation rule resides inside a Ruby class i.e. a class inheriting
ActiveRecord::Base, which is already there in memory to even be
functional.
It doesn’t make any other sense to cache this class (or rules inside the
class). Further, Rails automatically caches all the classes in
production
environment. Do you correctly understand this ?

In short, your say “I am thinking to move validation rule to cache
level”
itself is incorrect because the validation is already in memory and is
already cached, you can not speed it up further by saying you want to
cache
it.

And for processing millions of rows with 15 columns each row, by any
chance
it surely is going to be a slow procedure even with memcached (unless
you
want to put a super computer in place for this task)… and the
practical
solution is to move this to background job.

or, I didn’t understand you at all.

regards,
Sur
crimson9.com

#12

Thanks @tamouse for reply.
Actually my table consist more than 15 columns and each column have lots
of
validation like presence validation,format validation.
Major issue is user will upload CSV file which can consist hundred’s of
rows.Through active record validating each row consuming lots of time.

That’s why I am thinking to move validation rule to cache level. It will
fast process.

Hope you understood situation …

  • show quoted text -

On Nov 4, 2013, at 4:32 AM, Fahim P. [email protected] wrote:

@sur and @tam thanks for reply…

@sur - Currently I am using rescue …I am using worker’s but still I
am thinking to use memcache for fast performance…What you say can I
achieve validation on cache level.

@tam - I even showing processing image till process done…

Friends …can i achieve fast performance through putting validation on
cache level …or I have to assume that this is max performance code.

Heres what I dont get: what could you possibly be caching here? If whats
happening is a user is uploading a CSV file that is going to inserted
into
table, what is going to be cached?

  • show quoted text -

To view this discussion on the web visit
https://groups.google.com/d/msgid/rubyonrails-talk/7902ebce-ee88-42f7-bd2c-cbe9b7374d18%40googlegroups.com
.

  • show quoted text -
#13

Thanks @sur for reply…
Yes you understood correctly my issue, and I even understood your
solution…
let me cross check your solution and will update here.

If I understood you correctly,

Validation rule resides inside a Ruby class i.e. a class inheriting
ActiveRecord::Base, which is already there in memory to even be
functional.
It doesn’t make any other sense to cache this class (or rules inside the
class). Further, Rails automatically caches all the classes in
production
environment. Do you correctly understand this ?

In short, your say “I am thinking to move validation rule to cache
level”
itself is incorrect because the validation is already in memory and is
already cached, you can not speed it up further by saying you want to
cache
it.

And for processing millions of rows with 15 columns each row, by any
chance
it surely is going to be a slow procedure even with memcached (unless
you
want to put a super computer in place for this task)… and the
practical
solution is to move this to background job.

or, I didn’t understand you at all.

regards,
Sur
crimson9.com

#14

@tam - I am even not sure how can experiment it. Collecting data and
studying in progress.
@sur - I am still cross checking… If you provide any reference link or
data to proof that validation already get cached.

Friends and @sur please clear me on below question-
From rails app, when user insert data then each time data hitted on
database for validation or validation is present in cache ?

If validation get cached then we don’t need move validation on cache.

#15

On Nov 4, 2013, at 6:02 AM, Fahim P. [email protected] wrote:

Thanks @tamouse for reply.
Actually my table consist more than 15 columns and each column have lots of
validation like presence validation,format validation.
Major issue is user will upload CSV file which can consist hundred’s of
rows.Through active record validating each row consuming lots of time.

That’s why I am thinking to move validation rule to cache level. It will fast
process.

How do you move a validation rule to memcache?

#16

This is a great use case for a background job. Have you tried Resque?

If I have a user uploading thousands of rows, all of which need to be
validated and processed/saved, Id do it like this:
Let the user upload the file, save it somewhere. Trigger a Resque job.
Read the file one or more rows at a time, make a ActiveRecord or
ActiveModel object out of them and run the validators. Collate the
errors and include the row number.
If all the rows were valid, continue processing or saving.
If not, send an email to the uploader and/or show in his dashboard the
failure message, along with a list of all errors with row numbers.

Ive used this method in our app, and is working out great for our users.

–
Dheeraj K.

#17

@dheeraj - thanks for reply.I am using resque & redis. You exactly got
use
case .
can you please give answer for below point:-

  1. From rails app, when user insert data then each time data hitted on
    database for validation or validation is present in cache ?
#18

First validate the data, and then insert it to the database. Till then,
its read from the file and still in-memory.

Im not bringing the cache into the picture at all.

–
Dheeraj K.

#19

Well, this is a classic use case when we jump too early on Rails not
understanding the Ruby (or programming in general) to at least a decent
level… let alone the meta-programming stuff.

One should get a good hold of Ruby at least to a level where s/he
understands how the memory mapping is working behind the scenes…
otherwise you can end up writing the piles of junk which is eating up
all
your server’s memory.

@tam - I am even not sure how can experiment it. Collecting data and

studying in progress.
@sur - I am still cross checking… If you provide any reference link or
data to proof that validation already get cached.

“If you provide any reference link or data to proof that validation
already
get cached.”
I got a smile when I read the above line. :smiley:

a humble suggestion:
before finding any proof, you gotta study more about programming and how
the memory space is managed on the runtime.

Do you know what is the difference between a string and a symbol and why
and where symbols are preferred over strings ?

try this:

a = “some string”
b = “some string”
b.object_id == a.object_id (watch for the result of this expression)

sym_a = :some_symbol
sym_b = :some_symbol
sym_b.object_id == sym_a.object_id (watch for the result of this
expression)

Now, you don’t need to go too far for finding the proof for the
validation/class caching:
(let alone the Rails production environment, in one process of Ruby, if
Ruby can’t cache the class why would you even use it ?)

user_class = class User; end
another_user_class = class User; end

user_class.object_id == another_user_class.object_id (watch for the
result
of this expression)

Note:- object_id is the method available to every single object in Ruby
(remember every smallest/biggest entity in Ruby you have is an
object)…
and it returns the id of that object from the memory map.

Play as much as possible with the above snippets in IRB. IRB is
interactive
Ruby console, you can initiate it by typing “irb” in your terminal and
then
can type the above snippets to test there.
Try tasting the flavor of Ruby to a bit of the next levels.

regards,
Sur
crimson9.com

#20

On Tue, Nov 5, 2013 at 12:07 AM, Sur [email protected] wrote:

Do you know what is the difference between a string and a symbol and why and
where symbols are preferred over strings ?

try this:

a = “some string”
b = “some string”
b.object_id == a.object_id (watch for the result of this expression)

sym_a = :some_symbol
sym_b = :some_symbol
sym_b.object_id == sym_a.object_id (watch for the result of this
expression)

Preach on about this bullshit when Ruby isn’t primarily used for web
applications and when Ruby allows my system to reclaim memory, until
then lets leave the symbol argument bullshit out of this. Even if
they are constants they can lead to trouble if you just blindly state
they are better without explaining the dangers in a web application,
or people will repeat history, history people like me thought was
already common knowledge.