Forum: Ruby on Rails Can we keep validation in memcached ?

576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-02 21:18
(Received via mailing list)
Hi All ,

Can we validate request input data using memcache ? Actually I am facing
scenario in which user uploading CSV file which can have hundreds of
rows
and to validate each row using ActiveRecord taking long time.

I have below question:-

1. So can we cache our validation rule in memcache ?
2. If it is possible to store validation rules in memcache but can I
able
to perform
validation check ?
C993c6aaffc6cf7b477a5075099e1d36?d=identicon&s=25 Peter Hickman (Guest)
on 2013-11-02 22:10
(Received via mailing list)
Memcached is only useful for storing the responses and unless the
validation rules are incredibly complicated using memcached is not going
to
speed anything up.

Simply not the right tool for the job, the real question is why are your
validation rules so slow.
576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-02 23:10
(Received via mailing list)
Thanks Peter to reply.
Actually my table consist more than 15 columns and each column have lots
of
validation like presence validation,format validation.
Major issue is user will upload CSV file which can consist hundred's of
rows.Through active record validating each row consuming lots of time.

That's why I am thinking to move validation rule to cache level. It will
fast process.

Is this can be possible to validate data on cache level ?



On Sun, Nov 3, 2013 at 2:38 AM, Peter Hickman <
576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-02 23:13
(Received via mailing list)
Thanks Peter to reply.
Actually my table consist more than 15 columns and each column have lots
of
validation like presence validation,format validation.
Major issue is user will upload CSV file which can consist hundred's of
rows.Through active record validating each row consuming lots of time.

That's why I am thinking to move validation rule to cache level. It will
fast process.

Is this can be possible to validate data on cache level ?
5b50fec1e5bf39bb4d75b199130882f7?d=identicon&s=25 Josh Jordan (Guest)
on 2013-11-03 19:40
(Received via mailing list)
Why do you think memcached is only useful for storing response data?
That
is simply not true. Memcache has been around longer than Rails, and is
useful as a general purpose, fast, volatile data store. You can use it
for
anything you like, and it applies well in any case where there is
deterministic output given a certain set of inputs.
576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-03 19:58
(Received via mailing list)
Thanks for reply.
@josh - So you are saying that we can validate request on memcache level
also. Please correct me If I am wrong ?
Ce43b11af679aaa142f5ba7e35e64c4b?d=identicon&s=25 Sur Max (sur)
on 2013-11-04 09:44
(Received via mailing list)
@Fahim

In my opinion memcached is not a suitable tool for the purpose you've
mentioned. In fact it'd be waste of resources in such a case.
I would go for processing that thing in background using any of the
available gem for processing the background job.
Showing an interim screen to the user mentioning that your uploaded file
is
being processed and process it in background with the help of
ActiveRecord
+ DelayedJob.
Aa082c8b00a50928e5860dcd70bf2368?d=identicon&s=25 tamouse m. (tamouse_m)
on 2013-11-04 10:48
(Received via mailing list)
On Nov 4, 2013, at 2:42 AM, Sur Max <sur.max@gmail.com> wrote:

> @Fahim
>
> In my opinion memcached is not a suitable tool for the purpose you've mentioned.
In fact it'd be waste of resources in such a case.
> I would go for processing that thing in background using any of the available
gem for processing the background job.
> Showing an interim screen to the user mentioning that your uploaded file is
being processed and process it in background with the help of ActiveRecord +
DelayedJob.
>

I tend to agree. ActiveRecord does not do well for mass insert/updates,
and better to offload this task and not keep the Rails stack busy with
it.

I also wouldnt hold up the user with a wait screen. Id put a little
indicator in an area that shows the processing is being done but does
not restrict further use of the site, except for that new data. When the
background processing is finished, the indicator can be removed.
576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-04 11:34
(Received via mailing list)
@sur and @tam thanks for reply....

@sur -  Currently I am using rescue ....I am using worker's but still I
am
thinking to use memcache for fast performance....What you say can I
achieve
validation on cache level.

@tam - I even showing processing image till process done....

Friends ....can i achieve fast performance through putting validation on
cache level .....or I have to assume that this is max performance
code....
On Nov 4, 2013, at 2:42 AM, Sur Max <sur...@gmail.com> wrote:

> @Fahim
>
> In my opinion memcached is not a suitable tool for the purpose you've
mentioned. In fact it'd be waste of resources in such a case.
> I would go for processing that thing in background using any of the
available gem for processing the background job.
> Showing an interim screen to the user mentioning that your uploaded file
is being processed and process it in background with the help of
ActiveRecord + DelayedJob.
>

I tend to agree. ActiveRecord does not do well for mass insert/updates,
and
better to offload this task and not keep the Rails stack busy with it.

I also wouldnt hold up the user with a wait screen. Id put a little
indicator in an area that shows the processing is being done but does
not
restrict further use of the site, except for that new data. When the
background processing is finished, the indicator can be removed.

>
> On Monday, 4 November 2013 00:26:08 UTC+5:30, Fahim Patel wrote:
> Thanks for reply.
> @josh - So you are saying that we can validate request on memcache level
also. Please correct me If I am wrong ?
>
>
> On Sunday, November 3, 2013 1:46:20 AM UTC+5:30, Fahim Patel wrote:
> Hi All ,
>
> Can we validate request input data using memcache ? Actually I am facing
scenario in which user uploading CSV file which can have hundreds of
rows
and to validate each row using ActiveRecord taking long time.
>
> I have below question:-
>
> 1. So can we cache our validation rule in memcache ?
> 2. If it is possible to store validation rules in memcache but can I able
to perform
> validation check ?
>
>
> --
> You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to rubyonrails-ta...@googlegroups.com.
> To post to this group, send email to rubyonra...@googlegroups.com.
> To view this discussion on the web visit
https://groups.google.com/d/msgid/rubyonrails-talk...
.
Aa082c8b00a50928e5860dcd70bf2368?d=identicon&s=25 tamouse m. (tamouse_m)
on 2013-11-04 11:47
(Received via mailing list)
On Nov 4, 2013, at 4:32 AM, Fahim Patel <pafahim@gmail.com> wrote:

> @sur and @tam thanks for reply....
>
> @sur -  Currently I am using rescue ....I am using worker's but still I am
thinking to use memcache for fast performance....What you say can I achieve
validation on cache level.
>
> @tam - I even showing processing image till process done....
>
> Friends ....can i achieve fast performance through putting validation on cache
level .....or I have to assume that this is max performance code.

Heres what I dont get: what could you possibly be caching here? If whats
happening is a user is uploading a CSV file that is going to inserted
into table, what is going to be cached?
576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-04 13:04
(Received via mailing list)
Thanks @tamouse for  reply.
Actually my table consist more than 15 columns and each column have lots
of
validation like presence validation,format validation.
Major issue is user will upload CSV file which can consist hundred's of
rows.Through active record validating each row consuming lots of time.

That's why I am thinking to move validation rule to cache level. It will
fast process.

Hope you understood situation ....
- show quoted text -

On Nov 4, 2013, at 4:32 AM, Fahim Patel <paf...@gmail.com> wrote:

> @sur and @tam thanks for reply....
>
> @sur -  Currently I am using rescue ....I am using worker's but still I
am thinking to use memcache for fast performance....What you say can I
achieve validation on cache level.
>
> @tam - I even showing processing image till process done....
>
> Friends ....can i achieve fast performance through putting validation on
cache level .....or I have to assume that this is max performance code.

Heres what I dont get: what could you possibly be caching here? If whats
happening is a user is uploading a CSV file that is going to inserted
into
table, what is going to be cached?
- show quoted text -
> To view this discussion on the web visit
https://groups.google.com/d/msgid/rubyonrails-talk...
.
- show quoted text -
Ce43b11af679aaa142f5ba7e35e64c4b?d=identicon&s=25 Sur Max (sur)
on 2013-11-04 13:17
(Received via mailing list)
> Thanks @tamouse for  reply.
> Actually my table consist more than 15 columns and each column have lots
> of validation like presence validation,format validation.
> Major issue is user will upload CSV file which can consist hundred's of
> rows.Through active record validating each row consuming lots of time.
>
> That's why I am thinking to move validation rule to cache level. It will
> fast process.
>
>
If I understood you correctly,

Validation rule resides inside a Ruby class i.e. a class inheriting
ActiveRecord::Base, which is already there in memory to even be
functional.
It doesn't make any other sense to cache this class (or rules inside the
class). Further, Rails automatically caches all the classes in
production
environment. Do you correctly understand this ?

In short, your say "I am thinking to move validation rule to cache
level"
itself is incorrect because the validation is already in memory and is
already cached, you can not speed it up further by saying you want to
cache
it.

And for processing millions of rows with 15 columns each row, by any
chance
it surely is going to be a slow procedure even with memcached (unless
you
want to put a super computer in place for this task)... and the
practical
solution is to move this to background job.

or, I didn't understand you at all.


regards,
Sur
crimson9.com
576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-04 13:29
(Received via mailing list)
Thanks @sur for reply.....
Yes you understood correctly my issue, and I even understood your
solution....
let me cross check your solution and will update here.


If I understood you correctly,

Validation rule resides inside a Ruby class i.e. a class inheriting
ActiveRecord::Base, which is already there in memory to even be
functional.
It doesn't make any other sense to cache this class (or rules inside the
class). Further, Rails automatically caches all the classes in
production
environment. Do you correctly understand this ?

In short, your say "I am thinking to move validation rule to cache
level"
itself is incorrect because the validation is already in memory and is
already cached, you can not speed it up further by saying you want to
cache
it.

And for processing millions of rows with 15 columns each row, by any
chance
it surely is going to be a slow procedure even with memcached (unless
you
want to put a super computer in place for this task)... and the
practical
solution is to move this to background job.

or, I didn't understand you at all.


regards,
Sur
crimson9.com
Aa082c8b00a50928e5860dcd70bf2368?d=identicon&s=25 tamouse m. (tamouse_m)
on 2013-11-04 23:50
(Received via mailing list)
On Nov 4, 2013, at 6:02 AM, Fahim Patel <pafahim@gmail.com> wrote:

> Thanks @tamouse for  reply.
> Actually my table consist more than 15 columns and each column have lots of
validation like presence validation,format validation.
> Major issue is user will upload CSV file which can consist hundred's of
rows.Through active record validating each row consuming lots of time.
>
> That's why I am thinking to move validation rule to cache level. It will fast
process.

How do you move a validation rule to memcache?
576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-05 06:17
(Received via mailing list)
@tam - I am even not sure how can experiment it.  Collecting data and
studying in progress.
@sur - I am still  cross checking.. If you provide any reference link or
data to proof that validation already get cached.

Friends and @sur  please clear me on below question-
From rails app, when user insert data then each time data hitted on
database for validation or validation is present in cache ?

If validation get cached then we don't need move validation on cache.
Ce5e201eca55d4f79d0ead9acc91d544?d=identicon&s=25 Dheeraj Kumar (Guest)
on 2013-11-05 06:25
(Received via mailing list)
This is a great use case for a background job. Have you tried Resque?

If I have a user uploading thousands of rows, all of which need to be
validated and processed/saved, Id do it like this:
Let the user upload the file, save it somewhere. Trigger a Resque job.
Read the file one or more rows at a time, make a ActiveRecord or
ActiveModel object out of them and run the validators. Collate the
errors and include the row number.
If all the rows were valid, continue processing or saving.
If not, send an email to the uploader and/or show in his dashboard the
failure message, along with a list of all errors with row numbers.

Ive used this method in our app, and is working out great for our users.



--
Dheeraj Kumar
576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-05 06:55
(Received via mailing list)
@dheeraj - thanks for reply.I am using resque & redis. You  exactly got
use
case .
can you please give answer for below point:-
1. From rails app, when user insert data then each time data hitted on
database for validation or validation is present in cache ?
Ce5e201eca55d4f79d0ead9acc91d544?d=identicon&s=25 Dheeraj Kumar (Guest)
on 2013-11-05 07:05
(Received via mailing list)
First validate the data, and then insert it to the database. Till then,
its read from the file and still in-memory.

Im not bringing the cache into the picture at all.

--
Dheeraj Kumar
Ce43b11af679aaa142f5ba7e35e64c4b?d=identicon&s=25 Sur Max (sur)
on 2013-11-05 07:09
(Received via mailing list)
Well, this is a classic use case when we jump too early on Rails not
understanding the Ruby (or programming in general) to at least a decent
level.... let alone the meta-programming stuff.

One should get a good hold of Ruby at least to a level where s/he
understands how the memory mapping is working behind the scenes...
otherwise you can end up writing the piles of junk which is eating up
all
your server's memory.

@tam - I am even not sure how can experiment it.  Collecting data and
> studying in progress.
> @sur - I am still  cross checking.. If you provide any reference link or
> data to proof that validation already get cached.
>

"If you provide any reference link or data to proof that validation
already
get cached."
I got a smile when I read the above line. :D

a humble suggestion:
before finding any proof, you gotta study more about programming and how
the memory space is managed on the runtime.

Do you know what is the difference between a string and a symbol and why
and where symbols are preferred over strings ?

try this:

a = "some string"
b = "some string"
b.object_id == a.object_id  (watch for the result of this expression)

sym_a = :some_symbol
sym_b = :some_symbol
sym_b.object_id == sym_a.object_id  (watch for the result of this
expression)


Now, you don't need to go too far for finding the proof for the
validation/class caching:
(let alone the Rails production environment, in one process of Ruby, if
Ruby can't cache the class why would you even use it ?)

user_class = class User; end
another_user_class = class User; end

user_class.object_id == another_user_class.object_id  (watch for the
result
of this expression)


*Note:- object_id is the method available to every single object in Ruby
(remember every smallest/biggest entity in Ruby you have is an
object)...
and it returns the id of that object from the memory map.*


Play as much as possible with the above snippets in IRB. IRB is
interactive
Ruby console, you can initiate it by typing "irb" in your terminal and
then
can type the above snippets to test there.
Try tasting the flavor of Ruby to a bit of the next levels.



regards,
Sur
crimson9.com
52f3528c40e9cf28ad0900886eecb128?d=identicon&s=25 Jordon Bedwell (Guest)
on 2013-11-05 07:22
(Received via mailing list)
On Tue, Nov 5, 2013 at 12:07 AM, Sur <sur.max@gmail.com> wrote:
> Do you know what is the difference between a string and a symbol and why and
> where symbols are preferred over strings ?
>
> try this:
>
> a = "some string"
> b = "some string"
> b.object_id == a.object_id  (watch for the result of this expression)

> sym_a = :some_symbol
> sym_b = :some_symbol
> sym_b.object_id == sym_a.object_id  (watch for the result of this
> expression)

Preach on about this bullshit when Ruby isn't primarily used for web
applications and when Ruby allows my system to reclaim memory, until
then lets leave the symbol argument bullshit out of this.  Even if
they are constants they can lead to trouble if you just blindly state
they are better without explaining the dangers in a web application,
or people will repeat history, history people like me thought was
already common knowledge.
576908ab294e24dadb3303a6667b8776?d=identicon&s=25 fahim patel (fahim_patel)
on 2013-11-05 07:42
(Received via mailing list)
@sur - will check as you said.
@jordan - not understood your reply..
Ce43b11af679aaa142f5ba7e35e64c4b?d=identicon&s=25 Sur Max (sur)
on 2013-11-05 07:44
(Received via mailing list)
> Preach on about this bullshit when Ruby isn't primarily used for web
> applications and when Ruby allows my system to reclaim memory, until
> then lets leave the symbol argument bullshit out of this.  Even if
> they are constants they can lead to trouble if you just blindly state
> they are better without explaining the dangers in a web application,
> or people will repeat history, history people like me thought was
> already common knowledge.


:)

well it'd be interesting to know why you're frustrated out of Symbol
concepts of Ruby ?
I didn't say symbols are better than strings, in fact I asked that it's
important to know for a developer that why and where they can be used!

when did I say Ruby is best/primary used for web ? I use it in chip
programming and it's been great so far :)

It's an individual's choice to select a language/framework to achieve
the
solution to a problem, the main aim is to resolve a problem, now once
decided to go with a language, one hast to know the limits and pros and
cons of that language which certainly are there the way it's creators
have
made it to.

If someone doesn't know how the class is being loaded once in memory
clearly means there is huge lack of knowledge about how memory mappings
going on there. And getting insights of symbols is a great way to get to
know more about this.


regards,
Sur
crimson9.com
52f3528c40e9cf28ad0900886eecb128?d=identicon&s=25 Jordon Bedwell (Guest)
on 2013-11-05 08:00
(Received via mailing list)
On Tue, Nov 5, 2013 at 12:42 AM, Sur <sur.max@gmail.com> wrote:
> well it'd be interesting to know why you're frustrated out of Symbol
> concepts of Ruby ?

It's hard to explain, fight the good fight long enough you might see
it or you might not, I won't force my views on you.

> I didn't say symbols are better than strings, in fact I asked that it's
> important to know for a developer that why and where they can be used!

It was implied (to me.)

> It's an individual's choice to select a language/framework to achieve the
> solution to a problem, the main aim is to resolve a problem, now once
> decided to go with a language, one hast to know the limits and pros and cons
> of that language which certainly are there the way it's creators have made
> it to.

That's a fallacy as most of the time people do not have this sort of
flexibility, just because you hear all about it on Twitter and from
CTO's and startup's does not mean this is the case most of the time,
if it was then none of us would still be on Rails 2.3 with Ruby 1.8.
While this is not me, I know more developers who get told what they
are going to use then telling what they would like to use and then
using it.

> If someone doesn't know how the class is being loaded once in memory clearly
> means there is huge lack of knowledge about how memory mappings going on
> there. And getting insights of symbols is a great way to get to know more
> about this.

I think this leaves a lot to be explained because you explicitly
mention classes and I do believe this "loaded once in memory" only
applies to the singleton and since classes are meant to be initialized
there could remain many copies of that object with their own state at
the end of the day.  I could be wrong though.
Aa082c8b00a50928e5860dcd70bf2368?d=identicon&s=25 tamouse m. (tamouse_m)
on 2013-11-05 20:24
(Received via mailing list)
On Nov 5, 2013, at 12:20 AM, Jordon Bedwell <envygeeks@gmail.com> wrote:

>> sym_a = :some_symbol
> already common knowledge.
>
>

Wow, is this response ever out of context. The example Sur supplied had
to do with how code is stored in memory, IN RESPONSE TO the question of
whether code is in memory or not (see "caching").
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.