I need to make an “architecture” choice for my app. I believe it
corresponds to a fairly common case, yet I can’t find much input
online. I was hoping you could give me some recommendations or point
me in the right direction.
The app is a kind of mashup, so it does two things:
- handle users requests. I call it the “live” part.
- at regular intervals, build cache information, then merge it with
the main info. It is my “background” part.
The background part will run once a day every day, and it takes a lot
of time and resources (it connects to many web services).
So here are my initial thoughts:
- create a background process for the “background” part
- have the main process handle the “live” part.
But if I do this, a few things are not obvious to me, especially since
it involves threads:
1- in the background process, I will have to run threads (indeed I
need to run the requests to the web services in parallel, otherwise it
will take too long). Is that doable or is it “dangerous”?
2- can the cache tables be in the same database as the main one, or
shall I create a totally different DB?
3- when the merging happens, will it “work” (since both the background
process and the main one will want to deal with the same database)?
Please do let me know if I am getting the whole thing wrong (or if I
see problems where they don’t exist). I keep reading that I need to be
very careful with threads/processes and connections to the DB, but I
am not quite sure where is the limit and what are very bad design
Thanks a lot!