# Redirect on specific threshold! **URL:** \ **Category:** NGINX\ **Created:** [June 14, 2015, 7:13pm UTC](https://www.ruby-forum.com/t/redirect-on-specific-threshold/242310 "2015-06-14T19:13:07Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![musicdenotation](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ea666f/32.png) [@musicdenotation](https://www.ruby-forum.com/u/musicdenotation)\ **Post date:** [June 14, 2015, 7:13pm UTC](https://www.ruby-forum.com/t/redirect-on-specific-threshold/242310/1 "2015-06-14T19:13:07Z")
Hi, We’re using Nginx to serve videos on one of our Storage server(contains mp4 videos) and due to high amount of requests we’re planning to have a separate caching Node based on Fast SSD drives to serve “Hot” content in order to reduce load from Storage. We’re planning to have following method for caching : If there are exceeding 1K requests for [http://storage.domain.com/test.mp4](http://storage.domain.com/test.mp4) , nginx should construct a Redirect URL for rest of the requests related to test.mp4 i.e [http://cache.domain.com/test.mp4](http://cache.domain.com/test.mp4) and entertain the rest of requests for test.mp4 from Caching Node while long tail would still be served from storage. So, can we achieve this approach with nginx or other like varnish ? Thanks in advance. Regards. Shahzaib ---
**Author:** ![shahzaib12](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/90db22/32.png) [@shahzaib12](https://www.ruby-forum.com/u/shahzaib12)\ **Post date:** [June 14, 2015, 11:21pm UTC](https://www.ruby-forum.com/t/redirect-on-specific-threshold/242310/2 "2015-06-14T23:21:58Z")
Hi, On 15/06/15 05:12, shahzaib shahzaib wrote: > Redirect URL for rest of the requests related to test.mp4 i.e On the assumption that you’re hosting with a linux infrastructure, this is simply done by just adding more memory! By default any spare memory will be used to cache common files. If you want more control over it, then set your cache area up on a tmpfs backed partition. However, you’ll then have to manage what you cache yourself. With setups like this, it’s normally the bandwidth of the network that becones the bottleneck. Maybe a bit of round-robin DNS would help with this? Steve – Steve H. BSc(Hons) MIITP > **[Linux & Admin Support Services | "Services provided by people who are proud...](https://www.greengecko.co.nz/)** > > Offers linux support services, systems and database administration, consultancy and the full development lifecycle for open source - based solutions Linkedin: [Steve Holdoway - Robotron New Zealand Limited | LinkedIn](http://www.linkedin.com/in/steveholdoway) Skype: sholdowa ---
**Author:** ![shahzaib12](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/90db22/32.png) [@shahzaib12](https://www.ruby-forum.com/u/shahzaib12)\ **Post date:** [June 15, 2015, 6:15am UTC](https://www.ruby-forum.com/t/redirect-on-specific-threshold/242310/3 "2015-06-15T06:15:04Z")
Does a nginx reverse proxy with cache fit you need? Client → Caching server (with SSD and nginx proxy cache configured) → Storage server(s) (Slow) You can add even more storage server by utilizing nginx upstream module. On Sun, Jun 14, 2015 at 1:12 PM shahzaib shahzaib [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) ---
**Author:** ![shahzaib12](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/90db22/32.png) [@shahzaib12](https://www.ruby-forum.com/u/shahzaib12)\ **Post date:** [June 15, 2015, 12:46pm UTC](https://www.ruby-forum.com/t/redirect-on-specific-threshold/242310/4 "2015-06-15T12:46:28Z")
On Sunday 14 June 2015 22:12:37 shahzaib shahzaib wrote: > test.mp4 i.e [http://cache.domain.com/test.mp4](http://cache.domain.com/test.mp4) and entertain the rest of > requests for test.mp4 from Caching Node while long tail would still be > served from storage. > > So, can we achieve this approach with nginx or other like varnish ? [..] You can use limit\_conn and limit\_req modules to set limits: > **[Module ngx\_http\_limit\_conn\_module](http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html)** > **[Module ngx\_http\_limit\_req\_module](http://nginx.org/en/docs/http/ngx_http_limit_req_module.html)** and the error\_page directive to construct the redirect. wbr, Valentin V. Bartenev ---
**Author:** ![shahzaib12](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/90db22/32.png) [@shahzaib12](https://www.ruby-forum.com/u/shahzaib12)\ **Post date:** [June 15, 2015, 9:07am UTC](https://www.ruby-forum.com/t/redirect-on-specific-threshold/242310/5 "2015-06-15T09:07:59Z")
Hi, Thanks for the help guys. Regarding @ryd994 suggestion, the reason we don’t want to deploy this structure is that the Caching node will have to respond for each client’s request and even it will be only doing proxy for most of the requests(without caching them), high i/o will still be required to serve the big proxy request(700MB mp4) to the user and that way caching node will eventually become the bottleneck between user and storage node, isn’t it ? @steve thanks for tmpfs point. but we’re using caching node with 1TB+ SSD storage and will prefer SSD cache over RAM(though RAM is faster but not as big as SSD). Using redirect URL we believe would be only pointing specific requests towards the cachind node and than this node will fetch requested file using proxy\_cache. Regards. Shahzaib. ---
**Author:** ![shahzaib12](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/90db22/32.png) [@shahzaib12](https://www.ruby-forum.com/u/shahzaib12)\ **Post date:** [August 29, 2015, 1:58pm UTC](https://www.ruby-forum.com/t/redirect-on-specific-threshold/242310/6 "2015-08-29T13:58:02Z")
Hi, Sorry got back to this thread after long time. First of all, thanks to all for suggestions. Alright, i have also checked with rate\_limit module, should this work as well or it should be only limit\_conn (to parse error\_log and constructing redirect URL). P.S : Actuall looks like limit\_conn needs to recompile nginx as it is not included in default yum install nginx repo. So i tried with rate\_limit which is built-in within nginx repo. [http://greenroom.com.my/blog/2014/10/rate\_limit-with-nginx-on-ubuntu/](http://greenroom.com.my/blog/2014/10/rate_limit-with-nginx-on-ubuntu/) Regards. Shahzaib ---
**Author:** ![shahzaib12](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/90db22/32.png) [@shahzaib12](https://www.ruby-forum.com/u/shahzaib12)\ **Post date:** [June 16, 2015, 11:31pm UTC](https://www.ruby-forum.com/t/redirect-on-specific-threshold/242310/7 "2015-06-16T23:31:01Z")
On Mon, Jun 15, 2015 at 01:45:42PM +0300, Valentin V. Bartenev wrote: > On Sunday 14 June 2015 22:12:37 shahzaib shahzaib wrote: Hi there, > > If there are exceeding 1K requests for [http://storage.domain.com/test.mp4](http://storage.domain.com/test.mp4) , > > nginx should construct a Redirect URL for rest of the requests related to > > test.mp4 i.e [http://cache.domain.com/test.mp4](http://cache.domain.com/test.mp4) and entertain the rest of > > requests for test.mp4 from Caching Node while long tail would still be > > served from storage. > You can use limit\_conn and limit\_req modules to set limits: > [Module ngx\_http\_limit\_conn\_module](http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html) > [Module ngx\_http\_limit\_req\_module](http://nginx.org/en/docs/http/ngx_http_limit_req_module.html) > > and the error\_page directive to construct the redirect. limit\_conn and limit\_req are the right answer if you care about concurrent requests. (For example: rate=1r/m with burst=1000 might do most of what you want, without too much work on your part.) I think you might care about historical requests, instead – so if a url is ever accessed 1K times, then it is “popular” and future requests should be redirected. To do that, you probably will find it simpler to do it outside of nginx, at least initially. Have something read the recent-enough log files[\*], and whenever there are more that 1K requests for the same resource, add a fragment like location = /test.mp4 { return 301 [http://cache.domain.com/test.mp4](http://cache.domain.com/test.mp4); } to nginx.conf (and remove similar fragments that are no longer currently popular-enough, if appropriate), and do a no-downtime config reload. You can probably come up with a module or a code config that does the same thing, but I think it would take me longer to do that. [\*] or accesses the statistics by a method of your choice ## f Francis D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) ---
**Author:** ![shahzaib12](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/90db22/32.png) [@shahzaib12](https://www.ruby-forum.com/u/shahzaib12)\ **Post date:** [September 1, 2015, 11:37pm UTC](https://www.ruby-forum.com/t/redirect-on-specific-threshold/242310/8 "2015-09-01T23:37:39Z")
On Sat, Aug 29, 2015 at 04:57:19PM +0500, shahzaib shahzaib wrote: Hi there, > Sorry got back to this thread after long time. First of all, thanks to > all for suggestions. Alright, i have also checked with rate\_limit module, > should this work as well or it should be only limit\_conn (to parse > error\_log and constructing redirect URL). I think the answers already given were different, depending on different understanding of your requirements. Perhaps if you can re-state (or clarify) them, you will get a more specific answer. For what it’s worth, what I think you want is a tool to read the access logs from [storage.domain.com](http://storage.domain.com), copy files to [cache.domain.com](http://cache.domain.com), change the nginx config on [storage.domain.com](http://storage.domain.com), and restart nginx on [storage.domain.com](http://storage.domain.com). None of which involves any special modules or config within nginx. ## f Francis D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)