Forum: Mongrel problem in downloading large file with apache+mongrel

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 15:01
hii all,
am using apache as frontend to mongrel and loaded xsendfile
module in apache ,am looking for 1 GB of download .this is download
routine

def download

@stream = Stream.find(params[:id])

send_file(@stream.location,:filename => @stream.name,:disposition >
'attachment',:x_sendfile => true )

and

And this is how it is processed

Processing StreamsController#download (for 127.0.0.1 at 2010-09-22
12:32:10) [GET] Parameters: {"id"=>"6596"}

<-[4;36;1mSQL (0.0ms)?[0m ?[0;1mSET NAMES 'utf8'?[0m

<-[4;35;1mSQL (0.0ms)?[0m ?[0mSET SQL_AUTO_IS_NULL=0?[0m

<-[4;36;1mStream Columns (15.0ms)?[0m ?[0;1mSHOW FIELDS FROM streams?[0m

<-[4;35;1mStream Load (0.0ms)?[0m ?[0mSELECT * FROM streams WHERE
(streams.id = 6596) ?[0m

<-[4;36;1mCACHE (0.0ms)?[0m ?[0;1mSELECT * FROM streams WHERE
(streams.id = 596) ? ?[0m

Sending X-Sendfile header
d:/dm/predator_720x480_5mbps_30fps_17minclip.264.filepart Completed in
57513ms (View: 0, DB: 15) | 200 OK [http://src/streams/download/6596]

very first times its okk i getting full 1.1 GB of file getting
downloaded,but when i am trying to download this 1.1 gb of file again,
am getting failed to allocate memroy what could be soultion?? and can i
use cache.clear before send_file function??
and more thing to again download 1.1 gb of file i have to restart my
servers..

thanks??
E7cff3cfd41c495e1012227d7dc24202?d=identicon&s=25 Luis Lavena (luislavena)
on 2010-09-22 15:06
(Received via mailing list)
On Wed, Sep 22, 2010 at 10:01 AM, Amit Tomar <lists@ruby-forum.com>
wrote:
> hii all,
> am using apache as frontend to mongrel and loaded xsendfile
> module in apache ,am looking for 1 GB of download .this is download
> routine
>

Replied your previous message to the list also. Please provide the
answers to my questions.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 15:50
Luis Lavena wrote:
> On Wed, Sep 22, 2010 at 10:01 AM, Amit Tomar <lists@ruby-forum.com>
> wrote:
>> hii all,
>> am using apache as frontend to mongrel and loaded xsendfile
>> module in apache ,am looking for 1 GB of download .this is download
>> routine
>>
>
> Replied your previous message to the list also. Please provide the
> answers to my questions.
> --
> Luis Lavena
> AREA 17
> -
> Perfection in design is achieved not when there is nothing more to add,
> but rather when there is nothing more to take away.
> Antoine de Saint-Exupé²¹


just now i seen your message...
i am running apache 2.2.11 that comes with XAMPP 1.7.1 and mongrel
version is
1.1.2,
ruby version is ruby 1.8.6

luis you are saying



Rails is serving the request, the 1.1GB, not Apache
but how do i know rails is serving the request..or Apache


2. d:/dm  is not a network drive..its folder on d drive
how do i know whther it available  to apache ??

i am sending you access logs of apache after few mintues
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 15:58

when i am trying to downnload following happen in logs of apache

127.0.0.1 - - [22/Sep/2010:19:11:12 +0530] "GET /streams/download/6598
HTTP/1.1" 500 1024 "http://src/streams/6598" "Mozilla/5.0 (Windows; U;
Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko)
Chrome/6.0.472.33 Safari/534.3"

where 6598 is id of file to be downloaded
E7cff3cfd41c495e1012227d7dc24202?d=identicon&s=25 Luis Lavena (luislavena)
on 2010-09-22 16:06
(Received via mailing list)
On Wed, Sep 22, 2010 at 10:50 AM, Amit Tomar <lists@ruby-forum.com>
wrote:
> just now i seen your message...
> i am running apache 2.2.11 that comes with XAMPP 1.7.1 and mongrel
> version is
> 1.1.2,
> ruby version is ruby 1.8.6
>
>
> Rails is serving the request, the 1.1GB, not Apache
> but how do i know rails is serving the request..or Apache
>

Please answer all the questions I put, analyze the process memory usage.

If you don't see in apache serving the X-Sendfile request, then is not.


Try setting the headers on your own and render nothing:

Implement the first code example of this page:

http://john.guen.in/past/2007/4/17/send_files_fast...

Not the plugin.

> 2. d:/dm  is not a network drive..its folder on d drive
> how do i know whther it available  to apache ??
>

Try to setup a virtual server configuration that access D: and see if it
works.


--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 16:22

soory but how do i setup s virtual server configuration??
and i have to install that plugin??
E7cff3cfd41c495e1012227d7dc24202?d=identicon&s=25 Luis Lavena (luislavena)
on 2010-09-22 16:37
(Received via mailing list)
On Wed, Sep 22, 2010 at 11:22 AM, Amit Tomar <lists@ruby-forum.com>
wrote:
>
> soory but how do i setup s virtual server configuration??

Read apache documentation.

> and i have to install that plugin??

No, in your controller action set the headers, *do not* install the
plugin.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 17:00
Luis Lavena wrote:
> On Wed, Sep 22, 2010 at 11:22 AM, Amit Tomar <lists@ruby-forum.com>
> wrote:
>>
>> soory but how do i setup s virtual server configuration??
>
> Read apache documentation.
>
>> and i have to install that plugin??
>
> No, in your controller action set the headers, *do not* install the
> plugin.
> --
> Luis Lavena
> AREA 17
> -
> Perfection in design is achieved not when there is nothing more to add,
> but rather when there is nothing more to take away.
> Antoine de Saint-Exupé²¹

luis i change my code as you said

filename =  "#{@stream.location}"
response.headers['Content-Type'] = "application/force-download"
 response.headers['Content-Disposition'] = "attachment;
filename=\"#{File.basename(filename)}\""
response.headers["X-Sendfile"] = filename
response.headers['Content-length'] = File.size(filename)
render :nothing => true


this is how it is processed...
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 17:01
Amit Tomar wrote:
> Luis Lavena wrote:
>> On Wed, Sep 22, 2010 at 11:22 AM, Amit Tomar <lists@ruby-forum.com>
>> wrote:
>>>
>>> soory but how do i setup s virtual server configuration??
>>
>> Read apache documentation.
>>
>>> and i have to install that plugin??
>>
>> No, in your controller action set the headers, *do not* install the
>> plugin.
>> --
>> Luis Lavena
>> AREA 17
>> -
>> Perfection in design is achieved not when there is nothing more to add,
>> but rather when there is nothing more to take away.
>> Antoine de Saint-Exupé²¹
>
> luis i change my code as you said
>
> filename =  "#{@stream.location}"
> response.headers['Content-Type'] = "application/force-download"
>  response.headers['Content-Disposition'] = "attachment;
> filename=\"#{File.basename(filename)}\""
> response.headers["X-Sendfile"] = filename
> response.headers['Content-length'] = File.size(filename)
> render :nothing => true
>
>
> this is how it is processed...

Processing StreamsController#download (for 127.0.0.1 at 2010-09-22
20:22:06) [GET]
  Parameters: {"id"=>"6600"}
  ?[4;36;1mSQL (0.0ms)?[0m   ?[0;1mSET NAMES 'utf8'?[0m
  ?[4;35;1mSQL (0.0ms)?[0m   ?[0mSET SQL_AUTO_IS_NULL=0?[0m
  ?[4;36;1mStream Columns (0.0ms)?[0m   ?[0;1mSHOW FIELDS FROM
`streams`?[0m
  ?[4;35;1mStream Load (0.0ms)?[0m   ?[0mSELECT * FROM `streams` WHERE
(`streams`.`id` = 6600) ?[0m
  ?[4;36;1mCACHE (0.0ms)?[0m   ?[0;1mSELECT * FROM `streams` WHERE
(`streams`.`id` = 6600) ?[0m
0) ?[0m
Completed in 65347ms (View: 0, DB: 0) | 200 OK
[http://src/streams/download/6600]
E7cff3cfd41c495e1012227d7dc24202?d=identicon&s=25 Luis Lavena (luislavena)
on 2010-09-22 17:15
(Received via mailing list)
On Wed, Sep 22, 2010 at 12:00 PM, Amit Tomar <lists@ruby-forum.com>
wrote:
>
> this is how it is processed...
>

This? How? can you provide answers to my previous questions about
process memory usage? Please?

--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 17:17
Amit Tomar wrote:
> Amit Tomar wrote:
>> Luis Lavena wrote:
>>> On Wed, Sep 22, 2010 at 11:22 AM, Amit Tomar <lists@ruby-forum.com>
>>> wrote:
>>>>
>>>> soory but how do i setup s virtual server configuration??
>>>
>>> Read apache documentation.
>>>
>>>> and i have to install that plugin??
>>>
>>> No, in your controller action set the headers, *do not* install the
>>> plugin.
>>> --
>>> Luis Lavena
>>> AREA 17
>>> -
>>> Perfection in design is achieved not when there is nothing more to add,
>>> but rather when there is nothing more to take away.
>>> Antoine de Saint-Exupé²¹
>>
>> luis i change my code as you said
>>
>> filename =  "#{@stream.location}"
>> response.headers['Content-Type'] = "application/force-download"
>>  response.headers['Content-Disposition'] = "attachment;
>> filename=\"#{File.basename(filename)}\""
>> response.headers["X-Sendfile"] = filename
>> response.headers['Content-length'] = File.size(filename)
>> render :nothing => true
>>
>>
>> this is how it is processed...
>
> Processing StreamsController#download (for 127.0.0.1 at 2010-09-22
> 20:22:06) [GET]
>   Parameters: {"id"=>"6600"}
>   ?[4;36;1mSQL (0.0ms)?[0m   ?[0;1mSET NAMES 'utf8'?[0m
>   ?[4;35;1mSQL (0.0ms)?[0m   ?[0mSET SQL_AUTO_IS_NULL=0?[0m
>   ?[4;36;1mStream Columns (0.0ms)?[0m   ?[0;1mSHOW FIELDS FROM
> `streams`?[0m
>   ?[4;35;1mStream Load (0.0ms)?[0m   ?[0mSELECT * FROM `streams` WHERE
> (`streams`.`id` = 6600) ?[0m
>   ?[4;36;1mCACHE (0.0ms)?[0m   ?[0;1mSELECT * FROM `streams` WHERE
> (`streams`.`id` = 6600) ?[0m
> 0) ?[0m
> Completed in 65347ms (View: 0, DB: 0) | 200 OK
> [http://src/streams/download/6600]

and these are the lines i added in httpd.conf
<Directory "d:/dm">
Options Indexes FollowSymLinks
AllowOverride FileInfo
Order allow,deny
Allow from all
</Directory>


for virtual server configuration to  access D:
what should i do now
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 17:19
yaa am looking to provide answer for that also memory usage
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 17:29
Amit Tomar wrote:
>
> yaa am looking to provide answer for that also memory usage

Luis when i download file
httpd.exe  is varied from at 49,892k to 50,264k
while ruby.exe                934800 k to 1,208688k
E7cff3cfd41c495e1012227d7dc24202?d=identicon&s=25 Luis Lavena (luislavena)
on 2010-09-22 17:36
(Received via mailing list)
On Wed, Sep 22, 2010 at 12:01 PM, Amit Tomar <lists@ruby-forum.com>
wrote:
>  ?[4;36;1mCACHE (0.0ms)?[0m   ?[0;1mSELECT * FROM `streams` WHERE
> (`streams`.`id` = 6600) ?[0m
> 0) ?[0m
> Completed in 65347ms (View: 0, DB: 0) | 200 OK
> [http://src/streams/download/6600]

Rails log do not help, we need to know Apache headers, responses and
also mongrel headers sent to apache via the proxy.

Please use curl --head on mongrel to request the URL and provide the
complete output
Also do the same from apache.

Beyond that, I cannot help you more because I don't have time to
attempt replicate your environment and scenario.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 17:37
Amit Tomar wrote:
> Amit Tomar wrote:
>>
>> yaa am looking to provide answer for that also memory usage
>
> Luis when i download file
> httpd.exe  is varied from at 49,892k to 50,264k
> while ruby.exe                934800 k to 1,208688k

and louis as soon while getting downloded memoryuse of httd.exe is
getting decrerase to 768 k and the when the file completely downloded
memory use get down to 324 k
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-22 17:39
Amit Tomar wrote:
> Amit Tomar wrote:
>> Amit Tomar wrote:
>>>
>>> yaa am looking to provide answer for that also memory usage
>>
>> Luis when i download file
>> httpd.exe  is varied from at 49,892k to 50,264k
>> while ruby.exe                934800 k to 1,208688k
>
> and louis as soon while getting downloded memoryuse of httd.exe is
> getting decrerase to 768 k and the when the file completely downloded
> memory use get down to 324 k

yaa louis tahnks for your help.....
you must be very good person...
E7cff3cfd41c495e1012227d7dc24202?d=identicon&s=25 Luis Lavena (luislavena)
on 2010-09-22 17:47
(Received via mailing list)
On Wed, Sep 22, 2010 at 12:29 PM, Amit Tomar <lists@ruby-forum.com>
wrote:
> Amit Tomar wrote:
>>
>> yaa am looking to provide answer for that also memory usage
>
> Luis when i download file
> httpd.exe  is varied from at 49,892k to 50,264k
> while ruby.exe                934800 k to 1,208688k

That makes then clear that Ruby is trying to read the file, not apache.

Sorry, I can't help you more if you don't keep an structure
conversation and you investigate also.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
B19b02a49b433c9e2e6e6c43785d2bfb?d=identicon&s=25 James Tucker (Guest)
on 2010-09-22 17:47
(Received via mailing list)
On 22 Sep 2010, at 12:00, Amit Tomar wrote:
> luis i change my code as you said
>
> filename =  "#{@stream.location}"
> response.headers['Content-Type'] = "application/force-download"
> response.headers['Content-Disposition'] = "attachment;
> filename=\"#{File.basename(filename)}\""

The problem is here ^^^^

> response.headers["X-Sendfile"] = filename
> response.headers['Content-length'] = File.size(filename)

File.size should fail, as rails will be in Dir.pwd == Rails.root, and as
such, File.size(filename) for filename with no path should fail. This
should be raising Errno::ENOENT.
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-23 07:40
but louis,how do i force apache to read the file and this problem can
occur with Apache also??
4b1d69e2ac9037081497056f70196369?d=identicon&s=25 Amit Tomar (amit-singh)
on 2010-09-23 07:42
James Tucker wrote:
> On 22 Sep 2010, at 12:00, Amit Tomar wrote:
>> luis i change my code as you said
>>
>> filename =  "#{@stream.location}"
>> response.headers['Content-Type'] = "application/force-download"
>> response.headers['Content-Disposition'] = "attachment;
>> filename=\"#{File.basename(filename)}\""
>
> The problem is here ^^^^
>
>> response.headers["X-Sendfile"] = filename
>> response.headers['Content-length'] = File.size(filename)
>
> File.size should fail, as rails will be in Dir.pwd == Rails.root, and as
> such, File.size(filename) for filename with no path should fail. This
> should be raising Errno::ENOENT.

but tucker, i am getting failed to allocate memory and that is also of
second time....
B19b02a49b433c9e2e6e6c43785d2bfb?d=identicon&s=25 James Tucker (Guest)
on 2010-09-23 19:02
(Received via mailing list)
On 23 Sep 2010, at 02:42, Amit Tomar wrote:

>>
>>> response.headers["X-Sendfile"] = filename
>>> response.headers['Content-length'] = File.size(filename)
>>
>> File.size should fail, as rails will be in Dir.pwd == Rails.root, and as
>> such, File.size(filename) for filename with no path should fail. This
>> should be raising Errno::ENOENT.
>
> but tucker, i am getting failed to allocate memory and that is also of
> second time....

You confuse me for someone that doesn't read. You also clearly don't
understand "should", or what I've just told you.
This topic is locked and can not be replied to.