# Ngx\_upstream do not cleanup file buffers in subrequest

**URL:** <https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265>\
**Category:** NGINX\
**Created:** [March 8, 2011, 9:26pm UTC](https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265 "2011-03-08T21:26:06Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\
**Post date:** [March 8, 2011, 9:26pm UTC](https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265/1 "2011-03-08T21:26:06Z")

</div>

Hello,

Is it reasonable to add following check to not clean/discard fd in case  
if it’s sub request?

I developed nginx module and when it started to use tmp buffers under  
load I had following error:

[alert] 29663#0: \*45789 sendfile() failed (9: Bad file descriptor) while  
sending request to upstream, client: 10.1.24.14, server: hss, request:  
"POST

So that sub-request cleared file before main request reached it.  
Probably this bug should not appear if request body was discarded for  
sub request. But in case if you need request body for sub-request you  
should be sure that sub-request does not erases main request buffers.

– src/http/ngx\_http\_upstream.c\_orig 2011-03-08 08:01:20.000000000  
-0800  
+++ src/http/ngx\_http\_upstream.c 2011-03-08 08:01:41.000000000 -0800  
@@ -2033,7 +2033,7 @@ ngx\_http\_upstream\_send\_response(ngx\_http

```
 u->header_sent = 1;

```

- if (r-\>request\_body && r-\>request\_body-\>temp\_file) {

- if (r-\>request\_body && r-\>request\_body-\>temp\_file && r == r-\>main)  
{  
ngx\_pool\_run\_cleanup\_file(r-\>pool,  
r-\>request\_body-\>temp\_file-\>file.fd);  
r-\>request\_body-\>temp\_file-\>file.fd = NGX\_INVALID\_FILE;  
}

Posted at Nginx Forum:

> **[\[PATCH\] ngx\_upstream do not cleanup file buffers in subrequest](https://forum.nginx.org/read.php?2,181400,181400#msg-181400)**
>
> Hello, Is it reasonable to add following check to not clean/discard fd in case if it's sub request? I developed nginx module and when it started to use tmp buffers under load I had following error: \[alert\] 29663#0: \*45789 sendfile() failed (9: Bad...

---

<div class="post-metadata">

**Author:** ![magz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/47e85d/32.png) [@magz](https://www.ruby-forum.com/u/magz)\
**Post date:** [March 9, 2011, 12:17pm UTC](https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265/2 "2011-03-09T12:17:58Z")

</div>

Hello!

On Tue, Mar 08, 2011 at 03:24:47PM -0500, magz wrote:

> "POST
> 
> So that sub-request cleared file before main request reached it.  
> Probably this bug should not appear if request body was discarded for  
> sub request. But in case if you need request body for sub-request you  
> should be sure that sub-request does not erases main request buffers.

In official nginx, subrequests can’t appear before main  
request.

On the other hand, main request may clear body as well, and it  
won’t be available for subrequests as a result. This problem may  
be easily seen with ssi module. The other possible case when this  
problem may appear is filter finalization (e.g. as done by image  
filter module).

That is, just disabling body cleanup for subrequests is clearly  
not enough.

I’ve already posted the patch which removes the code in question  
at all (as a reply to some real problems seen with ssi and POSTs,  
somewhere in russian mailing list). Probably it should be made  
controllable via some configuration directive if there are some  
real use cases which may be seriously affected by removing this  
code. I would like to hear something from Igor about this.

BTW, just for history: the code in question was introduced in  
nginx 0.3.3 with the following log message:

```
*) Bugfix: a temporary file with client request body now is removed
   just after the response header was transferred to a client.

```

Maxim D.

---

<div class="post-metadata">

**Author:** ![magz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/47e85d/32.png) [@magz](https://www.ruby-forum.com/u/magz)\
**Post date:** [March 9, 2011, 7:48pm UTC](https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265/3 "2011-03-09T19:48:59Z")

</div>

## Maxim D. Wrote:

> > So that sub-request cleared file before main  
> > request.
> 
> controllable via some configuration directive if  
> \*) Bugfix: a temporary file with client  
> request body now is removed  
> just after the response header was  
> transferred to a client.
> 
> Maxim D.

additional flag to ngx\_http\_subrequest() has to work for this issue.

> > 
> 
> NGX\_INVALID\_FILE;
> 
> > [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx)
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx)

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,181400,181710#msg-181710](http://forum.nginx.org/read.php?2,181400,181710#msg-181710)

---

<div class="post-metadata">

**Author:** ![magz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/47e85d/32.png) [@magz](https://www.ruby-forum.com/u/magz)\
**Post date:** [March 9, 2011, 8:02pm UTC](https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265/4 "2011-03-09T20:02:48Z")

</div>

Hello!

On Wed, Mar 09, 2011 at 01:46:56PM -0500, magz wrote:

> > clean/discard fd in case
> > 
> > > "POST  
> > > In official nginx, subrequests can’t appear before  
> > > done by image  
> > > somewhere in russian mailing list). Probably it  
> > > nginx 0.3.3 with the following log message:
> > 
> > ```
> > *) Bugfix: a temporary file with client
> > 
> > ```
> > 
> > request body now is removed  
> > just after the response header was  
> > transferred to a client.
> > 
> > Maxim D.
> 
> additional flag to ngx\_http\_subrequest() has to work for this issue.

Additional flag to ngx\_http\_subrequest() can’t help if you aren’t  
even going to execute subrequests (as in filter finalization case)  
and can’t help with ssi either, as subrequests are executed after  
main request already cleared request body.

Maxim D.

---

<div class="post-metadata">

**Author:** ![magz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/47e85d/32.png) [@magz](https://www.ruby-forum.com/u/magz)\
**Post date:** [March 9, 2011, 8:24pm UTC](https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265/5 "2011-03-09T20:24:43Z")

</div>

Something like that. So we can do:  
rc = ngx\_http\_subrequest(r, &subreq\_loc , &args,  
&sr, psr, NGX\_HTTP\_REQUEST\_KEEP\_TEMP\_FILE);

# cat ngx\_keep\_tempfile\_flag.patch

diff -Nuarp nginx-0.9.5/src/http/ngx\_http\_core\_module.c  
nginx-0.9.5\_new/src/http/ngx\_http\_core\_module.c  
— nginx-0.9.5/src/http/ngx\_http\_core\_module.c 2011-01-20  
07:31:24.000000000 -0800  
+++ nginx-0.9.5\_new/src/http/ngx\_http\_core\_module.c 2011-03-09  
07:22:45.000000000 -0800  
@@ -2230,6 +2230,7 @@ ngx\_http\_subrequest(ngx\_http\_request\_t \*

```
 sr->subrequest_in_memory = (flags & NGX_HTTP_SUBREQUEST_IN_MEMORY)

```

!= 0;  
sr-\>waited = (flags & NGX\_HTTP\_SUBREQUEST\_WAITED) != 0;

- sr-\>keep\_temp\_file = (flags & NGX\_HTTP\_REQUEST\_KEEP\_TEMP\_FILE) !=  
0;

- 

#define NGX\_HTTP\_OK 200  
@@ -461,6 +463,7 @@ struct ngx\_http\_request\_s {

```
 unsigned subrequest_in_memory:1;
 unsigned waited:1;

```

- unsigned keep\_temp\_file:1;

#if (NGX\_HTTP\_CACHE)  
unsigned

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,181400,181736#msg-181736](http://forum.nginx.org/read.php?2,181400,181736#msg-181736)

---

<div class="post-metadata">

**Author:** ![magz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/47e85d/32.png) [@magz](https://www.ruby-forum.com/u/magz)\
**Post date:** [March 9, 2011, 8:30pm UTC](https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265/6 "2011-03-09T20:30:22Z")

</div>

Forgot include the check to the patch 🙂

cat ngx\_keep\_tempfile\_flag.patch  
diff -Nuarp nginx-0.9.5/src/http/ngx\_http\_core\_module.c  
nginx-0.9.5\_new/src/http/ngx\_http\_core\_module.c  
— nginx-0.9.5/src/http/ngx\_http\_core\_module.c 2011-01-20  
07:31:24.000000000 -0800  
+++ nginx-0.9.5\_new/src/http/ngx\_http\_core\_module.c 2011-03-09  
07:22:45.000000000 -0800  
@@ -2230,6 +2230,7 @@ ngx\_http\_subrequest(ngx\_http\_request\_t \*

```
 sr->subrequest_in_memory = (flags & NGX_HTTP_SUBREQUEST_IN_MEMORY)

```

!= 0;  
sr-\>waited = (flags & NGX\_HTTP\_SUBREQUEST\_WAITED) != 0;

- sr-\>keep\_temp\_file = (flags & NGX\_HTTP\_REQUEST\_KEEP\_TEMP\_FILE) !=  
0;

- 

#define NGX\_HTTP\_OK 200  
@@ -461,6 +463,7 @@ struct ngx\_http\_request\_s {

```
 unsigned subrequest_in_memory:1;
 unsigned waited:1;

```

- unsigned keep\_temp\_file:1;

#if (NGX\_HTTP\_CACHE)  
unsigned cached:1;  
— nginx-0.9.5/src/http/ngx\_http\_upstream.c 2010-07-28  
08:56:56.000000000 -0700  
+++ nginx-0.9.5\_new/src/http/ngx\_http\_upstream.c 2011-03-09  
07:30:30.000000000 -0800  
@@ -2030,7 +2033,7 @@ ngx\_http\_upstream\_send\_response(ngx\_http

```
 u->header_sent = 1;

```

- if (r-\>request\_body && r-\>request\_body-\>temp\_file) {

- if (r-\>request\_body && r-\>request\_body-\>temp\_file &&  
!r-\>keep\_temp\_file) {  
ngx\_pool\_run\_cleanup\_file(r-\>pool,  
r-\>request\_body-\>temp\_file-\>file.fd);  
r-\>request\_body-\>temp\_file-\>file.fd = NGX\_INVALID\_FILE;  
}

Posted at Nginx Forum:

> **[Re: \[PATCH\] ngx\_upstream do not cleanup file buffers in subrequest](https://forum.nginx.org/read.php?2,181400,181740#msg-181740)**
>
> Hello, Is it reasonable to add following check to not clean/discard fd in case if it's sub request? I developed nginx module and when it started to use tmp buffers under load I had following error: \[alert\] 29663#0: \*45789 sendfile() failed (9: Bad...

---

<div class="post-metadata">

**Author:** ![magz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/47e85d/32.png) [@magz](https://www.ruby-forum.com/u/magz)\
**Post date:** [March 10, 2011, 3:57am UTC](https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265/7 "2011-03-10T03:57:16Z")

</div>

> Additional flag to ngx\_http\_subrequest() can’t help if you aren’t  
> even going to execute subrequests (as in filter finalization case)  
> and can’t help with ssi either, as subrequests are executed after  
> main request already cleared request body.

> Maxim D.

Hm. I skipped this part. Sorry.

If it’s flag in struct ngx\_http\_request\_s() then you can control inside  
your module who will do the cleaning stuff, right?  
So that you can set it to main request or to sub-request.

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,181400,181863#msg-181863](http://forum.nginx.org/read.php?2,181400,181863#msg-181863)

---

<div class="post-metadata">

**Author:** ![magz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/47e85d/32.png) [@magz](https://www.ruby-forum.com/u/magz)\
**Post date:** [March 10, 2011, 1:29am UTC](https://www.ruby-forum.com/t/ngx-upstream-do-not-cleanup-file-buffers-in-subrequest/204265/8 "2011-03-10T01:29:05Z")

</div>

Hello!

On Wed, Mar 09, 2011 at 02:28:32PM -0500, magz wrote:

> Forgot include the check to the patch 🙂

[…]

Please re-read what I wrote. Sometimes reading make sense, not  
only writing. Thank you.

Maxim D.