# Quick start on writing a module?

**URL:** <https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217>\
**Category:** NGINX\
**Created:** [June 17, 2010, 4:22pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217 "2010-06-17T16:22:23Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:22pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/1 "2010-06-17T16:22:23Z")

</div>

I’d like to convert an apache module I’ve written to work  
under nginx. Can somebody loan me a clue for getting started?

## Background

The logic is pretty simple:

1. based on the input URI, determine what file will be  
sent. this will possibly involved a (cached) db lookup.

2a. set these headers:

```
     Content-disposition:
     ETag:
     X-mod_asset-version: (my custom header)

```

2b. set mime type.

1. allow nginx to send the bytes of the file. Hopefully  
it will handle these headers:

## Questions

For 2b, this is the proper way?

```
 r->headers_out.content_type.len = sizeof("image/gif") - 1;
 r->headers_out.content_type.data = (u_char *) "image/gif";

 Is there a special way to malloc the .data so that it
 will be freed automatically when the request is complete?

```

For 3, should I be using X-Accel-Redirect?

```
 add_header("X-Accel-Redirect: /protected/iso.img");

```

If so, then I can simply set my headers and pass along to the next  
phase?

Any simple example modules I can use as a starting point?

Many TIA!!  
Mark

–  
Mark H.  
Pixar Animation Studios

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:22pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/2 "2010-06-17T16:22:45Z")

</div>

Hello!

On Wed, Jun 16, 2010 at 02:29:43AM -0700, Mark H. wrote:

> I’d like to convert an apache module I’ve written to work  
> under nginx. Can somebody loan me a clue for getting started?

It’s probably good idea to start from here:

[http://nginx.org/en/links.html](http://nginx.org/en/links.html)

> **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)**
>
> ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com

> **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)**
>
> ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com

Note well: in most cases it’s good idea to rewrite module from  
scratch instead of trying to adopt something written for Apache.

> ```
> Content-disposition:
> Accept-Ranges:
> Content-Range:
> Content-Length:
> 
> ```

Problematic part is “db lookup”. In nginx you have to do it in  
non-blocking way, and this isn’t something readily available in  
database client libraries. You may want to deligate this part to  
some backend or already available modules like ngx\_postgres  
([GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/Nginx3rdPartyModules#Postgres_Module)).

> ## Questions
> 
> For 2b, this is the proper way?
> 
> ```
> r->headers_out.content_type.len = sizeof("image/gif") - 1;
> r->headers_out.content_type.data = (u_char *) "image/gif";
> 
> ```

No.

For 0.7.\* use:

```
r->headers_out.content_type_len = sizeof("image/gif") - 1;
r->headers_out.content_type.len = sizeof("image/gif") - 1;
r->headers_out.content_type.data = (u_char *) "image/gif";

```

For 0.8.37+ use ngx\_str\_set() macro to set string, i.e.

```
r->headers_out.content_type_len = sizeof("image/gif") - 1;
ngx_str_set(&r->headers_out.content_type, "image/gif");

```

> ```
> Is there a special way to malloc the .data so that it
> will be freed automatically when the request is complete?
> 
> ```

In the above case you don’t need to allocate anything as it uses  
constants strings. For dynamic allocations use  
ngx\_pool\_alloc(r-\>pool).

> For 3, should I be using X-Accel-Redirect?
> 
> ```
> add_header("X-Accel-Redirect: /protected/iso.img");
> 
> ```
> 
> If so, then I can simply set my headers and pass along to the next  
> phase?

No. X-Accel-Redirect is special header understood by upstream  
module (proxy, fastcgi) in backend responses. In your own module  
you have to use ngx\_http\_internal\_redirect() function for this.

> Any simple example modules I can use as a starting point?

For simple content generation see ngx\_http\_empty\_gif\_module.c,  
ngx\_http\_static\_module.c, ngx\_http\_flv\_module.c.

For simple examples of internal redirect usage see  
ngx\_http\_index\_module.c, ngx\_http\_random\_index\_module.c.

Maxim D.

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:22pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/3 "2010-06-17T16:22:25Z")

</div>

Hi,  
Maxim already answered most of your questions regarding writing a module  
like this from scratch. I would like to add that you can already do  
this,  
without writing new module, by misusing Maxim’s excellent  
ngx\_http\_auth\_request\_module [1] and ngx\_postgres [2].

There are two “but”:

1. Required “postgres\_set” directive is still available only in my local  
repository. I intend to push it to GitHub today or tomorrow, after  
adding  
some test cases, so stay tuned 🙂
2. You need to use “proxy\_pass” for now. Ideally, you should create your  
own  
mini-module that would serve static files based on complex\_value  
($location). This should be mostly copy & paste of  
“ngx\_http\_static\_handler”  
from nginx’s ngx\_http\_static\_module.c.

[1] [ngx\_http\_auth\_request\_module: log](http://mdounin.ru/hg/ngx_http_auth_request_module/)  
[2] [FRiCKLE Labs / nginx / ngx\_postgres](http://labs.frickle.com/nginx_ngx_postgres/)

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

— configuration —  
upstream pgsql {  
postgres\_server 127.0.0.1 dbname=test user=test password=test;  
}

server {  
listen 80;

location = /auth {  
internal;  
postgres\_pass pgsql;  
postgres\_query “SELECT ‘c-d’, ‘123456’, ‘0.1’, ‘/etc/passwd’”;  
postgres\_set $pg\_disposition 0 0 required;  
postgres\_set $pg\_etag 0 1 required;  
postgres\_set $pg\_version 0 2 required;  
postgres\_set $pg\_location 0 3 required;  
}

location / {  
auth\_request /auth;  
auth\_request\_set $disposition $pg\_disposition;  
auth\_request\_set $etag $pg\_etag;  
auth\_request\_set $version $pg\_version;  
auth\_request\_set $location $pg\_location;  
add\_header “Content-Disposition” $disposition;  
add\_header “Etag” $etag;  
add\_header “X-mod\_asset-version” $version;  
proxy\_pass [http://127.0.0.1:81/$location](http://127.0.0.1:81/$location);  
}  
}

# proxy for serving static files

server {  
listen 81;

location / {  
root /;  
}  
}

— output —  
$ curl -v [http://localhost/](http://localhost/)

- About to connect() to localhost port 80 (#0)
- Trying 127.0.0.1… connected
- Connected to localhost (127.0.0.1) port 80 (#0)

> GET / HTTP/1.1  
> User-Agent: curl/7.19.5  
> Host: localhost  
> Accept: _/_

\< HTTP/1.1 200 OK  
\< Server: nginx/0.8.41  
\< Date: Wed, 16 Jun 2010 16:55:30 GMT  
\< Content-Type: text/plain  
\< Connection: keep-alive  
\< Content-Length: 3078  
\< Last-Modified: Sat, 15 May 2010 01:27:40 GMT  
\< Accept-Ranges: bytes  
\< Content-Disposition: c-d  
\< Etag: 123456  
\< X-mod\_asset-version: 0.1  
\<  
root:\*:0:0:Charlie &:/root:/bin/ksh  
(..)

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:26pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/4 "2010-06-17T16:26:59Z")

</div>

Hello!

On Wed, Jun 16, 2010 at 05:13:40PM +0200, Piotr S. wrote:

> Hi,  
> Maxim already answered most of your questions regarding writing a  
> module like this from scratch. I would like to add that you can  
> already do this, without writing new module, by misusing Maxim’s  
> excellent ngx\_http\_auth\_request\_module [1] and ngx\_postgres [2].

I believe Valery K.'s eval module  
([http://grid.net.ru/nginx/eval.en.html](http://grid.net.ru/nginx/eval.en.html)) would fit better here than  
auth request module.

> There are two “but”:
> 
> 1. Required “postgres\_set” directive is still available only in my  
> local repository. I intend to push it to GitHub today or tomorrow,  
> after adding some test cases, so stay tuned 🙂

I believe this may be handled by eval as long as postgres module  
is able to return something it understands.

> 1. You need to use “proxy\_pass” for now. Ideally, you should create  
> your own mini-module that would serve static files based on  
> complex\_value ($location). This should be mostly copy & paste of  
> “ngx\_http\_static\_handler” from nginx’s ngx\_http\_static\_module.c.

Something like alias /path/to/$location; should do the trick.

Also, with eval it’s possible to use regular rewrites instead.

Maxim D.

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:22pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/5 "2010-06-17T16:22:46Z")

</div>

On Wed, Jun 16, 2010 at 5:29 PM, Mark H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> 1. based on the input URI, determine what file will be  
> sent. this will possibly involved a (cached) db lookup.

Well, (cached) db lookup can be done via ngx\_drizzle/ngx\_postgres +  
ngx\_srcache + ngx\_memc + memcached 🙂

> 2a. set these headers:
> 
> ```
> Content-disposition:
> ETag:
> X-mod_asset-version: (my custom header)
> 
> ```

ngx\_headers and/or ngx\_headers\_more may just do the work or at least  
can serve as good examples.

Cheers,  
-agentzh

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:27pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/6 "2010-06-17T16:27:03Z")

</div>

Hi,

> I believe Valery K.'s eval module  
> ([http://grid.net.ru/nginx/eval.en.html](http://grid.net.ru/nginx/eval.en.html)) would fit better here than  
> auth request module.  
> (..)  
> I believe this may be handled by eval as long as postgres module  
> is able to return something it understands.

I can’t agree with you here. ngx\_postgres can either return single value  
in  
clear text or whole result-set in RDS format (which can be converted to  
JSON  
using ngx\_rds\_json [1]). Since we need more than one value, we would  
need to  
either do multiple queries for each value or use JSON. Of course, we  
could  
extract values from JSON using ngx\_lua [2], but this seems like quite an  
overhead for something that could be easily achieved by misusing your  
module.

> Something like alias /path/to/$location; should do the trick.

I tried that before suggesting “proxy\_pass”.

Unfortunately both “alias $location;” and “alias /$location;” results in  
“/etc/passwdindex.html” is not a directory while sending to client".

[1] [GitHub - openresty/rds-json-nginx-module: An nginx output filter that formats Resty DBD Streams generated by ngx\_drizzle and others to JSON · GitHub](http://github.com/agentzh/rds-json-nginx-module)  
[2] [GitHub - openresty/lua-nginx-module: Embed the Power of Lua into NGINX HTTP servers · GitHub](http://github.com/chaoslawful/lua-nginx-module)

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:35pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/7 "2010-06-17T16:35:18Z")

</div>

Hello!

On Wed, Jun 16, 2010 at 05:46:32PM +0200, Piotr S. wrote:

> value in clear text or whole result-set in RDS format (which can be  
> converted to JSON using ngx\_rds\_json [1]). Since we need more than  
> one value, we would need to either do multiple queries for each  
> value or use JSON. Of course, we could extract values from JSON  
> using ngx\_lua [2], but this seems like quite an overhead for  
> something that could be easily achieved by misusing your module.

I wrote “as long as …” on purpose, as I have no idea what  
postrgess module is able to produce, sorry.

But as long as it’s able to return single value in clear text it  
should be possible to write select which will return all needed  
data separated by newlines. This should be understood by eval  
module easily.

> > Something like alias /path/to/$location; should do the trick.
> 
> I tried that before suggesting “proxy\_pass”.
> 
> Unfortunately both “alias $location;” and “alias /$location;”  
> results in “/etc/passwdindex.html” is not a directory while sending  
> to client".

This is caused by index module and only applies when uri ends with  
‘/’. Workarounds are obvious, but I doubt they are actually  
needed in this particular case.

Maxim D.

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:51pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/8 "2010-06-17T16:51:47Z")

</div>

> Oops. You’re right. I didn’t notice that eval module allowed multiple  
> variables until now. In that case, it might be indeed a better fit than  
> ngx\_http\_auth\_request\_module.

…or not. It seems that eval module evaluates multiple variables only  
for  
“application/x-www-form-urlencoded” content, so it won’t work with clear  
text values separated by new lines.

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:51pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/9 "2010-06-17T16:51:47Z")

</div>

Hello!

On Wed, Jun 16, 2010 at 06:47:29PM +0200, Piotr S. wrote:

[…]

> }
> 
> “GET /aaa” results in:

[…]

> \*4 open() “/etc/passwdaaa” failed (2: No such file or directory)"

No, it’s not bug, it works as documented. Alias specifies full  
path only in regexp locations (with captures). For normal locations  
it specifies path to point matched by location (even if alias  
contains variables).

Maxim D.

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 4:35pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/10 "2010-06-17T16:35:20Z")

</div>

Hi,

> I wrote “as long as …” on purpose, as I have no idea what  
> postrgess module is able to produce, sorry.
> 
> But as long as it’s able to return single value in clear text it  
> should be possible to write select which will return all needed  
> data separated by newlines. This should be understood by eval  
> module easily.

Oops. You’re right. I didn’t notice that eval module allowed multiple  
variables until now. In that case, it might be indeed a better fit than  
ngx\_http\_auth\_request\_module.

One note though, you will need to use agentzh’s fork of the eval module  
[1],  
because ngx\_postgres currently doesn’t work with subrequests in memory.

> This is caused by index module and only applies when uri ends with  
> ‘/’. Workarounds are obvious, but I doubt they are actually  
> needed in this particular case.

Yes, my previous error message was produced by index module,  
but the issue exists and should be probably considered a bug:

location / {  
set $file “/etc/passwd”;  
alias $file;  
}

“GET /aaa” results in:

\*4 http script var: “/etc/passwd”  
\*4 http filename: “/etc/passwdaaa”  
\*4 add cleanup: 000000020E509C88  
\*4 open() “/etc/passwdaaa” failed (2: No such file or directory)"  
\*4 http finalize request: 404, “/aaa?” a:1, c:1

This can indeed be worked-around by using:

location ~ (/.\*) {  
set $file “/etc/passwd”;  
alias $file;  
}

which results in:

\*2 http script var: “/etc/passwd”  
\*2 http filename: “/etc/passwd”  
\*2 add cleanup: 00000002097A0CA0  
\*2 http static fd: 12

> From what I can tell “ngx\_http\_map\_uri\_to\_path()” always adds URI to the  
> path, unless you’re using captures in the location.

[1] [GitHub - openresty/nginx-eval-module: A module for evaluating memcached or proxy response into variable · GitHub](http://github.com/agentzh/nginx-eval-module)

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

---

<div class="post-metadata">

**Author:** ![markh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@markh](https://www.ruby-forum.com/u/markh)\
**Post date:** [June 17, 2010, 5:23pm UTC](https://www.ruby-forum.com/t/quick-start-on-writing-a-module/190217/11 "2010-06-17T17:23:26Z")

</div>

Hi,

> No, it’s not bug, it works as documented. Alias specifies full  
> path only in regexp locations (with captures). For normal locations  
> it specifies path to point matched by location (even if alias  
> contains variables).

Uhm… I stand corrected. Russian documentation indeed says that.

Thanks!

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>