# Problem with PCRE matching, utf-8, Greek, rewrite

**URL:** <https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962>\
**Category:** NGINX\
**Created:** [July 1, 2010, 5:34pm UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962 "2010-07-01T17:34:56Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [July 1, 2010, 5:34pm UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/1 "2010-07-01T17:34:56Z")

</div>

Dear all,  
I try to implement some rewrites using regular expressions and my URIs  
will contain Greek characters.

Trials of the REs are going ok when tested with pcretest:

```auto
[root@localhost ~]# pcretest
PCRE version 8.10 2010-06-25

  re> #^[\x{0386}-\x{03FF}]+$#8
data> bv
No match
data> Τηλέ
 0: \x{3a4}\x{3b7}\x{3bb}\x{3ad}

```

note the 8 modifier that actually tells PCRE to do a UTF-8 matching.

Having the RE in nginx.config complains about

```auto
[emerg]: pcre_compile() failed: character value in \x{...} sequence is
too large in
```

which I guess means that somehow nginx calls PCRE without the PCRE\_UTF8  
option flag

Am I right? How can I implement these Greek character URL rewrites?

The system environment is:

- CentOS 5.4
- PCRE 8.10 with utf-8 and utf-properties enabled
- nginx 0.8.42

Cheers  
Tilemahos

Posted at Nginx Forum:

> **[problem with PCRE matching, utf-8, Greek, rewrite](https://forum.nginx.org/read.php?2,104357,104357#msg-104357)**
>
> Dear all, I try to implement some rewrites using regular expressions and my URIs will contain Greek characters. Trials of the REs are going ok when tested with pcretest: \[code\] \[root@localhost ~\]# pcretest PCRE version 8.10 2010-06-25

---

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [July 1, 2010, 7:12pm UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/2 "2010-07-01T19:12:44Z")

</div>

Hello!

On Thu, Jul 01, 2010 at 11:33:49AM -0400, tmanolat wrote:

> re\> #\[1\]+$#8
> 
> ```auto
> [emerg]: pcre_compile() failed: character value in \x{...} sequence is
> too large in 
> ```
> 
> which I guess means that somehow nginx calls PCRE without the PCRE\_UTF8  
> option flag
> 
> Am I right? How can I implement these Greek character URL rewrites?

Using (\*UTF8) to switch pcre into utf-8 mode should work find in  
both nginx and pcretest. See man pcresyntax for details.

Maxim D.

* * *

1. \x{0386}-\x{03FF}

---

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [July 2, 2010, 4:13am UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/3 "2010-07-02T04:13:05Z")

</div>

tmanolat at 2010-7-1 23:33 wrote:

> re\> #\[1\]+$#8
> 
> ```auto
> [emerg]: pcre_compile() failed: character value in \x{...} sequence is
> too large in 
> ```
> 
> which I guess means that somehow nginx calls PCRE without the PCRE\_UTF8  
> option flag
> 
> Am I right? How can I implement these Greek character URL rewrites?

I use the raw bits in Chinese character substitution in my subscitution  
module([Google Code Archive - Long-term storage for Google Code Project Hosting.](http://code.google.com/p/substitutions4nginx/wiki/ChineseCharacterSubsitution))

I think you could convert the Greek cahracter like this:

‘\x3a\x43\xb7\x3b\xb3\xad’

> Posted at Nginx Forum: [http://forum.nginx.org/read.php?2,104357,104357#msg-104357](http://forum.nginx.org/read.php?2,104357,104357#msg-104357)
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx)

–  
Weibin Y.

* * *

1. \x{0386}-\x{03FF}

---

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [September 23, 2010, 9:46am UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/4 "2010-09-23T09:46:51Z")

</div>

tmanolat Wrote:

> FYI I put in nginx.conf:
> 
> …  
> if ($request\_uri ~  
> “(_UTF8)^(._)[\?|&]filename=([%  
> ,a-zA-Z0-9\x{386}-\x{3ff}\_-.]+)(&.\*)?$”) {  
> …

I’ve got a very similar problem in nginx but I dont really understand  
your solution. Could you please post your nginx.conf or at least some  
more lines related to UTF8 filenames conversions? It would be a life  
saver!

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,104357,133258#msg-133258](http://forum.nginx.org/read.php?2,104357,133258#msg-133258)

---

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [July 2, 2010, 9:49am UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/5 "2010-07-02T09:49:38Z")

</div>

Dear Maxim,  
it works like a charm now.

FYI I put in nginx.conf:

```auto
...
      if ($request_uri ~ "(*UTF8)^(.*)[\\?|&]filename=([%
,a-zA-Z0-9\x{386}-\x{3ff}_\-\.]+)(&.*)?$") {
...
```

Kindest regards,  
Tilemahos Manolatos

PS. @Weibin Y.: I would like to avoid “fixed” character lists, I wanted  
to use ranges of characters, so the above solution seems, in my opinion,  
better for matching all Greek characters

Posted at Nginx Forum:

> **[Re: problem with PCRE matching, utf-8, Greek, rewrite](https://forum.nginx.org/read.php?2,104357,104628#msg-104628)**
>
> Dear all, I try to implement some rewrites using regular expressions and my URIs will contain Greek characters. Trials of the REs are going ok when tested with pcretest: \[code\] \[root@localhost ~\]# pcretest PCRE version 8.10 2010-06-25

---

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [September 23, 2010, 11:14am UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/6 "2010-09-23T11:14:05Z")

</div>

initially this worked well: (\x{386}-\x{3ff} for Greek chars)

```auto
    location ~ "^(/optionalwebappname)?/ProcessImageServlet.*$" {
        root /opt/myfilerepository/;
        rewrite ^(.+)$ http://static-dev.myhost.eu/$arg_hotel_id/$th$fn
break;

      set $hid '';
      set $filename '';
      set $th '';

      if ($request_uri ~ "^(.*)[\\?|&]hotel_id=([0-9]+)(&.*)?$") {
        set $hid $2;
      }
      if ($request_uri ~ "(*UTF8)^(.*)[\\?|&]filename=([%
,a-zA-Z0-9\x{386}-\x{3ff}_\-\.]+)(&.*)?$") {
        set $fn $2;
      }
      if ($request_uri ~ "^(.*)[\\?|&]type=th(&.*)?$") {
        set $th 'th_';
      }
      rewrite ^(.+)$ http://static-dev.myhost.eu/$hid/$th$fn break;
      access_log logs/site-pis.log main;
      expires 1h;
    }

```

however, later I found this to work better, including of course utf8  
arguments - you would better check this out first… much more elegant

```auto
                location ~
"^(/optionalwebappname)?/ProcessImageServlet.*$" {
      set $th '';

      if ($request_uri ~ "^(.*)[\\?|&]type=th(&.*)?$") {
        set $th 'th_';
      }
      rewrite ^(.+)$
http://static-dev.myhost.eu/$arg_hotel_id/$th$arg_filename break;
      expires 1d;
    }
```

Posted at Nginx Forum:

> **[Re: problem with PCRE matching, utf-8, Greek, rewrite](https://forum.nginx.org/read.php?2,104357,133276#msg-133276)**
>
> Dear all, I try to implement some rewrites using regular expressions and my URIs will contain Greek characters. Trials of the REs are going ok when tested with pcretest: \[code\] \[root@localhost ~\]# pcretest PCRE version 8.10 2010-06-25

---

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [September 23, 2010, 9:58am UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/7 "2010-09-23T09:58:34Z")

</div>

I would really like a wiki page on UTF-8 support as well.

\*UTF8 doesnt work for me though, ive tried.

---

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [September 25, 2010, 3:45am UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/8 "2010-09-25T03:45:10Z")

</div>

When attempting to use \*UTF8 I always receive.  
[emerg]: pcre\_compile() failed: (_VERB) not recognized in  
“(UTF8)^/([^/^.]+)(?:/?)(?:index([0-9]).html?)?$” at  
"8)^/([^/^.]+)(?:/?)(?:index([0-9]_).html?)?$" in  
/etc/nginx/sites-enabled/nexusddl.com:85

Yet my PCRE has UTF-8 support, tested it in PHP (both nginx and php  
compiled against PCRElib included in debian)

---

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [September 25, 2010, 11:18am UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/9 "2010-09-25T11:18:44Z")

</div>

Hello!

On Sat, Sep 25, 2010 at 11:43:48AM +1000, mat h wrote:

> When attempting to use \*UTF8 I always receive.  
> [emerg]: pcre\_compile() failed: (_VERB) not recognized in  
> “(UTF8)^/([^/^.]+)(?:/?)(?:index([0-9]).html?)?$” at  
> "8)^/([^/^.]+)(?:/?)(?:index([0-9]_).html?)?$" in  
> /etc/nginx/sites-enabled/nexusddl.com:85
> 
> Yet my PCRE has UTF-8 support, tested it in PHP (both nginx and php  
> compiled against PCRElib included in debian)

You need at least pcre 7.9 for (\*UTF8) support.

[http://www.pcre.org/changelog.txt](http://www.pcre.org/changelog.txt)

[…]

> > š š š š š š š š š š š šif ($request\_uri ~ “(_UTF8)^(._)[\?|&]filename=([%  
> > ,a-zA-Z0-9\x{386}-\x{3ff}\_-.]+)(&.\*)?$”) {  
> > š š š š š š š š š š š š š š š šset $fn $2;  
> > š š š š š š š š š š š š}

Note that (\*UTF8) is meaningless here as $request\_uri doesn’t  
contain utf-8 characters, it’s urlencoded.

Maxim D.

---

<div class="post-metadata">

**Author:** ![tmanolat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@tmanolat](https://www.ruby-forum.com/u/tmanolat)\
**Post date:** [September 25, 2010, 4:48pm UTC](https://www.ruby-forum.com/t/problem-with-pcre-matching-utf-8-greek-rewrite/190962/10 "2010-09-25T16:48:16Z")

</div>

so, seems that the following is the most correct, (at least for me  
worked well so far)

```auto
        location ~ "^(/optionalwebappname)?/ProcessImageServlet.*$" {
      set $th '';

      if ($request_uri ~ "^(.*)[\\?|&]type=th(&.*)?$") {
        set $th 'th_';
      }
      rewrite ^(.+)$
http://static-dev.myhost.eu/$arg_hotel_id/$th$arg_filename break;
      expires 1d;
    }
```

Again, my problem was to rewrite some urls with GET vars that were  
expected to contain utf8 characters.  
So, the $arg\_filename is passed correctly to the rewrite.

Posted at Nginx Forum:

> **[Re: problem with PCRE matching, utf-8, Greek, rewrite](https://forum.nginx.org/read.php?2,104357,133963#msg-133963)**
>
> Dear all, I try to implement some rewrites using regular expressions and my URIs will contain Greek characters. Trials of the REs are going ok when tested with pcretest: \[code\] \[root@localhost ~\]# pcretest PCRE version 8.10 2010-06-25