# Unicode roadmap? **URL:** \ **Category:** Ruby\ **Created:** [June 13, 2006, 11:12pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296 "2006-06-13T23:12:48Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 4:17am UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/41 "2006-06-15T04:17:30Z")
On 15-jun-2006, at 3:50, Charles O Nutter wrote: > operations work correctly treating those characters as string, > would that be a better ideal? Where are the breaking points in such > a design? What’s to stop the underlying implementation from > actually using a UTF-16 character, passing UTF-8 to libraries and > IO streams but still allowing you to access everything as UTF-16 or > your encoding of choice? (Of course this is somewhat rhetorical; we > do this currently with JRuby since Java’s scrints are UTF-16…we > just don’t have any way to provide access to UTF-16 characters, and > we normalize everything to UTF-8 for Ruby’s sake…but what if we > didn’t normalize and adjusted string functions to compensate?) This is more appropriate for ruby-talk – Julian ‘Julik’ Tarkhanov please send all personal mail to me at julik.nl ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 3:52am UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/42 "2006-06-15T03:52:43Z")
I agree it’s a very attractive solution. I have two questions related (perhaps you are out there to answer, Julik): 1. How does performance look with the unicode string add-on versus native strings? 2. Is this the ideal way to support unicode strings in ruby? And I explain the second as follows…if we could assume that switching from treating a string as an array of bytes to a list of characters of arbitrary width, and have all existing string operations work correctly treating those characters as string, would that be a better ideal? Where are the breaking points in such a design? What’s to stop the underlying implementation from actually using a UTF-16 character, passing UTF-8 to libraries and IO streams but still allowing you to access everything as UTF-16 or your encoding of choice? (Of course this is somewhat rhetorical; we do this currently with JRuby since Java’s scrints are UTF-16…we just don’t have any way to provide access to UTF-16 characters, and we normalize everything to UTF-8 for Ruby’s sake…but what if we didn’t normalize and adjusted string functions to compensate?) ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 4:24am UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/43 "2006-06-15T04:24:30Z")
I believe that Julik’s way of solving the unicode problem (String#u providing access to a unicode helper) is very attractive. I have two questions related, for Julik and the rest of the peanut gallery: 1. How does performance look with the unicode string add-on versus native strings (or as compared to icu4r, which is C-based)? 2. Is this the ideal way to support unicode strings in ruby? And I explain the second as follows…if we could assume switching from treating a string as an array of bytes to a list of characters of arbitrary width, and have all existing string operations work correctly treating those characters as indexed elements of that string, would that be a better ideal? Where are the breaking points in such a design? What’s to stop the underlying implementation from actually using a UTF-16 character, passing UTF-8 to libraries and IO streams but still allowing you to access everything as UTF-16 or your encoding of choice? Is it simply libraries or core APIs that explicitly need _byte_ counts? (Of course this is somewhat rhetorical; we do this currently with JRuby since Java’s strings are UTF-16…we just don’t have any uniform way to provide access to UTF-16 character strings, and we normalize everything to UTF-8 for Ruby’s sake…but what if we didn’t normalize and adjusted string functions to compensate?) ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 4:28am UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/44 "2006-06-15T04:28:22Z")
Fair enough; redirected. If any other rails-core folks want to chime in, please do so…I would expect unicode and multibyte are key issues for worldwide rails deployments. ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 4:41am UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/45 "2006-06-15T04:41:36Z")
On 6/14/06, Charles O Nutter [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I believe that Julik’s way of solving the unicode problem (String#u > providing access to a unicode helper) is very attractive. I have two > questions related, for Julik and the rest of the peanut gallery: > 1. How does performance look with the unicode string add-on versus native > strings (or as compared to icu4r, which is C-based)? > 2. Is this the ideal way to support unicode strings in ruby? No. In fact, I believe that Matz has the right idea for M17N strings in Ruby 2.0. The _reality_ is that there’s a _lot_ of data out there that isn’t Unicode. I would suggest that JRuby could offer a JavaString that acts in every way like a String except that it provides access to the native UTF-16 implementation. -austin ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 4:55am UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/46 "2006-06-15T04:55:09Z")
On 15-jun-2006, at 4:40, Austin Z. wrote: > No. In fact, I believe that Matz has the right idea for M17N strings > in Ruby 2.0. The _reality_ is that there’s a _lot_ of data out there > that isn’t Unicode. It’s very difficult for me to understand the implementation. What if we concat a Mojikyo string to a UTF8String? UnicodeDecodeError, ordinal not in range? I think Python folks proved that it’s terrible (it is). Nothing is ideal. > I would suggest that JRuby could offer a JavaString that acts in every > way like a String except that it provides access to the native UTF-16 > implementation. Just what the ICU4R extension does. It’s unusable to the point that you cannot concat a native string with a UString. To the point that you have to use special Regexp class for it. You end up having half of your Ruby script doing typecasting from one to the other. There is alot of data that isn’t Unicode, indeed. Converted on input and converted on output if necessary - just as in any other case when the encoding of your system doesn’t match your input or output. I don’t know if it can be possible to have the “internal” encoding of a system switchable (seems to me this is what Matz wants) - then you can’t safely refer to anything other than bytes. And then you get software that you can’t use, because they had a different assumtpion than you had as to what encoding the user will be using. ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 5:01am UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/47 "2006-06-15T05:01:21Z")
On 6/14/06, Austin Z. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > in Ruby 2.0. The _reality_ is that there’s a _lot_ of data out there > that isn’t Unicode. Yes, we all understand that Ruby 2.0 will be the coolest thing since sliced bread, but those of us that are currently developing international websites with Rails don’t have the luxury of waiting until Christmas of 2007. -PJ Hyett [http://pjhyett.com](http://pjhyett.com) ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 5:10am UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/48 "2006-06-15T05:10:42Z")
On 6/14/06, PJ Hyett [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > that isn’t Unicode. > > Yes, we all understand that Ruby 2.0 will be the coolest thing since > > sliced bread, but those of us that are currently developing > > international websites with Rails don’t have the luxury of waiting > > until Christmas of 2007. _shrug_ As far as I can tell, there will be no implementation of Ruby before then that has a “native” m17n string. So whether you have the luxury of waiting or not, Ruby 1.8.x will not _ever_ have a “Unicode string”. Adding a “Unicode string” would _break_ behaviour, and no example is better than the extension that was proposed which would change the meaning of #size and #length to mean two different things. So, there’s a point where patience is going to be necessary, whether you “have the luxury” or not. -austin ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 1:02pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/49 "2006-06-15T13:02:37Z")
On 6/14/06, Austin Z. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > individual string level so that you could meaningfully have Unicode > (probably UTF-8) and ShiftJIS strings in the same data and still > meaningfully call #length on them. > > You will _always_ have to care about the encoding. As well as, > ultimately, your locale. No. Since I have locale stdin can be marked with the proper encoding information so that all stings originating there have the proper encoding information. The string methods should not just blindly operate on bytes but use the encoding information to operate on characters rather than bytes. Sure something like byte\_length is needed when the string is stored somewhere outside Ruby but standard string methods should work with character offsets and characters, not byte offsets nor bytes. Since my stdout can be also marked with correct encoding the strings that are output there can be converted to that encoding. Even if it originates from a source file that happens to be in a different encoding. Hmm, prehaps it will be necessary to mark source files with encoding tags as well. It could be quite tedious to assingn the tag manually to every string in a source file. When strings are compared, concatenated, … the encoding is known so the methods should do the right thing. I do not have to care about encoding. You may make a string implemenation that forces me to care (such a the current one). But I do not have to. I can always turn to perl if I get really desperate. Thanks Michal ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 10:47am UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/50 "2006-06-15T10:47:13Z")
IIRC, Matz has said that internally String won’t change, and I suspect that a CharString class (or smth like) won’t be ever added. Maybe just introducing String#encoding flag and addig new methods to String with prefixes, like char\_array, char\_slice, char\_length, char\_index, char\_downcase, char\_strcoll, char\_strip, etc. that will internally look at encoding flag and process respectively bytes in this particular string without conversion (just maybe some hidden), and leaving old byte-processing methods intact, would be the way to keep older code working and enjoy M17N? Though, as for me, it is still unclear, what should happen, if one tries to perform operation on two strings with different String#encoding… ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 1:22pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/51 "2006-06-15T13:22:51Z")
On 6/15/06, Julian ‘Julik’ Tarkhanov [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > 1. Preferably separate (and strictly purposed) Bytestring that you > get out of Sockets and use in Servers etc. - or the ability to > “force” all strings recieved from external resources to be flagged > uniformly as being of a certain encoding in _your_ program, not > somewhere in someone’s library. If flags have to be set by libraries, > they won’t be set because most developers sadly don’t care: > > [http://www.zackvision.com/weblog/2005/11/mt-unicode-mysql.html](http://www.zackvision.com/weblog/2005/11/mt-unicode-mysql.html) > [unicodification](http://thraxil.org/users/anders/posts/2005/11/01/unicodification/) Where else should the strings be flagged? If you get a web page through http request, and the library parses the response for you, it should set enconding on the web page. You would never know since you only received the page, not the header. > setting such as $KCODE. > I do not see why libraries should be always wrong. After all, you can > always fix them. And setting the encoding globally is a bad thing. You > cannot have strings encoded in different encodings in one process > then. It looks quite limiting. For one, the web pages that you get > from various servers (and even the same server) can be in varoius > encodings. Thanks Michal ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 1:51pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/52 "2006-06-15T13:51:55Z")
On 15-jun-2006, at 13:21, Michal S. wrote: > > [http://www.zackvision.com/weblog/2005/11/mt-unicode-mysql.html](http://www.zackvision.com/weblog/2005/11/mt-unicode-mysql.html) > > [unicodification](http://thraxil.org/users/anders/posts/2005/11/01/unicodification/) > > Where else should the strings be flagged? > They should nog be flagged, because some strings will be flagged and > some won’t and exactly > in the wrong places at the wrong time. See _is\_uf\_8_ in Perl to > witness the terrible ugliness of this. > If you get a web page > through http request, and the library parses the response for you, it > should set enconding on the web page. You would never know since you > only received the page, not the header. That’s why you should distinguish between a ByteArray and a String. > > libraries I use will be getting it wrong - see above) or by a global > > setting such as $KCODE. > > I do not see why libraries should be always wrong. After all, you can > always fix them. And setting the encoding globally is a bad thing. You > cannot have strings encoded in different encodings in one process > then. It looks quite limiting. For one, the web pages that you get > from various servers (and even the same server) can be in varoius > encodings. Of course they can (and will). When I have to approach this I usually just snif the encoding of the strings I recieved and then feed them to iconv and friends before doing any processing. A library that downloads stuff off the Internet should be (IMO) aware of the charset madness and decode the strings for me. Trust me, when multibyte/Unicode handling is optional, 80% of libraries do it wrong. Re-read the links above if you don’t believe. Actually it seems that the solution with an accessor is quite nice, but that I had to figure out the hard way after breaking the String class with my hacks and seeing stuff collapse. Apparently the poster of a parallel thread finds it inspiring to repeat my experiment _in vitro_ just for the academic sake of it. ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 15, 2006, 3:13pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/53 "2006-06-15T15:13:16Z")
On 6/15/06, Julian ‘Julik’ Tarkhanov [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > somewhere in someone’s library. If flags have to be set by libraries, > > > they won’t be set because most developers sadly don’t care: > > > > > > [http://www.zackvision.com/weblog/2005/11/mt-unicode-mysql.html](http://www.zackvision.com/weblog/2005/11/mt-unicode-mysql.html) > > > [unicodification](http://thraxil.org/users/anders/posts/2005/11/01/unicodification/) > > > > Where else should the strings be flagged? > > They should nog be flagged, because some strings will be flagged and > > some won’t and exactly > > in the wrong places at the wrong time. See _is\_uf\_8_ in Perl to > > witness the terrible ugliness of this. You can certainly get the things wrong. But if you get a string that is wrongly flagged you have the choice to fix the code where the string originates or work arond it by flagging it right. If you have a code that gets the encoding wrong, and it tries to convert the string to some ‘universal’ encoding you want to use everywhere in your application, you get a broken string. > > If you get a web page > > through http request, and the library parses the response for you, it > > should set enconding on the web page. You would never know since you > > only received the page, not the header. > > That’s why you should distinguish between a ByteArray and a String. How does it help you here? > > > All of this can be controlled either per String (then 99 out of 100 > > > Of course they can (and will). When I have to approach this I usually > > > just snif the encoding of the strings I recieved and then feed them > > > to iconv and friends before doing any processing. A library that > > > downloads stuff off the Internet should be (IMO) aware of > > > the charset madness and decode the strings for me. If it can decode them, it can flag them. It has to be aware - that’s it. > Trust me, when multibyte/Unicode handling is optional, 80% of > libraries do it wrong. Re-read the links above if you don’t believe. But they get the very foundation wrong. In Python functions that take multiple strings can only thake them in one encoding. It is impossible to concatenate differently encoded strings. Of course, this is bound to fail. In the other case they use a database with poor support for unicode, and mysql that does exactly the same thing ruby does right now - works with strings as arrays of bytes. Of course, this is going to break. Neither is the case when the strings carry information about their encoding, and the string functions can handle strings encoded differently. The fact that there are libraries and languages with poor unicode support does not mean it must be always poor. Thanks Michal ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 17, 2006, 1:11pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/54 "2006-06-17T13:11:22Z")
On Thu, Jun 15, 2006 at 07:59:54PM +0900, Michal S. wrote: > > stream is encoded. This will sort of be like $KCODE but on an > > The string methods should not just blindly operate on bytes but use > the encoding information to operate on characters rather than bytes. > Sure something like byte\_length is needed when the string is stored > somewhere outside Ruby but standard string methods should work with > character offsets and characters, not byte offsets nor bytes. I empathically agree. I’ll even repeat and propose a new Plan for Unicode Strings in Ruby 2.0 in 10 points: 1. Strings should deal in characters (code points in Unicode) and not in bytes, and the public interface should reflect this. 2. Strings should neither have an internal encoding tag, nor an external one via $KCODE. The internal encoding should be encapsulated by the string class completely, except for a few related classes which may opt to work with the gory details for performance reasons. The internal encoding has to be decided, probably between UTF-8, UTF-16, and UTF-32 by the String class implementor. 3. Whenever Strings are read or written to/from an external source, their data needs to be converted. The String class encapsulates the encoding framework, likely with additional helper Modules or Classes per external encoding. Some methods take an optional encoding parameter, like #char(index, encoding=:utf8), or #to\_ary(encoding=:utf8), which can be used as helper Class or Module selector. 4. IO instances are associated with a (modifyable) encoding. For stdin, stdout this can be derived from the locale settings. String-IO operations work as expected. 5. Since the String class is quite smart already, it can implement generally useful and hard (in the domain of Unicode) operations like case folding, sorting, comparing etc. 6. More exotic operations can easily be provided by additional libraries because of Ruby’s open classes. Those operations may be coded depending on on String’s public interface for simplicissity, or work with the internal representation directly for performance. 7. This approach leaves open the possibility of String subclasses implementing different internal encodings for performance/space tradeoff reasons which work transparently together (a bit like FixInt and BigInt). 8. Because Strings are tightly integrated into the language with the source reader and are used pervasively, much of this cannot be provided by add-on libraries, even with open classes. Therefore the need to have it in Ruby’s canonical String class. This will break some old uses of String, but now is the right time for that. 9. The String class does not worry over character representation on-screen, the mapping to glyphs must be done by UI frameworks or the terminal attached to stdout. 10. Be flexible. This approach has several advantages and a few disadvantages, and I’ll try to bring in some new angles to this now too: _Advantages_ -POL, Encapsulation- All Strings behave exactly the same everywhere, are predictable, and do the hard work for their users. -Cross Library Transparency- No String user needs to worry which Strings to pass to a library, or worry which Strings he will get from a library. With Web-facing libraries like rails returning encoding-tagged Strings, you would be likely to get Strings of all possible encodings otherwise, and isthe String user prepared to deal with this properly? This is a _big_ deal IMNSHO. -Limited Conversions- Encoding conversions are limited to the time Strings are created or written or explicitly transformed to an external representation. -Correct String Operations- Even basic String operations are very hard in the world of Unicode. If we leave the String users to look at the encoding tags and sort it out themselves, they are bound to make mistakes because they don’t care, don’t know, or have no time. And these mistakes may be _security_ _sensitive_, since most often credentials are represented as Strings too. There already have been exploits related to Unicode. _Disadvantages_ (with mitigating reasoning of course) - String users need to learn that #byte\_length(encoding=:utf8) \>= #size, but that’s not too hard, and applies everywhere. Users do not need to learn about an encoding tag, which is surely worse to handle for them. - Strings cannot be used as simple byte buffers any more. Either use an array of bytes, or an optimized ByteBuffer class. If you need regular expresson support, RegExp can be extended for ByteBuffers or even more. - Some String operations may perform worse than might be expected from a naive user, in both the time or space domain. But we do this so the String user doesn’t need to himself, and are problably better at it than the user too. - For very simple uses of String, there might be unneccessary conversions. If a String is just to be passed through somewhere, without inspecting or modifying it at all, in- and outwards conversion will still take place. You could and should use a ByteBuffer to avoid this. - This ties Ruby’s String to Unicode. A safe choice IMHO, or would we really consider something else? Note that we don’t commit to a particular encoding of Unicode strongly. - More work and time to implement. Some could call it over-engineered. But it will save a lot of time and troubles when shit hits the fan and users really do get unexpected foreign characters in their Strings. I could offer help implementing it, although I have never looked at ruby’s source, C-extensions, or even done a lot of ruby programming yet. Close to the start of this discussion Matz asked what the problem with current strings really was for western users. Somewhere later he concluded case folding. I think it is more than that: we are lazy and expect character handling to be always as easy as with 7 bit ASCII, or as close as possible. Fixed 8-bit codepages worked quite fine most of the time in this regard, and breakage was limited to special characters only. Now let’s ask the question in reverse: are eastern programmers so used to doing elaborate byte-stream to character handling by hand they don’t recognize how hard this is any more? Surely it is a target for DRY if I ever saw one. Or are there actual problems not solveable this way? I looked up the mentioned Han-Unification issue, and as far as I understood this could be handled by future Unicode revisions allocating more characters, outside of Ruby, but I don’t see how it requires our Strings to stay dumb byte buffers. Jürgen ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 17, 2006, 3:51pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/55 "2006-06-17T15:51:46Z")
On Saturday 17 June 2006 13:08, Juergen S. wrote: > On Thu, Jun 15, 2006 at 07:59:54PM +0900, Michal S. wrote: > […] > > > The string methods should not just blindly operate on bytes but > > use the encoding information to operate on characters rather than > > bytes. Sure something like byte\_length is needed when the string > > is stored somewhere outside Ruby but standard string methods > > should work with character offsets and characters, not byte > > offsets nor bytes. > > I empathically agree. I’ll even repeat and propose a new Plan for > Unicode Strings in Ruby 2.0 in 10 points: Juergen, I agree with most of what you have written. I will add my thoughts. > 1. Strings should deal in characters (code points in Unicode) and > not in bytes, and the public interface should reflect this. > > 2. Strings should neither have an internal encoding tag, nor an > external one via $KCODE. The internal encoding should be > encapsulated by the string class completely, except for a few > related classes which may opt to work with the gory details for > performance reasons. The internal encoding has to be decided, > probably between UTF-8, UTF-16, and UTF-32 by the String class > implementor. Full ACK. Ruby programs shouldn’t need to care about the _internal_ string encoding. External string data is treated as a sequence of bytes and is converted to Ruby strings through an encoding API. > 1. Whenever Strings are read or written to/from an external source, > their data needs to be converted. The String class encapsulates the > encoding framework, likely with additional helper Modules or > Classes per external encoding. Some methods take an optional > encoding parameter, like #char(index, encoding=:utf8), or > #to\_ary(encoding=:utf8), which can be used as helper Class or > Module selector. I think the encoding/decoding API should be separated from the String class. IMO, the most important change is to strictly differentiate between arbitrary binary data and character data. Character data is represented by an instance of the String class. I propose adding a new core class, maybe call it ByteString (or ByteBuffer, or Buffer, whatever) to handle strings of bytes. Given a specific encoding, the encoding API converts ByteStrings to Strings and vice versa. This could look like: ``` my_character_str = Encoding::UTF8.encode(my_byte_buffer) buffer = Encoding::UTF8.decode(my_character_str) ``` > 1. IO instances are associated with a (modifyable) encoding. For > stdin, stdout this can be derived from the locale settings. > String-IO operations work as expected. I propose one of: 1. A low level IO API that reads/writes ByteBuffers. String IO can be implemented on top of this byte-oriented API. 2. The File class/IO module as of current Ruby just gets additional methods for binary IO (through ByteBuffers) and an encoding attribute. The methods that do binary IO don’t need to care about the encoding attribute. I think 1) is cleaner. > 1. Since the String class is quite smart already, it can implement > generally useful and hard (in the domain of Unicode) operations > like case folding, sorting, comparing etc. If the strings are represented as a sequence of Unicode codepoints, it is possible for external libraries to implement more advanced Unicode operations. Since IMO a new “character” class would be overkill, I propose that the String class provides codepoint-wise iteration (and indexing) by representing a codepoint as a Fixnum. AFAIK a Fixnum consists of 31 bits on a 32 bit machine, which is enough to represent the whole range of unicode codepoints. > 1. More exotic operations can easily be provided by additional > libraries because of Ruby’s open classes. Those operations may be > coded depending on on String’s public interface for simplicissity, > or work with the internal representation directly for performance. > > 2. This approach leaves open the possibility of String subclasses > implementing different internal encodings for performance/space > tradeoff reasons which work transparently together (a bit like > FixInt and BigInt). I think providing different internal String representations would be too much work, especially for maintenance in the long run. > 1. Be flexible. > The advantages of this proposal over the current situation and > tagging a string with an encoding are: - There is only one internal string (where string means a string of characters) representation. String operations don’t need to be written for different encodings. - No need for $KCODE. - Higher abstraction. - Separation of concerns. I always found it strange that most dynamic languages simply mix handling of character and arbitrary binary data (just think of pack/unpack). - Reading of character data in one encoding and representing it in other encoding(s) would be easy. It seems that the main argument against using Unicode strings in Ruby is because Unicode doesn’t work well for eastern countries. Perhaps there is another character set that works better that we could use instead of Unicode. The important point here is that there is only _one_ representation of character data Ruby. If Unicode is choosen as character set, there is the question which encoding to use internally. UTF-32 would be a good choice with regards to simplicity in implementation, since each codepoint takes a fixed number of bytes. Consider indexing of Strings: ``` "some string"[4] ``` If UTF-32 is used, this operation can internally be implemented as a simple, constant array lookup. If UTF-16 or UTF-8 is used, this is not possible to implement as an array lookup, since any codepoint before the fifth could occupy more than one (8 bit or 16 bit) unit. Of course there is the argument against UTF-32 that it takes to much memory. But I think that most text-processing done in Ruby spends much more memory on other data structures than in actual character data (just consider an REXML document), but I haven’t measured that 😉 An advantage of using UTF-8 would be that for pure ASCII files no conversion would be necessary for IO. Thank you for reading so far. Just in case Matz decides to implement something similar to this proposal, I am willing to help with Ruby development (although I don’t know much about Ruby’s internals and not too much about Unicode either). I do not have a CS degree and I’m not a Unicode expert, so perhaps the proposal is garbage, in this case please tell me what is wrong about it or why it is not realistic to implement it. ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 17, 2006, 4:16pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/56 "2006-06-17T16:16:36Z")
On 17-jun-2006, at 15:52, Austin Z. wrote: > > 1. Because Strings are tightly integrated into the language with the > > source reader and are used pervasively, much of this cannot be > > provided by add-on libraries, even with open classes. Therefore the > > need to have it in Ruby’s canonical String class. This will break > > some > > old uses of String, but now is the right time for that. > > “Now” isn’t; Ruby 2.0 is. Maybe Ruby 1.9.1. Most probably wise, but I need casefolding and character classes to work since yesteryear. Oniguruma is there but even if you complie with it (which is not the default, still) you don’t get char classes (AFAIK) and you don’t get casefolding. Case-insensitive search/replace quickly becomes bondage. I am maintaining a gem whose test fails due to different regexps in Oniguruma, but I would be able to quickly fix it knowing that Oniguruma is in stable now. > > 1. Be flexible. > > And little is more flexible than Matz’s m17n String. I couldn’t find a proper description of that - as I told already, the thing I’d least prefer would be # get a string from the database p str + my\_unicode\_chars # Ok, bail out with an ugly exception because the author of the DB adaptor didn’t care to send me proper Strings… If strings in the system are allowed to have varying encodings, I don’t understand how the engine is going to upgrade/downgrade strings automatically. Especially remembering that the receiver is on the left, so I actually might get different exceptions going as I do p my\_unicode\_chars + mojikyo\_str # who wins? or p mojikyo\_str + my\_unicode\_chars # who wins? or (especially) p mojikyo\_str + bytestring\_that\_i\_just\_grabbed\_by\_http\_and\_i\_know\_it\_is\_mojikyo\_but\_its\_ not # who wins? ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 17, 2006, 3:54pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/57 "2006-06-17T15:54:54Z")
On 6/17/06, Juergen S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I empathically agree. I’ll even repeat and propose a new Plan for > Unicode Strings in Ruby 2.0 in 10 points: > > 1. Strings should deal in characters (code points in Unicode) and not > in bytes, and the public interface should reflect this. Agree, mostly. Strings should have a way to indicate the buffer size of the String. > 1. Strings should neither have an internal encoding tag, nor an > external one via $KCODE. The internal encoding should be encapsulated > by the string class completely, except for a few related classes which > may opt to work with the gory details for performance reasons. > The internal encoding has to be decided, probably between UTF-8, > UTF-16, and UTF-32 by the String class implementor. Completely disagree. Matz has the right choice on this one. You can’t think in just terms of a pure Ruby implementation – you _must_ think in terms of the Ruby/C interface for extensions as well. > 1. Whenever Strings are read or written to/from an external source, > their data needs to be converted. The String class encapsulates the > encoding framework, likely with additional helper Modules or Classes > per external encoding. Some methods take an optional encoding > parameter, like #char(index, encoding=:utf8), or > #to\_ary(encoding=:utf8), which can be used as helper Class or Module > selector. Conversion should be possible at any time. An “external source” may be an extension that your Ruby program can’t distinguish. Again, this point fails because your #2 is unacceptable. > 1. IO instances are associated with a (modifyable) encoding. For > stdin, stdout this can be derived from the locale settings. String-IO > operations work as expected. Agree, realising that the internal implementation of String must be completely different than you’ve suggested. It is also important to retain _raw_ reading; a JPEG should not be interpreted as Unicode. > 1. Since the String class is quite smart already, it can implement > generally useful and hard (in the domain of Unicode) operations like > case folding, sorting, comparing etc. Agreed, but this would be expected regardless of the actual encoding of a String. > 1. More exotic operations can easily be provided by additional > libraries because of Ruby’s open classes. Those operations may be > coded depending on on String’s public interface for simplicissity, or > work with the internal representation directly for performance. Agreed. > 1. This approach leaves open the possibility of String subclasses > implementing different internal encodings for performance/space > tradeoff reasons which work transparently together (a bit like FixInt > and BigInt). Um. Disagree. Matz’s proposed approach does this; yours does not. Yours, in fact, makes things _much_ harder. > 1. Because Strings are tightly integrated into the language with the > source reader and are used pervasively, much of this cannot be > provided by add-on libraries, even with open classes. Therefore the > need to have it in Ruby’s canonical String class. This will break some > old uses of String, but now is the right time for that. “Now” isn’t; Ruby 2.0 is. Maybe Ruby 1.9.1. > 1. The String class does not worry over character representation > on-screen, the mapping to glyphs must be done by UI frameworks or the > terminal attached to stdout. The String class doesn’t worry about that now. > 1. Be flexible. And little is more flexible than Matz’s m17n String. > This approach has several advantages and a few disadvantages, and I’ll > try to bring in some new angles to this now too: > > _Advantages_ > > -POL, Encapsulation- > > All Strings behave exactly the same everywhere, are predictable, > and do the hard work for their users. Remember: POLS is not an acceptable reason for anything. Matz’s m17n Strings would be predictable, too. a + b would be possible if and only if a and b are the same encoding or one of them is “raw” (which would mean that the other is treated as the defined encoding) _or_ there is a built-in conversion for them. > -Cross Library Transparency- > No String user needs to worry which Strings to pass to a library, or > worry which Strings he will get from a library. With Web-facing > libraries like rails returning encoding-tagged Strings, you would be > likely to get Strings of all possible encodings otherwise, and isthe > String user prepared to deal with this properly? This is a _big_ deal > IMNSHO. This will be true with m17n strings. However, your proposal does _not_ work for Ruby/C interfaced items. Sorry. > -Limited Conversions- > > Encoding conversions are limited to the time Strings are created or > written or explicitly transformed to an external representation. This is a mistake. I may need to know the internal representation of a particular encoding of a String inside of a program. Trust me on this one: I _have_ done some low-level encoding work. Additionally, even though I might have marked a network object as “UTF-8”, I may not know whether it’s _actually_ UTF-8 or not until I get HTTP headers – or worse, a tag. Assuming UTF-8 reading in today’s world is doomed to failure. > -Correct String Operations- > Even basic String operations are very hard in the world of Unicode. If > we leave the String users to look at the encoding tags and sort it out > themselves, they are bound to make mistakes because they don’t care, > don’t know, or have no time. And these mistakes may be _security_ > _sensitive_, since most often credentials are represented as Strings > too. There already have been exploits related to Unicode. This is a misunderstanding on your part. Nothing about Matz’s m17n Strings suggests that String users would have to look at the encoding tags. Merely that they _could_. I suspect that there will be pragma- like behaviours to enforce a particular internal representation at all times. > _Disadvantages_ (with mitigating reasoning of course) > > - String users need to learn that #byte\_length(encoding=:utf8) \>= > #size, but that’s not too hard, and applies everywhere. Users do not > need to learn about an encoding tag, which is surely worse to handle > for them. True, but the encoding tag is not worse. Anyone who assumes that developers can ignore encoding at any time simply _doesn’t_ know about the level of problems that can be encountered. > - Strings cannot be used as simple byte buffers any more. Either use > an array of bytes, or an optimized ByteBuffer class. If you need > regular expresson support, RegExp can be extended for ByteBuffers or > even more. I see no reason for this. > - Some String operations may perform worse than might be expected from > a naive user, in both the time or space domain. But we do this so the > String user doesn’t need to himself, and are problably better at it > than the user too. This is a wash. > - For very simple uses of String, there might be unneccessary > conversions. If a String is just to be passed through somewhere, > without inspecting or modifying it at all, in- and outwards conversion > will still take place. You could and should use a ByteBuffer to avoid > this. This is a wash. > - This ties Ruby’s String to Unicode. A safe choice IMHO, or would we > really consider something else? Note that we don’t commit to a > particular encoding of Unicode strongly. This is a wash. I think that it’s better to leave the options open. After all, it _is_ a hope of mine to have Ruby running on iSeries (AS/400) and _that_ still uses EBCDIC. > - More work and time to implement. Some could call it over-engineered. > But it will save a lot of time and troubles when shit hits the fan and > users really do get unexpected foreign characters in their Strings. I > could offer help implementing it, although I have never looked at > ruby’s source, C-extensions, or even done a lot of ruby programming > yet. I would call it the amount of work necessary. But the work needs to be done for a _variety_ of encodings, and not just Unicode. _Especially_ because of C extensions. > Close to the start of this discussion Matz asked what the problem with > current strings really was for western users. Somewhere later he > concluded case folding. I think it is more than that: we are lazy and > expect character handling to be always as easy as with 7 bit ASCII, or > as close as possible. Fixed 8-bit codepages worked quite fine most of > the time in this regard, and breakage was limited to special > characters only. > Now let’s ask the question in reverse: are eastern programmers so used > to doing elaborate byte-stream to character handling by hand they > don’t recognize how hard this is any more? Surely it is a target for > DRY if I ever saw one. Or are there actual problems not solveable this > way? I looked up the mentioned Han-Unification issue, and as far as I > understood this could be handled by future Unicode revisions > allocating more characters, outside of Ruby, but I don’t see how it > requires our Strings to stay dumb byte buffers. No one has ever suggested that Ruby Strings stay byte buffers. However, blindly choosing Unicode _adds_ unnecessary complexity to the situation. -austin ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 17, 2006, 4:19pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/58 "2006-06-17T16:19:57Z")
On 6/17/06, Stefan L. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Full ACK. Ruby programs shouldn’t need to care about the > _internal_ string encoding. External string data is treated as > a sequence of bytes and is converted to Ruby strings through > an encoding API. This is incorrect. _Most_ Ruby programs won’t need to care about the internal string encoding. Experience suggests, however, that it is _most_. Definitely not all. > Given a specific encoding, the encoding API converts > ByteStrings to Strings and vice versa. > > This could look like: > > ``` > my_character_str = Encoding::UTF8.encode(my_byte_buffer) > buffer = Encoding::UTF8.decode(my_character_str) > > ``` Unnecessarily complex and inflexible. Before you go too much further, I _really_ suggest that you look in the archives and Google to find more about Matz’s m17n String proposal. It’s a really good one, as it allows developers (both pure Ruby and extension) to choose what is appropriate with the ability to transparently convert as well. > > 1. IO instances are associated with a (modifyable) encoding. For > > stdin, stdout this can be derived from the locale settings. > > String-IO operations work as expected. > > I propose one of: > > 1. A low level IO API that reads/writes ByteBuffers. String IO > can be implemented on top of this byte-oriented API. […] > 1. The File class/IO module as of current Ruby just gets > additional methods for binary IO (through ByteBuffers) and > an encoding attribute. The methods that do binary IO don’t > need to care about the encoding attribute. > > I think 1) is cleaner. I think neither is necessary and both would be a mistake. It is, as I indicated to Juergen, sometimes _impossible_ to determine the encoding to be used for an IO until you have some data from the IO already. > > 1. Since the String class is quite smart already, it can implement > > generally useful and hard (in the domain of Unicode) operations like > > case folding, sorting, comparing etc. > > If the strings are represented as a sequence of Unicode codepoints, it > > is possible for external libraries to implement more advanced Unicode > > operations. This would be true regardless of the encoding. > Since IMO a new “character” class would be overkill, I propose that > the String class provides codepoint-wise iteration (and indexing) by > representing a codepoint as a Fixnum. AFAIK a Fixnum consists of 31 > bits on a 32 bit machine, which is enough to represent the whole range > of unicode codepoints. This does not match what Matz will be doing. str = “Fran\303\247ais” str[5] # → “\303\247” This is better than doing a Fixnum representation. It is character iteration, but each character is, itself, a String. > > 1. This approach leaves open the possibility of String subclasses > > implementing different internal encodings for performance/space > > tradeoff reasons which work transparently together (a bit like > > FixInt and BigInt). > > I think providing different internal String representations > > would be too much work, especially for maintenance in the long > > run. If you’re depending on classes to do that, especially given that Ruby’s String, Array, and Hash classes don’t inherit well, you’re right. > The advantages of this proposal over the current situation and > tagging a string with an encoding are: The problem, of course, is that this proposal – and your take on it – don’t account for the m17n String that Matz has planned. The current situation is a mess. But the current situation is _not_ what is planned. I’ve had to do some encoding work for work in the last two years, and while I _prefer_ a UTF-8/UTF-16 internal representation, I also know that’s _impossible_ in some situations and you have to be flexible. I also know that POSIX handles this situation worse than any other setup. With the work that I’ve done on this, Matz is _right_ about this, and the people claiming that Unicode is the Only Way … are wrong. In an ideal world, Unicode would be the correct and only way. In the real world, however, it’s a lot messier, and Ruby has to be aware of that. We can _still_ make it as easy as possible for the common case (which will be UTF-8 encoding data and filenames). But we shouldn’t make the mistake of assuming that the common case is all that Ruby should handle. > - There is only one internal string (where string means a > string of characters) representation. String operations > don’t need to be written for different encodings. This is still (mostly) correct under the m17n String proposal. > - No need for $KCODE. This is true under the m17n String. > - Higher abstraction. This is true under the m17n String. > - Separation of concerns. I always found it strange that most dynamic > languages simply mix handling of character and arbitrary binary data > (just think of pack/unpack). The separation makes things harder most of the time. > - Reading of character data in one encoding and representing it in > other encoding(s) would be easy. This is true under the m17n String. > It seems that the main argument against using Unicode strings in Ruby > is because Unicode doesn’t work well for eastern countries. Perhaps > there is another character set that works better that we could use > instead of Unicode. The important point here is that there is only > _one_ representation of character data Ruby. This is a mistake. > If Unicode is choosen as character set, there is the question which > encoding to use internally. UTF-32 would be a good choice with regards > to simplicity in implementation, since each codepoint takes a fixed > number of bytes. Consider indexing of Strings: Yes, but this would be very hard on memory requirements. There are people who are trying to get Ruby to fit into small-memory environments. This would destroy any chance of that. […] > Thank you for reading so far. Just in case Matz decides to implement > something similar to this proposal, I am willing to help with Ruby > development (although I don’t know much about Ruby’s internals and not > too much about Unicode either). I would suggest that you look for discussions about m17n Strings in Ruby. Matz has this one right. > I do not have a CS degree and I’m not a Unicode expert, so perhaps the > proposal is garbage, in this case please tell me what is wrong about > it or why it is not realistic to implement it. I don’t have a CS degree either, but I have been in the business for a _long_ time and I’ve been immersed in Unicode and encoding issues for the last two years. If everyone used Unicode – and POSIX weren’t stupid – your proposal would be much more realistic. I _agree_ that Ruby should encourage the use of Unicode as much as is practical. But it also shouldn’t tie our hands like other programming languages do. -austin ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 17, 2006, 4:26pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/59 "2006-06-17T16:26:19Z")
On 6/17/06, Julian ‘Julik’ Tarkhanov [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > (AFAIK) and you don’t get casefolding. Case-insensitive search/replace > quickly becomes bondage. I don’t disagree. But you’re _not_ going to get those features, in all likelihood, in a Ruby 1.8.x release. It would be a breaking release. Oniguruma is the default for Ruby 1.9+. If there are things missing, work with the developer. > I am maintaining a gem whose test fails due to different regexps in > Oniguruma, but I would be able to quickly fix it knowing that > Oniguruma is in stable now. I don’t think that Oniguruma is in stable (1.8.x); I _don’t_ think it will be enabled as default in stable. Again, it’s a breaking change. > > > 1. Be flexible. > > > And little is more flexible than Matz’s m17n String. > > > I couldn’t find a proper description of that - as I told already, the > > > thing I’d least prefer would be > # get a string from the database > > p str + my\_unicode\_chars # Ok, bail out with an ugly exception > because the author of the DB adaptor didn’t care to send me proper > Strings… The DB adaptor, of course, will have to look at the encoding that the DB is using. > p mojikyo\_str + my\_unicode\_chars # who wins? > > or (especially) > > p mojikyo\_str + > bytestring\_that\_i\_just\_grabbed\_by\_http\_and\_i\_know\_it\_is\_mojikyo\_but\_its\_ > not # who wins? Consider coersion in Numerics (ri Numeric#coerce). A similar framework can be built for Strings. -austin ---
**Author:** ![rhaus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e68b1a/32.png) [@rhaus](https://www.ruby-forum.com/u/rhaus)\ **Post date:** [June 17, 2006, 6:04pm UTC](https://www.ruby-forum.com/t/unicode-roadmap/63296/60 "2006-06-17T18:04:47Z")
On 17/06/06, Austin Z. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > - This ties Ruby’s String to Unicode. A safe choice IMHO, or would we > > really consider something else? Note that we don’t commit to a > > particular encoding of Unicode strongly. > > This is a wash. I think that it’s better to leave the options open. > After all, it _is_ a hope of mine to have Ruby running on iSeries > (AS/400) and _that_ still uses EBCDIC. Not to mention that Matz has explicitly stated in the past that he wants Ruby to support other encodings (TRON, Mojikyo, etc.) that aren’t compatible with a Unicode internal representation. Not tying String to Unicode is also the right thing to do: it allows for future developments. Java’s weird encoding system is entirely down to the fact that it standardised on UCS-2; when codepoints beyond 65535 arrived, they had to be shoehorned in via an ugly hack. As far as possible, Ruby should avoid that trap. Paul. [Previous page](https://www.ruby-forum.com/t/unicode-roadmap/63296.md?page=2) [Next page](https://www.ruby-forum.com/t/unicode-roadmap/63296.md?page=4)