# Unicode Implementation - Thread: https://externals.io/message/19448 - Original archive: https://news-web.php.net/php.internals/19448 - Started by Derick Rethans on 2005-10-06T17:56:34+00:00 - Messages: 31 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/19448.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #19448 — Derick Rethans — 2005-10-06T17:56:34+00:00 URL: https://externals.io/message/19448#19448 Hello! I am thinking that we're doing something with the unicode implementation and that's that we're now getting duplicate implementations of quite some things: functions, internal functions, hash implementations, two ways for storing identifiers... only because we need to support both IS_STRING and IS_UNICODE and unicode=off mode. I think I would prefer an IS_UNICODE/unicode=on only PHP. This would mean that: - no duplicate functionality for tons of functions that will make maintaining the thing very hard - a cleaner (and a bit faster) Unicode implementation - we have a bit less BC. Internally we would only see IS_UNICODE and IS_BINARY, where we can have a small layer around extensions which return IS_STRING where we automatically convert it to and from unicode for those extensions. IS_STRING strings will still exist, but should not be there for the "user level". For things like: $str = unicode_convert($unicode, 'iso-2022'); and $unicode being "IS_UNICODE". $str will now be an IS_BINARY string, with all the restrictions that we already have on those strings (like no automatic conversions). Functions that work on binary strings can be quite limited (we wouldn't need a strtolower for that f.e.), so we are cutting down in a lot of duplicated code. The same goes for not having to support both unicode=off and unicode=on mode, as that can make things a bit complicated too. This will limit functionality on binary strings a bit though, but I think this is 10 times better than an unmaintainable PHP with Unicode support. Besides this, I ran some micro benchmarks on about 600 characters of text with a few functions and benchmarked their behavior between unicode=1 and unicode=0 mode. Results: strrev (100.000 iterations over 600 characters of normalized latin text): unicode off: 1.8secs unicode on: 5.0secs strtoupper (100.000 iterations over the same text): unicode off: 2.2secs unicode on: 7.9secs substr(50, 100) (1.000.000 over the same text): unicode off: 3.9secs unicode on: 11.9secs This is something I find quite not acceptable, and we need to figure out a way on how to optimize this - for substr the penalty is probably what we are using an iterator and not a direct memcpy (because of surrogates), I am not so sure about the others. regards, Derick -- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org --- ## #19451 — Marcus Boerger — 2005-10-06T18:14:29+00:00 URL: https://externals.io/message/19448#19451 · In reply to: #19448 Hello Derick, Thursday, October 6, 2005, 7:56:34 PM, you wrote: > Hello! > I am thinking that we're doing something with the unicode implementation and > that's that we're now getting duplicate implementations of quite some things: > functions, internal functions, hash implementations, two ways for storing > identifiers... only because we need to support both IS_STRING and IS_UNICODE > and unicode=off mode. > I think I would prefer an IS_UNICODE/unicode=on only PHP. +infinity (...) Best regards, Marcus --- ## #19459 — Andrei Zmievski — 2005-10-06T19:55:29+00:00 URL: https://externals.io/message/19448#19459 · In reply to: #19448 > I am thinking that we're doing something with the unicode > implementation and > that's that we're now getting duplicate implementations of quite some > things: > functions, internal functions, hash implementations, two ways for > storing > identifiers... only because we need to support both IS_STRING and > IS_UNICODE > and unicode=off mode. > > I think I would prefer an IS_UNICODE/unicode=on only PHP. > > This would mean that: > - no duplicate functionality for tons of functions that will make > maintaining > the thing very hard This is true. > - a cleaner (and a bit faster) Unicode implementation This is true too. > - we have a bit less BC. "A bit less"? I'd say it would break BC in a major way. People who want to upgrade to PHP 6 would need to rewrite a lot of their scripts. > Internally we would only see IS_UNICODE and IS_BINARY, where we can > have a > small layer around extensions which return IS_STRING where we > automatically > convert it to and from unicode for those extensions. IS_STRING strings > will > still exist, but should not be there for the "user level". > > For things like: > $str = unicode_convert($unicode, 'iso-2022'); > and $unicode being "IS_UNICODE". $str will now be an IS_BINARY string, > with all > the restrictions that we already have on those strings (like no > automatic > conversions). > > Functions that work on binary strings can be quite limited (we > wouldn't need a > strtolower for that f.e.), so we are cutting down in a lot of > duplicated code. > The same goes for not having to support both unicode=off and > unicode=on mode, > as that can make things a bit complicated too. This will limit > functionality on > binary strings a bit though, but I think this is 10 times better than > an > unmaintainable PHP with Unicode support. Sure, if you remove requirement for BC and merge the string/binary semantics, you can use IS_BINARY for all that stuff. > Besides this, I ran some micro benchmarks on about 600 characters of > text with > a few functions and benchmarked their behavior between unicode=1 and > unicode=0 > mode. Results: > > strrev (100.000 iterations over 600 characters of normalized latin > text): > unicode off: 1.8secs > unicode on: 5.0secs > > strtoupper (100.000 iterations over the same text): > unicode off: 2.2secs > unicode on: 7.9secs > > substr(50, 100) (1.000.000 over the same text): > unicode off: 3.9secs > unicode on: 11.9secs > > This is something I find quite not acceptable, and we need to figure > out a way > on how to optimize this - for substr the penalty is probably what we > are using > an iterator and not a direct memcpy (because of surrogates), I am not > so sure > about the others. We can try switching to _UNSAFE versions of the iterator macros - they assume well-formed UTF-16, so they will be somewhat faster. -Andrei --- ## #19483 — Derick Rethans — 2005-10-07T20:35:00+00:00 URL: https://externals.io/message/19448#19483 · In reply to: #19459 > > > […] > > This is true. > > > […] > > This is true too. > > > […] > > "A bit less"? I'd say it would break BC in a major way. People who want to > upgrade to PHP 6 would need to rewrite a lot of their scripts. Can you please specify which things you think that will break? I've gave it some thoughts but couldn't really think of anything serious... > > […] > > We can try switching to _UNSAFE versions of the iterator macros - they > assume well-formed UTF-16, so they will be somewhat faster. That's worth a try - I'll put that on my todo list somewhere. Derick -- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org --- ## #19485 — Ilia Alshanetsky — 2005-10-07T20:52:26+00:00 URL: https://externals.io/message/19448#19485 · In reply to: #19459 Andrei Zmievski wrote: > > […] > > > "A bit less"? I'd say it would break BC in a major way. People who want > to upgrade to PHP 6 would need to rewrite a lot of their scripts. I think most large applications will be in this boat anyway, we may as well do it properly once, so we don't end up hacks on top of hacks just for the sake of BC. > We can try switching to _UNSAFE versions of the iterator macros - they > assume well-formed UTF-16, so they will be somewhat faster. We definitely need to look at that since if upgrading to 6.0 means a 3x slower operation very few people will even consider upgrading. Ilia --- ## #19487 — Rasmus Lerdorf — 2005-10-07T20:55:33+00:00 URL: https://externals.io/message/19448#19487 · In reply to: #19485 Ilia Alshanetsky wrote: > Andrei Zmievski wrote: > > > […] > > > I think most large applications will be in this boat anyway, we may as > well do it properly once, so we don't end up hacks on top of hacks just > for the sake of BC. > > > > […] > > > We definitely need to look at that since if upgrading to 6.0 means a 3x > slower operation very few people will even consider upgrading. Which is why we need the unicode=off switch. I don't think there is any way we can make Unicode PHP as fast as non-Unicode PHP. For people who need Unicode support, Unicode PHP will be faster and easier than any other way for them to get there, but for people who have no need for Unicode it would be really nice to maintain the fast non-Unicode mode. -Rasmus --- ## #19488 — Derick Rethans — 2005-10-07T21:05:07+00:00 URL: https://externals.io/message/19448#19488 · In reply to: #19487 > > […] > > Which is why we need the unicode=off switch. I don't think there is any > way we can make Unicode PHP as fast as non-Unicode PHP. For people who > need Unicode support, Unicode PHP will be faster and easier than any > other way for them to get there, but for people who have no need for > Unicode it would be really nice to maintain the fast non-Unicode mode. What is wrong with PHP 5.1? People don't *have* to upgrade to the unicode enabled PHP if they don't want to. And it would probably be "nice" to have that mode for some users, but should that be over our own back with multiple implementations of everything? regards, Derick -- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org --- ## #19489 — Rasmus Lerdorf — 2005-10-07T21:09:06+00:00 URL: https://externals.io/message/19448#19489 · In reply to: #19488 Derick Rethans wrote: > > > > > […] > > > What is wrong with PHP 5.1? People don't *have* to upgrade to the > unicode enabled PHP if they don't want to. And it would probably be > "nice" to have that mode for some users, but should that be over our own > back with multiple implementations of everything? The "don't upgrade" argument doesn't work. Unless we commit to having two major versions forever where we will add new features. That is a possibility as well of course. Have 2 trees. Unicode-PHP and non-Unicode-PHP and everything that is not Unicode-related will need to be committed to both. -Rasmus --- ## #19490 — George Schlossnagle — 2005-10-07T21:11:16+00:00 URL: https://externals.io/message/19448#19490 · In reply to: #19488 > > > […] > > What is wrong with PHP 5.1? People don't *have* to upgrade to the > unicode enabled PHP if they don't want to. And it would probably be > "nice" to have that mode for some users, but should that be over > our own > back with multiple implementations of everything? Are you suggesting that people who don't want unicode should stick with 5.1 for perpetuity? George --- ## #19491 — Derick Rethans — 2005-10-07T21:24:34+00:00 URL: https://externals.io/message/19448#19491 · In reply to: #19489 > > […] > > The "don't upgrade" argument doesn't work. Unless we commit to having > two major versions forever where we will add new features. That is a > possibility as well of course. Have 2 trees. Unicode-PHP and > non-Unicode-PHP and everything that is not Unicode-related will need to > be committed to both. yeah, and if that allows us to cleanup PHP 6 for real, then I am not even sure if that is such a bad solution. Ofcourse, the new cool features would go to PHP 6 only, but that's now the same wih 4.4 and 5.1 too anyway. But I don't want to jump too quickly to conclusions about the UNicode speed and BC breakage. All I tested is raw string functions and there is no way that the whole application slows down as much as we saw in the few benchmarks that I run. Besides that, we can probably optimize the string functions somewhat anyway. I'd like to see the speed decrease for real applications too. regards, Derick -- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org --- ## #19492 — Ilia Alshanetsky — 2005-10-07T21:38:06+00:00 URL: https://externals.io/message/19448#19492 · In reply to: #19490 George Schlossnagle wrote: > > […] > > > Are you suggesting that people who don't want unicode should stick with > 5.1 for perpetuity? Assuming that 5.1 would be actively maintained and not just for bug fixes, I'd say that is a viable approach. There are plenty of sites that have no use for Unicode as nice as it may be, and much rather retain performance over useless (for them) functionality. Ilia --- ## #19493 — Rasmus Lerdorf — 2005-10-07T21:41:25+00:00 URL: https://externals.io/message/19448#19493 · In reply to: #19492 Ilia Alshanetsky wrote: > George Schlossnagle wrote: > > > […] > > > Assuming that 5.1 would be actively maintained and not just for bug > fixes, I'd say that is a viable approach. There are plenty of sites that > have no use for Unicode as nice as it may be, and much rather retain > performance over useless (for them) functionality. So, you are saying that something like the namespace patch would be added between 5.1.2 and 5.1.3, for example? That doesn't make much sense to me. -Rasmus --- ## #19494 — Derick Rethans — 2005-10-07T21:43:08+00:00 URL: https://externals.io/message/19448#19494 · In reply to: #19493 > > […] > > So, you are saying that something like the namespace patch would be > added between 5.1.2 and 5.1.3, for example? That doesn't make much > sense to me. True, but doing it between 5.1.1 and 5.2.0 would work. Derick -- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org --- ## #19495 — George Schlossnagle — 2005-10-07T21:44:56+00:00 URL: https://externals.io/message/19448#19495 · In reply to: #19493 > Ilia Alshanetsky wrote: > > > […] > > So, you are saying that something like the namespace patch would be > added between 5.1.2 and 5.1.3, for example? That doesn't make much > sense to me. Perhaps we need a separate version fork for the unicode support. I'm thinking one of those nifty unicode glyphs. It could be called 'the language formerly known as PHP'. George --- ## #19496 — Rasmus Lerdorf — 2005-10-07T21:45:59+00:00 URL: https://externals.io/message/19448#19496 · In reply to: #19494 Derick Rethans wrote: > > > > > […] > > > True, but doing it between 5.1.1 and 5.2.0 would work. Right, so effectively the 5.x and 6.x trees would mimic each other. Even major versions = unicode, odd version = non-unicode. I suppose it is an approach, but I think it is a confusing one. -Rasmus --- ## #19497 — Rasmus Lerdorf — 2005-10-07T21:47:01+00:00 URL: https://externals.io/message/19448#19497 · In reply to: #19495 George Schlossnagle wrote: > Perhaps we need a separate version fork for the unicode support. I'm > thinking one of those nifty unicode glyphs. It could be called 'the > language formerly known as PHP'. The Phillipine currency is called PHP. Do they perhaps have a currency symbol? -Rasmus --- ## #19499 — John Coggeshall — 2005-10-07T21:47:14+00:00 URL: https://externals.io/message/19448#19499 · In reply to: #19489 > The "don't upgrade" argument doesn't work. Unless we commit to having > two major versions forever where we will add new features. That is a > possibility as well of course. Have 2 trees. Unicode-PHP and > non-Unicode-PHP and everything that is not Unicode-related will need to > be committed to both. I think this would lead to PHP 6.2 with two different sets of functionality (Joe writes some super-string-manipulating thing for PHP 6.2, never find the time to port it to 6.2-unicode, and now no one else has time to do it either so we release one with the function, one without yet they are both 6.2). While I understand the argument that PHP itself will be slower with Unicode support, and there isn't much we can do about that (we're just doing a heck of a lot more work across the board), I think the best option is to have PHP 6 be completely Unicode. It will break applications, but they can be ported. It will mean PHP is going to be slower, but other languages have found ways to make up for that speed decrease in other places and I am sure we could learn from looking at them for answers to our problem. Cheers, John --- ## #19498 — Ilia Alshanetsky — 2005-10-07T21:47:18+00:00 URL: https://externals.io/message/19448#19498 · In reply to: #19493 Rasmus Lerdorf wrote: > > […] > > > So, you are saying that something like the namespace patch would be > added between 5.1.2 and 5.1.3, for example? That doesn't make much > sense to me. No, we bug fix in 5.1.X and put big features into 5.2, in 5.2.X work on fixing bugs and when we have enough, move to 5.3 and repeat... Ilia --- ## #19503 — Marcus Boerger — 2005-10-07T22:54:53+00:00 URL: https://externals.io/message/19448#19503 · In reply to: #19499 Hello John, Friday, October 7, 2005, 11:47:14 PM, you wrote: > > > […] > I think this would lead to PHP 6.2 with two different sets of > functionality (Joe writes some super-string-manipulating thing for PHP > 6.2, never find the time to port it to 6.2-unicode, and now no one else > has time to do it either so we release one with the function, one > without yet they are both 6.2). > While I understand the argument that PHP itself will be slower with > Unicode support, and there isn't much we can do about that (we're just > doing a heck of a lot more work across the board), I think the best > option is to have PHP 6 be completely Unicode. It will break > applications, but they can be ported. It will mean PHP is going to be > slower, but other languages have found ways to make up for that speed > decrease in other places and I am sure we could learn from looking at > them for answers to our problem. For instance PHP 6 could undergo a design face for some clearing which should imho includecase-sensitivity every where and ups, why is PHP 6 suddenly faster? We could also add a mode that disalloas dynamic class rewriting which is very clever once we have the build in apc or which ever bytecode compiler we chosse and ups! no it is even much faster when using objects. But to me the best thing would be to finally haver a plan. A real plan we stick to. And then outline how long we plan to support 5.x series. Because if we do so we can finally take us some time and change a few things in 6. But maybe some companies like it more to get 6 out asap and have little BC issues every now and then. Those companies with their private code - they don't have a real problem with that. It is the majority of small sites. People that want new features every now and then. And then it is a problem because the majotity of servers come with exactly one PHP version installed. Perhaps it is a good idea to suggest to have PHP 4.*, PHP 5.* and PHP 6.* installed in parallel. Because only if we suggest so that might happen. And if it happens there is no such thing as BC. But well is any body still reading or caring for what i write? Best regards, Marcus --- ## #19504 — Jani Taskinen — 2005-10-07T23:51:15+00:00 URL: https://externals.io/message/19448#19504 · In reply to: #19492 > > George Schlossnagle wrote: > > […] > > Assuming that 5.1 would be actively maintained and not just for bug > fixes, I'd say that is a viable approach. There are plenty of sites that > have no use for Unicode as nice as it may be, and much rather retain > performance over useless (for them) functionality. 5.1, 5.2, 5.3..where was it said that 5.1 has to stop at 5.1 ? :) --Jani --- ## #19506 — David Robley — 2005-10-08T03:23:05+00:00 URL: https://externals.io/message/19448#19506 · In reply to: #19497 Rasmus Lerdorf wrote: > George Schlossnagle wrote: > > […] > > The Phillipine currency is called PHP. Do they perhaps have a currency > symbol? > > -Rasmus http://www.fileformat.info/info/unicode/char/20b1/index.htm An uppercase P with two horizontal bars through the top :-) Cheers -- David Robley Shotgun wedding: a case of wife or death. --- ## #19528 — Adam Maccabee Trachtenberg — 2005-10-09T04:55:45+00:00 URL: https://externals.io/message/19448#19528 · In reply to: #19488 > > > > […] > > What is wrong with PHP 5.1? People don't *have* to upgrade to the > unicode enabled PHP if they don't want to. And it would probably be > "nice" to have that mode for some users, but should that be over our own > back with multiple implementations of everything? I got to agree with Derick. I certainly understand that speed is important, but people somehow manage to develop Web sites in Java, C#, and Perl, and all those languages are Unicode only. We seem to be under the impression that the Unicode speed penalty will be so harsh that a Unicode-only PHP 6 will be too slow for use. However, we don't know that for sure. Yes, it will be slower, when you look at the whole request cycle, it may not matter. Therefore, all the extra work we're doing today seems to be premature optimization (a.k.a. "the root of all evil") that only adds to code complexity and delays PHP 6. We'd be better off building PHP 6 as Unicode-only and spending the extra time that we're currently using on the non-Unicode mode tuning PHP 6 the best we can. If we somehow fail to get it fast enough for a certain subset of users, and those users cannot afford to buy better machines, then and only then should we figure out how solve the problem of making PHP 6 support a non-Unicode mode. -adam -- adam@trachtenberg.com | http://www.trachtenberg.com author of o'reilly's "upgrading to php 5" and "php cookbook" avoid the holiday rush, buy your copies today! --- ## #19546 — Peter Brodersen — 2005-10-10T10:57:05+00:00 URL: https://externals.io/message/19448#19546 · In reply to: #19528 On Sun, 9 Oct 2005 00:55:45 -0400 (EDT), in php.internals adam@trachtenberg.com (Adam Maccabee Trachtenberg) wrote: >We seem to be under the impression that the Unicode speed penalty will >be so harsh that a Unicode-only PHP 6 will be too slow for >use. However, we don't know that for sure. Yes, it will be slower, >when you look at the whole request cycle, it may not matter. I agree - the speed penalty comparsion seems mostly related to php scripts that does nothing else than handling strings. I recall the old discussions (echo vs print vs ?>... schreef in bericht news:s7hkk1l00v0gerlccnsc0937ov977j5og4@4ax.com... On Sun, 9 Oct 2005 00:55:45 -0400 (EDT), in php.internals adam@trachtenberg.com (Adam Maccabee Trachtenberg) wrote: >We seem to be under the impression that the Unicode speed penalty will >be so harsh that a Unicode-only PHP 6 will be too slow for >use. However, we don't know that for sure. Yes, it will be slower, >when you look at the whole request cycle, it may not matter. I agree - the speed penalty comparsion seems mostly related to php scripts that does nothing else than handling strings. I recall the old discussions (echo vs print vs ?>... To: Sent: Thursday, October 13, 2005 8:27 PM Subject: [php] Re: [PHP-DEV] Unicode Implementation > Well, if you want my 2 cents as well, the 2 cents a PHP user is very willing > to share with you guys... > > PHP6 is a major release. BC is a priority, but as far as I'm concerned not > the top priority. I wouldn't mind a unicode-only PHP at all. Like a few > others here, I think the speed penalty won't be huge at all. It's not like > my scripts are 90% string handling (and I assume the unicode benchmark > scripts are). I wouldn't want you guys having a messy codebase just because > you want to support non-unicode and unicode. I also wouldn't want you guys > branching to a non-unicode line of PHP releases. It would just be too much. > As far as I'm concerned, I'd be fine with a unicode only PHP6. Unicode is > progress. I think a lot of the people who don't want it, don't yet see the > positive things it brings. > > Ron > > > > "Peter Brodersen" schreef in bericht > news:s7hkk1l00v0gerlccnsc0937ov977j5og4@4ax.com... > On Sun, 9 Oct 2005 00:55:45 -0400 (EDT), in php.internals > adam@trachtenberg.com (Adam Maccabee Trachtenberg) wrote: > > > […] > > I agree - the speed penalty comparsion seems mostly related to php > scripts that does nothing else than handling strings. > > I recall the old discussions (echo vs print vs ?>... benchmarks. It is true when you compare the specific > functions/language constructs the relative difference in speed is > apparent. But compared to a bunch of other functions that you'll > usually use (database connections, file access, data sorting, etc.) I > imagine that the speed penalty is simply marginal. > > Just my 0.02 \x20B1. > > -- > - Peter Brodersen --- Next page: https://externals.io/message/19448.md?page=2