# why we must get rid of unicode.semantics switch ASAP - Thread: https://externals.io/message/34841 - Original archive: https://news-web.php.net/php.internals/34841 - Started by Antony Dovgal on 2008-01-21T14:38:00+00:00 - Messages: 47 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/34841.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #34841 — Antony Dovgal — 2008-01-21T14:38:00+00:00 URL: https://externals.io/message/34841#34841 6 reasons why we must to get rid of The Switch ASAP ---------------------------------------------------- 1) it gives users false sense of "compatibility" when no compatibility is even planned; 2) it's supposed to mean compatibility, but can be changed only in php.ini, which means users would still have to maintain 2 versions of their software: one for On and second for Off. 3) 2+ bigger codebase [1] (with lots of duplicates because we have to do same things in native and unicode modes); 4) increases the maintenance costs a lot [2]; 5) this is yet another reincarnation of ze1_compatibility switch. I believe most of the people here agree it was a total failure - untested, unneeded and, most important, not working thing that complicates user's and developer's lives. Those who want compatibility may and will stay with PHP5 forever, those who need Unicode support will use PHP6. 6) we need to remove the switch ASAP and make PHP6 Unicode-only before people spend their time doing useless "compatibility ports" of their applications. --- [1] http://cvs.php.net/viewvc.cgi/php-src/ext/standard/string.c?revision=1.664&view=markup Don't click this link if you want to sleep well today. [2] Here is a typical problem: http://bugs.php.net/bug.php?id=42861 Try to fix it without looking at the solution and you'll see what I mean. -- Wbr, Antony Dovgal --- ## #34842 — Marco — 2008-01-21T14:42:21+00:00 URL: https://externals.io/message/34841#34842 · In reply to: #34841 > 6 reasons why we must to get rid of The Switch ASAP Couldn't agree more! Regards Marco --- ## #34843 — Geoffrey Sneddon — 2008-01-21T15:01:52+00:00 URL: https://externals.io/message/34841#34843 · In reply to: #34841 > 2) it's supposed to mean compatibility, but can be changed only in > php.ini, which > means users would still have to maintain 2 versions of their software: > one for On and second for Off. I think this is the biggest issue for anyone writing software is the fact that is can only be changed in php.ini — it may well be fine if it can be set on a per request basis (though there will still be issues with that (software libraries that have to cope with both types of request, for example)). > 3) 2+ bigger codebase [1] (with lots of duplicates because we have > to do > same things in native and unicode modes); From the cross-reference I assume you mean PHP's codebase. We still need binary string support — Unicode is only suitable for textual content. Images, for example, are binary data and we still need binary strings for them. Not everything people deal with in PHP is a textual string. -- Geoffrey Sneddon 6 reasons why we must to get rid of The Switch ASAP > ---------------------------------------------------- > 1) it gives users false sense of "compatibility" when no compatibility is even planned; > 2) it's supposed to mean compatibility, but can be changed only in php.ini, which > means users would still have to maintain 2 versions of their software: > one for On and second for Off. > 3) 2+ bigger codebase [1] (with lots of duplicates because we have to do > same things in native and unicode modes); > 4) increases the maintenance costs a lot [2]; > 5) this is yet another reincarnation of ze1_compatibility switch. Which is the worst mistake ever imo - If you wanted PHP 4 you would simply use PHP 4. Now if you want PHP 5 just damn use PHP 5. > I believe most of the people here agree it was a total failure - untested, unneeded and, > most important, not working thing that complicates user's and developer's lives. > Those who want compatibility may and will stay with PHP5 forever, > those who need Unicode support will use PHP6. > 6) we need to remove the switch ASAP and make PHP6 Unicode-only before people spend > their time doing useless "compatibility ports" of their applications. > --- > [1] > http://cvs.php.net/viewvc.cgi/php-src/ext/standard/string.c?revision=1.664&view=markup > Don't click this link if you want to sleep well today. > [2] Here is a typical problem: http://bugs.php.net/bug.php?id=42861 > Try to fix it without looking at the solution and you'll see what I mean. > -- > Wbr, > Antony Dovgal Best regards, Marcus --- ## #34851 — Tomas Kuliavas — 2008-01-21T17:22:32+00:00 URL: https://externals.io/message/34841#34851 · In reply to: #34850 > > […] > Which is the worst mistake ever imo - If you wanted PHP 4 you would simply > use PHP 4. Now if you want PHP 5 just damn use PHP 5. And if you don't control PHP version used by end user? Only bad in-house apps are written for one specific PHP version and setup. -- Tomas --- ## #34853 — Marcus Boerger — 2008-01-21T18:55:43+00:00 URL: https://externals.io/message/34841#34853 · In reply to: #34851 Hello Tomas, you're point being? Without the requested change here you would have one more version, resulting in PHP 5.*, PHP 6.*-unicode, PHP6.*-native. marcus Monday, January 21, 2008, 6:22:32 PM, you wrote: > > […] > And if you don't control PHP version used by end user? Only bad in-house > apps are written for one specific PHP version and setup. > -- > Tomas Best regards, Marcus --- ## #34854 — Tomas Kuliavas — 2008-01-21T19:38:19+00:00 URL: https://externals.io/message/34841#34854 · In reply to: #34853 > > […] > > > […] > > you're point being? Without the requested change here you would have one > more version, resulting in PHP 5.*, PHP 6.*-unicode, PHP6.*-native. there is only a little difference between php5 and php6 with unicode.semantics off. php6 with unicode.semantics on design is broken. It replaces standard functions that worked same way through all PHP4-5 versions and forces use of code that is totally backwards incompatible. binary and unicode typecasting triggers `E_PARSE` errors in older PHP versions. I can't mix PHP6 and older PHP code in one script or library or function. PHP introduced changes similar to unicode.semantics=on with mbstring function overloading. When I learned about it, I've stopped trusting ereg and string functions. With mbstring overloading I still have options to disable broken design. With unicode semantics I am forced to use features provided by interpreter instead doing things the way I want and having better controls over script. PHP with unicode.semantics on is more suitable for novice developers who are not familiar with character sets and lazy developers, who want their PHP4-5 code to become Unicode aware without any changes on their side. If PHP4-5 code works with multiple charsets and 8bit data, it will break in PHP6. I don't care if you remove this setting. I'll find the way to make my code work. but don't expect me to remain silent if you say that it is a good thing. It is good for PHP codebase. It is not good for portable PHP script developers. Removal of setting forces developers to drop 5.2.0 and older versions or to maintain two library versions. If setting remains, developers can ask to turn it off. PHP_INI_SYSTEM is php.ini and httpd.conf. -- Tomas --- ## #34856 — Jeremy Privett — 2008-01-21T19:56:15+00:00 URL: https://externals.io/message/34841#34856 · In reply to: #34854 Tomas Kuliavas wrote: > > […] > > there is only a little difference between php5 and php6 with > unicode.semantics off. php6 with unicode.semantics on design is broken. It > replaces standard functions that worked same way through all PHP4-5 > versions and forces use of code that is totally backwards incompatible. > binary and unicode typecasting triggers `E_PARSE` errors in older PHP > versions. I can't mix PHP6 and older PHP code in one script or library or > function. > > PHP introduced changes similar to unicode.semantics=on with mbstring > function overloading. When I learned about it, I've stopped trusting ereg > and string functions. With mbstring overloading I still have options to > disable broken design. With unicode semantics I am forced to use features > provided by interpreter instead doing things the way I want and having > better controls over script. > > PHP with unicode.semantics on is more suitable for novice developers who > are not familiar with character sets and lazy developers, who want their > PHP4-5 code to become Unicode aware without any changes on their side. If > PHP4-5 code works with multiple charsets and 8bit data, it will break in > PHP6. > > I don't care if you remove this setting. I'll find the way to make my code > work. but don't expect me to remain silent if you say that it is a good > thing. It is good for PHP codebase. It is not good for portable PHP script > developers. Removal of setting forces developers to drop 5.2.0 and older > versions or to maintain two library versions. If setting remains, > developers can ask to turn it off. PHP_INI_SYSTEM is php.ini and > httpd.conf. > > And most people on shared servers don't have access to httpd.conf. As long as it's not PHP_INI_PERDIR, unicode.semantics will never be an acceptable solution. In my opinion, even if it were PERDIR, it still wouldn't be an acceptable solution as you'll still have portability problems, either way. -- Jeremy Privett C.E.O. & C.S.A. Omega Vortex Corporation Web: http://www.omegavortex.net E-Mail: jeremy@omegavortex.net Please note: This message has been sent with information that could be confidential and meant only for the intended recipient. If you are not the intended recipient, please delete all copies and inform us of the error as soon as possible. Thank you for your cooperation. --- ## #34857 — Geoffrey Sneddon — 2008-01-21T21:01:37+00:00 URL: https://externals.io/message/34841#34857 · In reply to: #34854 > > […] > > there is only a little difference between php5 and php6 with > unicode.semantics off. php6 with unicode.semantics on design is > broken. It > replaces standard functions that worked same way through all PHP4-5 > versions and forces use of code that is totally backwards > incompatible. I wholly agree that this is idiotic — I'd much rather see an approach to Unicode strings like there is in Python, namely that any string is binary data unless explicitly prefixed by u (e.g., gettype('a') == 'binary' && gettype(u'a') == 'unicode'). We already have support for similar syntax in the form of b'…'. This alternative, unlike the current situation, allows code written for PHP < 6 to run on PHP 6 without issue. > binary and unicode typecasting triggers `E_PARSE` errors in older PHP > versions. I can't mix PHP6 and older PHP code in one script or > library or > function. (binary) is fine on PHP 5.2.1 and above, though it's still a far from ideal situation (as it relies on unicode.runtime_encoding unlike b'…'). It is most certainly possible to get code working on both PHP 5 and PHP 6, but it requires a lot of verbose code (that can, on the upside, all be abstracted into unicode/binary classes that each deal with a different string type (which also gives an excuse for a PHP 5 userland implementation of unicode, so you can rely on unicode support on both PHP 5 and PHP 6)). > PHP introduced changes similar to unicode.semantics=on with mbstring > function overloading. When I learned about it, I've stopped trusting > ereg > and string functions. With mbstring overloading I still have options > to > disable broken design. With unicode semantics I am forced to use > features > provided by interpreter instead doing things the way I want and having > better controls over script. I already refuse to support mbstring.func_overload as there is absolutely no way to get back to dealing with binary strings. If PHP 6 continues on how it is I may well end up not supporting it either. If PHP 6 breaks backwards compatibility I'd rather it did so far more sweepingly, without options to give the allusion of it being compatible with previous versions. -- Geoffrey Sneddon > 6 reasons why we must to get rid of The Switch ASAP > ---------------------------------------------------- Amen! Derick --- ## #34862 — Jan Schneider — 2008-01-22T00:19:20+00:00 URL: https://externals.io/message/34841#34862 · In reply to: #34841 Zitat von Antony Dovgal : > 6 reasons why we must to get rid of The Switch ASAP Having maintained a huge Unicode compatible codebase in PHP4 for the last few years, I know which PITA it already is today, having to consider the availability of mbstring and iconv, or dealing with different input, processing, backend, and output charsets. I don't event want to start thinking about adding unicode semantics to that equation. Drop it. Jan. -- Do you need professional PHP or Horde consulting? http://horde.org/consulting/ --- ## #34870 — Tim Starling — 2008-01-22T04:21:52+00:00 URL: https://externals.io/message/34841#34870 · In reply to: #34841 Antony Dovgal wrote: > 6 reasons why we must to get rid of The Switch ASAP > ---------------------------------------------------- > 1) it gives users false sense of "compatibility" when no compatibility is even planned; > > 2) it's supposed to mean compatibility, but can be changed only in php.ini, which > means users would still have to maintain 2 versions of their software: > one for On and second for Off. > +1, I'm not looking forward to implementing support for both values in MediaWiki. As for PHP 6 generally: there needs to be a solid migration path, such as forwards-compatible syntax introduced to PHP 5. MediaWiki has extensive support for unicode in PHP 5, including a pure PHP implementation of NFC, cross-script and confusable character checks, extensive parsing of UTF-8 text using regexes both with and without /u, and megabytes of localisations in the form of PHP source files with UTF-8 string literals. Porting all this to a UTF-16-based environment would be a hassle, and we don't gain anything from it in terms of features for our users. I'd hate to end up in an adversarial situation, where developers working in PHP are forced to boycott or fork PHP 6. That's why a simple migration path is important. -- Tim Starling --- ## #34872 — Andrei Zmievski — 2008-01-22T04:25:06+00:00 URL: https://externals.io/message/34841#34872 · In reply to: #34870 > As for PHP 6 generally: there needs to be a solid migration path, > such as forwards-compatible syntax introduced to PHP 5. MediaWiki > has extensive support for unicode in PHP 5, including a pure PHP > implementation of NFC, cross-script and confusable character > checks, extensive parsing of UTF-8 text using regexes both with and > without /u, and megabytes of localisations in the form of PHP > source files with UTF-8 string literals. > > Porting all this to a UTF-16-based environment would be a hassle, > and we don't gain anything from it in terms of features for our > users. I'd hate to end up in an adversarial situation, where > developers working in PHP are forced to boycott or fork PHP 6. > That's why a simple migration path is important. > I don't think that "porting to a UTF-16 environment" in your case is that hard at all. UTF-8 source files will work transparently with proper script encoding setting, PCRE regexes work the same way, and you could replace your own implementations of NFC, etc with PHP provided ones. -Andrei --- ## #34873 — David M. — 2008-01-22T04:31:27+00:00 URL: https://externals.io/message/34841#34873 · In reply to: #34870 Without repeating too much of what has already been said, phpBB3 runs with its own normalizer (NF[CD]K?) and a full implementation of case folding along with all sorts of other goodies. For us, it would be best if semantics were off. Then we could trivially determine whether or not we should use functions built into PHP or if we should use functions we have written. unicode.semantics saves us nothing... --- ## #34875 — Cristian Rodriguez — 2008-01-22T05:49:39+00:00 URL: https://externals.io/message/34841#34875 · In reply to: #34841 2008/1/21, Antony Dovgal : > 6) we need to remove the switch ASAP Yes :) I urge you to do this, the introduction of this setting is probably the worst design mistake in PHP history after safe_mode and register_globals . Please withdrawn this insanity before it is too late, if you dont, it will probably not work correctly anyway.. Deeply concerned, Cristian. -- http://www.cristianrodriguez.net --- ## #34879 — Pierre — 2008-01-22T10:29:15+00:00 URL: https://externals.io/message/34841#34879 · In reply to: #34841 > > 6 reasons why we must to get rid of The Switch ASAP I was +1 months ago, I'm still +1 now :) -- Pierre http://blog.thepimp.net | http://www.libgd.org --- ## #34880 — Gwynne Raskind — 2008-01-22T11:34:40+00:00 URL: https://externals.io/message/34841#34880 · In reply to: #34879 > > > […] > I was +1 months ago, I'm still +1 now :) I'll throw in my +1 too. That's right, I'm still alive! :) -- Gwynne, Daughter of the Code "This whole world is an asylum for the incurable." --- ## #34881 — Richard Lynch — 2008-01-23T02:32:28+00:00 URL: https://externals.io/message/34841#34881 · In reply to: #34841 > > 6 reasons why we must to get rid of The Switch ASAP > ---------------------------------------------------- I was +1... Until folks started posting that old PHP scripts won't run as-is in PHP 6?... That's just daft... When my webhost upgrades to PHP 6, I need all my old scripts to just keep on chugging away, as much as possible... I really think we're stuck with the default "string" being an old-school binary string, unless you want to lose a LOT of users in a hurry, or have PHP 5 stick around forever and ever. -- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/from/lynch Yeah, I get a buck. So? --- ## #34883 — Alexey Zakhlestin — 2008-01-23T07:20:40+00:00 URL: https://externals.io/message/34841#34883 · In reply to: #34841 +1 remove switch make unicode strings default > > 6 reasons why we must to get rid of The Switch ASAP > ---------------------------------------------------- > 1) it gives users false sense of "compatibility" when no compatibility is even planned; > > 2) it's supposed to mean compatibility, but can be changed only in php.ini, which > means users would still have to maintain 2 versions of their software: > one for On and second for Off. > > 3) 2+ bigger codebase [1] (with lots of duplicates because we have to do > same things in native and unicode modes); > > 4) increases the maintenance costs a lot [2]; > > 5) this is yet another reincarnation of ze1_compatibility switch. > I believe most of the people here agree it was a total failure - untested, unneeded and, > most important, not working thing that complicates user's and developer's lives. > Those who want compatibility may and will stay with PHP5 forever, > those who need Unicode support will use PHP6. > > 6) we need to remove the switch ASAP and make PHP6 Unicode-only before people spend > their time doing useless "compatibility ports" of their applications. > > > --- > [1] http://cvs.php.net/viewvc.cgi/php-src/ext/standard/string.c?revision=1.664&view=markup > Don't click this link if you want to sleep well today. > > [2] Here is a typical problem: http://bugs.php.net/bug.php?id=42861 > Try to fix it without looking at the solution and you'll see what I mean. > > -- > Wbr, > Antony Dovgal > > -- > > > > -- Alexey Zakhlestin http://blog.milkfarmsoft.com/ --- ## #34884 — Lokrain — 2008-01-23T07:30:48+00:00 URL: https://externals.io/message/34841#34884 · In reply to: #34883 +1 Remove switch. Make unicode strings default. --- ## #34890 — Chris Stockton — 2008-01-23T19:14:20+00:00 URL: https://externals.io/message/34841#34890 · In reply to: #34881 I partially agree, I have been watching this discussion and it's funny how we have such a class of high end developers saying to break old PHP code. But, the majority of the success of PHP is not due to this small class of high end developers, it's due to it's availability in a shared hosting environment, and the ease of use for beginners, and the oodles of fairly poor quality code that is easy to copy and paste onto peoples websites. Look at the adoption of php4, many webhosts haven't even updated to PHP5 completely due to things like register_globals and small backwards compatibility breakage. The list of problems is small and correctable, if you give system engineers at all of these hosting companies the choice of A. Upgrade to php6 and drive support calls through the roof, or B. Stay at PHP4/5 for eternity until a more (insert your complaints / rants here) language comes along to dethrone PHP. Problem is, PHP has been built to great success based on it's early foundation, but now a group of high class developers want it to be more then PHP was built onto. You will sacrifice it's success if backwards compatibility is not just, broke, but obliterated. Why change PHP's philosophy? Keep it easy for the new user, keep it successful, and make me work a little more when I want to implement my "high class" development methodologies. I don't mind, I do it already. I write this as a "high class" developer. -1 -Chris > > > […] > > I was +1... > > Until folks started posting that old PHP scripts won't run as-is in > PHP 6?... > > That's just daft... > > When my webhost upgrades to PHP 6, I need all my old scripts to just > keep on chugging away, as much as possible... > > I really think we're stuck with the default "string" being an > old-school binary string, unless you want to lose a LOT of users in a > hurry, or have PHP 5 stick around forever and ever. > > -- > Some people have a "gift" link here. > Know what I want? > I want you to buy a CD from some indie artist. > http://cdbaby.com/from/lynch > Yeah, I get a buck. So? --- ## #34891 — Rasmus Lerdorf — 2008-01-23T19:28:48+00:00 URL: https://externals.io/message/34841#34891 · In reply to: #34890 I don't disagree with this, and that is actually why I insisted on having the unicode-semantics switch from the early days of the Unicode discussions, so you can blame me, again, if you consider it a bad design decision. My take on it was that just about all ISPs would run with Unicode semantics off and that the Unicode semantics on mode was more geared for large standalone applications and sites that wanted the luxury of working natively in their chosen character set without needing to always jump through hoops. If we get rid of the switch, then I agree that we can't make the default string IS_UNICODE. We would be crippling the implementation and taking a step backwards in terms of leading the way in Unicode adoption. The longterm goal for just about everyone has got to be a "Unicode everywhere" approach. It used to be that the Web was primarily a Western single-byte charset phenomena, but that hasn't been the case for years. All major applications out there have implemented various hacks to deal with these issues, some with more success than others. This is what PHP does. We take common Web development pains and try to reduce them. Think back to the pains of XML parsing in PHP 3 and even in PHP 4 compared to today. Ultimately we need to get to Unicode everywhere, and the Unicode semantics switch was an acknowledgement that the world isn't quite ready for that yet. But it sounds like the world isn't ready for the switch either. Without it, I am afraid we will never get there, and that may just be something we have to live with. -Rasmus Chris Stockton wrote: > I partially agree, I have been watching this discussion and it's funny > how we have such a class of high end developers saying to break old > PHP code. But, the majority of the success of PHP is not due to this > small class of high end developers, it's due to it's availability in a > shared hosting environment, and the ease of use for beginners, and the > oodles of fairly poor quality code that is easy to copy and paste onto > peoples websites. > > Look at the adoption of php4, many webhosts haven't even updated to > PHP5 completely due to things like register_globals and small > backwards compatibility breakage. The list of problems is small and > correctable, if you give system engineers at all of these hosting > companies the choice of A. Upgrade to php6 and drive support calls > through the roof, or B. Stay at PHP4/5 for eternity until a more > (insert your complaints / rants here) language comes along to dethrone > PHP. > > Problem is, PHP has been built to great success based on it's early > foundation, but now a group of high class developers want it to be > more then PHP was built onto. You will sacrifice it's success if > backwards compatibility is not just, broke, but obliterated. Why > change PHP's philosophy? Keep it easy for the new user, keep it > successful, and make me work a little more when I want to implement my > "high class" development methodologies. I don't mind, I do it already. > > I write this as a "high class" developer. > > -1 > > -Chris > > > > […] --- ## #34892 — Richard Lynch — 2008-01-23T20:03:28+00:00 URL: https://externals.io/message/34841#34892 · In reply to: #34891 > I don't disagree with this, and that is actually why I insisted on > having the unicode-semantics switch from the early days of the Unicode > discussions, so you can blame me, again, if you consider it a bad > design > decision. Would the world really end for people who write NEW apps in a NEW version of PHP, #6, if they had to put u"foo" to get their nifty new-fangled Unicode strings?... Surely that is better than making a BC break of gigantic proportions for the unwashed masses that don't know a charset from a croquette and having NOBODY move to PHP 6 except a handful of large corporations... -- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/from/lynch Yeah, I get a buck. So? --- ## #34893 — Rasmus Lerdorf — 2008-01-23T20:21:17+00:00 URL: https://externals.io/message/34841#34893 · In reply to: #34892 Richard Lynch wrote: > > > […] > > Would the world really end for people who write NEW apps in a NEW > version of PHP, #6, if they had to put u"foo" to get their nifty > new-fangled Unicode strings?... > > Surely that is better than making a BC break of gigantic proportions > for the unwashed masses that don't know a charset from a croquette and > having NOBODY move to PHP 6 except a handful of large corporations... Like I said, without the unicode semantics switch, we can't make unicode strings default for BC reasons. The switch was there to allow not just large corporations, but also smaller companies and projects not restricted by portability or BC concerns to build stuff from the ground up entirely in Unicode. u"foo" is a hack that will eventually disappear from the various languages that have it or something similar. 10 years from now I doubt anybody could even imagine that you could have a string that didn't carry its character set with it. Unfortunately 10 years ago, I wasn't very concerned about that. -Rasmus --- ## #34894 — Richard Lynch — 2008-01-23T20:32:02+00:00 URL: https://externals.io/message/34841#34894 · In reply to: #34893 > Richard Lynch wrote: > > […] > > Like I said, without the unicode semantics switch, we can't make > unicode > strings default for BC reasons. The switch was there to allow not > just > large corporations, but also smaller companies and projects not > restricted by portability or BC concerns to build stuff from the > ground > up entirely in Unicode. u"foo" is a hack that will eventually > disappear > from the various languages that have it or something similar. 10 > years > from now I doubt anybody could even imagine that you could have a > string > that didn't carry its character set with it. Unfortunately 10 years > ago, I wasn't very concerned about that. Does the switch just change u"foo" to "foo" and "foo" to b"foo" or vice-versa? Or could it be MADE that simple instead of whatever it currently does?... Or maybe it is that simple on the surface, but causes all manner of grief under the hood? -- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/from/lynch Yeah, I get a buck. So? --- Next page: https://externals.io/message/34841.md?page=2