# Unicode progress [Was: unicode.semantics ad infinitum] - Thread: https://externals.io/message/37727 - Original archive: https://news-web.php.net/php.internals/37727 - Started by Andrei Zmievski on 2008-05-19T19:40:42+00:00 - Messages: 34 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/37727.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #37727 — Andrei Zmievski — 2008-05-19T19:40:42+00:00 URL: https://externals.io/message/37727#37727 Done. See latest commit: switch is removed and mode is defaulted to On. -Andrei > Hi all, > > Is there any chance of getting the unicode.semantics issue sorted soon? > > Done. See latest commit: switch is removed and mode is defaulted to On. Thank you , VERY MUCH Andrei, now we can **really** move forward.. -- "Progress is possible only if we train ourselves to think about programs without thinking of them as pieces of executable code.” - Edsger W. Dijkstra Cristian Rodríguez R. Platform/OpenSUSE - Core Services SUSE LINUX Products GmbH Research & Development http://www.opensuse.org/ --- ## #37729 — Lukas Kahwe Smith — 2008-05-19T21:42:58+00:00 URL: https://externals.io/message/37727#37729 · In reply to: #37728 > Andrei Zmievski escribió: > > […] > > Thank you , VERY MUCH Andrei, now we can **really** move forward.. well the next question is binary or unicode as the default for strings .. regards, Lukas --- ## #37730 — Andrei Zmievski — 2008-05-19T21:54:03+00:00 URL: https://externals.io/message/37727#37730 · In reply to: #37729 Yes, but we should run some sort of test suite and see what exactly breaks with unicode strings as the default ones. On Mon, May 19, 2008 at 2:42 PM, Lukas Kahwe Smith wrote: > > > > Andrei Zmievski escribió: > > […] > > well the next question is binary or unicode as the default for strings .. > > regards, > Lukas --- ## #37732 — Andi Gutmans — 2008-05-20T06:03:21+00:00 URL: https://externals.io/message/37727#37732 · In reply to: #37730 The idea here was to find volunteers who'd try and run their apps and some standard apps on this so we can figure out what the migration path looks like. That'd also give us the necessary data point to make a decision on default for strings. I think none of us right now truly knows what to expect. Andi -----Original Message----- From: Andrei Zmievski [mailto:andrei@gravitonic.com] Sent: Monday, May 19, 2008 2:54 PM To: Lukas Kahwe Smith Cc: Cristian Rodríguez; Steph Fox; internals Subject: Re: [PHP-DEV] Unicode progress [Was: unicode.semantics ad infinitum] Yes, but we should run some sort of test suite and see what exactly breaks with unicode strings as the default ones. On Mon, May 19, 2008 at 2:42 PM, Lukas Kahwe Smith wrote: > > > > Andrei Zmievski escribió: > > […] > > well the next question is binary or unicode as the default for strings .. > > regards, > Lukas --- ## #37733 — Lars Strojny — 2008-05-20T06:42:35+00:00 URL: https://externals.io/message/37727#37733 · In reply to: #37732 Hi Andi, et. al, Am Montag, den 19.05.2008, 23:03 -0700 schrieb Andi Gutmans: > The idea here was to find volunteers who'd try and run their apps and > some standard apps on this so we can figure out what the migration > path looks like. That'd also give us the necessary data point to make > a decision on default for strings. I think none of us right now truly > knows what to expect. This might be the time, where we need to think about are more substantial solution for this problem. What I have in mind is to provide a system, with the current development versions of PHP (today is would be 5_2, 5_3 and HEAD) installed. We would liberally grant Shell-Accounts and DBs to those who are willing to test their web applications there. Say the S9Y-developers get an account to test their blog system, they get three Vhosts: s9y.php6.func.php.net, s9y.php53.func.php.net, s9y.php52.func.php.net. We would continously build CVS snapshots on this systems to make sure they test against the latest version. Projects with more resources could run a selenium grid against the server for their functional tests, the rest would simply test the applications by hand. We could even install a cruisecontrol (resp. phpUnderControl) instance to run projects unit tests if the have some. cu, Lars --- ## #37734 — Antony Dovgal — 2008-05-20T08:07:10+00:00 URL: https://externals.io/message/37727#37734 · In reply to: #37730 > Yes, but we should run some sort of test suite and see what exactly breaks > with unicode strings as the default ones. Right, but in order to do this we need to add filtered streams support to `stream_select()`. At the moment all tests in HEAD fail because of this: Warning: `stream_select()`: cannot cast a filtered stream on this system in /local/qa/head/run-tests.php on line 938 Warning: `stream_select()`: No stream arrays were passed in /local/qa/head/run-tests.php on line 938 -- Wbr, Antony Dovgal --- ## #37740 — Steph Fox — 2008-05-20T13:49:20+00:00 URL: https://externals.io/message/37727#37740 · In reply to: #37734 > At the moment all tests in HEAD fail because of this: > Warning: `stream_select()`: cannot cast a filtered stream on this system in > /local/qa/head/run-tests.php on line 938 > Warning: `stream_select()`: No stream arrays were passed in > /local/qa/head/run-tests.php on line 938 Why not just use the 5_3 run-tests.php, which works? - Steph --- ## #37741 — Antony Dovgal — 2008-05-20T13:51:32+00:00 URL: https://externals.io/message/37727#37741 · In reply to: #37740 > > […] > > Why not just use the 5_3 run-tests.php, which works? run-tests.php in 5_3 uses `stream_select()`, too. And `stream_select()` in HEAD doesn't work because Unicode is now enabled by default. -- Wbr, Antony Dovgal --- ## #37742 — Steph Fox — 2008-05-20T13:58:04+00:00 URL: https://externals.io/message/37727#37742 · In reply to: #37741 > run-tests.php in 5_3 uses `stream_select()`, too. > And `stream_select()` in HEAD doesn't work because Unicode is now enabled by > default. Bizarre. I ran 400-odd tests with run-tests from 5_3 and never saw that error... - Steph > > -- > Wbr, Antony Dovgal --- ## #37743 — Steph Fox — 2008-05-20T14:18:56+00:00 URL: https://externals.io/message/37727#37743 · In reply to: #37742 > > […] > > Bizarre. I ran 400-odd tests with run-tests from 5_3 and never saw that > error... Ah, I have it. My TEST_PHP_EXECUTABLE isn't the same as the one running the script - it's 5.3.0-dev. So make test will fail, but my manual tests don't. I'm seeing 85% pass rate in Zend/tests, investigating. - Steph --- ## #37756 — Edward Z. Yang — 2008-05-20T20:04:03+00:00 URL: https://externals.io/message/37727#37756 · In reply to: #37732 Andi Gutmans wrote: > The idea here was to find volunteers who'd try and run their apps and > some standard apps on this so we can figure out what the migration > path looks like. That'd also give us the necessary data point to make > a decision on default for strings. I think none of us right now truly > knows what to expect. As I a PHP developer on Windows, I pay very close attention to development work on PHP and regularly test my code against PHP 5.3, hand-built from the CVS. Nearly all of the PECL extensions are broken on Windows for PHP 6. The http extension has a very nasty problem: its fatal error appears to loop nearly ad infinitum. This makes compiling snaps locally very difficult--I've given up. Why is the log file 4.4 MB long? Also, as stands it appears most non-trivial applications are fairly broken. Those issues need to be resolved first, before we toss Unicode onto the pile. -- Edward Z. Yang GnuPG: 0x869C48DA HTML Purifier Anti-XSS Filter [[ 3FA8 E9A9 7385 B691 A6FC B3CB A933 BE7D 869C 48DA ]] --- ## #37757 — Edward Z. Yang — 2008-05-20T20:16:33+00:00 URL: https://externals.io/message/37727#37757 · In reply to: #37756 Edward Z. Yang wrote: > Also, as stands it appears most non-trivial applications are fairly > broken. Those issues need to be resolved first, before we toss Unicode > onto the pile. I retract this statement; I didn't realize we had already turned on Unicode semantics. -- Edward Z. Yang GnuPG: 0x869C48DA HTML Purifier Anti-XSS Filter [[ 3FA8 E9A9 7385 B691 A6FC B3CB A933 BE7D 869C 48DA ]] --- ## #37758 — Steph Fox — 2008-05-20T20:21:32+00:00 URL: https://externals.io/message/37727#37758 · In reply to: #37756 Hi Edward, > As I a PHP developer on Windows, I pay very close attention to > development work on PHP and regularly test my code against PHP 5.3, > hand-built from the CVS. Nearly all of the PECL extensions are broken on > Windows for PHP 6. The http extension has a very nasty problem: its > fatal error appears to loop nearly ad infinitum. This makes compiling > snaps locally very difficult--I've given up. Why is the log file 4.4 MB > long? Because nobody's been trying to compile snaps against PHP 6? Basically most extension developers decided to stay with 5_3 for now and haven't migrated their code to work with Unicode on. I would hope, though, that if PHP 6 is Unicode-only there might be more enthusiasm for adapting code to work with it. > Also, as stands it appears most non-trivial applications are fairly > broken. Those issues need to be resolved first, before we toss Unicode > onto the pile. I think the test script and ZE problems should be resolved first... plus I just discovered that my php.ini isn't being found when I run commandline tests (e.g. php -r "var_dump(strcasecmp());" doesn't throw an error unless I explicitly set error_reporting from the commandline, despite having E_ALL|E_STRICT in the INI). These are basic things, nobody can really go much further until they're fixed. So I agree with you - it's far too soon for apps testing at this stage. - Steph --- ## #37769 — Antony Dovgal — 2008-05-21T10:45:47+00:00 URL: https://externals.io/message/37727#37769 · In reply to: #37758 > I think the test script and ZE problems should be resolved first... And to do that we need to drop the Unicode Switch =) > plus I just discovered that my php.ini isn't being found when I run commandline > tests (e.g. php -r "var_dump(strcasecmp());" doesn't throw an error unless I > explicitly set error_reporting from the commandline, despite having > E_ALL|E_STRICT in the INI). These are basic things, nobody can really go > much further until they're fixed. Your php.ini is being found & read allright, the problem is that the INI scanner looks for _string_ constants, while all constants are registered as _unicode_ constants in Unicode mode. So it fails to find `E_ALL` constant and ends up with atoi("E_ALL"), which is 0. I committed a fix for this one a minute ago. The broken function names in error messages are easy to fix too, as soon as we drop the Switch - functions in Unicode mode are registered using their Unicode names, but then a weird thing happens: zend_register_standard_ini_entries() makes Unicode copies of function/class/constant tables (ignoring the fact that they are already Unicode ones), this is why you see only the first character in function names. The patch is as easy as this: http://dev.daylessday.org/diff/fix_unicode_function_names.diff I didn't commit this one yet as I didn't investigate how it would affect non-Unicode mode (and whether I should worry about non-Unicode mode at all). -- Wbr, Antony Dovgal --- ## #37770 — Steph Fox — 2008-05-21T11:12:04+00:00 URL: https://externals.io/message/37727#37770 · In reply to: #37769 Hey Tony, First off, thanks for caring :) > And to do that we need to drop the Unicode Switch =) Not fix the streams issue? > I committed a fix for this one a minute ago. Thanks! > The patch is as easy as this: > http://dev.daylessday.org/diff/fix_unicode_function_names.diff This is the biggest problem, isn't it - that things like this obviously hadn't been tested until now. I fixed the same issue in Phar last week - it probably will need un-fixing there once your patch goes in :\ > I didn't commit this one yet as I didn't investigate how it would affect > non-Unicode mode (and whether I should worry about non-Unicode mode at > all). I would've hoped that PHP 6 would be Unicode throughout. Maybe jumping around when Andrei removed the physical switch was premature... obviously the UG(unicode) checks are still in place, whereas their binary alternatives shouldn't still exist. Then again some things are unclear. What should happen with `pack()`, for example? - Steph --- ## #37771 — Antony Dovgal — 2008-05-21T11:16:29+00:00 URL: https://externals.io/message/37727#37771 · In reply to: #37770 > Hey Tony, > > First off, thanks for caring :) > > > […] > > Not fix the streams issue? That, too =) -- Wbr, Antony Dovgal --- ## #37772 — Antony Dovgal — 2008-05-21T13:03:39+00:00 URL: https://externals.io/message/37727#37772 · In reply to: #37770 > Hey Tony, > > First off, thanks for caring :) > > > […] > > Not fix the streams issue? Felipe has taken care of that issue, so now we have only one uber-major issue left. -- Wbr, Antony Dovgal --- ## #37777 — Steph Fox — 2008-05-21T13:50:04+00:00 URL: https://externals.io/message/37727#37777 · In reply to: #37772 > > […] > > Felipe has taken care of that issue, so now we have only one uber-major > issue left. Your patch? Or the changed error message? FWIW the --EXPECT-- sections on those tests are different in 5_3 and would pass, so maybe they should just match those? - Steph --- ## #37778 — Antony Dovgal — 2008-05-21T13:51:13+00:00 URL: https://externals.io/message/37727#37778 · In reply to: #37777 > > […] > > Your patch? Or the changed error message? Yes, I mean that problem with storing string/Unicode function names. That patch seems to be no good, run-tests.php doesn't work after I apply it (some functions are not found). -- Wbr, Antony Dovgal --- ## #37779 — Steph Fox — 2008-05-21T13:58:27+00:00 URL: https://externals.io/message/37727#37779 · In reply to: #37778 > Yes, I mean that problem with storing string/Unicode function names. > That patch seems to be no good, run-tests.php doesn't work after I apply > it (some functions are not found). it'll be a lot easier to see what's going on if/when the UG(unicode) if'n'butting goes. There's obviously a conversion too many somewhere down the line @ present. - Steph --- ## #37780 — Antony Dovgal — 2008-05-21T14:00:29+00:00 URL: https://externals.io/message/37727#37780 · In reply to: #37779 > > […] > > it'll be a lot easier to see what's going on if/when the > UG(unicode) if'n'butting goes. There's obviously a conversion too many > somewhere down the line @ present. Right. That's exactly what I meant. -- Wbr, Antony Dovgal --- ## #37784 — Andrei Zmievski — 2008-05-21T18:49:55+00:00 URL: https://externals.io/message/37727#37784 · In reply to: #37770 Nothing premature about it. It was time. UG(unicode) checks are still secondary I think - they don't prevent us from doing tests and moving forward, although cleaning them up would be nice. `pack()` should take binary strings only, methinks. -Andrei Steph Fox wrote: > I would've hoped that PHP 6 would be Unicode throughout. Maybe jumping > around when Andrei removed the physical switch was premature... > obviously the UG(unicode) checks are still in place, whereas their > binary alternatives shouldn't still exist. Then again some things are > unclear. What should happen with `pack()`, for example? > > - Steph --- ## #37794 — Steph Fox — 2008-05-22T10:26:44+00:00 URL: https://externals.io/message/37727#37794 · In reply to: #37784 Hey Andrei, > UG(unicode) checks are still secondary I think - they don't prevent us > from doing tests and moving forward, although cleaning them up would be > nice. Cleaning them up would make it possible to find and fix the bugs we already know are there ;) There don't seem to be too many, but when it comes to tracking down conversions... bleh... > `pack()` should take binary strings only, methinks. It does - 'UG(ascii_conv)' - but sometimes there's a unicode-to-binary conversion warning thrown by that, and then you have to cast the variable to a binary empty string to suppress the warning before you assign the `pack()` result to it. (See ext/phar/tests/phar_test.inc.) There are similar issues with `unpack()`, and with `fwrite()`/fread() and friends when dealing with a purely binary string. What concerns me at this point is that we could and should be telling PHP users how to future-proof their code during the move from PHP 4 to PHP 5. If we can get a fix on that now and get word out, we'll have made their future migration path *much* smoother. - Steph --- ## #37806 — Edward Z. Yang — 2008-05-22T19:10:57+00:00 URL: https://externals.io/message/37727#37806 · In reply to: #37794 Steph Fox wrote: > What concerns me at this point is that we could and should be telling > PHP users how to future-proof their code during the move from PHP 4 to > PHP 5. If we can get a fix on that now and get word out, we'll have made > their future migration path *much* smoother. As far as I can tell, it's effectively impossible for anyone who wants to support "low" PHP 5 versions. That's because the b"" prefix wasn't introduced until PHP 5.2.1, and that prefix is *crucial* to future-proofing any code that deals with binary data. -- Edward Z. Yang GnuPG: 0x869C48DA HTML Purifier Anti-XSS Filter [[ 3FA8 E9A9 7385 B691 A6FC B3CB A933 BE7D 869C 48DA ]] --- Next page: https://externals.io/message/37727.md?page=2