# [RFC] Replace the flex-based scanner with an re2c [1] based lexer - Thread: https://externals.io/message/35891 - Original archive: https://news-web.php.net/php.internals/35891 - Started by Marcus Boerger on 2008-03-02T22:21:34+00:00 - Messages: 59 - Page 1 of 3: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/35891.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #35891 — Marcus Boerger — 2008-03-02T22:21:34+00:00 URL: https://externals.io/message/35891#35891 RFC: REPLACE THE FLEX-BASED SCANNER WITH AN RE2C [1] BASED LEXER Situation: The current flex-based lexer depends on an outdated and unsupported flex version. Alternatives include either updating to a newer version of flex or using re2c, which we already use for a variety of things (serializing, pdo sql scanning, date/time parsing). While moving towards a newer flex version would be much easier, switching to re2c promises a much faster lexer. Actually, without any specific re2c optimizations we already get around a 20% scanner performance increase. Running the tests gets an overall speedup of 2%. It is arguable whether this is enough, but re2c has more advantages. First of all, re2c allows one to scan any type of input (ASCII, UTF-8, UTF-16, UTF-32). Secondly, it allows for better integration with Lemon [2], which would be the next step. And thirdly we can switch to a reentrant scanner. Current state: Flex has been fully replaced by re2c in Zend. We have also switched to an mmap-based lexer approach for now. However, we had to drop multibyte support as well as the encoding declare. The current state can be checked out from Scott's subversion repository [3] and you can follow the development on his Trac setup [4]. When you want to build php with re2c, then you need to grab re2c from its sourceforge subversion repository [5]. You can also check out the changes in a patch created Sunday 2nd March against a PHP checkout from 14th February [6]. Further steps: Commit this to PHP 5.3. Synch to HEAD. Add pecl/intl to 5.3. Discuss/recreate multibyte support with libintl. Future steps: Replace bison with lemon in PHP 5.4 or HEAD. Time Frame: Commit to 5.3 between the 5th and the 15th of March. Synch to HEAD a couple of days later. Moving pecl/libintl to ext (depends on the 5.3 RMs decision). After that is done, decide about multibyte support. Along with the commit to the 5.3 branch there will be a new re2c version available. Marcus Boerger Nuno Lopes Scott MacVicar [1] http://re2c.org/ [2] http://www.hwaci.com/sw/lemon/ [3] svn://whisky.macvicar.net/php-re2c [4] http://trac.macvicar.net/php-re2c/ [5] https://re2c.svn.sourceforge.net/svnroot/re2c/trunk/re2c [6] http://php.net/~helly/php-re2c-20080302.diff.txt --- ## #35892 — Stanislav Malyshev — 2008-03-02T22:47:57+00:00 URL: https://externals.io/message/35891#35892 · In reply to: #35891 Hi! > be much easier, switching to re2c promises a much faster lexer. Actually, > without any specific re2c optimizations we already get around a 20% scanner I think 20% faster is very cool. However, as I understand re2c is not a standard tool found everywhere. So what happens if you wanted to use it on some exotic system where re2c is not readily available as manintainer-supported software? Also, flex is available on Windows for example as part of cygwin, while I don't see re2c there. I understand this can be of low importance since we keep generated files in our repositories, but I think we still have to keep it in mind. I understand also current patch requires non-release version of re2c - maybe we should have some release version at least until we make PHP depend on it? > Current state: > Flex has been fully replaced by re2c in Zend. We have also switched to an > mmap-based lexer approach for now. However, we had to drop multibyte support Were the stream support issues solved? > as well as the encoding declare. The current state can be checked out from > Scott's subversion repository [3] and you can follow the development on his > Trac setup [4]. When you want to build php with re2c, then you need to grab > re2c from its sourceforge subversion repository [5]. You can also check out > the changes in a patch created Sunday 2nd March against a PHP checkout from > 14th February [6]. > > Further steps: > Commit this to PHP 5.3. Synch to HEAD. Add pecl/intl to 5.3. Discuss/recreate > multibyte support with libintl. Note - pecl/intl does nothing towards multibyte support etc., at least for now. If there are voloteers to change that, it can be discussed, but so far it is for doing entirely other things (locale-dependent functionality mostly). So, I think before re2c parser can be merged the issue with multibyte compatibility must be solved - otherwise it will make the users that rely on it unable to use newer PHP. As cool as 20% faster is, I think we can't drop support for such feature, especially not in 5.3. > Commit to 5.3 between the 5th and the 15th of March. Synch to HEAD a couple > of days later. Moving pecl/libintl to ext (depends on the 5.3 RMs decision). > After that is done, decide about multibyte support. Along with the commit to > the 5.3 branch there will be a new re2c version available. I think we first need to figure out what happens to multibyte support, and not commit anything before we have it figured out. Multibyte support is important piece of functionality for some PHP users, and it works now. Breaking it without providing any alternative - especially that we have now 5.3 mostly ready for the release cycle, and solving multibyte problems with re2c may take undefined amount of time, as far as I understand. I do not think it would be acceptable to release 5.3 without multibyte support, so the option here either merge it now and have 5.3 waiting until MB is figured out, or try to figure it out before commit and if we can't in a reasonable term, go forward with 5.3 and defer the parser change for 5.4. Again, while I think the speedup is great and congratulate Marcus, Nuno and Scott on great work, I think we should keep in mind we have working parser right now and changing it in an incompatible way is very high-risk and should not be taken hastily. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #35893 — Rasmus Lerdorf — 2008-03-02T23:25:52+00:00 URL: https://externals.io/message/35891#35893 · In reply to: #35892 Stanislav Malyshev wrote: > Hi! > > > […] > > I think 20% faster is very cool. > However, as I understand re2c is not a standard tool found everywhere. > So what happens if you wanted to use it on some exotic system where > re2c is not readily available as manintainer-supported software? Also, > flex is available on Windows for example as part of cygwin, while I > don't see re2c there. I don't think this part is a concern since we have required re2c for quite a while now to build many critical parts of PHP. People who actually need to regenerate the parser files are the same people for whom it is trivial to figure out how to install re2c. And yes, it would of course be good to use a released version of re2c, but I think by the time 5.3 is ready to go the version of re2c we need will be out there. Since it is Marcus' baby, he can just push it out, I don't think this is a stumbling block either. Some of the new stuff in re2c was specifically added to make it easier to write a PHP parser, so I don't think backporting to an older version is really an option. -Rasmus --- ## #35894 — Marcus Boerger — 2008-03-02T23:26:51+00:00 URL: https://externals.io/message/35891#35894 · In reply to: #35892 Hello Stanislav, Sunday, March 2, 2008, 11:47:57 PM, you wrote: > Hi! > > […] > I think 20% faster is very cool. > However, as I understand re2c is not a standard tool found everywhere. > So what happens if you wanted to use it on some exotic system where re2c > is not readily available as manintainer-supported software? Also, flex > is available on Windows for example as part of cygwin, while I don't see > re2c there. > I understand this can be of low importance since we keep generated files > in our repositories, but I think we still have to keep it in mind. > I understand also current patch requires non-release version of re2c - > maybe we should have some release version at least until we make PHP > depend on it? Well, re2c works for on a very large amount of systems, can easily be build and comes with a read to download windows executable. Furthermore all major distributions have re2c packages. Along with storing the generated files in cvs i see no issue at all in these regards. > > […] > Were the stream support issues solved? We completely dropped multibyte support. The reason is that the way we were doing it, is that we constanlty switch between the full original and a recoded duplicate that simply ignores multibyte (or any encoding at all). Once we have finished the move to re2c, we can support all of those correctly. The multibyte support also duplicated the encoding tables otherwise available in ext/mbstring or ext/iconv or pecl/intl. > > […] > Note - pecl/intl does nothing towards multibyte support etc., at least > for now. If there are voloteers to change that, it can be discussed, but > so far it is for doing entirely other things (locale-dependent > functionality mostly). Yes I know. However pecl/intl brings in a php/icu bridge which we can build on. > So, I think before re2c parser can be merged the issue with multibyte > compatibility must be solved - otherwise it will make the users that > rely on it unable to use newer PHP. As cool as 20% faster is, I think we > can't drop support for such feature, especially not in 5.3. Rely on a not supported undocumented feature? I am rather able to build php and rewrite that support. > > […] > I think we first need to figure out what happens to multibyte support, > and not commit anything before we have it figured out. Multibyte support > is important piece of functionality for some PHP users, and it works > now. Breaking it without providing any alternative - especially that we > have now 5.3 mostly ready for the release cycle, and solving multibyte > problems with re2c may take undefined amount of time, as far as I > understand. I do not think it would be acceptable to release 5.3 without > multibyte support, so the option here either merge it now and have 5.3 > waiting until MB is figured out, or try to figure it out before commit > and if we can't in a reasonable term, go forward with 5.3 and defer the > parser change for 5.4. > Again, while I think the speedup is great and congratulate Marcus, Nuno > and Scott on great work, I think we should keep in mind we have working > parser right now and changing it in an incompatible way is very > high-risk and should not be taken hastily. You are free to contribute and make MB support working upfront. Best regards, Marcus --- ## #35895 — Pierre Joye — 2008-03-02T23:27:28+00:00 URL: https://externals.io/message/35891#35895 · In reply to: #35892 Hi Stan, > Hi! > > > > […] > > I think 20% faster is very cool. > However, as I understand re2c is not a standard tool found everywhere. > So what happens if you wanted to use it on some exotic system where re2c > is not readily available as manintainer-supported software? Also, flex > is available on Windows for example as part of cygwin, while I don't see > re2c there. A quick note about this non problem. re2c works pretty well on windows and they provide a .exe as far as I remember (much easier than flex which requires cygwin or gnuwin32, even if both work :). Besides the portability of re2c, we already use it in some extensions (if I remember correctly) and nobody complained. Cheers, -- Pierre http://blog.thepimp.net | http://www.libgd.org --- ## #35896 — Marcus Boerger — 2008-03-02T23:43:10+00:00 URL: https://externals.io/message/35891#35896 · In reply to: #35893 Hello Rasmus, Monday, March 3, 2008, 12:25:52 AM, you wrote: > Stanislav Malyshev wrote: > > […] > I don't think this part is a concern since we have required re2c for > quite a while now to build many critical parts of PHP. People who > actually need to regenerate the parser files are the same people for > whom it is trivial to figure out how to install re2c. And yes, it would > of course be good to use a released version of re2c, but I think by the > time 5.3 is ready to go the version of re2c we need will be out there. > Since it is Marcus' baby, he can just push it out, I don't think this is > a stumbling block either. Some of the new stuff in re2c was > specifically added to make it easier to write a PHP parser, so I don't > think backporting to an older version is really an option. Right. The current re2c development cycle is solely dedicated to be able to rewrite the PHP scanners. I will update re2c whenever necessary during the remaining development cycle and release a new stable release before we release PHP 5.3. Best regards, Marcus --- ## #35897 — Alan Knowles — 2008-03-02T23:48:44+00:00 URL: https://externals.io/message/35891#35897 · In reply to: #35891 Can you clarify the Multibyte issues: - I presume this means that it can handle ASCII/UTF8/16 etc. but will not handle things like BIG5/GB encoding in source code - this may be a bit of an issue around here.. Regards Alan Marcus Boerger wrote: > RFC: REPLACE THE FLEX-BASED SCANNER WITH AN RE2C [1] BASED LEXER > > Situation: > The current flex-based lexer depends on an outdated and unsupported flex > version. Alternatives include either updating to a newer version of flex or > using re2c, which we already use for a variety of things (serializing, pdo sql > scanning, date/time parsing). While moving towards a newer flex version would > be much easier, switching to re2c promises a much faster lexer. Actually, > without any specific re2c optimizations we already get around a 20% scanner > performance increase. Running the tests gets an overall speedup of 2%. It is > arguable whether this is enough, but re2c has more advantages. First of all, > re2c allows one to scan any type of input (ASCII, UTF-8, UTF-16, UTF-32). > Secondly, it allows for better integration with Lemon [2], which would be the > next step. And thirdly we can switch to a reentrant scanner. > > Current state: > Flex has been fully replaced by re2c in Zend. We have also switched to an > mmap-based lexer approach for now. However, we had to drop multibyte support > as well as the encoding declare. The current state can be checked out from > Scott's subversion repository [3] and you can follow the development on his > Trac setup [4]. When you want to build php with re2c, then you need to grab > re2c from its sourceforge subversion repository [5]. You can also check out > the changes in a patch created Sunday 2nd March against a PHP checkout from > 14th February [6]. > > Further steps: > Commit this to PHP 5.3. Synch to HEAD. Add pecl/intl to 5.3. Discuss/recreate > multibyte support with libintl. > > Future steps: > Replace bison with lemon in PHP 5.4 or HEAD. > > Time Frame: > Commit to 5.3 between the 5th and the 15th of March. Synch to HEAD a couple > of days later. Moving pecl/libintl to ext (depends on the 5.3 RMs decision). > After that is done, decide about multibyte support. Along with the commit to > the 5.3 branch there will be a new re2c version available. > > > Marcus Boerger > Nuno Lopes > Scott MacVicar > > > [1] http://re2c.org/ > [2] http://www.hwaci.com/sw/lemon/ > [3] svn://whisky.macvicar.net/php-re2c > [4] http://trac.macvicar.net/php-re2c/ > [5] https://re2c.svn.sourceforge.net/svnroot/re2c/trunk/re2c > [6] http://php.net/~helly/php-re2c-20080302.diff.txt --- ## #35898 — Stanislav Malyshev — 2008-03-03T04:39:35+00:00 URL: https://externals.io/message/35891#35898 · In reply to: #35894 Hi! > > […] > > We completely dropped multibyte support. The reason is that the way we were I wasn't asking about multibyte (that we discuss below), but about other streams - I think I mentioned it on IRC last time re2c parser was discussed. I remember re2c used mmap, and not all files PHP can run can be mmap'ed. Was it fixed? > Once we have finished the move to re2c, we can support all of those > correctly. The multibyte support also duplicated the encoding tables > otherwise available in ext/mbstring or ext/iconv or pecl/intl. pecl/intl per se doesn't have any encoding tables. ICU does, but that would mean you have to have ICU to run PHP. That might not be a big problem since ICU is supported by IBM (read: good chance more "exotic" systems would have support) it is still dependency on non-bundled 3rd party library in PHP 5 core. Of course, PHP 6 has this dependency, but we might want to not have such things in 5.x so that you won't have to change your system too much while staying on 5.x. > Rely on a not supported undocumented feature? I am rather able to build php > and rewrite that support. Being undocumented is nothing to be proud of, however as poorly documented as it is, it is used. I'm all for implementing it in a better way - and having new parser is a good time to do it. That's exactly the reason we shouldn't rush with it but do it right this time. There's no burning need to have a new parser right now, so we can have some moment to think - ok, how we want multibyte support there to work? And if we might need some modifications, we'd have time and flexibility to do it, not having the code in 5.3 which was supposed to go in RC in Q1 (ending 1 month from now). > You are free to contribute and make MB support working upfront. I know I'm free :) However, as much as I understand the eagerness of having it in the source tree, I repeat that I do not think dropping multibyte support in 5.3 is acceptable. Thus, if it is committed right now, 5.3 would have to be deferred until this is resolved. If this is resolved timely for 5.3 - great. If not, we better get it in 5.4 right than in 5.3 wrong. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #35899 — Stanislav Malyshev — 2008-03-03T04:40:33+00:00 URL: https://externals.io/message/35891#35899 · In reply to: #35893 > I don't think this part is a concern since we have required re2c for > quite a while now to build many critical parts of PHP. People who Ok, great then - only issue remaining is the multibyte support. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #35900 — Derick Rethans — 2008-03-03T08:28:01+00:00 URL: https://externals.io/message/35891#35900 · In reply to: #35891 > However, we had to drop multibyte support as well as the encoding > declare. Just wondering, why did you have to drop the "declare(encoding=...)" ? It's just ignored in PHP 5.x - and it is useful to have for migrating php 5.3 apps to 6. So can you atleast make the new parser just ignore this statement? regards, Derick -- Derick Rethans http://derickrethans.nl | http://ezcomponents.org | http://xdebug.org --- ## #35903 — johannes@php.net — 2008-03-03T10:21:21+00:00 URL: https://externals.io/message/35891#35903 · In reply to: #35900 Hi Derick, > > > > […] > > Just wondering, why did you have to drop the "declare(encoding=...)" ? > It's just ignored in PHP 5.x - and it is useful to have for migrating > php 5.3 apps to 6. So can you atleast make the new parser just ignore > this statement? It is not ignored in PHP 5 as Marcus and I found out while reading the code :-) If you compile with --enable-zend-multibyte you can change the encoding using declare even multiple times per file using declare it seems. johannes --- ## #35905 — Marcus Boerger — 2008-03-03T10:22:14+00:00 URL: https://externals.io/message/35891#35905 · In reply to: #35900 Hello Derick, actually you get a message (E_COMPILE_WARNING) that this is not supported. Maybe we could turn this into an `E_NOTICE` though. marcus Monday, March 3, 2008, 9:28:01 AM, you wrote: > > > […] > Just wondering, why did you have to drop the "declare(encoding=...)" ? > It's just ignored in PHP 5.x - and it is useful to have for migrating > php 5.3 apps to 6. So can you atleast make the new parser just ignore > this statement? > regards, > Derick > -- > Derick Rethans > http://derickrethans.nl | http://ezcomponents.org | http://xdebug.org Best regards, Marcus --- ## #35906 — Derick Rethans — 2008-03-03T10:28:31+00:00 URL: https://externals.io/message/35891#35906 · In reply to: #35905 > actually you get a message (E_COMPILE_WARNING) that this is not > supported. Maybe we could turn this into an `E_NOTICE` though. No, I don't get any warning/notice/ whatever with PHP 5.3: derick@kossu:~$ php-5.3dev -derror_reporting=65535 foo Please don't break this. regards, Derick --- ## #35907 — Marcus Boerger — 2008-03-03T10:33:13+00:00 URL: https://externals.io/message/35891#35907 · In reply to: #35898 Hello Stanislav, Monday, March 3, 2008, 5:39:35 AM, you wrote: > Hi! > > […] > I wasn't asking about multibyte (that we discuss below), but about other > streams - I think I mentioned it on IRC last time re2c parser was > discussed. I remember re2c used mmap, and not all files PHP can run can > be mmap'ed. Was it fixed? Ah, you didn't write that so I got confused. Anyway, what we are doing is the following order: 1) If mmap is supported, then use it 2) If mmap is not supported or does not work then read the whole stream 3) If that is not possible read char by char The flex based scanner reads in smaller chunks or char by char, so it is more or less always like case 3. > > […] > pecl/intl per se doesn't have any encoding tables. ICU does, but that > would mean you have to have ICU to run PHP. That might not be a big > problem since ICU is supported by IBM (read: good chance more "exotic" > systems would have support) it is still dependency on non-bundled 3rd > party library in PHP 5 core. Of course, PHP 6 has this dependency, but > we might want to not have such things in 5.x so that you won't have to > change your system too much while staying on 5.x. Are you saying we cannot depend on ICU in PHP 6 and have to redo it completely or what? > > […] > Being undocumented is nothing to be proud of, however as poorly > documented as it is, it is used. I'm all for implementing it in a better > way - and having new parser is a good time to do it. That's exactly the > reason we shouldn't rush with it but do it right this time. There's no > burning need to have a new parser right now, so we can have some moment > to think - ok, how we want multibyte support there to work? And if we > might need some modifications, we'd have time and flexibility to do it, > not having the code in 5.3 which was supposed to go in RC in Q1 (ending > 1 month from now). > > […] > I know I'm free :) However, as much as I understand the eagerness of > having it in the source tree, I repeat that I do not think dropping > multibyte support in 5.3 is acceptable. Thus, if it is committed right > now, 5.3 would have to be deferred until this is resolved. If this is > resolved timely for 5.3 - great. If not, we better get it in 5.4 right > than in 5.3 wrong. I don't see a problem with redoing multibyte support in a useable way. Actually we better redo it anyway because it is a very bad solution as it is right now. That is the current solution duplicates the input and uses a flattening filter to always scan an eight bit input stream. Then when something needs to get pushed to the output, we recalculate the position on the original input and use that part. Changing to re2c we can do a very easy solution. When requested or detected per BOM, we switch to a second version of the scanner that works on unsigned int and supports the full unicode character set (only thing to do for re2c is to switch the input type and guess what, this is already in production on a lot of systems). Other approaches are to natively support UTF-8 and UTF-16 besides 8 bit and UTF-32. Further more we can apply any kind of filtering correctly on top of the UTF-* scanner. I Know there is some work left but when we do not apply the work now then we basically have two engines. In that case I'll just rewrite the engine completely and replace it. Best regards, Marcus --- ## #35908 — Marcus Boerger — 2008-03-03T10:34:29+00:00 URL: https://externals.io/message/35891#35908 · In reply to: #35897 Hello Alan, be my hero then :-) Could you generate a few tests for the multibyte support so that we know how it is used right now and what we need to take care of? marcus Monday, March 3, 2008, 12:48:44 AM, you wrote: > Can you clarify the Multibyte issues: > - I presume this means that it can handle ASCII/UTF8/16 etc. but will > not handle things like BIG5/GB encoding in source code - this may be a > bit of an issue around here.. > Regards > Alan > Marcus Boerger wrote: > > […] Best regards, Marcus --- ## #35909 — Marcus Boerger — 2008-03-03T10:38:02+00:00 URL: https://externals.io/message/35891#35909 · In reply to: #35906 Hello Derick, ok, for now I changed to not issue any error at all. marcus Monday, March 3, 2008, 11:28:31 AM, you wrote: > > > […] > No, I don't get any warning/notice/ whatever with PHP 5.3: > derick@kossu:~$ php-5.3dev -derror_reporting=65535 > declare(encoding="utf-8"); > echo "foo\n"; ?>> > foo > Please don't break this. > regards, > Derick Best regards, Marcus --- ## #35910 — johannes@php.net — 2008-03-03T10:41:01+00:00 URL: https://externals.io/message/35891#35910 · In reply to: #35892 Hi, > Hi! > > > […] > > I think 20% faster is very cool. > However, as I understand re2c is not a standard tool found everywhere. > So what happens if you wanted to use it on some exotic system where re2c > is not readily available as manintainer-supported software? Also, flex > is available on Windows for example as part of cygwin, while I don't see > re2c there. > I understand this can be of low importance since we keep generated files > in our repositories, but I think we still have to keep it in mind. > I understand also current patch requires non-release version of re2c - > maybe we should have some release version at least until we make PHP > depend on it? We need a change there anyways, flex 2.5.4 is bundled with less systems, even my Solaris 20 box has 2.5.33 instead of 2.5.4 by default. And I think changing to something which is maintained by one of our main contributors might be beneficial for us. > Note - pecl/intl does nothing towards multibyte support etc., at least > for now. If there are voloteers to change that, it can be discussed, but > so far it is for doing entirely other things (locale-dependent > functionality mostly). > So, I think before re2c parser can be merged the issue with multibyte > compatibility must be solved - otherwise it will make the users that > rely on it unable to use newer PHP. As cool as 20% faster is, I think we > can't drop support for such feature, especially not in 5.3. > > > […] > > I think we first need to figure out what happens to multibyte support, > and not commit anything before we have it figured out. Multibyte support > is important piece of functionality for some PHP users, and it works > now. Breaking it without providing any alternative - especially that we > have now 5.3 mostly ready for the release cycle, and solving multibyte > problems with re2c may take undefined amount of time, as far as I > understand. I do not think it would be acceptable to release 5.3 without > multibyte support, so the option here either merge it now and have 5.3 > waiting until MB is figured out, or try to figure it out before commit > and if we can't in a reasonable term, go forward with 5.3 and defer the > parser change for 5.4. Since there's no documentation about zend-multibyte stuff I spent some time searching for other resources about it, but except bug reports I found nothing whee it was required. I'm sure there are some but comments like "TODO: support widechars" in the code give me the impression that it doesn't really work... and I guess many people just enable it sinceit sounds important not due to the fact that hey really need it. Of course I might be wrong so I'd be interested in use cases for --enable-zend-multibyte stuff. Maybe we can fullfill the needs without the switch. If there are good use cases for that switch I won't like to replace some small engine thingy with a huge external library like ICU. And I doubt that more than just a few people know what it really does - Marcus and I just found out while working on that stuff over the weekend. > Again, while I think the speedup is great and congratulate Marcus, Nuno > and Scott on great work, I think we should keep in mind we have working > parser right now and changing it in an incompatible way is very > high-risk and should not be taken hastily. Right, it's great work they did there but a broken scanner would be one of the worst things we might ship. So I'd invite everybody to checkout that version from SVN (see Marcus's mail) and test it using the worst stuff you can think off :-) johannes --- ## #35911 — Lukas Kahwe Smith — 2008-03-03T10:58:13+00:00 URL: https://externals.io/message/35891#35911 · In reply to: #35897 > Can you clarify the Multibyte issues: > - I presume this means that it can handle ASCII/UTF8/16 etc. but > will not handle things like BIG5/GB encoding in source code - this > may be a bit of an issue around here.. > At first I also thought that this had something to do with ext/ mbstring, but since then I have learned that this is not the case. However this confusion is likely what causes many people to enable zend mb support. So the question to Stas (Alan and the rest of the world) is if they really have a script in the wild that actually requires this switch and would break if its would be disabled. And if there is such a script what exactly are the needs and how can these be filled in 5.3 using re2c. regards, Lukas --- ## #35915 — Alan Knowles — 2008-03-03T15:19:24+00:00 URL: https://externals.io/message/35891#35915 · In reply to: #35908 a few replaces with this file should be a good testcase - probably worth testing * comments with these character in them. both /* and // * string with these characters in them. lynx -source 'http://smontagu.damowmow.com/genEncodingTest.cgi?family=windows&codepage=950' | grep test | grep -v testcase I have definatly seen code with chinese characters in comments and strings and a few times function names and variable names with chinese characters... Regards Alan Marcus Boerger wrote: > Hello Alan, > > be my hero then :-) Could you generate a few tests for the multibyte > support so that we know how it is used right now and what we need to take > care of? > > marcus > > Monday, March 3, 2008, 12:48:44 AM, you wrote: > > > > […] > > > > […] > > > > > […] > > > > > > Best regards, > Marcus --- ## #35917 — Stanislav Malyshev — 2008-03-03T18:58:55+00:00 URL: https://externals.io/message/35891#35917 · In reply to: #35910 Hi! > Since there's no documentation about zend-multibyte stuff I spent some > time searching for other resources about it, but except bug reports I > found nothing whee it was required. I'm sure there are some but comments > like "TODO: support widechars" in the code give me the impression that > it doesn't really work... and I guess many people just enable it sinceit It does work and there are people using it, even though I imagine it can have some bugs. I guess it would be best to talk to mbstring maintainer on code details, etc. > If there are good use cases for that switch I won't like to replace some > small engine thingy with a huge external library like ICU. The use cases are scripts written in encodings like shift-JIS, etc. > And I doubt that more than just a few people know what it really does - > Marcus and I just found out while working on that stuff over the > weekend. So I guess documentation is important :) Let it be a lesson to us all. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #35918 — Stanislav Malyshev — 2008-03-03T18:59:41+00:00 URL: https://externals.io/message/35891#35918 · In reply to: #35907 Hi! > 1) If mmap is supported, then use it > 2) If mmap is not supported or does not work then read the whole stream > 3) If that is not possible read char by char Why should it read the whole stream into memory? The file could be very big, maybe it would make more sense to read it is some chunks? BTW, I see there are some substantial changes in streams there - new file type ZEND_HANDLE_MAPPED, new "fsizer" handler. Will they be dsecribed somewhere so other places which deal with streams could be updated if there's need? Does it have any implication on extensions etc. or it's isolated? > > […] > > Are you saying we cannot depend on ICU in PHP 6 and have to redo it > completely or what? Just curious who you were answering to... Anyway, to be clear: 1. PHP 6 is major version with its major feature being Unicode support. 2. PHP 5.x is same-major branch, where you are not expected to have to change your system in order to upgrade. 3. We do not expect people to take PHP 6 and have absolutely everything work instantly from PHP 5. We try to minimize upgrade path, but major version upgrades can take some adjustments. 4. We expect people to upgrade from 5.2.x to 5.3.x without changing their systems. Is it clearer why I think PHP 5.x and 6 are different and why I think ICU dependency in the 5.3 core might be a problem? > I Know there is some work left but when we do not apply the work now then > we basically have two engines. In that case I'll just rewrite the engine > completely and replace it. I think better to have two engines one of which we can release than have no release-able engine at all. Again, if you say you are 100% sure that can be figured out quickly and not delay 5.3 release cycle to Q2 or Q3 - great, let's do it. Since we were not going to have any major changes in the engine until 5.3 is out anyway - the gap between your code and CVS code won't be substantial. You could use a branch to make it even easier. As for rewriting the engine - I think that would be just a waste of effort. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #35919 — Pierre Joye — 2008-03-03T19:42:55+00:00 URL: https://externals.io/message/35891#35919 · In reply to: #35918 Hi, > Just curious who you were answering to... Anyway, to be clear: > 1. PHP 6 is major version with its major feature being Unicode support. > 2. PHP 5.x is same-major branch, where you are not expected to have to > change your system in order to upgrade. > 3. We do not expect people to take PHP 6 and have absolutely everything > work instantly from PHP 5. We try to minimize upgrade path, but major > version upgrades can take some adjustments. > 4. We expect people to upgrade from 5.2.x to 5.3.x without changing > their systems. > > Is it clearer why I think PHP 5.x and 6 are different and why I think > ICU dependency in the 5.3 core might be a problem? It is clearer but it is not a problem. New features may introduce new dependencies. Having a dependency on libicu while we introduce intl and other features related to unicode or i18n. I would agree if we were talking about 5.2.x. -- Pierre http://blog.thepimp.net | http://www.libgd.org --- ## #35920 — Marcus Boerger — 2008-03-03T19:44:12+00:00 URL: https://externals.io/message/35891#35920 · In reply to: #35918 Hello Stanislav, Monday, March 3, 2008, 7:59:41 PM, you wrote: > Hi! > > […] > Why should it read the whole stream into memory? The file could be very > big, maybe it would make more sense to read it is some chunks? > BTW, I see there are some substantial changes in streams there - new > file type ZEND_HANDLE_MAPPED, new "fsizer" handler. Will they be > dsecribed somewhere so other places which deal with streams could be > updated if there's need? Does it have any implication on extensions etc. > or it's isolated? The streams stuff is meant to integrate with the PHP streams layer which we take care of. In the beginning we thought we could live with the old interface but that wasn't really a good idea. So I came up with a new interface and all that this would break is stuff like Phar (well there is as far as i know nothing else like it) and guess I'll take care of it. Actually pohar already works nearly 100%. And since Phar is a moving target right now I wouldn't worry too much about it. Also if an application decides to do less than it can provide less handlers than it had to earlier. And well I could document stuff. But since when is that the PHP way? In fact when I document stuff inside the engine I tend to get strange reactions. > > […] > Just curious who you were answering to... Anyway, to be clear: > 1. PHP 6 is major version with its major feature being Unicode support. > 2. PHP 5.x is same-major branch, where you are not expected to have to > change your system in order to upgrade. Oh since when? Where did you read that? > 3. We do not expect people to take PHP 6 and have absolutely everything > work instantly from PHP 5. We try to minimize upgrade path, but major > version upgrades can take some adjustments. > 4. We expect people to upgrade from 5.2.x to 5.3.x without changing > their systems. Since when? And when did we ever say anything like that? Since when do we tell people anything about their system? > Is it clearer why I think PHP 5.x and 6 are different and why I think > ICU dependency in the 5.3 core might be a problem? I don't care at all. So far the plan was to bring in ICU and there is no need to start a new discussion here. All I said is that we might be able to do things better. But maybe taht is the complete wrong approach and I am all wrong. > > […] > I think better to have two engines one of which we can release than have > no release-able engine at all. Again, if you say you are 100% sure that > can be figured out quickly and not delay 5.3 release cycle to Q2 or Q3 - > great, let's do it. In software there is nothing like 100% Or is everything you work on bug free? Mine isn't. It took me more than two years to make re2c ready for this task (take this in whatever way you feel). > Since we were not going to have any major changes in > the engine until 5.3 is out anyway - the gap between your code and CVS > code won't be substantial. You could use a branch to make it even easier. How so? We don't use git or any other CMS that allows merging at all. I would love to switch to GIT or Subversion but I don't think this is reasonable given our development group/cycles. > As for rewriting the engine - I think that would be just a waste of effort. Good, then let's not do that :-) Best regards, Marcus --- ## #35921 — Stanislav Malyshev — 2008-03-03T19:48:38+00:00 URL: https://externals.io/message/35891#35921 · In reply to: #35919 Hi! > It is clearer but it is not a problem. New features may introduce new > dependencies. Having a dependency on libicu while we introduce intl > and other features related to unicode or i18n. I would agree if we > were talking about 5.2.x. pecl/intl is an extension, there's no surprise that you need external library when you enable extension. However, adding dependency in core that you can not rid of has a lot of consequences (think distributions, builds on non-Linux systems, etc., etc.). -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #35922 — George Schlossnagle — 2008-03-03T19:55:59+00:00 URL: https://externals.io/message/35891#35922 · In reply to: #35921 > Hi! > > > […] > > pecl/intl is an extension, there's no surprise that you need > external library when you enable extension. However, adding > dependency in core that you can not rid of has a lot of consequences > (think distributions, builds on non-Linux systems, etc., etc.). *steps in from nowhere* It's just a build dependency, right? And one that's already required if you want to generate all the internal parsers by hand as part of your build. If it's really that huge a concern you could ship a precompiled scanner/lexer. George --- Next page: https://externals.io/message/35891.md?page=2