Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:35908
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: unknown (pb1.pair.com: domain php.net does not designate 85.214.94.56 as permitted sender)
Date: Mon, 3 Mar 2008 11:34:29 +0100
Reply-To: Marcus Boerger <helly@php.net>
Message-ID: <1789182684.20080303113429@marcus-boerger.de>
To: Alan Knowles <alan@akbkhome.com>
CC: internals@lists.php.net
In-Reply-To: <47CB3CDC.8050006@akbkhome.com>
References: <1706278209.20080302232134@marcus-boerger.de> <47CB3CDC.8050006@akbkhome.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-15
Content-Transfer-Encoding: 8bit
Subject: Re: [PHP-DEV] [RFC] Replace the flex-based scanner with an re2c [1] based lexer
From: helly@php.net (Marcus Boerger)

Hello Alan,

  be my hero then :-) Could you generate a few tests for the multibyte
support so that we know how it is used right now and what we need to take
care of?

marcus

Monday, March 3, 2008, 12:48:44 AM, you wrote:

> Can you clarify the Multibyte issues:
> - I presume this means that it can handle ASCII/UTF8/16 etc. but will 
> not handle things like BIG5/GB encoding in source code - this may be a 
> bit of an issue around here..

> Regards
> Alan


> Marcus Boerger wrote:
>> RFC: REPLACE THE FLEX-BASED SCANNER WITH AN RE2C [1] BASED LEXER
>>
>> Situation:
>> The current flex-based lexer depends on an outdated and unsupported flex
>> version. Alternatives include either updating to a newer version of flex or
>> using re2c, which we already use for a variety of things (serializing, pdo sql
>> scanning, date/time parsing). While moving towards a newer flex version would
>> be much easier, switching to re2c promises a much faster lexer. Actually,
>> without any specific re2c optimizations we already get around a 20% scanner
>> performance increase. Running the tests gets an overall speedup of 2%. It is
>> arguable whether this is enough, but re2c has more advantages. First of all,
>> re2c allows one to scan any type of input (ASCII, UTF-8, UTF-16, UTF-32).
>> Secondly, it allows for better integration with Lemon [2], which would be the
>> next step. And thirdly we can switch to a reentrant scanner.
>>
>> Current state:
>> Flex has been fully replaced by re2c in Zend. We have also switched to an
>> mmap-based lexer approach for now. However, we had to drop multibyte support
>> as well as the encoding declare. The current state can be checked out from
>> Scott's subversion repository [3] and you can follow the development on his
>> Trac setup [4]. When you want to build php with re2c, then you need to grab
>> re2c from its sourceforge subversion repository [5]. You can also check out
>> the changes in a patch created Sunday 2nd March against a PHP checkout from 
>> 14th February [6].
>>
>> Further steps:
>> Commit this to PHP 5.3. Synch to HEAD. Add pecl/intl to 5.3. Discuss/recreate
>> multibyte support with libintl.
>>
>> Future steps:
>> Replace bison with lemon in PHP 5.4 or HEAD.
>>
>> Time Frame:
>> Commit to 5.3 between the 5th and the 15th of March. Synch to HEAD a couple
>> of days later. Moving pecl/libintl to ext (depends on the 5.3 RMs decision).
>> After that is done, decide about multibyte support. Along with the commit to
>> the 5.3 branch there will be a new re2c version available.
>>
>>
>> Marcus Boerger
>> Nuno Lopes
>> Scott MacVicar
>>
>>
>> [1] http://re2c.org/
>> [2] http://www.hwaci.com/sw/lemon/
>> [3] svn://whisky.macvicar.net/php-re2c
>> [4] http://trac.macvicar.net/php-re2c/
>> [5] https://re2c.svn.sourceforge.net/svnroot/re2c/trunk/re2c
>> [6] http://php.net/~helly/php-re2c-20080302.diff.txt
>>
>>
>>
>>   





Best regards,
 Marcus