Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:36361
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain zend.com designates 63.205.162.117 as permitted sender)
Message-ID: <47E5D75D.9010108@zend.com>
Date: Sat, 22 Mar 2008 21:06:53 -0700
Organization: Zend Technologies
User-Agent: Thunderbird 2.0.0.12 (Windows/20080213)
MIME-Version: 1.0
To: Marcus Boerger <helly@php.net>
Cc: internals@lists.php.net, Alan Knowles <alan@akbkhome.com>,
	Andi Gutmans <andi@zend.com>, Rui Hirokawa <hirokawa@php.net>,
	Johannes Schlueter <johannes@php.net>
References: <1706278209.20080302232134@marcus-boerger.de> <47CB3CDC.8050006@akbkhome.com> <1789182684.20080303113429@marcus-boerger.de> <47CC16FC.7080802@akbkhome.com> <497326103.20080322152343@marcus-boerger.de>
In-Reply-To: <497326103.20080322152343@marcus-boerger.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [PHP-DEV] [RFC] Replace the flex-based scanner with an re2c [1]
 based lexer
From: stas@zend.com (Stanislav Malyshev)

> is broken code and not a single test. If this is not going to change as in
> we are not getting any .phpt files for this feature then there are two

As I understand the theory of the thing should be pretty simple, you set 
input encoding (by config or declare) and internal encoding, and then 
when script is being read, you convert it from input to internal.
However, it appears that since flex couldn't stomach certain encodings, 
there's also a hack there - script is translated from input to some 
"safe" encoding for flex, and then strings are translated back to 
"internal" encoding after flex processes them. If re2c can deal with 
encodings like SJIS without trouble then some of the hacks might be 
unnecessary. I think encodings that need to be checked are those in 
zend_multibyte.c that have "compatible" flag off.

Here's a short script example I found that shows what's the problem there:

<?php echo 'ソ'; ?>

Character echoed there is U+30BD "Katakana letter SO". Now if you run it 
in UTF-8, works good. However, if you recode it to Shift-JIS, it won't 
run, since this script looks to the parser this way:

<?php echo '<83>\'; ?>
(that's dump of VI output, so replace <83> with actual 0x83 if you 
compose it). That's parse error for the parser, if parsed "naively". So 
somehow the parser needs to know 0x83+\ is actually U+30BD and at the 
same time the user still might want it as 0x83+\ in a zval (or maybe as 
utf-8 - it depends on him).
-- 
Stanislav Malyshev, Zend Software Architect
stas@zend.com   http://www.zend.com/
(408)253-8829   MSN: stas@zend.com