Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:55326
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 74.125.83.53 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <CAF+90c9qb-Cp1e+stxxWztxLjjRHfLUVeOjU6XGZyNMRnbMPsA@mail.gmail.com>
References: <CAF+90c9qb-Cp1e+stxxWztxLjjRHfLUVeOjU6XGZyNMRnbMPsA@mail.gmail.com>
Date: Fri, 9 Sep 2011 10:01:59 +0200
Message-ID: <CAH-PCH6wxC0wR6F4PTwMQi2KEsGP_GDS2MZ3b1LyBGN89yFCZQ@mail.gmail.com>
To: Nikita Popov <nikita.ppv@googlemail.com>
Cc: PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] Revert Tokenizer behavior for 5.4
From: tyra3l@gmail.com (Ferenc Kovacs)

On Fri, Sep 9, 2011 at 9:15 AM, Nikita Popov <nikita.ppv@googlemail.com> wr=
ote:
> In Bug #54089 [1] a patch was applied that cuts of token_get_all()
> output after a T_HALT_COMPILER token. This was done because otherwise
> PHP would keep on lexing after that and would generate errors because
> of binary data (which is not valid PHP, mostly.)
>
> The problem with the patch is, that there are some tokens after
> T_HALT_COMPILER that are of interest, namely the '(' ')' ';'. After
> the patch it is impossible to get those tokens, without either
> relexing the code after T_HALT_COMPILER (that way you get the binary
> data problem back, just with much more complex code) or writing a
> regular expression to match it (which is really hard, as there may be
> any token dropped by the PHP parser in there, i.e. whitespace,
> comments, PHP tags).
>
> This issue was pointed out by the creator of the bug report, but was
> those comments were ignored for some reason.
>
> I would like this patch to be reverted on the 5.4 and trunk branches.
> I assume it's too late to revert it on 5.3, as it has been there for
> some time already. It is just counterproductive. (Alternatively one
> could fix token_get_all to return the (); tokens after
> __halt_compiler, too, but that would be hard, probably.)

I think that it wouldn't be too hard.
From a quick glance on the code, we should introduce a new local
variable, set that to true where we break now (
http://svn.php.net/viewvc/php/php-src/trunk/ext/tokenizer/tokenizer.c?view=
=3Dmarkup#l155
) and don't break there but for the next ';'. another maybe less
confusing solution would be to explicitly add '(', ')' and ';' to the
result in the T_HALT_COMPILER condition before breking out of the
loop.
I will create a patch for this afternoon.

or could there be other important tokens after the __halt_compiler()
which should be present in the token_get_all() result?

--=20
Ferenc Kov=C3=A1cs
@Tyr43l - http://tyrael.hu