Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:55325
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain googlemail.com designates 209.85.215.176 as permitted sender)
MIME-Version: 1.0
Date: Fri, 9 Sep 2011 09:15:21 +0200
Message-ID: <CAF+90c9qb-Cp1e+stxxWztxLjjRHfLUVeOjU6XGZyNMRnbMPsA@mail.gmail.com>
To: PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=ISO-8859-1
Subject: [PHP-DEV] Revert Tokenizer behavior for 5.4
From: nikita.ppv@googlemail.com (Nikita Popov)

In Bug #54089 [1] a patch was applied that cuts of token_get_all()
output after a T_HALT_COMPILER token. This was done because otherwise
PHP would keep on lexing after that and would generate errors because
of binary data (which is not valid PHP, mostly.)

The problem with the patch is, that there are some tokens after
T_HALT_COMPILER that are of interest, namely the '(' ')' ';'. After
the patch it is impossible to get those tokens, without either
relexing the code after T_HALT_COMPILER (that way you get the binary
data problem back, just with much more complex code) or writing a
regular expression to match it (which is really hard, as there may be
any token dropped by the PHP parser in there, i.e. whitespace,
comments, PHP tags).

This issue was pointed out by the creator of the bug report, but was
those comments were ignored for some reason.

I would like this patch to be reverted on the 5.4 and trunk branches.
I assume it's too late to revert it on 5.3, as it has been there for
some time already. It is just counterproductive. (Alternatively one
could fix token_get_all to return the (); tokens after
__halt_compiler, too, but that would be hard, probably.)

Nikita


[1]: https://bugs.php.net/bug.php?id=54089