Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:98613
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: softfail (pb1.pair.com: domain golemon.com does not designate 74.125.82.49 as permitted sender)
MIME-Version: 1.0
Sender: php@golemon.com
In-Reply-To: <op.yxkq9ac9zsg2np@hnb2>
References: <CAF+90c9vdsLuh1GAB4VPbYzJp=XiH6a5w79t5Kt=JLV6=xkdDA@mail.gmail.com>
 <op.yxkq9ac9zsg2np@hnb2>
Date: Thu, 23 Mar 2017 18:33:31 -0500
Message-ID: <CAESVnVqcJ3TxPPL3nea3Z_YYjQocG6_hzPGMSA0_bsqG1CWN1g@mail.gmail.com>
To: =?UTF-8?Q?Jan_Tvrd=C3=ADk?= <j.tvr@centrum.cz>
Cc: PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] Re: TOKEN_AS_OBJECT for token_get_all()
From: pollita@php.net (Sara Golemon)

On Thu, 23 Mar 2017 18:16:31 +0100, Nikita Popov <nikita.ppv@gmail.com>
> I'd like to add a new TOKEN_AS_OBJECT flag to token_get_all(), which
> returns an array of PhpToken objects, rather than the mix of plain string=
s
> and arrays we currently have. The PhpToken class is defined as:
>
> class PhpToken {
>     public $type;
>     public $text;
>     public $line;
> }
>
> This has been previously suggested and implemented by Rouven We=C3=9Fling=
 [1],
> I've just ported this feature to master and optimized the implementation
> [2].
>
>
Yeah, IIRC the more recent discussion ended at "Oh, and this! And
this! And this!" which eventually went nowhere (largely my ADD).  IMO
there's no harm in adding this, and the class format seems entirely
reasonable.

If I may bikeshed a TINY bit, I'd ask that all tokens return as
objects, rather than char|PhpToken similar to the current char|array
format we have. (Maybe that's in the PR, I haven't looked at either)


On Thu, Mar 23, 2017 at 5:25 PM, Jan Tvrd=C3=ADk <j.tvr@centrum.cz> wrote:
> Regarding memory - would it be possible to return iterator instead of arr=
ay?
>
That has some hairy edges to it since the lexer isn't actually
reentrant within a given thread. Image the following:

foreach (token_get_all($code, TOKEN_AS_ITERATOR) as $token) {
  $moretokens =3D token_get_all($morecode, $whateverflags); // <--- Here
be broken dreams and crying unicorns
}

Worse still if you have parallel iterators.

That's probably fixable, but it's a much heavier refactor and one that
should probably have an RFC.

-Sara