Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:90572
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: softfail (pb1.pair.com: domain golemon.com does not designate 209.85.217.173 as permitted sender)
MIME-Version: 1.0
Sender: php@golemon.com
In-Reply-To: <B0800012-C341-468D-AE1C-BF6551AB356F@rouvenwessling.de>
References: <CAESVnVoJBRF3VfKrnWGNCah3KAbxwjacPAwR1HZUC-T-vN=KDQ@mail.gmail.com>
	<77.F6.12097.81D2C865@pb1.pair.com>
	<B0800012-C341-468D-AE1C-BF6551AB356F@rouvenwessling.de>
Date: Tue, 12 Jan 2016 10:58:42 -0800
Message-ID: <CAESVnVobg-woJ2eHff_RSFkkMPZLyqOia1U+agSSTNxCjYc_tg@mail.gmail.com>
To: =?UTF-8?Q?Rouven_We=C3=9Fling?= <me@rouvenwessling.de>
Cc: Andrea Faulds <ajf@ajf.me>, PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] Re: [RFC] Normalize token_get_all() output (with flag)
From: pollita@php.net (Sara Golemon)

On Tue, Jan 12, 2016 at 9:07 AM, Rouven We=C3=9Fling <me@rouvenwessling.de>=
 wrote:
>> On 05 Jan 2016, at 21:52, Andrea Faulds <ajf@ajf.me> wrote:
>> This is more of a side-note, but maybe it's worth bringing up. Since tok=
en_get_all gives an array with subarrays of a regular structure, might it b=
e worthwhile returning an array of objects instead? It would probably reduc=
e memory usage (assuming they're objects of a Token class or something, not=
 StdClass), but I don't know if it's that useful.
>>
>
> I liked the idea so I went ahead and tested it. Code is here: https://git=
hub.com/php/php-src/pull/1727
>
> Doing some incredibly simplistic testing using a modified version of Niki=
ta=E2=80=99s PHP-Parser.
> It saved ~10% in memory (just parsing the parser itself and immediately d=
iscarding the
> result) and speed up within the margin of error. The proposal from Sara w=
ould probably
> take slightly more memory than the current code since there would be more=
 arrays allocated.
>
Nice! I'll say one thing though: If we're going to introduce a class,
I'm tempted to leave token_get_all alone (or at least limit its
changes to what's in the current porposal), and have the PhpToken
class implement its own static method to do the same thing.  This
avoid overloading token_get_all() with a bunch of behaviors and allows
us to have some broader choice of actions with PhpToken::scan().  i.e.
An iteration version which would probably have nice perf wins, or a
callback version, etc...  This also minimizes the bike-shedding risk
while we try out ideas.

How does this sound?
1. Keep the current RFC basically as is.  It's a minor addition to
token_get_all() which can be slotted into existing code to improve
readability, but offers little advantage beyond that.
2. Make a new extension to prototype this PhpToken class outside of
the php-src tree, add all the extra goodies we want (array of token
return, iterator of token return, extra info, limited info, etc...)
3. When this extension is good and solid, propose merging into core.

I had reserved the github.com/phplang project for the specification,
but didn't end up using it.  We can share a repo there if you'd like.

-Sara