Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:90213
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: softfail (pb1.pair.com: domain golemon.com does not designate 209.85.215.48 as permitted sender)
MIME-Version: 1.0
Sender: php@golemon.com
In-Reply-To: <CAF+90c8y4BupOrkstxE58LSZu86O6rG8Tm2LZpbULCvXuNzUcQ@mail.gmail.com>
References: <CAESVnVoJBRF3VfKrnWGNCah3KAbxwjacPAwR1HZUC-T-vN=KDQ@mail.gmail.com>
	<CAF+90c_2zxzeQZ8rwPwX5ncBUa=dDXGGngCc97Cokr+UB4_Kkg@mail.gmail.com>
	<CAESVnVqbo0kTSaJfPKHt_uT_mUJT3bzy2cDc8A5zOnaOBzvD7A@mail.gmail.com>
	<CAF+90c8y4BupOrkstxE58LSZu86O6rG8Tm2LZpbULCvXuNzUcQ@mail.gmail.com>
Date: Wed, 6 Jan 2016 11:01:39 -0800
Message-ID: <CAESVnVrTuS75xdrGfo9LbGawVdfG=OGCqxoyTx6j_zO4UMj2CQ@mail.gmail.com>
To: Nikita Popov <nikita.ppv@gmail.com>
Cc: PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=UTF-8
Subject: Re: [PHP-DEV] [RFC] Normalize token_get_all() output (with flag)
From: pollita@php.net (Sara Golemon)

On Wed, Jan 6, 2016 at 1:43 AM, Nikita Popov <nikita.ppv@gmail.com> wrote:
> Basically: token_get_all() is rather slow. I think it says something that
> getting the tokens of a script is about as slow as lexing it, parsing it
> into an internal AST and constructing an object-based userland AST for it.
> If you use token_get_all() in a matter that only requires one lookahead
> token at a time, you don't really care about how nice the token format is,
> you're only interested in it being efficient. I was hoping that we can
> optimize it by dropping the line numbers (which is the most volatile part of
> the structure) and try to reuse the same array for tokens which have the
> same ID and content (but likely different lineno). It's very likely that a
> script contains the T_WHITESPACE( ) token more than one and similarly labels
> and variables tend to repeat, etc. No idea if that would actually work/help,
> just an idea.
>
Ah, I see what you mean.  That'll take some thinking, and is well
outside the scope of this RFC, but I'll give it some thought and maybe
you or I or someone can gist something up later maybe...

-Sara