Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:90679
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: softfail (pb1.pair.com: domain golemon.com does not designate 209.85.215.53 as permitted sender)
MIME-Version: 1.0
Sender: php@golemon.com
In-Reply-To: <3F.20.05337.CEE68965@pb1.pair.com>
References: <CAESVnVoJBRF3VfKrnWGNCah3KAbxwjacPAwR1HZUC-T-vN=KDQ@mail.gmail.com>
	<77.F6.12097.81D2C865@pb1.pair.com>
	<B0800012-C341-468D-AE1C-BF6551AB356F@rouvenwessling.de>
	<CAESVnVobg-woJ2eHff_RSFkkMPZLyqOia1U+agSSTNxCjYc_tg@mail.gmail.com>
	<3F.20.05337.CEE68965@pb1.pair.com>
Date: Sat, 16 Jan 2016 20:23:41 -0800
Message-ID: <CAESVnVr=RjyJX272rgZT-3gJpmxwLkkHYAuUCLxb4wWtLFG-bw@mail.gmail.com>
To: Andrea Faulds <ajf@ajf.me>
Cc: PHP internals <internals@lists.php.net>
Content-Type: text/plain; charset=UTF-8
Subject: Re: [PHP-DEV] Re: [RFC] Normalize token_get_all() output (with flag)
From: pollita@php.net (Sara Golemon)

On Thu, Jan 14, 2016 at 7:59 PM, Andrea Faulds <ajf@ajf.me> wrote:
>> 2. Make a new extension to prototype this PhpToken class outside of
>> the php-src tree, add all the extra goodies we want (array of token
>> return, iterator of token return, extra info, limited info, etc...)
>> 3. When this extension is good and solid, propose merging into core.
>>
>> I had reserved the github.com/phplang project for the specification,
>> but didn't end up using it.  We can share a repo there if you'd like.
>
>
> Like Stas, I think this sounds good. I might even look over what's created,
> if I find the time.
>
> Regarding arrays versus iterators, you can create the former from the
> latter, but not the other way around. I'd go for just having an iterator and
> no array.. It maps better to how PHP actually works, anyway.
>
> But enough discussion here, I shall await your git repository...
>
Okay, i've got a starting point up at
https://github.com/phplang/php-token .  It's basically just
token_get_all() and token_name() as they currently stand (I didn't
even bother overriding the behavior in token_name() yet).  I did
change some of the codeflow layout a bit (and add support for scanning
files in the process), but that's it.

Unfortunately... I realized a few issues with my plans to add support
for callbacks and iterators...  The tokenizer uses a global state that
is REALLY easy to invalidate, so once we allow userspace code to
resume, we're going to have to be careful about how we
preserve/restore/guard the tokenizer's state.  This is usually not an
issue since there are no scanner callbacks in PHP currently.

I've added TazeTSchnitzel(Andrea),  smalyshev(Stas), and nikic(Nikita)
to the ACL so far.  Poke me here or on twitter with your github id if
you're interested in contributing...

-Sara