Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:90173
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain fredemmott.co.uk from 209.85.220.49 cause and error)
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-ID: <F479FC33-F639-4CA6-8E0B-8F33E8137B26@fredemmott.co.uk>
Date: Tue, 5 Jan 2016 15:57:55 -0800
To: PHP internals <internals@lists.php.net>
Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\))
Subject: token_get_all(): additional location information, and raw tokens
From: fred@fredemmott.co.uk (Fred Emmott)

I=E2=80=99m planning on adding this functionality in some form to HHVM, =
however if it=E2=80=99s also wanted in PHP, I=E2=80=99d rather not add =
something HHVM-specific and will be happy to put up RFCs :)

Location Information
=E2=80=94=E2=80=94=E2=80=94=E2=80=94

token_get_all() returns a line number for some tokens. I propose adding =
an additional TOKEN_EXTENDED_LOCATION flag, that would include:

 - starting line and character number within that line
 - ending line and character number within that line

T_ENCAPSED_AND_WHITESPACE and T_INLINE_HTML seem to be the most common =
cases of start line !=3D=3D end line.

Raw Tokens
=E2=80=94=E2=80=94=E2=80=94=E2=80=94

While token_get_all() is documented as returning whatever the lexer =
sees, in practice third-party software frequently depends on specific =
output. This gives you 3 options:

1. limit changes you make to the lexer to preserve BC
2. lie about the tokens to preserve BC
3. break BC

In our experience, #3 is not practical and #1 can lead to much more =
complicated solutions for problems that would be easily fixable in the =
lexer - so we went for #2. For example, HHVM converts:

 - T_HASHBANG to T_INLINE_HTML
 - T_ELSEIF to T_ELSE T_WHITESPACE T_IF

However, this means that there=E2=80=99s not currently a way to get the =
real lexer tokens. I propose adding a TOKEN_RAW flag, which should =
explicitly allow implementation-specific tokens and no guarantees about =
output stability.
For now, this would be a no-op in PHP, however it would give you more =
freedom in modifying the lexer in the future (in combination with #2 if =
the flag isn=E2=80=99t specified).

With thanks,
 - Fred=