token_get_all(): additional location information, and raw tokens

I’m planning on adding this functionality in some form to HHVM, however if it’s also wanted in PHP, I’d rather not add something HHVM-specific and will be happy to put up RFCs :)

Location Information
————

token_get_all() returns a line number for some tokens. I propose adding an additional TOKEN_EXTENDED_LOCATION flag, that would include:

• starting line and character number within that line
• ending line and character number within that line

T_ENCAPSED_AND_WHITESPACE and T_INLINE_HTML seem to be the most common cases of start line !== end line.

Raw Tokens
————

While token_get_all() is documented as returning whatever the lexer sees, in practice third-party software frequently depends on specific output. This gives you 3 options:

1. limit changes you make to the lexer to preserve BC
2. lie about the tokens to preserve BC
3. break BC

In our experience, #3 is not practical and #1 can lead to much more complicated solutions for problems that would be easily fixable in the lexer - so we went for #2. For example, HHVM converts:

• T_HASHBANG to T_INLINE_HTML
• T_ELSEIF to T_ELSE T_WHITESPACE T_IF

However, this means that there’s not currently a way to get the real lexer tokens. I propose adding a TOKEN_RAW flag, which should explicitly allow implementation-specific tokens and no guarantees about output stability.
For now, this would be a no-op in PHP, however it would give you more freedom in modifying the lexer in the future (in combination with #2 if the flag isn’t specified).

With thanks,

• Fred