# [VOTE] Object-based token_get_all() alternative

- Thread: https://externals.io/message/108885
- Original archive: https://news-web.php.net/php.internals/108885
- Started by Nikita Popov on 2020-03-06T09:33:30+00:00
- Messages: 10
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #108885 — Nikita Popov — 2020-03-06T09:33:30+00:00

URL: https://externals.io/message/108885#108885

Hi internals,

I've opened voting on https://wiki.php.net/rfc/token_as_object, which adds
a new object-based interface to the tokenizer. Voting closes on 2020-03-20.

Regards,
Nikita

---

## #108940 — come.chilliet@fusiondirectory.org — 2020-03-10T11:36:11+00:00

URL: https://externals.io/message/108885#108940 · In reply to: #108885

Hello,

I know I’m late, but shouldn't these tokens have a __toString() method? 
It’s not listed in rejected features.

-- 
Côme Chilliet
FusionDirectory - https://www.fusiondirectory.org

---

## #108952 — Nikita Popov — 2020-03-10T14:53:41+00:00

URL: https://externals.io/message/108885#108952 · In reply to: #108940

On Tue, Mar 10, 2020 at 12:36 PM Côme Chilliet <
come.chilliet@fusiondirectory.org> wrote:

> Hello,
>
> I know I’m late, but shouldn't these tokens have a __toString() method?
> It’s not listed in rejected features.
>

Would __toString() return $token->text? I'm not strictly opposed to this,
but I'd be hard pressed to think of a situation where I would prefer
writing (string) $token over $token->text. Generally, the less magic the
better :)

Nikita

---

## #108956 — come.chilliet@fusiondirectory.org — 2020-03-10T16:18:54+00:00

URL: https://externals.io/message/108885#108956 · In reply to: #108952

Le mardi 10 mars 2020, 15:53:41 CET Nikita Popov a écrit :
> Would __toString() return $token->text? I'm not strictly opposed to this,
> but I'd be hard pressed to think of a situation where I would prefer
> writing (string) $token over $token->text. Generally, the less magic the
> better :)

My idea was that since tokens are parsed from a string it should be easy to write a string back with them.
I do not have an obvious usecase but the idea is doing stuff like

if ($token.is('function')) {
	echo "public $token";
} else {
	echo $token;
}

Am I right to assume that with __toString the same as ->text we’d have implode('', PhpToken::getAll($code)) == $code ?

This would allow:
$tokens = PhpToken::getAll($code);
// Modifications on $tokens
$code = implode('', $tokens);

-- 
Côme Chilliet
FusionDirectory - https://www.fusiondirectory.org

---

## #108972 — Nikita Popov — 2020-03-11T09:57:16+00:00

URL: https://externals.io/message/108885#108972 · In reply to: #108956

On Tue, Mar 10, 2020 at 5:19 PM Côme Chilliet <
come.chilliet@fusiondirectory.org> wrote:

> Le mardi 10 mars 2020, 15:53:41 CET Nikita Popov a écrit :
> > […]
>
> My idea was that since tokens are parsed from a string it should be easy
> to write a string back with them.
> I do not have an obvious usecase but the idea is doing stuff like
>
> if ($token.is('function')) {
>         echo "public $token";
> } else {
>         echo $token;
> }
>
> Am I right to assume that with __toString the same as ->text we’d have
> implode('', PhpToken::getAll($code)) == $code ?
>
> This would allow:
> $tokens = PhpToken::getAll($code);
> // Modifications on $tokens
> $code = implode('', $tokens);
>

Yes, that would work. It's not much harder without __toString though:

$code = implode(array_column($tokens, 'text'));

Does anyone else have thoughts on whether PhpToken should implement
__toString() or not?

Nikita

---

## #108982 — Larry Garfield — 2020-03-11T15:59:19+00:00

URL: https://externals.io/message/108885#108982 · In reply to: #108972

> On Tue, Mar 10, 2020 at 5:19 PM Côme Chilliet <
> come.chilliet@fusiondirectory.org> wrote:
> 
> > […]
> 
> Yes, that would work. It's not much harder without __toString though:
> 
> $code = implode(array_column($tokens, 'text'));
> 
> Does anyone else have thoughts on whether PhpToken should implement
> __toString() or not?
> 
> Nikita

I forget who it was but someone on the list a while back offered a heuristic of "__toString should be used iff there is one and only one logical string representation of a value, because the value is conceptually just a string with extra metadata on it".

That sounds like a really solid heuristic to me.  In this case, I think yes, ->text is the only reasonable thing to produce for __toString().  

Or perhaps this is a good use case for "you can use a subclass, do that" if there's any good reason to not have it just be ->text.

--Larry Garfield

---

## #108994 — come.chilliet@fusiondirectory.org — 2020-03-12T11:07:39+00:00

URL: https://externals.io/message/108885#108994 · In reply to: #108982

Le mercredi 11 mars 2020, 10:59:19 CET Larry Garfield a écrit :
> I forget who it was but someone on the list a while back offered a heuristic of "__toString should be used iff there is one and only one logical string representation of a value, because the value is conceptually just a string with extra metadata on it".
> 
> That sounds like a really solid heuristic to me.  In this case, I think yes, ->text is the only reasonable thing to produce for __toString().  

Yes that was my reasoning, a token is essentially a substring, so I was expecting __toString to reflect this and allow me to use it as a string.

-- 
Côme Chilliet
FusionDirectory - https://www.fusiondirectory.org

---

## #109229 — Nikita Popov — 2020-03-23T14:07:46+00:00

URL: https://externals.io/message/108885#109229 · In reply to: #108885

> Hi internals,
>
> I've opened voting on https://wiki.php.net/rfc/token_as_object, which
> adds a new object-based interface to the tokenizer. Voting closes on
> 2020-03-20.
>

This RFC has been accepted unanimously, with 47 votes in favor.

Based on the additional feedback in this thread, I'll also add a
__toString() method that returns the value of $token->text, unless anyone
has objects to that...

Regards,
Nikita

---

## #109232 — Larry Garfield — 2020-03-23T16:03:35+00:00

URL: https://externals.io/message/108885#109232 · In reply to: #109229

> 
> 
> > […]
> 
> This RFC has been accepted unanimously, with 47 votes in favor.
> 
> Based on the additional feedback in this thread, I'll also add a
> __toString() method that returns the value of $token->text, unless anyone
> has objects to that...
> 
> Regards,
> Nikita

As long as __toString() is overridable in subclasses, that seems fine to me.

--Larry Garfield

---

## #109285 — Alexander Lisachenko — 2020-03-25T10:50:51+00:00

URL: https://externals.io/message/108885#109285 · In reply to: #109232

Hello, internals team!

I would like to propose to rename the PhpToken::getAll() method into the
PhpToken::tokenize(string $source) or PhpToken::lex(string $source) to use
a more meaningful verb word.

After renaming, it will sound more natural: $tokenStream =
PhpToken::tokenize($sourceCode)

Best regards,
Alexander


пн, 23 мар. 2020 г. в 19:04, Larry Garfield <larry@garfieldtech.com>:

> 
> > […]
> wrote:
> > […]
>
> As long as __toString() is overridable in subclasses, that seems fine to
> me.
>
> --Larry Garfield