Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124533
Message-ID: <a9b373d7-eb4c-4a17-b921-636ab4445aff@bastelstu.be>
Date: Sun, 21 Jul 2024 13:28:57 +0200
Precedence: bulk
MIME-Version: 1.0
Subject: Re: [PHP-DEV] Request for opinions: bug vs feature -
 changeintokenization of yield from
To: "Christoph M. Becker" <cmbecker69@gmx.de>,
 Juliette Reinders Folmer <php-internals_nospam@adviesenzo.nl>,
 internals@lists.php.net
References: <66984FD0.5090805@adviesenzo.nl>
 <AM8P250MB0170FFCA0014FB9EC272DB4FE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <c9736385-94b7-4064-911f-e5fc1df7e2bd@gmx.de>
 <AM8P250MB0170B4C59B4313A565CD029AE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <6699F817.8070806@adviesenzo.nl>
 <9571bb82-9873-4319-9bd1-0361748335be@bastelstu.be>
 <669BDB00.70507@adviesenzo.nl>
 <fa7db246-1577-43b7-a117-c18a73140091@bastelstu.be>
 <669BE870.2050908@adviesenzo.nl>
 <b7b47e7a-fc20-4774-bc49-fff271386504@bastelstu.be>
 <f4925fe4-15fb-432d-9dac-451240c5ea52@gmx.de>
Content-Language: en-US
In-Reply-To: <f4925fe4-15fb-432d-9dac-451240c5ea52@gmx.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
From: tim@bastelstu.be (=?UTF-8?Q?Tim_D=C3=BCsterhus?=)

Hi

On 7/20/24 20:31, Christoph M. Becker wrote:
> If the tokenizer would tokenize a whole file as a single token, would
> that also be correct?  Of course, I'm exaggerating, but

The function is documented as:

> token_get_all() parses the given code string into PHP language tokens using the Zend engine's lexical scanner.

So if Zend engine's lexical scanner would tokenize a complete PHP file 
as a single token, then the output would be correct, yes.

> <https://3v4l.org/qIf2c> doesn't look correct to me – "yield /* comment
> */ from" shouldn't be a single token.
> 

Looking at this from a user expectation perspective and not a technical 
perspective, neither should `yield from` be a single token in:

     var_dump(token_get_all('<?php yield from $foo;'));

Best regards
Tim Düsterhus