Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124520
Message-ID: <b7b47e7a-fc20-4774-bc49-fff271386504@bastelstu.be>
Date: Sat, 20 Jul 2024 18:51:33 +0200
Precedence: bulk
MIME-Version: 1.0
Subject: Re: [PHP-DEV] Request for opinions: bug vs feature - change
 intokenization of yield from
To: Juliette Reinders Folmer <php-internals_nospam@adviesenzo.nl>,
 internals@lists.php.net
References: <66984FD0.5090805@adviesenzo.nl>
 <AM8P250MB0170FFCA0014FB9EC272DB4FE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <c9736385-94b7-4064-911f-e5fc1df7e2bd@gmx.de>
 <AM8P250MB0170B4C59B4313A565CD029AE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <6699F817.8070806@adviesenzo.nl>
 <9571bb82-9873-4319-9bd1-0361748335be@bastelstu.be>
 <669BDB00.70507@adviesenzo.nl>
 <fa7db246-1577-43b7-a117-c18a73140091@bastelstu.be>
 <669BE870.2050908@adviesenzo.nl>
Content-Language: en-US
In-Reply-To: <669BE870.2050908@adviesenzo.nl>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
From: tim@bastelstu.be (=?UTF-8?Q?Tim_D=C3=BCsterhus?=)

Hi

On 7/20/24 18:40, Juliette Reinders Folmer wrote:
> Tim, you're making my point for me. This is *exactly* why the current
> change should be reverted.

I am not sure how you read "PHP users have no idea what a token is" as 
an argument in favor of reverting the change, because reverting the 
change means that completely reasonable code suddenly stops working with 
a parser error in a patch version and PHP users will rightfully come to 
PHP's issue tracker to complain.

> And not, like it is now, an undocumented, random change creating an
> inconsistency in the Tokenizer.

The tokenizer is doing the right thing: It tokenizes the PHP source 
code. It is absolutely normal that PHP first and second-digit updates 
make changes to the token stream. New tokens are added, old tokens are 
removed, tokens may appear in places where they previously could not 
appear for well-formed PHP programs. Tools working on the token stream 
need to adapt and this change is no different from any other change to 
PHP's syntax in that regard (except that documenting the change was 
forgotten).

Best regards
Tim Düsterhus