Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124527
Feedback-ID: i8414410d:Fastmail
User-Agent: Cyrus-JMAP/3.11.0-alpha0-568-g843fbadbe-fm-20240701.003-g843fbadb
Precedence: bulk
MIME-Version: 1.0
Message-ID: <86865df3-999e-4811-83de-eb35f7db167c@app.fastmail.com>
In-Reply-To: <b7b47e7a-fc20-4774-bc49-fff271386504@bastelstu.be>
References: <66984FD0.5090805@adviesenzo.nl>
 <AM8P250MB0170FFCA0014FB9EC272DB4FE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <c9736385-94b7-4064-911f-e5fc1df7e2bd@gmx.de>
 <AM8P250MB0170B4C59B4313A565CD029AE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <6699F817.8070806@adviesenzo.nl>
 <9571bb82-9873-4319-9bd1-0361748335be@bastelstu.be>
 <669BDB00.70507@adviesenzo.nl>
 <fa7db246-1577-43b7-a117-c18a73140091@bastelstu.be>
 <669BE870.2050908@adviesenzo.nl>
 <b7b47e7a-fc20-4774-bc49-fff271386504@bastelstu.be>
Date: Sat, 20 Jul 2024 17:16:13 -0500
To: "php internals" <internals@lists.php.net>
Subject: Re: [PHP-DEV] Request for opinions: bug vs feature - change intokenization of
 yield from
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: larry@garfieldtech.com ("Larry Garfield")

On Sat, Jul 20, 2024, at 11:51 AM, Tim D=C3=BCsterhus wrote:
> Hi
>
> On 7/20/24 18:40, Juliette Reinders Folmer wrote:
>> Tim, you're making my point for me. This is *exactly* why the current
>> change should be reverted.
>
> I am not sure how you read "PHP users have no idea what a token is" as=20
> an argument in favor of reverting the change, because reverting the=20
> change means that completely reasonable code suddenly stops working wi=
th=20
> a parser error in a patch version and PHP users will rightfully come t=
o=20
> PHP's issue tracker to complain.
>
>> And not, like it is now, an undocumented, random change creating an
>> inconsistency in the Tokenizer.
>
> The tokenizer is doing the right thing: It tokenizes the PHP source=20
> code. It is absolutely normal that PHP first and second-digit updates=20
> make changes to the token stream. New tokens are added, old tokens are=20
> removed, tokens may appear in places where they previously could not=20
> appear for well-formed PHP programs. Tools working on the token stream=20
> need to adapt and this change is no different from any other change to=20
> PHP's syntax in that regard (except that documenting the change was=20
> forgotten).
>
> Best regards
> Tim D=C3=BCsterhus

Yes, any syntax change means tools need to adapt, but that doesn't mean =
tokenization can change randomly and accidentally.  Syntax changes requi=
re an RFC.  If an RFC passes that necessitates SA tools update, so be it=
.  (That happens almost every version.)  But that's still an RFC change.

I would agree with reverting this change for now.  The odds of it breaki=
ng something for someone are vanishingly small, and it's a bug, not a fe=
ature.  If we want it to be a feature, we can make an RFC for it.

--Larry Garfield