Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124518
Message-ID: <fa7db246-1577-43b7-a117-c18a73140091@bastelstu.be>
Date: Sat, 20 Jul 2024 18:04:52 +0200
Precedence: bulk
MIME-Version: 1.0
Subject: Re: [PHP-DEV] Request for opinions: bug vs feature - change
 intokenization of yield from
To: Juliette Reinders Folmer <php-internals_nospam@adviesenzo.nl>,
 internals@lists.php.net
References: <66984FD0.5090805@adviesenzo.nl>
 <AM8P250MB0170FFCA0014FB9EC272DB4FE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <c9736385-94b7-4064-911f-e5fc1df7e2bd@gmx.de>
 <AM8P250MB0170B4C59B4313A565CD029AE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <6699F817.8070806@adviesenzo.nl>
 <9571bb82-9873-4319-9bd1-0361748335be@bastelstu.be>
 <669BDB00.70507@adviesenzo.nl>
Content-Language: en-US
In-Reply-To: <669BDB00.70507@adviesenzo.nl>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
From: tim@bastelstu.be (=?UTF-8?Q?Tim_D=C3=BCsterhus?=)

Hi

On 7/20/24 17:42, Juliette Reinders Folmer wrote:
> I already mentioned cast tokens before. Whitespace is perfectly
> acceptable within the parentheses of these. Comments are not:
> https://3v4l.org/A6Sgj and https://3v4l.org/nE9H8
> 
> Now you may argue that cast tokens "feel like" a single operator, but
> that's subjective and there's even a sniff to enforce no spacing within
> cast parentheses as apparently people do pad them with spaces - and
> doing so is allowed in PHP. *
> _* Qualifier: spaces and tabs are allowed inside cast parentheses, but
> new lines are not..._

I stand corrected. Though that indeed is another odd special case with 
not allowing newlines. Looking at the lexer, there's another case where 
tabs and spaces may appear, but not other whitespace. Between '<<<' and 
the delimiter of a heredoc.

> So to get back to my original point, as of PHP 8.3 is the **only** token
> which allows for a comment to be tokenized as part of the token. There
> is no other token which allows that.

PHP users have no idea what a token is internally. I'm looking at this 
from a PHP user perspective. It looks like two keywords, it walks like 
two keywords and it quacks like two keywords. I find it reasonable for 
users to consider this as two keywords and not care about how it's 
implemented internally.

> So, what "feels" natural (whitespace-wise) to one person may not be the
> same for the next, but comments _within_ tokens is different thing and
> should in my opinion, not be allowed.

As I've said: I agree that the current situation is unfortunate. But the 
correct solution is not "disallow comments", but "split the T_YIELD_FROM 
into T_YIELD T_WHITESPACE T_FROM_FOR_YIELD_FROM".

Best regards
Tim Düsterhus