Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:124516
Subject: Re: [PHP-DEV] Request for opinions: bug vs feature - change
 intokenization of yield from
To: internals@lists.php.net
References: <66984FD0.5090805@adviesenzo.nl>
 <AM8P250MB0170FFCA0014FB9EC272DB4FE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <c9736385-94b7-4064-911f-e5fc1df7e2bd@gmx.de>
 <AM8P250MB0170B4C59B4313A565CD029AE2AC2@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
 <6699F817.8070806@adviesenzo.nl>
 <9571bb82-9873-4319-9bd1-0361748335be@bastelstu.be>
Message-ID: <669BDB00.70507@adviesenzo.nl>
Date: Sat, 20 Jul 2024 17:42:56 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.7.0
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <9571bb82-9873-4319-9bd1-0361748335be@bastelstu.be>
Content-Type: multipart/alternative;
 boundary="------------050008020602030606080801"
From: php-internals_nospam@adviesenzo.nl (Juliette Reinders Folmer)

This is a multi-part message in MIME format.
--------------050008020602030606080801
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

On 20-7-2024 16:51, Tim Düsterhus wrote:
> Hi
>
> On 7/19/24 07:22, Juliette Reinders Folmer wrote:
>> More than anything, I find it concerning that this change sets a
>> precedent for tokens to include comments.
>>
>> Just as an example: what does this mean for the PHP 8.0 nullsafe object
>> operator ? Should we now suddenly allow that to be written as `?
>> /*comment*/ ->` ?
>> Or what about a cast token ? Should that be allowed to be `(string /*for
>> reasons*/)` ?
>
> The difference between `yield from` and `?->` is that the former looks 
> and feels like it would be two separate keywords, because of the 
> *required* whitespace between the `yield` and the `from`. The fact 
> that a `yield` keyword actually exists also contributes to that. `?->` 
> on the other hand looks and feels like a single operator, just like 
> `++`, `!==`, `<=>` and others.
>
> Except for `yield from` the rule where comments may be placed as far 
> as I can tell is "comments may appear where whitespace may appear", 
> which is easy enough to explain and understand. So it makes sense to 
> allow for comments between `yield` and `from`, but I agree that 
> ideally those would be emitted as separate tokens.

Tim,

"comments may appear where whitespace may appear" ?

You'd think so, except it isn't true.

I already mentioned cast tokens before. Whitespace is perfectly 
acceptable within the parentheses of these. Comments are not: 
https://3v4l.org/A6Sgj and https://3v4l.org/nE9H8

Now you may argue that cast tokens "feel like" a single operator, but 
that's subjective and there's even a sniff to enforce no spacing within 
cast parentheses as apparently people do pad them with spaces - and 
doing so is allowed in PHP. *
_* Qualifier: spaces and tabs are allowed inside cast parentheses, but 
new lines are not..._

Along the same lines and I'm beginning to repeat myself, the PHP 8.0 RFC 
which changed the tokenization of namespaced names explicitly disallowed 
whitespace and comments _inside_ namespaced names tokenized as a single 
token, while in the previous, multi-token situation, whitespace and 
comments were allowed in namespaced names.

So to get back to my original point, as of PHP 8.3 is the **only** token 
which allows for a comment to be tokenized as part of the token. There 
is no other token which allows that.

Whitespace is one thing, comments is a different matter. And even the 
whitespace is an interesting one as I've seen bug reports in PHPCS about 
a sniff breaking on `yield from` with a new line and indentation between 
the keywords. PHP allows this, the sniff in question does not handle it 
correctly.

So, what "feels" natural (whitespace-wise) to one person may not be the 
same for the next, but comments _within_ tokens is different thing and 
should in my opinion, not be allowed.

Smile,
Juliette






--------------050008020602030606080801
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 20-7-2024 16:51, Tim Düsterhus
      wrote:<br>
    </div>
    <blockquote
      cite="mid:9571bb82-9873-4319-9bd1-0361748335be@bastelstu.be"
      type="cite">Hi
      <br>
      <br>
      On 7/19/24 07:22, Juliette Reinders Folmer wrote:
      <br>
      <blockquote type="cite">More than anything, I find it concerning
        that this change sets a
        <br>
        precedent for tokens to include comments.
        <br>
        <br>
        Just as an example: what does this mean for the PHP 8.0 nullsafe
        object
        <br>
        operator ? Should we now suddenly allow that to be written as `?
        <br>
        /*comment*/ -&gt;` ?
        <br>
        Or what about a cast token ? Should that be allowed to be
        `(string /*for
        <br>
        reasons*/)` ?
        <br>
      </blockquote>
      <br>
      The difference between `yield from` and `?-&gt;` is that the
      former looks and feels like it would be two separate keywords,
      because of the *required* whitespace between the `yield` and the
      `from`. The fact that a `yield` keyword actually exists also
      contributes to that. `?-&gt;` on the other hand looks and feels
      like a single operator, just like `++`, `!==`, `&lt;=&gt;` and
      others.
      <br>
      <br>
      Except for `yield from` the rule where comments may be placed as
      far as I can tell is "comments may appear where whitespace may
      appear", which is easy enough to explain and understand. So it
      makes sense to allow for comments between `yield` and `from`, but
      I agree that ideally those would be emitted as separate tokens.
      <br>
    </blockquote>
    <br>
    Tim,<br>
    <br>
    "comments may appear where whitespace may appear" ?<br>
    <br>
    You'd think so, except it isn't true.<br>
    <br>
    I already mentioned cast tokens before. Whitespace is perfectly
    acceptable within the parentheses of these. Comments are not: <a
      href="https://3v4l.org/A6Sgj" rel="nofollow"><a class="moz-txt-link-freetext" href="https://3v4l.org/A6Sgj">https://3v4l.org/A6Sgj</a></a>
    and <a href="https://3v4l.org/nE9H8" rel="nofollow">https://3v4l.org/nE9H8</a><br>
    <br>
    Now you may argue that cast tokens "feel like" a single operator,
    but that's subjective and there's even a sniff to enforce no spacing
    within cast parentheses as apparently people do pad them with spaces
    - and doing so is allowed in PHP. *<br>
    _* Qualifier: spaces and tabs are allowed inside cast parentheses,
    but new lines are not..._<br>
    <br>
    Along the same lines and I'm beginning to repeat myself, the PHP 8.0
    RFC which changed the tokenization of namespaced names explicitly
    disallowed whitespace and comments _inside_ namespaced names
    tokenized as a single token, while in the previous, multi-token
    situation, whitespace and comments were allowed in namespaced names.<br>
    <br>
    So to get back to my original point, as of PHP 8.3 is the **only**
    token which allows for a comment to be tokenized as part of the
    token. There is no other token which allows that.<br>
    <br>
    Whitespace is one thing, comments is a different matter. And even
    the whitespace is an interesting one as I've seen bug reports in
    PHPCS about a sniff breaking on `yield from` with a new line and
    indentation between the keywords. PHP allows this, the sniff in
    question does not handle it correctly.<br>
    <br>
    So, what "feels" natural (whitespace-wise) to one person may not be
    the same for the next, but comments _within_ tokens is different
    thing and should in my opinion, not be allowed.<br>
    <br>
    Smile,<br>
    Juliette<br>
    <br>
    <br>
    <br>
    <br>
    <br>
  </body>
</html>

--------------050008020602030606080801--