Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:127084
Feedback-ID: ifab94697:Fastmail
Precedence: bulk
MIME-Version: 1.0
Date: Wed, 09 Apr 2025 07:56:57 +0200
To: internals@lists.php.net
Message-ID: <d89f2547-a983-4038-bfc3-a1779749804b@app.fastmail.com>
In-Reply-To: 
 <CAPyj-LBBsHoPZ+Ueu8s8cm5tSi6L5tJbAbOk=eOdHbGje_A3nQ@mail.gmail.com>
References: <de8140ab-0941-45cd-ba20-4aece2f69aea@app.fastmail.com>
 <CAPyj-LBvmUfhakS+Z9wDYQ-J7Bypk9bbTupPqv-kvfPp8r_qnA@mail.gmail.com>
 <5efa2f02-dd1d-4d59-ae07-c75f193b4096@app.fastmail.com>
 <92b7f1ea-900b-4438-bed7-3fd766bb2d61@rwec.co.uk>
 <7e2a3dea-aaaf-4427-b1b2-32c568af8b77@app.fastmail.com>
 <51df1d77-33ce-414e-b489-8a62f9768811@rwec.co.uk>
 <78cb31b7-ac23-4ee6-8317-5ba265db8de2@app.fastmail.com>
 <CAPyj-LBBsHoPZ+Ueu8s8cm5tSi6L5tJbAbOk=eOdHbGje_A3nQ@mail.gmail.com>
Subject: Re: [PHP-DEV] [RFC] Pipe Operator (again)
Content-Type: multipart/alternative;
 boundary=4161ad2e7aa3448f8a680ddcbc528409
From: rob@bottled.codes ("Rob Landers")

--4161ad2e7aa3448f8a680ddcbc528409
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On Wed, Apr 9, 2025, at 01:29, Ilija Tovilo wrote:
> Hi Larry
>=20
> Sorry again for the delay.
>=20
> On Fri, Apr 4, 2025 at 6:37=E2=80=AFAM Larry Garfield <larry@garfieldt=
ech.com> wrote:
> >
> > * A new iterable API is absolutely a good thing and we should do it.
> > * That said, we *need* to split Sequence, Set, and Dictionary into s=
eparate types.  We are the only language I reviewed that didn't have the=
m as separate constructs with their own APIs.
> > * The use of the same construct (arrays and iterables) for all three=
 types is a fundamental and core flaw in PHP's design that we should not=
 double-down on.  It's ergonomically awful, it's bad for performance, an=
d it invites major security holes.  (The "Drupageddon" remote exploit wa=
s caused by using an array and assuming it was sequential when it was ac=
tually a map.)
> >
> > So while I want a new iterable API, the more I think on it, the more=
 I think a bunch of map(iterable $it, callable $fn) style functions woul=
d not be the right way to do it.  That would be easy, but also ineffecti=
ve.
> >
> > The behavior of even basic operations like map and filter are subtly=
 different depending on which type you're dealing with.  Whether the inp=
ut is lazy or not is the least of the concerns.  The bigger issue is whe=
n to pass keys to the $fn; probably always in Dict, probably never in Se=
q, and certainly never in Set (as there are no meaningful keys).  Simila=
rly, when filtering a Dict, you would want keys preserved.  When filteri=
ng a Seq, you'd want the indexes re-zeroed.  (Or to seem like it, given =
or take implementation details.)  And then, yes, there's the laziness qu=
estion.
> >
> > So we'd effectively want three different versions of map(), filter()=
, etc. if we didn't want to perpetuate and further entrench the design f=
law and security hole that is "sequences and hashes are the same thing i=
f you squint."  And... frankly I'd probably vote against an interable/co=
llections API that didn't address that issue.
>=20
> I fundamentally disagree with this assessment. In most languages,
> including PHP, iterators are simply a sequence of values that can be
> consumed. Usually, the consumer should not be concerned with the data
> structure of the iterated value, this is abstracted away through the
> iterator. For most languages, both Sequences and Sets are translated
> 1:1 (i.e. Sequence<T> =3D> Iterator<T>, Set<T> =3D> Iterator<T>).
> Dictionaries usually result in a tuple, combining both the key and
> value into a single value pair (Dict<T, U> =3D> Iterator<(T, U)>). PHP
> is a bit different in that all iterators require a key. Semantically,
> this makes sense for both Sequences (which are logically indexed by
> the elements position in the sequence, so Sequence<T> =3D> Iterator<in=
t,
> T>) and Dicts (which have an explicit key, so Dict<T, U> =3D>
> Iterator<T, U>). Sets don't technically have a logical key, but IMO
> this is not enough of a reason to fundamentally change how iterators
> work. A sequential number would be fine, which is also what yield
> without providing a key does. If we really wanted to avoid it, we can
> make it return null, as this is already allowed for generators.
> https://3v4l.org/LvIjP
>=20
> The big upside of treating all iterators the same, regardless of their
> data source is 1. the code becomes more generic, you don't need three
> variants of a value map() functions when the one works on all of them.
> And 2. you can populate any of the data structures from a generic
> iterator without any data shuffling.
>=20
> $users
>     |> Iter\mapKeys(fn($u) =3D> $u->getId())
>     |> Iter\toDict();
>=20
> This will work if $users is a Sequence, Set or existing Dict with some
> other key. Actually, it works for any Traversable. If mapKeys() only
> applied to Dict iterators you would necessarily have to create a
> temporary dictionary first, or just not use the iterator API at all.
>=20
> > However, a simple "first arg" pipe wouldn't allow for that.  Or rath=
er, we'd need to implement seqMap(iterable $it, callable $fn), setMap(it=
erable $it, callable $fn), and dictMap(iterable $it, callable $fn).  And=
 the same split for filter, and probably a few other things.  That seems=
 ergonomically suspect, at best, and still wouldn't really address the i=
ssue since you would have no way to ensure you're using the "right" vers=
ion of each function. Similarly, a dict version of implode() would likel=
y need to take 2 separators, whereas the other types would take only one.
> >
> > So the more I think on it, the more I think the sort of iterable API=
 that first-arg pipes would make easy is... probably not the iterable AP=
I we want anyway.  There may well be other cases for Elixir-style first-=
arg pipes, but a new iterable API isn't one of them, at least not in thi=
s form.
>=20
> After having talked to you directly, it seemed to me that there is
> some confusion about the iterator API vs. the API offered by the data
> structure itself. For example:
>=20
> > $l =3D new List(1,2, 3);
> > $l2 =3D $l |> map(fn($x) =3D> $x*2);
> >
> > What is the type of $l2? I would expect it to be a List, but there's=
 currently
> > no way to write a map() that statically guarantees that. (And that's=
 before we
> > get into generics.)
>=20
> $l2 wouldn't be a List (or Sequence, to stick with the same
> terminology) but an iterator, specifically Iterator<int, int>. If you
> want to get back a sequence, you need to populate a new sequence from
> the iterator using Iter\toSeq(). We may also decide to introduce a
> Sequence::map() method that maps directly to a new sequence, which may
> be more efficient for single transformations. That said, the nice
> thing about the iterator API is that it generically applies to all
> data structures implementing Traversable. For example, an Iter\max()
> function would not need to care about the implementation details of
> the underlying data structure, nor do all data structures need to
> reimplement their own versions of max().
>=20
> > Which brings us then to extension functions.
>=20
> I have largely changed my mind on extension functions. Extension
> functions that are exclusively local, static and detached from the
> type system are rather useless. Looking at an example:
>=20
> > function PointEntity.toMessage(): PointMessage {
> >     return new PointMessage($this->x, $this->y);
> > }
> >
> > $result =3D json_encode($point->toMessage());
>=20
> If for some reason toMessage() cannot be implemented on PointEntity,
> there's arguably no benefit of $point->toMessage() over `$point |>
> PointEntityExtension\toMessage()` (with an optional import to make it
> almost as short). All the extension really achieves is changing the
> syntax, but we would already have the pipe operator for this.
> Technically, you can use such extensions for untyped, local
> polymorphism, but this does not seem like a good approach.
>=20
> function PointEntity.toMessage(): PointMessage { ... }
> function RectEntity.toMessage(): RectMessage { ... }
>=20
> $entities =3D [new Point, new Rect];
>=20
> foreach ($entities as $e) {
>     $e->toMessage(); // Technically works, but the type system is
> entirely unaware.
>     takesToMessage($e); // This breaks, because Point and Rect don't
> actually implement the ToMessage interface.
> }
>=20
> Where extensions would really shine is if they could hook into the
> type system by implementing interfaces on types that aren't in your
> control. Rust and Swift are two examples that take this approach.
>=20
> implement ToMessage for Rect { ... }
>=20
> takesToMessage(new Rect); // Now this actually works.
>=20
> However, this becomes even harder to implement than extension
> functions already would. I won't go into detail because this e-mail is
> already too long, but I'm happy to discuss it further off-list. All
> this to say, I don't think extensions will work well in PHP, but I
> also don't think they are necessary for the iterator API.
>=20
> Regards,
> Ilija
>=20

Hi Ilija and Larry,

This got me thinking: what if instead of "magically" passing a first val=
ue to a function, or partial applications, we create a new interface; so=
mething like:

interface PipeCompatible {
  function receiveContext(mixed $lastValue): void;
}

If the implementing type implements this interface, it will receive the =
last value via the interface before being called

This would then force userland to implement a bunch of functionality to =
take true advantage of the pipe operator, but at the same time, allow fo=
r extensions (or core / SPL) to also take full advantage of them.=20

I have no idea if such a thing works in practice, so I'm just spit balli=
ng here.

=E2=80=94 Rob
--4161ad2e7aa3448f8a680ddcbc528409
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html><html><head><title></title><style type=3D"text/css">p.Mso=
Normal,p.MsoNoSpacing{margin:0}</style></head><body><div>On Wed, Apr 9, =
2025, at 01:29, Ilija Tovilo wrote:<br></div><blockquote type=3D"cite" i=
d=3D"qt" style=3D""><div>Hi Larry<br></div><div><br></div><div>Sorry aga=
in for the delay.<br></div><div><br></div><div>On Fri, Apr 4, 2025 at 6:=
37=E2=80=AFAM Larry Garfield &lt;<a href=3D"mailto:larry@garfieldtech.co=
m">larry@garfieldtech.com</a>&gt; wrote:<br></div><div>&gt;<br></div><di=
v>&gt; * A new iterable API is absolutely a good thing and we should do =
it.<br></div><div>&gt; * That said, we *need* to split Sequence, Set, an=
d Dictionary into separate types.&nbsp; We are the only language I revie=
wed that didn't have them as separate constructs with their own APIs.<br=
></div><div>&gt; * The use of the same construct (arrays and iterables) =
for all three types is a fundamental and core flaw in PHP's design that =
we should not double-down on.&nbsp; It's ergonomically awful, it's bad f=
or performance, and it invites major security holes.&nbsp; (The "Drupage=
ddon" remote exploit was caused by using an array and assuming it was se=
quential when it was actually a map.)<br></div><div>&gt;<br></div><div>&=
gt; So while I want a new iterable API, the more I think on it, the more=
 I think a bunch of map(iterable $it, callable $fn) style functions woul=
d not be the right way to do it.&nbsp; That would be easy, but also inef=
fective.<br></div><div>&gt;<br></div><div>&gt; The behavior of even basi=
c operations like map and filter are subtly different depending on which=
 type you're dealing with.&nbsp; Whether the input is lazy or not is the=
 least of the concerns.&nbsp; The bigger issue is when to pass keys to t=
he $fn; probably always in Dict, probably never in Seq, and certainly ne=
ver in Set (as there are no meaningful keys).&nbsp; Similarly, when filt=
ering a Dict, you would want keys preserved.&nbsp; When filtering a Seq,=
 you'd want the indexes re-zeroed.&nbsp; (Or to seem like it, given or t=
ake implementation details.)&nbsp; And then, yes, there's the laziness q=
uestion.<br></div><div>&gt;<br></div><div>&gt; So we'd effectively want =
three different versions of map(), filter(), etc. if we didn't want to p=
erpetuate and further entrench the design flaw and security hole that is=
 "sequences and hashes are the same thing if you squint."&nbsp; And... f=
rankly I'd probably vote against an interable/collections API that didn'=
t address that issue.<br></div><div><br></div><div>I fundamentally disag=
ree with this assessment. In most languages,<br></div><div>including PHP=
, iterators are simply a sequence of values that can be<br></div><div>co=
nsumed. Usually, the consumer should not be concerned with the data<br><=
/div><div>structure of the iterated value, this is abstracted away throu=
gh the<br></div><div>iterator. For most languages, both Sequences and Se=
ts are translated<br></div><div>1:1 (i.e. Sequence&lt;T&gt; =3D&gt; Iter=
ator&lt;T&gt;, Set&lt;T&gt; =3D&gt; Iterator&lt;T&gt;).<br></div><div>Di=
ctionaries usually result in a tuple, combining both the key and<br></di=
v><div>value into a single value pair (Dict&lt;T, U&gt; =3D&gt; Iterator=
&lt;(T, U)&gt;). PHP<br></div><div>is a bit different in that all iterat=
ors require a key. Semantically,<br></div><div>this makes sense for both=
 Sequences (which are logically indexed by<br></div><div>the elements po=
sition in the sequence, so Sequence&lt;T&gt; =3D&gt; Iterator&lt;int,<br=
></div><div>T&gt;) and Dicts (which have an explicit key, so Dict&lt;T, =
U&gt; =3D&gt;<br></div><div>Iterator&lt;T, U&gt;). Sets don't technicall=
y have a logical key, but IMO<br></div><div>this is not enough of a reas=
on to fundamentally change how iterators<br></div><div>work. A sequentia=
l number would be fine, which is also what yield<br></div><div>without p=
roviding a key does. If we really wanted to avoid it, we can<br></div><d=
iv>make it return null, as this is already allowed for generators.<br></=
div><div><a href=3D"https://3v4l.org/LvIjP">https://3v4l.org/LvIjP</a><b=
r></div><div><br></div><div>The big upside of treating all iterators the=
 same, regardless of their<br></div><div>data source is 1. the code beco=
mes more generic, you don't need three<br></div><div>variants of a value=
 map() functions when the one works on all of them.<br></div><div>And 2.=
 you can populate any of the data structures from a generic<br></div><di=
v>iterator without any data shuffling.<br></div><div><br></div><div>$use=
rs<br></div><div>&nbsp;&nbsp;&nbsp; |&gt; Iter\mapKeys(fn($u) =3D&gt; $u=
-&gt;getId())<br></div><div>&nbsp;&nbsp;&nbsp; |&gt; Iter\toDict();<br><=
/div><div><br></div><div>This will work if $users is a Sequence, Set or =
existing Dict with some<br></div><div>other key. Actually, it works for =
any Traversable. If mapKeys() only<br></div><div>applied to Dict iterato=
rs you would necessarily have to create a<br></div><div>temporary dictio=
nary first, or just not use the iterator API at all.<br></div><div><br><=
/div><div>&gt; However, a simple "first arg" pipe wouldn't allow for tha=
t.&nbsp; Or rather, we'd need to implement seqMap(iterable $it, callable=
 $fn), setMap(iterable $it, callable $fn), and dictMap(iterable $it, cal=
lable $fn).&nbsp; And the same split for filter, and probably a few othe=
r things.&nbsp; That seems ergonomically suspect, at best, and still wou=
ldn't really address the issue since you would have no way to ensure you=
're using the "right" version of each function. Similarly, a dict versio=
n of implode() would likely need to take 2 separators, whereas the other=
 types would take only one.<br></div><div>&gt;<br></div><div>&gt; So the=
 more I think on it, the more I think the sort of iterable API that firs=
t-arg pipes would make easy is... probably not the iterable API we want =
anyway.&nbsp; There may well be other cases for Elixir-style first-arg p=
ipes, but a new iterable API isn't one of them, at least not in this for=
m.<br></div><div><br></div><div>After having talked to you directly, it =
seemed to me that there is<br></div><div>some confusion about the iterat=
or API vs. the API offered by the data<br></div><div>structure itself. F=
or example:<br></div><div><br></div><div>&gt; $l =3D new List(1,2, 3);<b=
r></div><div>&gt; $l2 =3D $l |&gt; map(fn($x) =3D&gt; $x*2);<br></div><d=
iv>&gt;<br></div><div>&gt; What is the type of $l2? I would expect it to=
 be a List, but there's currently<br></div><div>&gt; no way to write a m=
ap() that statically guarantees that. (And that's before we<br></div><di=
v>&gt; get into generics.)<br></div><div><br></div><div>$l2 wouldn't be =
a List (or Sequence, to stick with the same<br></div><div>terminology) b=
ut an iterator, specifically Iterator&lt;int, int&gt;. If you<br></div><=
div>want to get back a sequence, you need to populate a new sequence fro=
m<br></div><div>the iterator using Iter\toSeq(). We may also decide to i=
ntroduce a<br></div><div>Sequence::map() method that maps directly to a =
new sequence, which may<br></div><div>be more efficient for single trans=
formations. That said, the nice<br></div><div>thing about the iterator A=
PI is that it generically applies to all<br></div><div>data structures i=
mplementing Traversable. For example, an Iter\max()<br></div><div>functi=
on would not need to care about the implementation details of<br></div><=
div>the underlying data structure, nor do all data structures need to<br=
></div><div>reimplement their own versions of max().<br></div><div><br><=
/div><div>&gt; Which brings us then to extension functions.<br></div><di=
v><br></div><div>I have largely changed my mind on extension functions. =
Extension<br></div><div>functions that are exclusively local, static and=
 detached from the<br></div><div>type system are rather useless. Looking=
 at an example:<br></div><div><br></div><div>&gt; function PointEntity.t=
oMessage(): PointMessage {<br></div><div>&gt;&nbsp;&nbsp;&nbsp;&nbsp; re=
turn new PointMessage($this-&gt;x, $this-&gt;y);<br></div><div>&gt; }<br=
></div><div>&gt;<br></div><div>&gt; $result =3D json_encode($point-&gt;t=
oMessage());<br></div><div><br></div><div>If for some reason toMessage()=
 cannot be implemented on PointEntity,<br></div><div>there's arguably no=
 benefit of $point-&gt;toMessage() over `$point |&gt;<br></div><div>Poin=
tEntityExtension\toMessage()` (with an optional import to make it<br></d=
iv><div>almost as short). All the extension really achieves is changing =
the<br></div><div>syntax, but we would already have the pipe operator fo=
r this.<br></div><div>Technically, you can use such extensions for untyp=
ed, local<br></div><div>polymorphism, but this does not seem like a good=
 approach.<br></div><div><br></div><div>function PointEntity.toMessage()=
: PointMessage { ... }<br></div><div>function RectEntity.toMessage(): Re=
ctMessage { ... }<br></div><div><br></div><div>$entities =3D [new Point,=
 new Rect];<br></div><div><br></div><div>foreach ($entities as $e) {<br>=
</div><div>&nbsp;&nbsp;&nbsp; $e-&gt;toMessage(); // Technically works, =
but the type system is<br></div><div>entirely unaware.<br></div><div>&nb=
sp;&nbsp;&nbsp; takesToMessage($e); // This breaks, because Point and Re=
ct don't<br></div><div>actually implement the ToMessage interface.<br></=
div><div>}<br></div><div><br></div><div>Where extensions would really sh=
ine is if they could hook into the<br></div><div>type system by implemen=
ting interfaces on types that aren't in your<br></div><div>control. Rust=
 and Swift are two examples that take this approach.<br></div><div><br><=
/div><div>implement ToMessage for Rect { ... }<br></div><div><br></div><=
div>takesToMessage(new Rect); // Now this actually works.<br></div><div>=
<br></div><div>However, this becomes even harder to implement than exten=
sion<br></div><div>functions already would. I won't go into detail becau=
se this e-mail is<br></div><div>already too long, but I'm happy to discu=
ss it further off-list. All<br></div><div>this to say, I don't think ext=
ensions will work well in PHP, but I<br></div><div>also don't think they=
 are necessary for the iterator API.<br></div><div><br></div><div>Regard=
s,<br></div><div>Ilija<br></div><div><br></div></blockquote><div><br></d=
iv><div>Hi&nbsp;Ilija and Larry,<br></div><div><br></div><div>This got m=
e thinking: what if instead of "magically" passing a first value to a fu=
nction, or partial applications, we create a new interface; something li=
ke:<br></div><div><br></div><div>interface PipeCompatible {<br></div><di=
v>&nbsp; function receiveContext(mixed $lastValue): void;<br></div><div>=
}<br></div><div><br></div><div>If the implementing type implements this =
interface, it will receive the last value via the interface before being=
 called<br></div><div><br></div><div>This would then force userland to i=
mplement a bunch of functionality to take true advantage of the pipe ope=
rator, but at the same time, allow for extensions (or core / SPL) to als=
o take full advantage of them.&nbsp;</div><div><br></div><div>I have no =
idea if such a thing works in practice, so I'm just spit balling here.</=
div><div><br></div><div id=3D"sig121229152">=E2=80=94 Rob<br></div></bod=
y></html>
--4161ad2e7aa3448f8a680ddcbc528409--