Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:127081
Precedence: bulk
MIME-Version: 1.0
References: <de8140ab-0941-45cd-ba20-4aece2f69aea@app.fastmail.com>
 <CAPyj-LBvmUfhakS+Z9wDYQ-J7Bypk9bbTupPqv-kvfPp8r_qnA@mail.gmail.com>
 <5efa2f02-dd1d-4d59-ae07-c75f193b4096@app.fastmail.com> <92b7f1ea-900b-4438-bed7-3fd766bb2d61@rwec.co.uk>
 <7e2a3dea-aaaf-4427-b1b2-32c568af8b77@app.fastmail.com> <51df1d77-33ce-414e-b489-8a62f9768811@rwec.co.uk>
 <78cb31b7-ac23-4ee6-8317-5ba265db8de2@app.fastmail.com>
In-Reply-To: <78cb31b7-ac23-4ee6-8317-5ba265db8de2@app.fastmail.com>
Date: Wed, 9 Apr 2025 01:29:45 +0200
Message-ID: <CAPyj-LBBsHoPZ+Ueu8s8cm5tSi6L5tJbAbOk=eOdHbGje_A3nQ@mail.gmail.com>
Subject: Re: [PHP-DEV] [RFC] Pipe Operator (again)
To: php internals <internals@lists.php.net>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
From: tovilo.ilija@gmail.com (Ilija Tovilo)

Hi Larry

Sorry again for the delay.

On Fri, Apr 4, 2025 at 6:37=E2=80=AFAM Larry Garfield <larry@garfieldtech.c=
om> wrote:
>
> * A new iterable API is absolutely a good thing and we should do it.
> * That said, we *need* to split Sequence, Set, and Dictionary into separa=
te types.  We are the only language I reviewed that didn't have them as sep=
arate constructs with their own APIs.
> * The use of the same construct (arrays and iterables) for all three type=
s is a fundamental and core flaw in PHP's design that we should not double-=
down on.  It's ergonomically awful, it's bad for performance, and it invite=
s major security holes.  (The "Drupageddon" remote exploit was caused by us=
ing an array and assuming it was sequential when it was actually a map.)
>
> So while I want a new iterable API, the more I think on it, the more I th=
ink a bunch of map(iterable $it, callable $fn) style functions would not be=
 the right way to do it.  That would be easy, but also ineffective.
>
> The behavior of even basic operations like map and filter are subtly diff=
erent depending on which type you're dealing with.  Whether the input is la=
zy or not is the least of the concerns.  The bigger issue is when to pass k=
eys to the $fn; probably always in Dict, probably never in Seq, and certain=
ly never in Set (as there are no meaningful keys).  Similarly, when filteri=
ng a Dict, you would want keys preserved.  When filtering a Seq, you'd want=
 the indexes re-zeroed.  (Or to seem like it, given or take implementation =
details.)  And then, yes, there's the laziness question.
>
> So we'd effectively want three different versions of map(), filter(), etc=
. if we didn't want to perpetuate and further entrench the design flaw and =
security hole that is "sequences and hashes are the same thing if you squin=
t."  And... frankly I'd probably vote against an interable/collections API =
that didn't address that issue.

I fundamentally disagree with this assessment. In most languages,
including PHP, iterators are simply a sequence of values that can be
consumed. Usually, the consumer should not be concerned with the data
structure of the iterated value, this is abstracted away through the
iterator. For most languages, both Sequences and Sets are translated
1:1 (i.e. Sequence<T> =3D> Iterator<T>, Set<T> =3D> Iterator<T>).
Dictionaries usually result in a tuple, combining both the key and
value into a single value pair (Dict<T, U> =3D> Iterator<(T, U)>). PHP
is a bit different in that all iterators require a key. Semantically,
this makes sense for both Sequences (which are logically indexed by
the elements position in the sequence, so Sequence<T> =3D> Iterator<int,
T>) and Dicts (which have an explicit key, so Dict<T, U> =3D>
Iterator<T, U>). Sets don't technically have a logical key, but IMO
this is not enough of a reason to fundamentally change how iterators
work. A sequential number would be fine, which is also what yield
without providing a key does. If we really wanted to avoid it, we can
make it return null, as this is already allowed for generators.
https://3v4l.org/LvIjP

The big upside of treating all iterators the same, regardless of their
data source is 1. the code becomes more generic, you don't need three
variants of a value map() functions when the one works on all of them.
And 2. you can populate any of the data structures from a generic
iterator without any data shuffling.

$users
    |> Iter\mapKeys(fn($u) =3D> $u->getId())
    |> Iter\toDict();

This will work if $users is a Sequence, Set or existing Dict with some
other key. Actually, it works for any Traversable. If mapKeys() only
applied to Dict iterators you would necessarily have to create a
temporary dictionary first, or just not use the iterator API at all.

> However, a simple "first arg" pipe wouldn't allow for that.  Or rather, w=
e'd need to implement seqMap(iterable $it, callable $fn), setMap(iterable $=
it, callable $fn), and dictMap(iterable $it, callable $fn).  And the same s=
plit for filter, and probably a few other things.  That seems ergonomically=
 suspect, at best, and still wouldn't really address the issue since you wo=
uld have no way to ensure you're using the "right" version of each function=
. Similarly, a dict version of implode() would likely need to take 2 separa=
tors, whereas the other types would take only one.
>
> So the more I think on it, the more I think the sort of iterable API that=
 first-arg pipes would make easy is... probably not the iterable API we wan=
t anyway.  There may well be other cases for Elixir-style first-arg pipes, =
but a new iterable API isn't one of them, at least not in this form.

After having talked to you directly, it seemed to me that there is
some confusion about the iterator API vs. the API offered by the data
structure itself. For example:

> $l =3D new List(1,2, 3);
> $l2 =3D $l |> map(fn($x) =3D> $x*2);
>
> What is the type of $l2? I would expect it to be a List, but there's curr=
ently
> no way to write a map() that statically guarantees that. (And that's befo=
re we
> get into generics.)

$l2 wouldn't be a List (or Sequence, to stick with the same
terminology) but an iterator, specifically Iterator<int, int>. If you
want to get back a sequence, you need to populate a new sequence from
the iterator using Iter\toSeq(). We may also decide to introduce a
Sequence::map() method that maps directly to a new sequence, which may
be more efficient for single transformations. That said, the nice
thing about the iterator API is that it generically applies to all
data structures implementing Traversable. For example, an Iter\max()
function would not need to care about the implementation details of
the underlying data structure, nor do all data structures need to
reimplement their own versions of max().

> Which brings us then to extension functions.

I have largely changed my mind on extension functions. Extension
functions that are exclusively local, static and detached from the
type system are rather useless. Looking at an example:

> function PointEntity.toMessage(): PointMessage {
>     return new PointMessage($this->x, $this->y);
> }
>
> $result =3D json_encode($point->toMessage());

If for some reason toMessage() cannot be implemented on PointEntity,
there's arguably no benefit of $point->toMessage() over `$point |>
PointEntityExtension\toMessage()` (with an optional import to make it
almost as short). All the extension really achieves is changing the
syntax, but we would already have the pipe operator for this.
Technically, you can use such extensions for untyped, local
polymorphism, but this does not seem like a good approach.

function PointEntity.toMessage(): PointMessage { ... }
function RectEntity.toMessage(): RectMessage { ... }

$entities =3D [new Point, new Rect];

foreach ($entities as $e) {
    $e->toMessage(); // Technically works, but the type system is
entirely unaware.
    takesToMessage($e); // This breaks, because Point and Rect don't
actually implement the ToMessage interface.
}

Where extensions would really shine is if they could hook into the
type system by implementing interfaces on types that aren't in your
control. Rust and Swift are two examples that take this approach.

implement ToMessage for Rect { ... }

takesToMessage(new Rect); // Now this actually works.

However, this becomes even harder to implement than extension
functions already would. I won't go into detail because this e-mail is
already too long, but I'm happy to discuss it further off-list. All
this to say, I don't think extensions will work well in PHP, but I
also don't think they are necessary for the iterator API.

Regards,
Ilija