[RFC] Short Closures 2, aka auto-capture take 3

Hi,

On jeudi 9 juin 2022 18:46:53 CEST Marco Pivetta wrote:

nesting these functions within each other

What happens when/if we nest these functions? Take this minimal example:

$a = 'hello world';

(fn () {
    (fn () {
        echo $a;
    })();
})();

Capture bubbles up. When an inner function uses a variable, the outer
function
in fact uses it too, so it's captured by both functions, by-value.

This example prints "hello world": The inner function captures $a from the
outer function, which captures $a from its declaring scope.

This is equivalent to

(function () use ($a) {
    (function () use ($a) {
        echo $a;
    })();
})();

capturing $this

In the past (also present), I had to type static fn () => ... or
static function () { ... all over the place, to avoid implicitly binding
$this
to a closure, causing hidden memory leaks.

Assuming following:

• these new closures could capture $this automatically, once detected
• these new closures can optimize away unnecessary variables that aren't
  captured

Would that allow us to get rid of static fn () { declarations, when
creating one of these closures in an instance method context?

It would be great to get rid of this, but ideally this would apply to
Arrow
Functions and Anonymous Functions as well. This could be a separate RFC.

I've tried this in the past, and this is not possible due to implicit $this
uses. See
https://wiki.php.net/rfc/arrow_functions_v2#this_binding_and_static_arrow_functions
for a brief note on this. The tl;dr is that if your closure does "fn() =>
Foo::bar()" and Foo happens to be a parent of your current scope and bar()
a non-static method, then this performs a scoped instance call that
inherits $this. Not binding $this here would result in an Error exception,
but the compiler doesn't have any way to know that $this needs to be bound.

Regards,
Nikita

On Sat, Jun 11, 2022 at 11:14 PM Rowan Tommins rowan.collins@gmail.com
wrote:

Last year, Nuno Maduro and I put together an RFC for combining the
multi-line capabilities of long-closures with the auto-capture compactness
of short-closures ... Arnaud Le Blanc has now picked up the flag with an
improved implementation ... The RFC has therefore been overhauled
accordingly and is now ready for consideration.

https://wiki.php.net/rfc/auto-capture-closure

They may sound like the same thing, but to me "short closure syntax"
(and a lot of the current RFC) implies that the new syntax is better for
nearly all closures, and that once it is introduced, the old syntax
would only really be there for compatibility - similar to how the []
syntax replaces array() and list(). If that is the aim, it's not enough
to assert that "the majority" of closures are very short; the syntax
should stand up even when used for, say, a middleware handler in a
micro-framework. As such, I think we need additional features to opt
back out of capturing, and explicitly mark function- or block-scoped
variables.

The RFC does mention that the existing Anonymous Function Syntax remains
untouched and will not be deprecated. Whether the new syntax is better for
nearly all closures will be a personal choice. If the new syntax doesn't
suit, say, a middleware handler, then we still can:

• reach for the old syntax
• use invocable classes
• call another method or function which creates a brand new scope and then
  returns a function/callable.

Correct. If this RFC passes, there will be three equally supported syntaxes for creating closures:

function ($a) use ($b) {
return $a * $b;
};

fn ($a) => $a * $b;

fn ($a) {
return $a * $b;
};

Which one is appropriate in a given situation is left up to developer judgement.

My own personal position would be

• use fn => where possible
• use fn {} if going mult-line.
• if the body of the closure is more than ~3 lines and is not virtually the entire wrapping scope, it should be its own named function/method anyway, and the new first-class-callable syntax makes that nice and easy to use.

That is, I would probably not use the manual-capture syntax very often at all. However, if someone disagrees with me on case 3 it's still there for them if that's easier in context.

Whether the new syntax is viewed as "adding auto-capture to long closures" or "adding multi-line support to short closures" is, in the end, a mostly academic distinction with no practical difference. The resulting syntax is smack in the middle of the two existing ones. The original RFC from a year ago approached it from the perspective of the first; The rewritten RFC leans on the second perspective. The use of both names is mostly a historical artifact of reusing the old URL. :-) The net result is the same.

--Larry Garfield

The RFC does mention that the existing Anonymous Function Syntax
remains untouched and will not be deprecated. Whether the new syntax
is better for nearly all closures will be a personal choice.

I honestly don't think this is how it will be perceived. If this syntax
is approved, people will see "fn" as the "new, better way" and
"function" as the "old, annoying way".

To put it a different way: imagine we had no closure support at all, and
decided that we needed two flavours, one with explicit capture and one
with implicit capture. Would we choose "function" and "fn" as keywords?

The previous discussions talked about use() or use(...) and most
people I know that would love this RFC to pass would also dislike that
alternative. It does not have the greatest asset for short closure:
aesthetics. [...] All I can say is that use() is not a replacement
for the RFC.

I think you're trying to have it both ways here: if you really believed
that the two syntaxes were going to live side by side, there would be no
reason for "aesthetics" to be any more important for one than the other.

Some people are of the opinion that automatic capture should always have
been the default, and the current syntax is a mistake. I'm fine with
that opinion, but I want people to be honest about it rather than
pretending they're just adding a new option for a narrow use case.

Any attempt to make it explicit defeats the purpose of the RFC.

That depends what you think the purpose of the RFC is, which is what I
want people to be honest about.

If the purpose is to replace long lists of captured variables, an
explicit "capture all" syntax like "use(*)" achieves that purpose
perfectly fine.

Ultimately, I see fn() as "an opt-in to not create a separate scope
for a function".

I disagree with both parts of this:

1. I don't think users will see "fn" as an "opt-in", they'll see it as
   "the new normal", and "function" as a rare "opt-out" or a "legacy version".
2. It does still create a separate scope, it just creates a nested
   scope, which combines two sets of variables, in a way that PHP currently
   never does.

Regards,

--
Rowan Tommins
[IMSoP]

Last year, Nuno Maduro and I put together an RFC for combining the multi-line capabilities of long-closures with the auto-capture compactness of short-closures ... Arnaud Le Blanc has now picked up the flag with an improved implementation ... The RFC has therefore been overhauled accordingly and is now ready for consideration.

https://wiki.php.net/rfc/auto-capture-closure

First of all, thanks to all three of you for the work on this. Although
I'm not quite convinced yet, I know a lot of people have expressed
desire for this feature over the years.

To go back to the point about variable scope: right now, if you're in a
function, all variables are scoped to that function. With a tiny handful
of exceptions (e.g. superglobals), access to variables from any other
scope is always explicit - via parameters, "global", "use", "$this", and
so on. If we think that should change, we should make that decision
explicitly, not treat it as a side-effect of syntax.

I don't find the comparison to a foreach loop very convincing. Loops are
still only accessing variables while the function is running, not saving
them to be used at some indeterminate later time. And users don't "learn
to recognize" that a loop doesn't hide all variables from the parent
scope; it would be very peculiar if it did.

There are languages that do, however. Some languages have block-scoped variables by default (such as Rust), or partially blocked scoped depending on details. PHP is not one of them, but to someone coming from a language that does, PHP's way of doing things is just as weird and requires learning.

Working in Go now for several years I'd say one of its biggest foot guns that I consistently run into when doing code reviews and even in my own code is block-level scoping where one variable shadows the same named variable outside the block and the inner variable is updated when the intention was to update the outer variable.

In short, block level scoping is a convenience that does more harm than good. At least in my experience.

Thus I would highly recommend not adding block level variable scope to PHP where a block in the middle of a function shadows a variable outside the block, and that variable is used below the block, such as for a return value.

#jmtcw #fwiw

-Mike

... users don't "learn to recognize" that a loop doesn't hide all variables from the parent
scope; it would be very peculiar if it did.
There are languages that do, however. Some languages have block-scoped variables by default (such as Rust), or partially blocked scoped depending on details.

That's not what the RFC example implies, though; it implies that someone
might expect $guests and $guestsIds to not be usable inside the
foreach loop, because they were declared outside it. I don't know of
any language where entering a loop creates a completely empty symbol
table, do you?

Whether or not $guest, having been declared inside the loop, is
visible after the loop is a completely different question, and one
that doesn't apply to closures - the content of the closure hasn't been
executed yet, so it is inevitably a black box to the code after it.

PHP is fairly internally consistent: functions and classes create a scope, nothing else does. This RFC doesn't change that one way or another...

The RFC fundamentally changes the rule that a function always creates a
new, empty scope. Every variable that is not local to that scope has
to be explicitly imported, one way or another.

Every language I know where scopes do not start out empty has keywords
for marking which variables are definitely local to that block. That's
why I think a "var" or "let" equivalent is a natural accompaniment to
changing PHP's rules in this way.

Plus, as noted, the fn keyword becomes consistently the flag saying "auto-capture happens here, FYI", which is already the case as of 7.4.

It's a cute idea, but I don't think "if you miss most of the letters out
of a word, it means this special thing" is at all memorable. I've never
heard the syntax added in 7.4 called "fn functions", but I've frequently
heard it called "arrow functions", because what stands out to people is
the "=>". The keyword is only there because ($a)=>$b on its own would
collide with array syntax.

I would much rather see "fn" and "function" become synonyms, so that
"public fn foo() {}" is a valid method declaration, and "function() =>
$foo" is a valid arrow function.

Regards,

--
Rowan Tommins
[IMSoP]

The proposal is to extend the Arrow Functions syntax so that it allows
multiple statements.

That's one perspective. The other perspective is that the proposal is to
extend closure syntax to support automatic capture.

Currently the use() syntax co-exists with auto-capture, but we could change
it so that an explicit use() list disables auto-capture instead:

fn () use ($a) { } // Would capture $a and disable auto-capture
fn () use () { }   // Would capture nothing and disable auto-capture

That's an interesting idea. I was coming from the other direction, but
it might make sense I guess.

By the way, the current RFC implies you could write this:

fn() use (&$myRef, $a) { $myRef = $a * $b; }

It's clear that $myRef is captured by reference, and $a by value; but
what about $b? Is it local to the closure as it would be in a "long"
closure, or implicitly captured by value as it would be with no "use"
statement?

It might be best for such mixtures to raise an error.

Unfortunately, Arrow Functions already auto-capture today, so requiring a
use(*) to enable auto-capture would be a breaking change.

I'm not suggesting any change to arrow functions, just the ability to
write "use(*)" (or "use(...)") in all the place you can write
"use($foo)" today.

I don't think that introduces any problems, if you think of "fn" as an
alternative spelling of "function", and "=>" as expanding to "use(*) {
return"

I don't find the comparison to a foreach loop very convincing. Loops are
still only accessing variables while the function is running, not saving
them to be used at some indeterminate later time.
Do you have an example where this would be a problem?

I didn't say anything was a problem; I just said that the comparison
didn't make sense, because the scenarios are so different.

Auto-capture in PHP is by-value. This makes this impossible. It also makes
explicit declarations non-necessary and much less useful.

Live-variable analysis is mentioned in as part of implementation details. It
should not be necessary to understand these details to understand the behavior
of auto-capture.

As noted in my other e-mail, by-value capture can still have side
effects, so users may still want to ensure that their code is free of
such side effects.

Currently, the only way to do so is to understand the "implementation
details" of which variables will be captured, and perhaps add dummy
statements like "$foo = null;" or "unset($foo);" to make sure of it.

Regards,

--
Rowan Tommins
[IMSoP]

Hi Arnaud,

Arnaud Le Blanc arnaud.lb@gmail.com wrote:

Following your comment, I have clarified a few things in the "Auto-capture
semantics" section. This includes a list of way in which these effects can be
observed. These are really marginal cases that are not relevant for most
programs.

Cool, thanks.

Unfortunately, Arrow Functions already auto-capture today, so requiring a
use(*) to enable auto-capture would be a breaking change.

I think there are two things that making this conversation be more
adversial than it could be otherwise:

1. Some people really want implicit auto-capture, while others are
   deeply fearful of it. That comes more from the experience people have
   from writing/reading different types of code leading them to have
   different aesthetic preferences. Trying to persuade people their lived
   experience is wrong, is hard.

2. The current situation of having to list all variables is kind of
   annoying when it doesn't provide much value e.g. for stuff like:

function getCallback($foo, $bar, $quux)
{
return function($x) use ($foo, $bar, $quux)
{
return $quux($foo, $bar, $x);
}
}

Where the code that returns the closure is trivial having to list out
the full of captured variables does feel tedious, and doesn't provide
any value.

I realise it's annoying when people suggest expanding the scope of an
RFC, however...how would you feel about adding support for use(*) to
the current 'long closures'?

That way, people could choose between:

• Explicit capture of individual variables: function($x) use ($foo,
  $bar, $quux) {...}
• Explicit capture of all relevant variables: function($x) use (*) {...}
• Implicit capture of all relevant variables, and fewer letters: fn($x) {...}

People who don't want implicit capture would be able tell their code
quality analysis tools to warn on any use of short closures (or
possibly better, warn when a variable has been captured). People who
do want implicit capture can use the short closures which always have
implicit capture.

cheers
Dan
Ack

Hi Larry, Arnaud,

Auto-capture in PHP is by-value. This makes this impossible. It also makes
explicit declarations non-necessary and much less useful.

Separating off some pedantism from the hopefully constructive comment,
I think some of the words in the RFC are a bit inaccurate:

A by-value capture means that it is not possible to modify any variables from the outer scope:

Because variables are bound by-value, the confusing behaviors often associated with closures do not exist.

Because variables are captured by-value, Short Closures can not have unintended side effects.

Those statements are true for scalar values. They are not true for objects:

class Foo
{
function __construct(public string $value) {}

function __toString()
{
return $this->value;
}
}

$a = new Foo('bar');
$f = fn() {
$a->value = 'explicit scope is nice';
};

print $a; // prints "bar"
$f();
print $a; // prints 'explicit scope is nice';

Yes, I know you can avoid these types of problems by avoiding
mutability, and/or avoiding capturing variables that represent
services, but sometimes those things are needed.

When you are capturing objects that can have side effects, making that
capture be explicit is quite nice (imo). I think the different
emphasis on capturing scalar values or objects might come down to a
difference in style of how different people use closures.

cheers
Dan
Ack

Den 2022-06-13 kl. 14:57, skrev Arnaud Le Blanc:

On samedi 11 juin 2022 23:14:28 CEST Rowan Tommins wrote:

My main concern is summed up accidentally by your choice of subject line
for this thread: is the proposal to add short closure syntax or is it
to add auto-capturing closures?

The proposal is to extend the Arrow Functions syntax so that it allows
multiple statements. I wanted to give a name to the RFC, so that we could
refer to the feature by that name instead of the longer "auto-capture multi-
statement closures". But the auto-capture behavior is an important aspect we
want to inherit from Arrow Functions.

As such, I think we need additional features to opt
back out of capturing, and explicitly mark function- or block-scoped
variables.

Currently the use() syntax co-exists with auto-capture, but we could change
it so that an explicit use() list disables auto-capture instead:

fn () use ($a) { } // Would capture $a and disable auto-capture
fn () use () { }   // Would capture nothing and disable auto-capture

I like this idea very much. In the RFC two variables are captured
explicitly and one implicitly.
$c = 1;
fn () use ($a, &$b) { return $a + $b + $c; }

I don't see the UC / value for not specifying $c while specifying
$a. Think it's much clearer when capturing variables to implicitly
capture everything or list the ones that should be captured. One
only need to think about which variables are listed, not the ones
that might be implicitly captured.

Of course capturing by reference will always be required to list
and if combined with capturing variables by value, they also needs
to be listed.

The there is this other proposal to enhance traditional anonymous
functions by allowing the syntax use(*), meaning capture everything.
Even if it's outside the scope of this RFC it could be mentioned in
"What about Anonymous Functions?" or "Future scope".

r//Björn L

A little data:

I used Nikita's project analyzer on the top 1000 projects to get a rough sense of how long-closures are used now. All usual caveats apply about such survey data. I was specifically looking at how many use statements a closure typically had, and how many statements it typically had. Mainly, I am interested in how common "really long closures where the developer is likely to lose track of what is and isn't closed over" are.

Total closures: 20052
Total used variables: 11534

Did many of those closures use "pass by reference" in the use clause,
because that's one real differentiator between traditional closures and
short lambdas. There's also the fact that use values are bound at the
point where the closure is defined, not where it's called (if they even
exist at all at that point), although that's probably more difficult to
determine.

New run to check for that:

Total used variables: 11534
ByRef used variables: 1833

So around 13% of used variables are by-ref, and thus would need to be explicitly used even with the new syntax.

There's also the fact that use values are bound at the
point where the closure is defined, not where it's called (if they even
exist at all at that point), although that's probably more difficult to
determine.

I... don't see what relevance that has? The potential for confusion is at the definition point, not call point. If a closure is used inline then those are the same place, but if they're not, it's only the definition point that is relevant at the moment.

--Larry Garfield

On dimanche 12 juin 2022 19:54:06 CEST Mark Baker wrote:

Did many of those closures use "pass by reference" in the use clause,
because that's one real differentiator between traditional closures and
short lambdas. There's also the fact that use values are bound at the
point where the closure is defined, not where it's called (if they even
exist at all at that point), although that's probably more difficult to
determine.

Please note that auto-capture binds variables at function declaration. This is
the case in Arrow Functions, and is inherited by this RFC.

The primary target of this RFC is people writing 2-4 line closures that import 1-2 variables

The first half of that sentence I was expecting - although as I've
already said, I think the chosen syntax suggests strongly that the RFC
is really targeting all closures, not any subset of them.

The second half makes much less sense. If you are only importing 1 or 2
variables, is writing their names really that big a burden?

Several of the conversations I've had on this in the past have been very
explicitly about the burden of large numbers of captures; if that's
really as rare as you suggest, it makes me wonder why we're even bothering.

Regards,

Disclaimer: My own view, I cannot speak for Nuno or Arnaud.

If you're capturing a very large number of variables, then I would view that as a code smell. "Very large" is subjective, of course, and there's some context to it.

The two main use cases I see myself using are

A) 2-3 liners that use 1-3 variables from scope, so it's dead obvious what they are. In this case, the extra use clause doesn't really add much beyond visible noise.

B) An entire method body is a closure that is being returned, or inlined into an inTransction() call or something like that. In this case, basically all method parameters would be captured, and it would be on the very previous line, so no matter how many there are (more than ~5 is probably a problem with the method, not with the closure), they're redundant and don't tell you anything that isn't already self-evident.

So the burden is in having to think about redundant syntax at all, plus having more redundant text that has to be read in the future. Even with use(*) or use(...) or whatever, that's better than the status quo but is still just more boilerplate that would have to be added/removed when switching from a one line short lambda (side note: This is the term I always use; I basically never use "arrow function". I don't know how typical that is) to a 2-line closure when refactoring.

--Larry Garfield

Following your comment, I have clarified a few things in the "Auto-capture
semantics" section. This includes a list of way in which these effects can be
observed. These are really marginal cases that are not relevant for most
programs.

I'm not sure I agree that all of these are marginal, or with the way
you've characterised them...

Note that destructor timing is undefined in PHP, especially when
reference cycles exist.

Outside of reference cycles, which are pretty rare and generally easy to
avoid, PHP's destructors are entirely deterministic. Unlike in fully
garbage-collected languages, you can use a plain object to implement an
"RAII" pattern - e.g. the constructor locks a file and the destructor
unlocks it; or the constructor starts a transaction, and the destructor
rolls it back if not yet committed.

A related case is resource lifetime: file and network handles are
guaranteed to be closed when they go out of scope, and accidentally
taking an extra copy of their "value" can prevent that.

It ends up capturing the same variables that would have been captured
by a manually curated |use| list.

This slightly muddles two different questions:

1. Given a well-written closure, where all variables are either clearly
   local or clearly intended to be captured, does the implementation do a
   good job of distinguishing them?
2. Given a badly-written closure, where variables are accidentally
   ambiguous, what side-effects might the user experience?

The answer to question 1 seems to be yes, the implementation does a good
job, and that's good news, and thank you for working on it.

That is not the same, however, as saying that question 2 is never
relevant. Consider the following, adapted from an example in the RFC:

$filter = fn ($user) {
    if ( $user->id !== -1 ) {
        $guest = $repository->findByUserId($user->id);
    }
    return isset($guest) && in_array($guest->id, $guestsIds);
};

This is not particularly great code, but it works ... unless the parent
scope happens to have a variable named $guest, which will then be bound
to the closure, since there is a path where it is read before being
written. In this case, side effects include:

• The behaviour will change based on the captured value of $guest
• Any resources held by that value will be held until $filter is
  destructed, rather than when $guest is destructed

Whether the risk of these side effects is a big problem is up for
debate, but it's wrong to suggest they don't exist.

Regards,

--
Rowan Tommins
[IMSoP]