[RFC Discuss] Generator Delegation

Rowan Collins wrote on 02/03/2015 14:44:

Could you explain a bit more about how the generator return
functionality is necessary for this? It seems to me like it's still just
a "nice to have", and that the main functionality - delegating from one
generator to another - is completely separate. Or is there some common
use case that only makes sense if you combine the two features?

This is a good question and it's 100% justified because I haven't fully
addressed it in the body of the RFC. You'll notice there are a couple of
@TODO notes still outstanding. So ... I'll try to flesh some of that out
now and then copy/paste some of what follows over into the RFC body :)

Hi Daniel,

Thanks for taking the time to clarify. I think I almost understand now...

There are two main reasons why return value access is crucial for
generator delegation:

1. Refactoring/readability
2. Generators as lightweight "threads"

The primary impetus for return values in generator delegations is the same
reason why we need return values in class methods: to break up
functionality into smaller conceptual units. Imagine a class method where
no return value was possible. Yes, we could store the result in an
instance property and retrieve it from the context of the calling code
after execution has completed but this would be somewhat nonsensical (not
to mention this is exactly why return values exist).

Now also recognize that in functional contexts we don't have this extra
state from an object instance. If we don't actually return the result of
the computation there's no other way to access it.

Well, in a normal generator context, we have two-way communication using
"yield". The case for a distinct final value is one of convenience, until
we get to delegation. With delegation, we can't get data from the
subgenerator's "yield", because it will be automatically delegated. Is that
more or less right?

Now, one might argue that we shouldn't need to expose
Generator::getReturn() as we can simply return the results directly to
the delegation expression in the parent generator. The problem with this
argument is that without the ability to access the return value from
individual subgenerators outside the context of the delegating generator
we're left unable to do things like write unit tests for individual
generator functions to verify that they work as expected.

Is it unfortunate that we have to use a stateful "object" with actual
public methods to represent pausable functions in userland? Maybe. But I
don't (personally) see this as a sound reason to argue that generator
functions shouldn't behave like traditional serial functions. Beyond this:
other languages (python, javascript) also expose the ability to return
values from generator functions as mentioned.

The reason I ask is that "yield from" (or whatever syntax) sounds
useful, but I'm not that keen on allowing "return" in generators,
because I can see it further muddying the water between functions and
generators: when you call the following "function", it doesn't actually
return 1 as it would appear at a glance, because the existence of a
yield statement elsewhere in the body magically makes it return a
Generator object instead:

function foo() {
// potentially many lines of code
yield 1;
// potentially many more lines of code
return 1;
}

This confusion is not the result of a fundamental difference between
functions and generators; a generator is a function is a function is a
function. A generator is simply a function promoted to have traversable
characteristics. It should not be thought of as a traversable object that
happens to be statically instantiated via function call.

It just so happens that to make generators work cleanly in PHP we need to
create and return an iterable "placeholder" Generator object to encapsulate
the necessary state for pausing and resumption. This is unfortunate because
it leads to the kind of confusion above, but it's simply the only way to
expose this functionality. If it weren't a function it wouldn't be a
generator ... it would just be some traversable thing.

This part I don't understand. I can't use a generator where I'd normally
use a function, and I can't use a function where I'd use a generator. The
only way I can reason about them in the same way is to say that the
generator is a function whose return value is an instance of something.
That's why being able to specify "return" feels so wrong to me - the return
value of foo() should be what I get when I execute foo().

The fact that it's iterable aside, I can't see what other form a coroutine
could take. In order to be useful, you need to have a pointer to something
with identity, internal state, and behaviour. That sure sounds like an
object, which makes anything which produces that thing sound very like a
constructor. A function has none of those traits.

I would personally have preferred generators to have used a distinct
keyword rather than just looking like functions, but since that ship has
sailed, making them look even more like functions worries me. Is there
perhaps some other syntax that could be used for this "generator result
value"?

This is the fundamental mental block that most people have when they try
to wrap their heads around cooperative multitasking via coroutines. We need
to understand Generators are not iterators. They are functions with the
added capacity that they can be paused and resumed. The function keyword is
the only thing that makes sense here.

The fundamental difference is that functions are never instantiated - you
don't initialise a copy of strlen() then interrogate it for its value,
there is a single function, and the result of calling it is an integer you
can pass to other functions. Coroutines may have the same role as
functions, as units of executable code, but they have fundamentally
different behaviour, so I don't see how it's helpful to fold them
together.

Would it not be clearer to say this?

coroutine foo($bar) { blah(); blah(); yield; etc(); return $bar; }
$foo = new foo(42);
// $foo now points to an instance of the coroutine

You could even have coroutines with no yield statements that way, which the
current syntax doesn't allow.

However, since the existence of the word "yield" is the only thing that
marks a coroutine now, how about using a variant of that for the final
value, e.g. "yield final $foo"? That would remove the perceived ambiguity
between a value being returned in the normal way, and one accessible only
through Generator syntax.

Regards,

Rowan Collins
[IMSoP]