[RFC] Generators

Do you have a good example usage other than a file? I don't find fopen/fgets/fclose all that complicated. What are the other valid use cases for such a thing?

One fabulous use case is creating an iterator for a Binary Search
Tree. A post order done without generators looks like:
https://github.com/morrisonlevi/PHP-Datastructures/blob/master/src/Spl/PostOrderIterator.php.
An iterator using a generator looks something like:

public function getIterator() {
if ($this->left) yield* $this->left;
yield $this->value;
if ($this->right) yield* $this->right;
}

This is 5 lines. The fully commented version of the post-order
iterator previously mentioned is 106 lines of code and is considerably
harder to understand.

--

Each Iterator implementation in PHP is longer and require a class. The yield keyword can reduce the code and make things simpler. If my vote count, though I can't believe it is, I'd vote for it - clean syntax, make things simpler to develop after you understand the concept (for those who against adding because of the learning issue and beginners - we'll have beginners anytime that'll mess up with the concept, so I don't think that it should restrict us. Put that aside, we should make the documentation clear and easy as possible in order to help beginners understand the concept).

Do you have a good example usage other than a file? I don't find fopen/fgets/fclose all that complicated. What are the other valid use cases for such a thing?

One fabulous use case is creating an iterator for a Binary Search
Tree. A post order done without generators looks like:
https://github.com/morrisonlevi/PHP-Datastructures/blob/master/src/Spl/PostOrderIterator.php.
An iterator using a generator looks something like:

 public function getIterator() {
     if ($this->left) yield* $this->left;
     yield $this->value;
     if ($this->right) yield* $this->right;
 }

This is 5 lines. The fully commented version of the post-order
iterator previously mentioned is 106 lines of code and is considerably
harder to understand.

Well, it's 52 lines, not 106. The 5 lines above are not commented nor
are they spaced at all like the ones in the class. In the above example,
what sets $this->right? or $this->left? There has to be more calling
code around this. I don't consider this a very good example.

Brian.
brianlmoon@php.net

2012/8/13 Anthony Ferrara ircmaxell@gmail.com:

It's absolutely something that takes getting used to. But it's also quite
intuitive once you think about it. You're not "returning" back to the
parent scope (exiting your scope), you're "yielding" a value to the parent
scope, expecting to continue on later (think of it as a stop light where
you let the other code run for a while until you go to the next one).

I think for people, which are - like me - more used with databases,
the cursor-concept is a good comparison.

Perhaps this could help a little bit to explain it...

--
Alex Aulbach

2012/8/13 Nikita Popov nikita.ppv@gmail.com:

• After first yield: Recoverable fatal error

So this would allow you to call ->rewind() after creating the
generator, but will throw an error if you try to do so later.

Hm. I think this looks like a workaround over conceptual problems.
The (in my eyes) "right" behavior would be to call a new instance of
the generator if you rewind and destroy the current (= replace current
generator with a new).

I would prefer to say "this is not an real iterator"; if you want to
make it an real iterator, then implement it yourself.

Otherwise it's like "Ok, we have here a new type of function which
looks enough like a function to be a function, which implements an
iterator which looks enough like an iterator to be an iterator.". :)

I also recommend to say, that generators aren't a good idea if used on
resources, which can't be shared. For example: the file-read-example
doesn't lock. To implement locking new functions or a new class is
needed - hiding complexity.

Generators are (in my opinion) really nice for things which are
already in the memory. Not more or less.

--
Alex Aulbach

If I may chime in, not all iterators support rewind, at least not as outlined.

There exists also the NoRewindIterator which you can just use to decorate an iterator out of a generator to not make it do a fatal.

However as far as a generator is concerned, it probably should just behave like NoRewindIterator already so not give any errors on rewind. Because you can not rewind what you can not rewind. No need to error, no need to do something. The function is just there because of the interface, but it must not do anything.

If different behavior is needed it could be decorated, but right now I can not imagine why rewind would be called (apart from using in foreach() for the first time which implies a rewind and that one is already like I suggest.)

My 2 cents.

-- hakre

----- Ursprüngliche Message -----

Von: Nikita Popov nikita.ppv@gmail.com
An: Brian Moon brian@moonspot.net
CC: PHP internals internals@lists.php.net
Gesendet: 18:09 Montag, 13.August 2012
Betreff: Re: [PHP-DEV] [RFC] Generators

Also, not allowing rewinding is unintuitive for something that is an
iterator in PHP. If I can foreach() it and I can call next() on it, I
expect
to be able to reset() it as well. IMO, you would need to issue a FATAL PHP
error if that simply is not allowed. Or you have to have a second syntax
for
what to do in that case. At that point, you are implementing Iterator.

Currently I'm planning to implement the following behavior for rewind():

• If before first yield: Resume to first yield (this priming behavior
  is common to all the Iterator methods)
• At first yield: No-op
• After first yield: Recoverable fatal error

So this would allow you to call ->rewind() after creating the
generator, but will throw an error if you try to do so later.

Another thing that I'd like to do is drop the ->close() method. It
doesn't really make sense to explicitly close generators.

Any objects to those two things?

Nikita

Also, not allowing rewinding is unintuitive for something that is an
iterator in PHP. If I can foreach() it and I can call next() on it, I expect
to be able to reset() it as well. IMO, you would need to issue a FATAL PHP
error if that simply is not allowed. Or you have to have a second syntax for
what to do in that case. At that point, you are implementing Iterator.
Currently I'm planning to implement the following behavior for rewind():

• If before first yield: Resume to first yield (this priming behavior
  is common to all the Iterator methods)
• At first yield: No-op
• After first yield: Recoverable fatal error

So this would allow you to call ->rewind() after creating the
generator, but will throw an error if you try to do so later.

My perspective is that generators are intended to generate and yield a
result, not so much to iterate over a known set of results. Thus, the
results not really being known, there is nothing to rewind to. Rewinding
would also presumably require more state control, or keeping track of
previous results, and defeat the memory advantages of generators.
Perhaps this is erroneous?

That said, rewind() should behave consistently. I don't feel it makes
sense to have rewind() succeed at one point, and fail at another. It
would only cause confusion when not familiar with the behavior. Either
allow it, or don't. Not both.

Thanks,
Aaron Holmes

Since this is yet another area where 'one does not have to use it if one
does not want to' ... FLAGGING to the other users that a function is a
'special one' rather than just a normal function with a generator function
seems to me just a necessity? HAVING to work through a functions code
simply to see if it contains a 'yeild' so as to understand that it's not
going to give a normal return seems insane? Alright the function can be
properly documented with a docblock, but if it isn't ...

I don't see how that is any different from having to look through a
function to see if it is a void function or if it returns something, and
if so, what it returns. Or checking to see if it outputs something.

PHP is very dynamic. A function can be both void and non-void at the
same time. It can return mixed types and do pretty much anything you can
imagine. I don't really see what the generator keyword achieves. What
happens if I have a function that I mark as a generator but then I don't
put a yield in it? Is that a compile-level error? That might work, but
then what if I put a yield in it but the logic of the function never
gets to it:

if(cond) return [1,2,3];
else if(other_cond) yield $val;
else return;

There is no way to know at compile-time whether the yield will ever be
reached.

To me it seems odd to have a generator keyword that cannot be enforced
and doesn't mean anything other than stating a developer's intent.
Whether the developer actually carried through and implemented the
intended generator can only be verified by reading the code. This smells
much more like a source comment to me.

If we were going to go with a generator keyword, then I think the
feature needs to be completely re-thought. I would suggest dropping the
yield keyword entirely and having return be equivalent to a yield if the
function is marked as a generator. Then the keyword actually does something.

-Rasmus

Since this is yet another area where 'one does not have to use it if one
does not want to' ... FLAGGING to the other users that a function is a
'special one' rather than just a normal function with a generator function
seems to me just a necessity? HAVING to work through a functions code
simply to see if it contains a 'yeild' so as to understand that it's not
going to give a normal return seems insane? Alright the function can be
properly documented with a docblock, but if it isn't ...

I don't understand this argument. Generator functions are transparent
to the user. You use a generator function just like you would use a
function that returns an array. From a user point of view it does not
matter whether getLinesFromFile() is just a function returning an
array, or whether it is a generator (or maybe even returns a
hand-implemented iterator). It's just all the same. The fact that the
function uses yield internally is just an implementation detail, not
something that has to be part of the public API. Actually you can swap
between an array and generator implementation just by replacing one
line in the function body (yield <=> $array[])...

generator function getLinesFromFile($fileName) {
if (!$fileHandle = fopen($fileName, 'r')) {
return;
}

There is an existing generator implementation in HipHop PHP, which

uses automatic-detection. Using the asterix modifier would break
compatibility.

This should not be a concern, sure, it's annoying for the hiphop
developers but they chose to copy and then chance the PHP language for
their own effect.

yield: Yields the value null with an auto-incrementing integer key.
What is the usecase for this?
I can see some interesting porting problems with this ... one of the stock
changes when moving from a MySQL only setup to support other databases is
to remove the auto-increment magic, and replace it with proper sequence
code prior to inserting. I can see translating a 'MySQL' geared generator
into a more general one as being impossible if some tricks like this are
used.

Generators are supposed to act very similar to array-returning
functions, so they also have similar key semantics. If you do
$result[] = $foo; that will behave the same as doing yield $foo;. Both
will use an auto-incrementing key.

Nikita

Hi!

I am against this. This is even more magic in PHP. Is it really that
difficult to have to mark the function with a different keyword, such as
"generator":

You have a point here, but "public static final generator function
foo()" sounds a bit long-winded to me... Also, we'd have then to decide
which function can be marked generator and which can't (e.g., interface
probably can't, abstract probably can't, anonymous probably can, etc.)
which adds more complexity.

Also, I think that people that complain about having to scan through
huge functions to see if they're generators or not, forget one thing:
documentation. Yes, there is a way to make the purpose of the function
understandable to a human without having him to do computer's work.
That's documentation. Undocumented code is broken code. Broken code is
not a good example when we're talking about right design.

I would still like to understand what this generator keyword would
actually do. I don't see how it would work. Would a function marked
generator somehow not be allowed to return normally or to finish and not
return anything? How could this be enforced? I am completely against any
keyword that is essentially documentation-only.

-Rasmus