# Generators in PHP

- Thread: https://externals.io/message/60741
- Original archive: https://news-web.php.net/php.internals/60741
- Started by Nikita Popov on 2012-06-05T17:35:14+00:00
- Messages: 120
- Page 1 of 5: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/60741.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #60741 — Nikita Popov — 2012-06-05T17:35:14+00:00

URL: https://externals.io/message/60741#60741

Hi internals!

In the last few days I've created a proof of concept implementation
for generators in PHP. It's not yet complete, but the basic
functionality is there:
https://github.com/nikic/php-src/tree/addGeneratorsSupport

The implementation is outlined in the RFC-stub here:
https://wiki.php.net/rfc/generators

Before going any further I'd like to get some comments about what you
think of adding generator support to PHP.

If you don't know what generators are you should have a look at the
"Introduction" section in the above RFC or in the Python documentation
at http://wiki.python.org/moin/Generators.

Nikita

---

## #60742 — Kris Craig — 2012-06-05T18:32:31+00:00

URL: https://externals.io/message/60741#60742 · In reply to: #60741

On Tue, Jun 5, 2012 at 10:35 AM, Nikita Popov <nikita.ppv@googlemail.com>wrote:

> Hi internals!
>
> In the last few days I've created a proof of concept implementation
> for generators in PHP. It's not yet complete, but the basic
> functionality is there:
> https://github.com/nikic/php-src/tree/addGeneratorsSupport
>
> The implementation is outlined in the RFC-stub here:
> https://wiki.php.net/rfc/generators
>
> Before going any further I'd like to get some comments about what you
> think of adding generator support to PHP.
>
> If you don't know what generators are you should have a look at the
> "Introduction" section in the above RFC or in the Python documentation
> at http://wiki.python.org/moin/Generators.
>
> Nikita
>
> --
> 
> 
>
>
Some observations and questions:


   1. In the RFC, the top example claims to make use of the `file()`
   function, but in fact does not.  Did you mean `fopen()`?  Or did you mean
   that this to be an example of someone writing their own `file()` function in
   PHP for some reason (the "userland" reference is a bit confusing IMHO given
   the context).
   2. In what way(s) do you believe this approach would differ from inline
   functions and what advantage(s) do you see in those differences?
   3. What release version do you believe should be targetted for this?


--Kris

---

## #60747 — Laruence — 2012-06-06T02:15:14+00:00

URL: https://externals.io/message/60741#60747 · In reply to: #60741

Hi Nikita:

    the most important part to me is how did you implemented `yield`
keyword,   is there a whole patch file I can look into?

    what will happen if you use a `yield` in a normal function?

    actually,  I tried to implemented coroutine, but since I could not
find a way to make zend_execute interruptable, then I didn't make it.

    so, I am really interesting of this tech-specific :)

thanks


> Hi internals!
>
> In the last few days I've created a proof of concept implementation
> for generators in PHP. It's not yet complete, but the basic
> functionality is there:
> https://github.com/nikic/php-src/tree/addGeneratorsSupport
>
> The implementation is outlined in the RFC-stub here:
> https://wiki.php.net/rfc/generators
>
> Before going any further I'd like to get some comments about what you
> think of adding generator support to PHP.
>
> If you don't know what generators are you should have a look at the
> "Introduction" section in the above RFC or in the Python documentation
> at http://wiki.python.org/moin/Generators.
>
> Nikita
>
> --
> 
> 
>



-- 
Laruence  Xinchen Hui
http://www.laruence.com/

---

## #60748 — Laruence — 2012-06-06T02:27:47+00:00

URL: https://externals.io/message/60741#60748 · In reply to: #60747

> Hi Nikita:
>
>    the most important part to me is how did you implemented `yield`
> keyword,   is there a whole patch file I can look into?
Nervermind,  I will check the branch out later

thanks
>
>    what will happen if you use a `yield` in a normal function?
>
>    actually,  I tried to implemented coroutine, but since I could not
> find a way to make zend_execute interruptable, then I didn't make it.
>
>    so, I am really interesting of this tech-specific :)
>
> thanks
>
> 
> > […]
>
>
>
> --
> Laruence  Xinchen Hui
> http://www.laruence.com/



-- 
Laruence  Xinchen Hui
http://www.laruence.com/

---

## #60749 — Laruence — 2012-06-06T02:42:44+00:00

URL: https://externals.io/message/60741#60749 · In reply to: #60748

> 
> > […]
> Nervermind,  I will check the branch out later
>
> thanks

After a quick look,  I think the main idea should goes to two parts:

1. implement yield (Zend)
2. implement spl_generators but not generator class (Spl)

then we can implement spl_coroutine later base on this.  what do you think?

thanks
> > […]
>
>
>
> --
> Laruence  Xinchen Hui
> http://www.laruence.com/



-- 
Laruence  Xinchen Hui
http://www.laruence.com/

---

## #60752 — Rasmus Lerdorf — 2012-06-06T08:11:57+00:00

URL: https://externals.io/message/60741#60752 · In reply to: #60749

> 
> > […]
> 
> After a quick look,  I think the main idea should goes to two parts:
> 
> 1. implement yield (Zend)
> 2. implement spl_generators but not generator class (Spl)

yield is interesting. It is one of the main features Facebook added, so
it might be worthwhile poking someone there for their implementation
details in order to at least be somewhat compatible.

-Rasmus

---

## #60753 — Gustavo Lopes — 2012-06-06T08:55:28+00:00

URL: https://externals.io/message/60741#60753 · In reply to: #60741

>
> Before going any further I'd like to get some comments about what you
> think of adding generator support to PHP.
>

I approve.

A few comments on the current RFC:

* The RFC is too vague.
* You're violating the contract of Iterator left and right. `current()` 
and `key()` should return false when !valid(). `next()` is underspecified. 
valid() refers to a section that doesn't exist. And if you cannot 
implement `rewind()`, doing nothing is not an option. In fact, if you 
can't implement Iterator in full, you should implement Traversable 
instead. RewindableGenerator could perhaps implement Iterator though -- 
but I find the nature of RewindableGenerator  very strange. Whether an 
iterator is rewindable or not seems to be related to the nature of the 
generator (e.g., is it reading packets from the network). It's not 
something you can wrap a non-rewindable generator and expect it to work. 
So it's a sort of unsafe operation, like a cast in C.
* Overall, the RFC is very underspecified. We never have a formal 
description of what a generator is and exact semantics of it. There is 
no reference to exceptions. What to do if the generator returns 
function. If the generator can be a function method.
* There are missing sections

I suppose this is a work in progress and that you just want to gauge 
the general interest, and I hope you take these considerations into 
account.

-- 
Gustavo Lopes

---

## #60754 — Pierre Joye — 2012-06-06T09:39:47+00:00

URL: https://externals.io/message/60741#60754 · In reply to: #60752

hi Rasmus, Sara,

Adding Sara to the loop as she is around and may miss that thread. We
also tried to convince her to contribute that back to core :)


> 
> > […]
>
> yield is interesting. It is one of the main features Facebook added, so
> it might be worthwhile poking someone there for their implementation
> details in order to at least be somewhat compatible.
>
> -Rasmus
>
>
> --
> 
> 
>



-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #60757 — Ivan Enderlin @ Hoa — 2012-06-06T10:20:57+00:00

URL: https://externals.io/message/60741#60757 · In reply to: #60741

> Hi internals!
Hi Nikita,

> Before going any further I'd like to get some comments about what you
> think of adding generator support to PHP.
I have always hoped to see the “yield” keywork introduced in PHP. I even 
suggested that in this mailing-list at least one time but I'm very glad 
to see some patches coming out.

In addition to Gustavo's remarks, I wonder how the GC would collect a 
Generator object that is not use anymore (especially with a referenced 
yield). Does it fit to the current CG strategy or do we need an extra 
strategy? I would notice that the “Yield by reference” section does not 
exist but you have targeted it).
Moreover, I wonder how a “recursive yield” would act (something like 
“public function *f ( … ) { … yield $this->f(…); … }”). It is possible? 
Is it anticipated?

Thanks for you work.

Cheers.

-- 
Ivan Enderlin
Developer of Hoa
http://hoa.42/ or http://hoa-project.net/

PhD. student at DISC/Femto-ST (Vesontio) and INRIA (Cassis)
http://disc.univ-fcomte.fr/ and http://www.inria.fr/

Member of HTML and WebApps Working Group of W3C
http://w3.org/

---

## #60758 — Lazare Inepologlou — 2012-06-06T10:39:12+00:00

URL: https://externals.io/message/60741#60758 · In reply to: #60757

Hello Ivan,


Moreover, I wonder how a “recursive yield” would act (something like
> “public function *f ( … ) { … yield $this->f(…); … }”). It is possible? Is
> it anticipated?
>


According to the RFC, your syntax will yield an entire generator and not
its intividual values. Please consider this example:

function * f() { ... }
function * g() {
  yield f();
}
function * h() {
  foreach( f() as $key => $value )
    yield $key => value;
}

The pattern in function h() is quite common, and it is what you ask for. It
would be nice to have some syntactical sugar for it, maybe something like
that:

public function * h() {
  yield foreach f();
}



Good work, I am looking forward to having generators in php.

Lazare INEPOLOGLOU
Ingénieur Logiciel


2012/6/6 Ivan Enderlin @ Hoa <ivan.enderlin@hoa-project.net>

> 
>
> > […]
> Hi Nikita,
>
>  Before going any further I'd like to get some comments about what you
> > […]
> I have always hoped to see the “yield” keywork introduced in PHP. I even
> suggested that in this mailing-list at least one time but I'm very glad to
> see some patches coming out.
>
> In addition to Gustavo's remarks, I wonder how the GC would collect a
> Generator object that is not use anymore (especially with a referenced
> yield). Does it fit to the current CG strategy or do we need an extra
> strategy? I would notice that the “Yield by reference” section does not
> exist but you have targeted it).
> Moreover, I wonder how a “recursive yield” would act (something like
> “public function *f ( … ) { … yield $this->f(…); … }”). It is possible? Is
> it anticipated?
>
> Thanks for you work.
>
> Cheers.
>
> --
> Ivan Enderlin
> Developer of Hoa
> http://hoa.42/ or http://hoa-project.net/
>
> PhD. student at DISC/Femto-ST (Vesontio) and INRIA (Cassis)
> http://disc.univ-fcomte.fr/ and http://www.inria.fr/
>
> Member of HTML and WebApps Working Group of W3C
> http://w3.org/

---

## #60762 — Nikita Popov — 2012-06-06T17:09:14+00:00

URL: https://externals.io/message/60741#60762 · In reply to: #60742

> Some observations and questions:
>
> In the RFC, the top example claims to make use of the `file()` function, but
> in fact does not.  Did you mean `fopen()`?  Or did you mean that this to be an
> example of someone writing their own `file()` function in PHP for some reason
> (the "userland" reference is a bit confusing IMHO given the context).
It's an implementation of the `file()` function in userland code (as
opposed to the internal `file()` implementation). `file()` returns all
lines from a file as an array.

> In what way(s) do you believe this approach would differ from inline
> functions and what advantage(s) do you see in those differences?
I'm not sure what you mean by inline functions. PHP doesn't do
function inlining (and it doesn't seem related to this). Or do you
mean closures? Again, I'm not sure how closures are related to this.

> What release version do you believe should be targetted for this?
The next major version, i.e. PHP 5.5.

Nikita

---

## #60763 — Nikita Popov — 2012-06-06T17:54:36+00:00

URL: https://externals.io/message/60741#60763

> This is really awesome. Some remarks:
>
> 1. The * looks a bit hidden. How about reusing the yield keyword? "function
> yield getLines()"
I mainly chose the * modifier because this is how JavaScript
(ECMAScript Harmony) does it. It also seemed to convey the idea of
returning multiple values well (Cleene star). But I wouldn't have much
of a problem using some other keyword. As already written in the RFC
"generator" is problematic as it is not an uncommon word. I wouldn't
like reusing "yield" as it doesn't make much sense semantically there
(if you read "function yield xyz()" out). What seems plausible is
using a separate keyword "yielding", so you have a "yielding function
xyz()".

> 2. Any comments about iterator vs closure?
>
> function getLinesFromFile($fileName, Closure $yield) {
> //..
>
>     while (false !== $line = fgets($fileHandle)) {
>         $yield($line);
>     }
> // ..
> }
>

Yes, using callbacks is indeed a common approach to work around the
lack of generators. The problems with the callback approach (and
several other alternatives) is outlined very well in the first few
paragraphs of PEP 255 (http://www.python.org/dev/peps/pep-0255/).
Basically the issue is that without using generators either the
producer (here getLinesFromFile) or the consumer (here the main code)
has to manually keep track of it's current state. Depending on the
complexity of the producer/consumer this can become very hard, which
can be especially well seen in the tokenizer/parser examples given in
the aforementioned PEP. When using generators on the other hand the
state is implicitly maintained by the VM, so the programmer doesn't
have to worry about that anymore.

So basically, yes, callbacks can be used for the simpler applications
of generators, but with more complex problems they can quickly become
very messy.

Nikita

---

## #60764 — Nikita Popov — 2012-06-06T19:23:27+00:00

URL: https://externals.io/message/60741#60764 · In reply to: #60747

> Hi Nikita:
>
>    the most important part to me is how did you implemented `yield`
> keyword,   is there a whole patch file I can look into?
>
>    what will happen if you use a `yield` in a normal function?
>
>    actually,  I tried to implemented coroutine, but since I could not
> find a way to make zend_execute interruptable, then I didn't make it.
Yes I also faced that problem. The current execute() function accepts
an op_array and initializes the execution context from that
(execute_data + CVs + Ts).

So I added a new function execute_ex() which takes that execution
context directly. The execute() function now works by initializing the
execute_data using a new zend_create_execute_data_from_op_array()
function and then calls execute_ex() with it.

The most relevant commit for this change is
https://github.com/nikic/php-src/commit/f627be52540738e124da7cb1566d7f60a2b6a48b.

Nikita

---

## #60765 — Kris Craig — 2012-06-06T19:33:03+00:00

URL: https://externals.io/message/60741#60765 · In reply to: #60762

On Wed, Jun 6, 2012 at 10:09 AM, Nikita Popov <nikita.ppv@googlemail.com>wrote:

> 
> > […]
> but
> > […]
> be an
> > […]
> reason
> > […]
> It's an implementation of the `file()` function in userland code (as
> opposed to the internal `file()` implementation). `file()` returns all
> lines from a file as an array.
>
> > […]
> I'm not sure what you mean by inline functions. PHP doesn't do
> function inlining (and it doesn't seem related to this). Or do you
> mean closures? Again, I'm not sure how closures are related to this.
>

Yes sorry, I meant closures.  I have the nasty habit of calling them
"inline functions," forgetting that that term already means something
completely different lol.

I guess the point I was making is that *closures* and generators are very
similar in many ways.  This isn't to say that generators are a bad idea (I
actually think it's an awesome idea), but the question occurred to me as to
whether or not generators will add any functionality that can't already be
accomplished (albeit somewhat more clumsily) via closures.  I tend to think
the answer to that probably lies in yields, but I think it would be
beneficial to address this-- along with some examples to illustrate this
advantage over closures in appropriate use-cases-- in the RFC.  I agree
with Stas that, while it's a great idea, the RFC itself is far from
complete.


>
> > […]
> The next major version, i.e. PHP 5.5.
>

Ok good.  Just wanted to make sure we weren't talking about squeezing this
into 5.4 lol.  Personally, I think RFCs should specify the target PHP
version when applicable.  I know it's not required but I think it'd be a
good idea to get in that habit anyway IMHO.

--Kris

>
> Nikita

---

## #60769 — Nikita Popov — 2012-06-06T23:10:52+00:00

URL: https://externals.io/message/60741#60769 · In reply to: #60753

> 
> > […]
>
> I approve.
>
> A few comments on the current RFC:
>
> * The RFC is too vague.
Yup, it's not yet complete.

> `current()` and `key()` should return false when !valid()
Are you sure about that? The Iterator::current() docs don't specify
anything, but the Iterator::key() docs say that it should return `NULL` 
on failure. Checking on the first spl class that came to my mind
SplFixedArray also returns `NULL` when it is out of elements.

My personal preference would be to throw exceptions if those two
methods are called after the generator was closed (as calling them is
clearly an error, isn't it?), but I see that this would be
inconsistent with the usual iterator behavior.

> `next()` is underspecified.
Not sure what exactly you mean. `next()` really doesn't do much more
than resuming the generator. What should I add additionally about it?

> valid() refers to a section that doesn't exist.
Yes, sorry, I hadn't yet written it. I now added it at
https://wiki.php.net/rfc/generators#closing_a_generator.

> And if you cannot implement `rewind()`, doing nothing is not an option.
I was thinking of the Generator as a NoRewindIterator, which also
simply does nothing when `rewind()` is called on it. This allows you to
start traversing the iterator in one loop and continue traversing it
in another loop:

    function *allNaturalNumbers() {
        for ($i = 0; ; ++$i) {
            yield $n;
        }
    }

    $numbers = allNaturalNumbers();

    echo 'First loop:', "\n";
    foreach ($numbers as $n) {
        echo $n, "\n";
        if ($n == 3) break;
    }

    echo 'Second loop:', "\n";
    foreach ($numbers as $n) {
        echo $n, "\n";
        if ($n == 6) break;
    }

This would output:

    First loop:
    1
    2
    3
    Second loop:
    4
    5
    6

Generators in Python behave the same way. (To be fair though Python
generally has no notion of rewinding an iterator, so all
iterators in Python act like that.)

I don't know whether that behavior is of any use, so I'll gladly
change the behavior to throwing an exception if that's more desirable.

> In fact, if you can't implement Iterator in full, you should implement Traversable instead.
Not sure whether the lack of rewinding behavior really disqualifies it
to use the Iterator interface. I think having the internal iteration
methods exposed is quite handy, especially if you consider their use
as coroutines, where you often want to call the iteration interface
manually and not using a foreach loop.

> RewindableGenerator could perhaps implement Iterator though -- but I find
> the nature of RewindableGenerator  very strange. Whether an iterator is
> rewindable or not seems to be related to the nature of the generator (e.g.,
> is it reading packets from the network). It's not something you can wrap a
> non-rewindable generator and expect it to work. So it's a sort of unsafe
> operation, like a cast in C.
Yes, agree with that.

> * Overall, the RFC is very underspecified. We never have a formal
> description of what a generator is and exact semantics of it. There is no
> reference to exceptions. What to do if the generator returns function. If
> the generator can be a function method.
Yes, the RFC is only a stub, it's not complete yet. But to answer your
questions:
 * Exceptions can be thrown from generators as always.
 * Not sure what you mean by "generator returns function". What would
be that problem with that?
 * Generators can be methods, as shown in the example in the
Introduction section. Closures can also be generators.

> I suppose this is a work in progress and that you just want to gauge the
> general interest, and I hope you take these considerations into account.
Yup, thanks for your comments!

Nikita

---

## #60770 — Anthony Ferrara — 2012-06-06T23:37:42+00:00

URL: https://externals.io/message/60741#60770 · In reply to: #60769

Nikita,

> I don't know whether that behavior is of any use, so I'll gladly
> change the behavior to throwing an exception if that's more desirable.

You can't throw an exception from rewind, since it's called before
foreach().  So it wouldn't be iterable then.

I think the noop on rewind is valid in this context, as long as it's
documented...

Anthony

---

## #60774 — Gustavo Lopes — 2012-06-07T08:18:09+00:00

URL: https://externals.io/message/60741#60774 · In reply to: #60769

> > […]
>
> Are you sure about that? The Iterator::current() docs don't specify
> anything, but the Iterator::key() docs say that it should return `NULL` 
> on failure. Checking on the first spl class that came to my mind
> SplFixedArray also returns `NULL` when it is out of elements.

My bad. I was under the impression the semantics were similar to those 
of `next()`, `key()`, etc. Instead the docs say under `current()` that the 
function can return anything, under key) that it returns `NULL` on failure 
(and issues an E_NOTICE) and for `next()` that "the return value is 
ignored" -- whatever that means; I'll interpret it as saying anything 
can be returned. I'm not sure how correct this documentation is, though.

> > […]
> Not sure what exactly you mean. `next()` really doesn't do much more
> than resuming the generator. What should I add additionally about it?

Sorry for not having been clearer. I mean you say "Resumes the 
generator (unless the generator is already closed)", but you don't 
specify what it returns (though apparently Iterator does not re

>
> > […]
> Yes, sorry, I hadn't yet written it. I now added it at
> https://wiki.php.net/rfc/generators#closing_a_generator.
>
> > […]
> I was thinking of the Generator as a NoRewindIterator, which also
> simply does nothing when `rewind()` is called on it. This allows you to
> start traversing the iterator in one loop and continue traversing it
> in another loop:
>
>     function *allNaturalNumbers() {
>         for ($i = 0; ; ++$i) {
>             yield $n;
>         }
>     }
>
>     $numbers = allNaturalNumbers();
>
>     echo 'First loop:', "\n";
>     foreach ($numbers as $n) {
>         echo $n, "\n";
>         if ($n == 3) break;
>     }
>
>     echo 'Second loop:', "\n";
>     foreach ($numbers as $n) {
>         echo $n, "\n";
>         if ($n == 6) break;
>     }
>
> This would output:
>
>     First loop:
>     1
>     2
>     3
>     Second loop:
>     4
>     5
>     6
>
> Generators in Python behave the same way. (To be fair though Python
> generally has no notion of rewinding an iterator, so all
> iterators in Python act like that.)
>
> I don't know whether that behavior is of any use, so I'll gladly
> change the behavior to throwing an exception if that's more 
> desirable.
>
> > […]
> Not sure whether the lack of rewinding behavior really disqualifies 
> it
> to use the Iterator interface. I think having the internal iteration
> methods exposed is quite handy, especially if you consider their use
> as coroutines, where you often want to call the iteration interface
> manually and not using a foreach loop.
>
> > […]
> Yes, agree with that.
>
> > […]
> Yes, the RFC is only a stub, it's not complete yet. But to answer 
> your
> questions:
>  * Exceptions can be thrown from generators as always.
>  * Not sure what you mean by "generator returns function". What would
> be that problem with that?
>  * Generators can be methods, as shown in the example in the
> Introduction section. Closures can also be generators.
>
> > […]
> Yup, thanks for your comments!
>
> Nikita

-- 
Gustavo Lopes

---

## #60775 — Gustavo Lopes — 2012-06-07T08:46:16+00:00

URL: https://externals.io/message/60741#60775 · In reply to: #60769

(Sorry, I pressed something that sent the message prematurely)



> > […]
>
> Are you sure about that? The Iterator::current() docs don't specify
> anything, but the Iterator::key() docs say that it should return `NULL` 
> on failure. Checking on the first spl class that came to my mind
> SplFixedArray also returns `NULL` when it is out of elements.

My bad. I was under the impression the semantics were similar to those 
of `next()`, `key()`, etc. Instead the docs say under `current()` that the 
function can return anything, under key) that it returns `NULL` on failure 
(and issues an E_NOTICE) and for `next()` that "the return value is 
ignored" -- whatever that means; I'll interpret it as saying anything 
can be returned. I'm not sure how correct this documentation is, though.

> > […]
> Not sure what exactly you mean. `next()` really doesn't do much more
> than resuming the generator. What should I add additionally about it?

Sorry for not having been clearer. I mean you say "Resumes the 
generator (unless the generator is already closed)", but you don't 
specify what it returns (though apparently Iterator does not require you 
to specify anything specific) and you don't say what happens when 
theiterator is already closed.

> > […]
> Not sure whether the lack of rewinding behavior really disqualifies 
> it
> to use the Iterator interface. I think having the internal iteration
> methods exposed is quite handy, especially if you consider their use
> as coroutines, where you often want to call the iteration interface
> manually and not using a foreach loop.
>

Well, the Iterator::rewind() docs say that `rewind()` rewinds the 
iterator. If this is not possible, then Iterator should not be 
implemented. See 
http://www.parashift.com/c++-faq-lite/proper-inheritance.html#faq-21.8

In the same order, your options are:

1. As changing Iterator is not possible, have a new interface, say, 
BasicIterator from which Iterator will inherit without `rewind()`.
2. Make all the Generators rewindable (possibly by fetching a new 
generator each time).
3. Do not implement Iterator.

(By the way, if you implement Traversable, one you can always use the 
unsafe IteratorIterator to get access to the methods)

>  * Exceptions can be thrown from generators as always.
>  * Not sure what you mean by "generator returns function". What would
> be that problem with that?

I mean how do you deal with "return $foo" in the body of the generator? 
Does it work like a final yield? Is the return value ignored?

>  * Generators can be methods, as shown in the example in the
> Introduction section. Closures can also be generators.

Can other callable classes be generators?

Are you planning on any internal API for functions to implement 
generators (though I really haven't thought how that would work)?

-- 
Gustavo Lopes

---

## #60780 — Nikita Popov — 2012-06-08T11:16:11+00:00

URL: https://externals.io/message/60741#60780 · In reply to: #60757

On Wed, Jun 6, 2012 at 12:20 PM, Ivan Enderlin @ Hoa
<ivan.enderlin@hoa-project.net> wrote:
> In addition to Gustavo's remarks, I wonder how the GC would collect a
> Generator object that is not use anymore (especially with a referenced
> yield). Does it fit to the current CG strategy or do we need an extra
> strategy? I would notice that the “Yield by reference” section does not
> exist but you have targeted it).
I added a new section which answers the question at least partially:
https://wiki.php.net/rfc/generators#closing_a_generator
If you don't need the generator object anymore you can either
explicitly ->close() it or wait until all references to it are removed
(typically when leaving the scope of the calling function). When the
generator is closed it releases all used resources, including the
suspended execution context and the currently yielded value. Whether
or not the value is yielded by reference shouldn't make a difference.
So, yes, the current GC strategy works well for generators too :)

> Moreover, I wonder how a “recursive yield” would act (something like “public
> function *f ( … ) { … yield $this->f(…); … }”). It is possible? Is it
> anticipated?
Your particular code would simply yield a generator object (as that's
what $this->f(…) returns).

If instead you wanted to yield all values from that object you could
wrap it in a foreach loop:

function *f() {
    // ...
    foreach ($this->f() as $value) {
        yield $value;
    }
    // ...
}

What this doesn't yet properly cover is the use of generators as
cofunctions. In this case it is desirable that ->send() calls are also
propagated (same applies to ->close() and the not yet implemented
->throw()).

To cover all those use-cases you'd have to come up with a rather big
and ugly block of code (you can find a Python same implementation in
http://www.python.org/dev/peps/pep-0380/#formal-semantics).

Thus it is clear that another expression is required which allows you
to delegate execution to another generator/cofunction. In Python this
is "yield from", in JavaScript "yield*".

A tree implementation using it could look like this:

class Tree implements IteratorAggregate {
    protected $value, $left, $right;

    public function __construct($value, Tree $left = null, Tree $right = null) {
        $this->value = $value;
        $this->left = $left;
        $this->right = $right;
    }

    function *getIterator() {
        if ($this->left) yield* $this->left;
        yield $this->value;
        if ($this->right) yield* $this->right;
    }
}

As you can see the iterator implementation is dead simple.

(Note though that the yield* expression described above isn't yet
implemented, but I plan to implement it.)

Nikita

---

## #60781 — Nikita Popov — 2012-06-08T11:24:49+00:00

URL: https://externals.io/message/60741#60781 · In reply to: #60775

> I mean how do you deal with "return $foo" in the body of the generator? Does
> it work like a final yield? Is the return value ignored?
Currently there will be a fatal error if return statements (with
values) are used in the generator. In the future I'd like to use the
return value as the result of a yield from / yield* expression, as it
is currently done in Python and JavaScript. This is particularly
useful if you are delegating control to another coroutine. This way
you can for example break down a coroutine-based parser into several
functions (instead of having one big ugly function).

> Are you planning on any internal API for functions to implement generators
> (though I really haven't thought how that would work)?
I don't think that this is possible. Generators require that the
execution context is suspended in some way and we have to that control
only over userland code, not internal C code.

Nikita

---

## #60782 — Gustavo Lopes — 2012-06-08T11:51:14+00:00

URL: https://externals.io/message/60741#60782 · In reply to: #60781

> > […]
> I don't think that this is possible. Generators require that the
> execution context is suspended in some way and we have to that 
> control
> only over userland code, not internal C code.
>

Couldn't we simulate this by saving the state in a heap allocated 
structure (whose exact form would depend on the generator 
implementation). Something like:

struct generator_context {
     zval *(*yield_next)(struct generator_context*);
}

struct spec_generator_context {
     struct generator_context parent;
     int foo;
}

PHP_FUNCTION(get_generator)
{
     struct spec_generator_context *ctx = emalloc(*ctx);
     ctx->parent.yield_next = foo_bar();
     return_value = make_internal_generator(ctx);
}

And you could also change the yield_next pointer in foo_bar() to avoid 
going through goto's or switch statements. Possibly yield_next could be 
take the arg with double indirection and you could also completely 
replace the context.

I understand this has the problem that the internal and userspace 
implementations would be markedly different.

-- 
Gustavo Lopes

---

## #60783 — Ivan Enderlin @ Hoa — 2012-06-08T11:53:11+00:00

URL: https://externals.io/message/60741#60783 · In reply to: #60780

Hi Nikita,


> On Wed, Jun 6, 2012 at 12:20 PM, Ivan Enderlin @ Hoa
> <ivan.enderlin@hoa-project.net> wrote:
> > […]
> I added a new section which answers the question at least partially:
> https://wiki.php.net/rfc/generators#closing_a_generator
Good. Thank you.

> If you don't need the generator object anymore you can either
> explicitly ->close() it or wait until all references to it are removed
> (typically when leaving the scope of the calling function). When the
> generator is closed it releases all used resources, including the
> suspended execution context and the currently yielded value. Whether
> or not the value is yielded by reference shouldn't make a difference.
> So, yes, the current GC strategy works well for generators too :)
Ok. A very naive question: would it be interesting to make the 
difference between a “yield” and a “weak referenced yield values” (same 
concept that https://wiki.php.net/rfc/weakreferences which is under 
voting phase as the RFC said).

Thank you for your clarifications.

-- 
Ivan Enderlin
Developer of Hoa
http://hoa.42/ or http://hoa-project.net/

PhD. student at DISC/Femto-ST (Vesontio) and INRIA (Cassis)
http://disc.univ-fcomte.fr/ and http://www.inria.fr/

Member of HTML and WebApps Working Group of W3C
http://w3.org/

---

## #60797 — Tom Boutell — 2012-06-11T21:12:03+00:00

URL: https://externals.io/message/60741#60797

Can you really use setjmp and longjmp in that way safely? I thought it
was only safe to longjmp "back," not "forward" - you can use them to
fake exception support but that's it because you'll smash the stack
otherwise. Something like that...

OK, I'm thinking of this:

"Similarly, C99 does not require that longjmp preserve the current
stack frame. This means that jumping into a function which was exited
via a call to longjmp is undefined.[6] However, most implementations
of longjmp leave the stack frame intact, allowing setjmp and longjmp
to be used to jump back-and-forth between two or more functions—a
feature exploited for multitasking."

It does not sound like something that can be done in portable C.


> Hi,
>
> it'd be really cool if the implementation would do the "real" context
> switch in c, which would be the true basis for spl_coroutine. The current
> implementation seems not to do that.
>
> Context switch in c would be comparable with threads. In fact coroutines
> would be a comensation for lacking threads functionality in php, as they
> are already implemented on most popular platforms. However, it would
> require some platform specific libs (fibers on windows, setjmp and longjmp
> on unix(like)).
>
> Coroutines in c are per se thread safe, so implementing them once would
> not hurt both ts and nts. Additionally, in some cases coroutines can even
> have advantages over the "usual" preemptive threads.
>
> Regards
>
> Anatoliy
>
> Am Mi, 6.06.2012, 04:42 schrieb Laruence:
> > […]
>
>
>
> --
> 
> 
>



-- 
Tom Boutell
P'unk Avenue
215 755 1330
punkave.com
window.punkave.com

---

## #60799 — Jevon Wright — 2012-06-12T03:32:33+00:00

URL: https://externals.io/message/60741#60799 · In reply to: #60741

I don't understand why you need to introduce two new keywords into the
language - * and yield. Could you not solve it like follows?

// Generator implements Iterable
class AllEvenNumbers extends Generator {
  private $i;

  public __construct() { $this->i = 0; }

  function generate() {
    return ($this->i++) * 2;
  }
}

That way I think you can persist state (as part of the class instance)
and implement rewind (per whichever way you do it - either caching the
output, or serialising the class instance) without having to introduce
yet more keywords.

If this is instead a discussion to specifically implement yield,
rather than simplifying the implementation of rewindable Iterators, it
might be more useful to frame the discussion in that way.

Jevon


> Hi internals!
>
> In the last few days I've created a proof of concept implementation
> for generators in PHP. It's not yet complete, but the basic
> functionality is there:
> https://github.com/nikic/php-src/tree/addGeneratorsSupport
>
> The implementation is outlined in the RFC-stub here:
> https://wiki.php.net/rfc/generators
>
> Before going any further I'd like to get some comments about what you
> think of adding generator support to PHP.
>
> If you don't know what generators are you should have a look at the
> "Introduction" section in the above RFC or in the Python documentation
> at http://wiki.python.org/moin/Generators.
>
> Nikita

---

## #60807 — keisial@gmail.com — 2012-06-12T21:07:32+00:00

URL: https://externals.io/message/60741#60807 · In reply to: #60797

> Can you really use setjmp and longjmp in that way safely? I thought it
> was only safe to longjmp "back," not "forward" - you can use them to
> fake exception support but that's it because you'll smash the stack
> otherwise. Something like that...
My first reaction was also "How do you return to the mid-running function?"

However, given that the running function is in PHP-land, I think you could
(in non-zts, it's direct in zts), save EG() contents and replace with
new values,
and then continue the execution.
As it's treating "threads" non-preemtively, that should work.
The C code would also view a different thread, but because it is viewing
different globals (some extensions might need changes). Not really
involving
setjmp() / longjmp().

You could make userland context switches (adding arch-specific code) for
the C code by switching the stack under your ESP, but it's highly likely to
produce some obscure bugs by missing to keep a register...

---

Next page: https://externals.io/message/60741.md?page=2