Refactoring our IO multiplexing layer

So basically, the first question will be do we need it ?

I would like it, yes. Having a IO abstraction layer internally would
be really nice. I'm not up-to-date on all the PHP extensions that
would facilitate this behavior, but perhaps some improved version of
the libuv extensions would be great. I am not sure on the health of
those extensions (I believe there are at least two of them) but it's
worth investigating at least. Potentially it could save us some work
as well as well as give us a bit more confidence.

Second question will be : do we use, and then link against libevent
(or any other lib that could fit the need) , or do we develop our own
layer, based on the FPM layer which is pretty nicely done (just not
shareable elsewhere at the moment).

I also favor libuv over libevent; to be completely fair I have only
used both of them a little bit but I liked libuv a bit more.

We definitely shouldn't roll our own. If we had corporate funding then
rolling our own would have the advantage of not having to wait for
upstream merges for bug fixes, but I don't think we have the manpower
or interest to build and maintain our own.

Hey :-)
Hello :-),

I spent few hours analyzing PHP's IO multiplexing parts.

IO multiplexing refers to calls to select() / poll() / epoll() or kqueue().

Simple notice : everything is sparse everywhere, we don't have an
IOmux layer designed.

I plan to write an RFC on the subject.
The goal is to centralize IO mux calls in a layer to be used
everywhere. My analyze leads to those points :

• FPM has something, but not designed to be shared
• User streams use hardcoded select() , with an ugly emulation for poll()
• We have nothing about epoll() or kqueue(), except in FPM
• curl_multi_select() uses hardcoded select()
• php_cli_server uses hardcoded select()
• mysqlnd uses hardcoded select() for its poll() mecanisms

Python already have a layer, at
http://hg.python.org/cpython/file/abcf17bc5eae/Modules/selectmodule.c

Apache's APR has a nice layer with adapters, at
https://github.com/apache/apr/blob/trunk/poll/unix/pollset.c

Still, there exist the excellent libevent http://libevent.org/
What about depending on libuv also https://github.com/joyent/libuv? It
would offer nice major features.
(Another related link: https://github.com/chobie/php-uv)

So basically, the first question will be do we need it ?
I would say yes. As we are talking about PHP6 and modernising our API,
it is time I think to implement IO mux correctly and don't use the old
select() everywhere where there is a better implementation (epoll and
kqueue are both faster, and they dont suffer from any FD_SETSIZE
limit)
Also, we didnt talk about things as websockets etc... yet, for PHP6 ,
but it is sure that nowadays 2014, IO mux is a must-have.

Second question will be : do we use, and then link against libevent
(or any other lib that could fit the need) , or do we develop our own
layer, based on the FPM layer which is pretty nicely done (just not
shareable elsewhere at the moment).

And then, will people be interested by the RFC ?
I do.

Cheers.

--
Ivan Enderlin
Developer of Hoa
http://hoa-project.net/

PhD. student at DISC/Femto-ST (Vesontio) and INRIA (Cassis)
http://disc.univ-fcomte.fr/ and http://www.inria.fr/

Member of HTML and WebApps Working Group of W3C
http://w3.org/

Hi!

Second question will be : do we use, and then link against libevent
(or any other lib that could fit the need) , or do we develop our own
layer, based on the FPM layer which is pretty nicely done (just not
shareable elsewhere at the moment).

And then, will people be interested by the RFC ?

This would be pretty interesting. I wonder though how libevent behaves
on systems like Windows or more exotic Unix variants PHP is know to run
on. In any case, basing it on some mature library would definitely make
sense.

Thanks,

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

CCing internals

On Tue, Jun 17, 2014 at 1:07 PM, Martin Pelikan
martin.pelikan@gmail.com wrote:

Still, there exist the excellent libevent http://libevent.org/

Speaking as only an observer to PHP development, I have been reinventing
this particular wheel in my project (SSH client to thousands of machines
to replace the painfully slow python's paramiko, as seen in my thesis at
http://qviz.storkhole.cz/).

My ~3 months of experience:

The questions you want to ask are:

• "what's wrong with libevent"
• "how would you do it differently"
• "why does PHP need it differently"

I strongly suggest not creating yet another way of doing I/O multiplexing.
nginx looks pretty neat and shows just how much work is needed to get things
right on all platforms. PHP developers have better things to do than this.

http://trac.nginx.org/nginx/browser/nginx/src#event/modules

If someone would say to me "learn and then read libevent" right away, my
backend would be finished now. Seriously. Go and have a look, it is
likely to save a lot of your time.

Yes, libevent seems to be the API we need.
Regading Nginx, I'm not sure we need file AIO, for PHP, network AIO
could be enough to implement. Still, our FPM IO multiplexing layer is
stable now.

File AIO would still be nice to have though, it's common to pipe data
between a network fd and a file fd - when the file system blocks, that
can be a sizeable performance hit.

I may get shot for suggesting this, but libuv provides all the things
libevent provides, plus filesystem support (also, the built-in async
DNS resolution is a nice bonus).

The problem of binding a lib is that you add one more dependency, you
have to check for licence compatibility, and check for the dependency
when building. Not talking about bugs in the lib impacting our
codebase. This is a tradeoff.

While this is very valid, in this particular case I believe it would
be worth it, due to the insane complexity of reinventing this
particular wheel.

I believe Daniel Lowrey and Joe Watkins were bouncing some ideas
around on exactly this subject off-list a few weeks ago, I'm not sure
whether they have anything tangible yet.

Thanks, Chris

I strongly suggest not creating yet another way of doing I/O multiplexing.
nginx looks pretty neat and shows just how much work is needed to get things
right on all platforms. PHP developers have better things to do than this.

Hey,
I like the idea and don't want to rain on your parade, but please do an
extensive cross-OS check beforehand, I do remember some problems with
this kind of stuff on NetBSD/OpenBSD for example (not in PHP,
elsewhere), and PHP supports a lot of OSs I haven't used in 10 years or
not at all. AFAIK both libevent and libuv have quite a good support, but
just after a quick look into the manual (Do we even have a definitive
list of supported OS there?) - Solaris, Illumos, HP-UX... no clue here.

Greetings,
Florian