# Refactoring our IO multiplexing layer - Thread: https://externals.io/message/74938 - Original archive: https://news-web.php.net/php.internals/74938 - Started by Julien Pauli on 2014-06-17T10:04:23+00:00 - Messages: 19 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #74938 — Julien Pauli — 2014-06-17T10:04:23+00:00 URL: https://externals.io/message/74938#74938 Hey :-) I spent few hours analyzing PHP's IO multiplexing parts. IO multiplexing refers to calls to select() / poll() / epoll() or kqueue(). Simple notice : everything is sparse everywhere, we don't have an IOmux layer designed. I plan to write an RFC on the subject. The goal is to centralize IO mux calls in a layer to be used everywhere. My analyze leads to those points : - FPM has something, but not designed to be shared - User streams use hardcoded select() , with an ugly emulation for poll() - We have nothing about epoll() or kqueue(), except in FPM - `curl_multi_select()` uses hardcoded select() - php_cli_server uses hardcoded select() - mysqlnd uses hardcoded select() for its poll() mecanisms Python already have a layer, at http://hg.python.org/cpython/file/abcf17bc5eae/Modules/selectmodule.c Apache's APR has a nice layer with adapters, at https://github.com/apache/apr/blob/trunk/poll/unix/pollset.c Still, there exist the excellent libevent http://libevent.org/ So basically, the first question will be do we need it ? I would say yes. As we are talking about PHP6 and modernising our API, it is time I think to implement IO mux correctly and don't use the old select() everywhere where there is a better implementation (epoll and kqueue are both faster, and they dont suffer from any FD_SETSIZE limit) Also, we didnt talk about things as websockets etc... yet, for PHP6 , but it is sure that nowadays 2014, IO mux is a must-have. Second question will be : do we use, and then link against libevent (or any other lib that could fit the need) , or do we develop our own layer, based on the FPM layer which is pretty nicely done (just not shareable elsewhere at the moment). And then, will people be interested by the RFC ? Thx, Julien --- ## #74940 — Julien Pauli — 2014-06-17T11:05:17+00:00 URL: https://externals.io/message/74938#74940 CCing internals On Tue, Jun 17, 2014 at 1:07 PM, Martin Pelikan wrote: > > […] > > Speaking as only an observer to PHP development, I have been reinventing > this particular wheel in my project (SSH client to thousands of machines > to replace the painfully slow python's paramiko, as seen in my thesis at > http://qviz.storkhole.cz/). > > My ~3 months of experience: > > The questions you want to ask are: > - "what's wrong with libevent" > - "how would you do it differently" > - "why does PHP need it differently" > > I strongly suggest not creating yet another way of doing I/O multiplexing. > nginx looks pretty neat and shows just how much work is needed to get things > right on all platforms. PHP developers have better things to do than this. > > http://trac.nginx.org/nginx/browser/nginx/src#event/modules > > If someone would say to me "learn and then read libevent" right away, my > backend would be finished now. Seriously. Go and have a look, it is > likely to save a lot of your time. Yes, libevent seems to be the API we need. Regading Nginx, I'm not sure we need file AIO, for PHP, network AIO could be enough to implement. Still, our FPM IO multiplexing layer is stable now. The problem of binding a lib is that you add one more dependency, you have to check for licence compatibility, and check for the dependency when building. Not talking about bugs in the lib impacting our codebase. This is a tradeoff. Julien --- ## #74941 — Chris Wright — 2014-06-17T11:33:58+00:00 URL: https://externals.io/message/74938#74941 · In reply to: #74940 > CCing internals > > On Tue, Jun 17, 2014 at 1:07 PM, Martin Pelikan > wrote: > > […] > > Yes, libevent seems to be the API we need. > Regading Nginx, I'm not sure we need file AIO, for PHP, network AIO > could be enough to implement. Still, our FPM IO multiplexing layer is > stable now. File AIO would still be nice to have though, it's common to pipe data between a network fd and a file fd - when the file system blocks, that can be a sizeable performance hit. I may get shot for suggesting this, but libuv provides all the things libevent provides, plus filesystem support (also, the built-in async DNS resolution is a nice bonus). > The problem of binding a lib is that you add one more dependency, you > have to check for licence compatibility, and check for the dependency > when building. Not talking about bugs in the lib impacting our > codebase. This is a tradeoff. While this is very valid, in this particular case I believe it would be worth it, due to the insane complexity of reinventing this particular wheel. I believe Daniel Lowrey and Joe Watkins were bouncing some ideas around on exactly this subject off-list a few weeks ago, I'm not sure whether they have anything tangible yet. Thanks, Chris --- ## #74942 — Julien Pauli — 2014-06-17T11:49:05+00:00 URL: https://externals.io/message/74938#74942 · In reply to: #74941 > > > […] > > File AIO would still be nice to have though, it's common to pipe data > between a network fd and a file fd - when the file system blocks, that > can be a sizeable performance hit. Libuv seems nice as well, yes. Not sure about its licencing compatibility with us though. > > I may get shot for suggesting this, but libuv provides all the things > libevent provides, plus filesystem support (also, the built-in async > DNS resolution is a nice bonus). Why not use this part as well in our DNS resolutions, could be nice. > > > […] > > While this is very valid, in this particular case I believe it would > be worth it, due to the insane complexity of reinventing this > particular wheel. +1 ! > > I believe Daniel Lowrey and Joe Watkins were bouncing some ideas > around on exactly this subject off-list a few weeks ago, I'm not sure > whether they have anything tangible yet. So, waiting for them to jump into the topic :-) Julien --- ## #74943 — Pierre Joye — 2014-06-17T12:01:42+00:00 URL: https://externals.io/message/74938#74943 · In reply to: #74942 hi, > So, waiting for them to jump into the topic :-) Not directly related, but highly recommend read about async, crossplatforms: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores -- Pierre @pierrejoye | http://www.libgd.org --- ## #74945 — Julien Pauli — 2014-06-17T12:35:48+00:00 URL: https://externals.io/message/74938#74945 CCing internals On Tue, Jun 17, 2014 at 2:37 PM, Martin Pelikan wrote: > > […] > > All these are very good with libevent compared to other libraries. > I try to avoid dependencies a lot (more than most people do) and I wrote > that e-mail because in this case that was the wrong thing to do. > > Doing it once is easy. Doing it for many systems quickly becomes a pain > in the arse, because you're just solving issues. That's not programming. > > But again, I don't know what exactly PHP needs. That is another question. Actually, from a user POV, only `stream_select()` is available. It is a "silly" wrapper over select(), and that's all. It works, but suffers from the same problems as select(). I'm not sure userland needs that perf so that select() becomes the bottleneck. It is known to become notably slow starting from several hundreds of fd. I cant believe one would use `stream_select()` in userland with that quantity of fds , better code in C such parts. But, where performance start becoming critical is into FPM. FPM is a fastcgi handler that needs to juggle with FastCGI clients. Here, the needs are barely the same as a webserver like NginX : fast, performant, the most CPU bound as possible, no blocking waits , etc... FPM developed for that its own layer, which actually (AFAIR) , fits the need. What I think is pitty, is that there is no consistence in PHP. FPM AIO layer is good (but obviously could be improved), but only FPM can use it actually , there is no shared API other parts of PHP could consume. Thus my questions. Julien --- ## #74950 — Arvids Godjuks — 2014-06-17T14:21:37+00:00 URL: https://externals.io/message/74938#74950 · In reply to: #74948 I think we do need this and RFC will is needed. And Daniel Lowrey just sold the libuv as the go to library with his message I think :) --- ## #74951 — Levi Morrison — 2014-06-17T14:28:52+00:00 URL: https://externals.io/message/74938#74951 · In reply to: #74938 > So basically, the first question will be do we need it ? I would like it, yes. Having a IO abstraction layer internally would be really nice. I'm not up-to-date on all the PHP extensions that would facilitate this behavior, but perhaps some improved version of the libuv extensions would be great. I am not sure on the health of those extensions (I believe there are at least two of them) but it's worth investigating at least. Potentially it could save us some work as well as well as give us a bit more confidence. > Second question will be : do we use, and then link against libevent > (or any other lib that could fit the need) , or do we develop our own > layer, based on the FPM layer which is pretty nicely done (just not > shareable elsewhere at the moment). I also favor libuv over libevent; to be completely fair I have only used both of them a little bit but I liked libuv a bit more. We definitely shouldn't roll our own. If we had corporate funding then rolling our own would have the advantage of not having to wait for upstream merges for bug fixes, but I don't think we have the manpower or interest to build and maintain our own. --- ## #74952 — Julien Pauli — 2014-06-17T14:57:46+00:00 URL: https://externals.io/message/74938#74952 · In reply to: #74951 > > > […] > > I would like it, yes. Having a IO abstraction layer internally would > be really nice. I'm not up-to-date on all the PHP extensions that > would facilitate this behavior, but perhaps some improved version of > the libuv extensions would be great. I am not sure on the health of > those extensions (I believe there are at least two of them) but it's > worth investigating at least. Potentially it could save us some work > as well as well as give us a bit more confidence. > > > […] > > I also favor libuv over libevent; to be completely fair I have only > used both of them a little bit but I liked libuv a bit more. > > We definitely shouldn't roll our own. If we had corporate funding then > rolling our own would have the advantage of not having to wait for > upstream merges for bug fixes, but I don't think we have the manpower > or interest to build and maintain our own. I'm glad there seem to be interest in the subject, I had a good feeling while writing the mail. Like Daniel said => IOs is the 2014 bottleneck , and as you can see from my analyze of the current situation, we have all but a clean IO API. I'm gonna start writing the draft RFC this week so that the ideas are tied into a single place. I'll be able to invest some time developping such a layer. Feel free to ping on IRC or intern@ls if one wants to join the effort. The idea will then be to create a branch for some experiment. I'm not confident about threads. Threads is another topic (which shares some comon technical ideas with IOs, such as locking) , and I think we should treat threads in an other RFC Julien Pauli --- ## #74948 — \"Ivan Enderlin\"@Hoaivan.enderlin"@hoa-project.net — 2014-06-17T15:00:54+00:00 URL: https://externals.io/message/74938#74948 · In reply to: #74938 > Hey :-) Hello :-), > I spent few hours analyzing PHP's IO multiplexing parts. > > IO multiplexing refers to calls to select() / poll() / epoll() or kqueue(). > > Simple notice : everything is sparse everywhere, we don't have an > IOmux layer designed. > > I plan to write an RFC on the subject. > The goal is to centralize IO mux calls in a layer to be used > everywhere. My analyze leads to those points : > > - FPM has something, but not designed to be shared > - User streams use hardcoded select() , with an ugly emulation for poll() > - We have nothing about epoll() or kqueue(), except in FPM > - `curl_multi_select()` uses hardcoded select() > - php_cli_server uses hardcoded select() > - mysqlnd uses hardcoded select() for its poll() mecanisms > > Python already have a layer, at > http://hg.python.org/cpython/file/abcf17bc5eae/Modules/selectmodule.c > > Apache's APR has a nice layer with adapters, at > https://github.com/apache/apr/blob/trunk/poll/unix/pollset.c > > Still, there exist the excellent libevent http://libevent.org/ What about depending on libuv also https://github.com/joyent/libuv? It would offer nice major features. (Another related link: https://github.com/chobie/php-uv) > > So basically, the first question will be do we need it ? > I would say yes. As we are talking about PHP6 and modernising our API, > it is time I think to implement IO mux correctly and don't use the old > select() everywhere where there is a better implementation (epoll and > kqueue are both faster, and they dont suffer from any FD_SETSIZE > limit) > Also, we didnt talk about things as websockets etc... yet, for PHP6 , > but it is sure that nowadays 2014, IO mux is a must-have. > > Second question will be : do we use, and then link against libevent > (or any other lib that could fit the need) , or do we develop our own > layer, based on the FPM layer which is pretty nicely done (just not > shareable elsewhere at the moment). > > And then, will people be interested by the RFC ? I do. Cheers. -- Ivan Enderlin Developer of Hoa http://hoa-project.net/ PhD. student at DISC/Femto-ST (Vesontio) and INRIA (Cassis) http://disc.univ-fcomte.fr/ and http://www.inria.fr/ Member of HTML and WebApps Working Group of W3C http://w3.org/ --- ## #74982 — Florian Anderiasch — 2014-06-19T07:05:46+00:00 URL: https://externals.io/message/74938#74982 · In reply to: #74940 > > […] Hey, I like the idea and don't want to rain on your parade, but please do an *extensive* cross-OS check beforehand, I do remember some problems with this kind of stuff on NetBSD/OpenBSD for example (not in PHP, elsewhere), and PHP supports a lot of OSs I haven't used in 10 years or not at all. AFAIK both libevent and libuv have quite a good support, but just after a quick look into the manual (Do we even have a definitive list of supported OS there?) - Solaris, Illumos, HP-UX... no clue here. Greetings, Florian --- ## #74987 — Chris Wright — 2014-06-19T09:22:10+00:00 URL: https://externals.io/message/74938#74987 · In reply to: #74982 > Hey, > I like the idea and don't want to rain on your parade, but please do an > *extensive* cross-OS check beforehand, I do remember some problems with > this kind of stuff on NetBSD/OpenBSD for example (not in PHP, > elsewhere), and PHP supports a lot of OSs I haven't used in 10 years or > not at all. AFAIK both libevent and libuv have quite a good support, but > just after a quick look into the manual (Do we even have a definitive > list of supported OS there?) - Solaris, Illumos, HP-UX... no clue here. Node.js (i.e. libuv) is available for all the common operating systems that PHP is used with, including Solaris and every BSD flavour I can think of. It's also possible to build Node on SmartOS, so Illumos is presumably viable. It doesn't seem to support HP-UX at the moment, but this fairly old thread [1] suggests that Joyent believe it's possible, and I'm sure they would be open to outside help to make it happen. You are absolutely right though, any major change in architecture such as this requires checks that we won't break support for an OS where PHP is used a lot - this is likely not to be the only such change that will be suggested for PHP6, so maybe job #0 should be to compile an "official" list of supported operating systems, rather than the anecdotal "hey, I made it work here" system that currently seems to exist for some of the less common systems. Or maybe there already is one and I just can't find it... [1] https://groups.google.com/forum/#!topic/nodejs/F2-AtY-YkEw --- ## #74988 — Julien Pauli — 2014-06-19T11:53:17+00:00 URL: https://externals.io/message/74938#74988 · In reply to: #74987 > > > […] > > Node.js (i.e. libuv) is available for all the common operating systems > that PHP is used with, including Solaris and every BSD flavour I can > think of. It's also possible to build Node on SmartOS, so Illumos is > presumably viable. > > It doesn't seem to support HP-UX at the moment, but this fairly old > thread [1] suggests that Joyent believe it's possible, and I'm sure > they would be open to outside help to make it happen. > > You are absolutely right though, any major change in architecture such > as this requires checks that we won't break support for an OS where > PHP is used a lot - this is likely not to be the only such change that > will be suggested for PHP6, so maybe job #0 should be to compile an > "official" list of supported operating systems, rather than the > anecdotal "hey, I made it work here" system that currently seems to > exist for some of the less common systems. We have no official list AFAIR, but the documentation's got sheets about specific OS recommandations, at http://www.php.net/manual/en/install.php I agree it could be a nice thing to list OSes we officially support nowadays, OSes we would like to support for PHP6, and then try to make this fit with a possible future lib we would link against. Julien --- ## #75004 — Stas Malyshev — 2014-06-19T23:37:48+00:00 URL: https://externals.io/message/74938#75004 · In reply to: #74938 Hi! > Second question will be : do we use, and then link against libevent > (or any other lib that could fit the need) , or do we develop our own > layer, based on the FPM layer which is pretty nicely done (just not > shareable elsewhere at the moment). > > And then, will people be interested by the RFC ? This would be pretty interesting. I wonder though how libevent behaves on systems like Windows or more exotic Unix variants PHP is know to run on. In any case, basing it on some mature library would definitely make sense. Thanks, -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #75007 — Stas Malyshev — 2014-06-20T00:15:08+00:00 URL: https://externals.io/message/74938#75007 · In reply to: #74988 Hi! > I agree it could be a nice thing to list OSes we officially support nowadays, > OSes we would like to support for PHP6, and then try to make this fit > with a possible > future lib we would link against. For a project like PHP, I'm not even sure what "officially support" means. I've seen PHP run on a lot of systems, from System i to Raspberry Pi, and on all kinds of OSes C compiler runs on. Of course, if it is a stand-alone module, most of the exotic setups would be OK living without it, but if we have anything in core depend on it we'd need to be careful not to introduce dependencies on something that won't work for people that could run PHP before. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #75016 — Chris Wright — 2014-06-20T08:26:30+00:00 URL: https://externals.io/message/74938#75016 · In reply to: #75007 > > Hi! > > > […] nowadays, > > […] > > For a project like PHP, I'm not even sure what "officially support" > means. I've seen PHP run on a lot of systems, from System i to Raspberry > Pi, and on all kinds of OSes C compiler runs on. Of course, if it is a > stand-alone module, most of the exotic setups would be OK living without > it, but if we have anything in core depend on it we'd need to be careful > not to introduce dependencies on something that won't work for people > that could run PHP before. "Officially support" would be "if feature X doesn't work or gracefully degrade in all these places, it will not be accepted". Of course, almost anything *can* be made to run almost anywhere, but that does not mean it is supported. "Supported" means that the PHP project is prepared to actively maintain current and future compatibility with it. This doesn't mean that you can't make it work on platform X, only that it's not guaranteed that you can. For example, raspberry pi is probably not one of those places that would be officially supported. PHP is used heavily in commercial settings, so the way to compile the list is probably to look at what people are actually using on a commercial scale and use that list. Suppot can also be dropped for ancient operating systems when it is reasonable (such as when a new feature is blocked by that OS but a newer version provides the means to implement it), as happened with win XP etc. This would obviously only apply to the core, PECL extensions would be free to support what they like. Thanks, Chris --- ## #75025 — johannes@schlueters.de — 2014-06-20T18:23:47+00:00 URL: https://externals.io/message/74938#75025 · In reply to: #75016 > For example, raspberry pi is probably not one of those places that would be > officially supported. Why not? - Currently we support everything which is close to POSIX and understands C89 with very few extensions. In that sense the Pi is just a Linux system (without much memory) I see no reason why we wouldn't support that. > PHP is used heavily in commercial settings, so the way to compile the list > is probably to look at what people are actually using on a commercial scale > and use that list. Suppot can also be dropped for ancient operating systems > when it is reasonable (such as when a new feature is blocked by that OS but > a newer version provides the means to implement it), as happened with win > XP etc. The primary reason for dropping XP is around providing binaries using specific windows runtime library versions. While it goes a bit hand in hand the reason is less that we need specific newer apis than we want to use later compilers for official binaries. On other platforms we don't ship binaries and thus are less version dependent. As long as the system is POSIX and has a mostly sane compiler we will support it. We won't be ale to test it but react on feedback we get. (overall we have no support guarantees in any way, as we're fully run y volunteers who might disappear tomorrow) It's a key property of the project and code base that we don't only look at those large scale x64 Linux deployments but also care for embedded devices etc. > This would obviously only apply to the core, PECL extensions would be free > to support what they like. Well, it's not only PECL but also some core extensions - if your system has no libxml2 you can't use PHP's XML features. If your system's libc has no (POSIX-compliant) chroot your PHP won't have it. The reminder works, though. johannes --- ## #75026 — Florian Anderiasch — 2014-06-20T18:33:18+00:00 URL: https://externals.io/message/74938#75026 · In reply to: #75025 > As long as the system is POSIX and has a mostly sane compiler we will > support it. We won't be ale to test it but react on feedback we get. > (overall we have no support guarantees in any way, as we're fully run y > volunteers who might disappear tomorrow) It's a key property of the > project and code base that we don't only look at those large scale x64 > Linux deployments but also care for embedded devices etc. > > > […] > > Well, it's not only PECL but also some core extensions - if your system > has no libxml2 you can't use PHP's XML features. If your system's libc > has no (POSIX-compliant) chroot your PHP won't have it. The reminder > works, though. Yes, I wish I'd phrased it as "commonly tested" instead of "officially supported" operating systems - but the point still stands, we have a handful that are supposed to work absolutely, all the time. I wouldn't bet that some of the lesser mainstream ones are tested on every release, that was where I was hinting at. I'm all for the revamp, just a bit wary of adding such a big external library in such a low-level place. Greetings, Florian --- ## #75043 — Julien Pauli — 2014-06-23T08:05:16+00:00 URL: https://externals.io/message/74938#75043 · In reply to: #74943 > hi, > > > > > […] > > Not directly related, but highly recommend read about async, crossplatforms: > > https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores Definetly a must-read. I learnt many things on the subject. Julien