# Porting PECL to userspace

- Thread: https://externals.io/message/52482
- Original archive: https://news-web.php.net/php.internals/52482
- Started by Larry Garfield on 2011-05-20T16:15:55+00:00
- Messages: 13
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #52482 — Larry Garfield — 2011-05-20T16:15:55+00:00

URL: https://externals.io/message/52482#52482

Hi all.

I'm working with a fellow developer on an experimental project.  There 
are some PECL modules that we want to try and use in an open source 
project where we cannot guarantee that PECL modules will be available, 
since it's intended for widespread distribution on both shared hosts and 
custom hosting. The thought we had was to do a user-space port of the 
PECL module to include in the project and rely on that.  Then if the 
PECL module is installed, we don't include the library (either via an 
 `extension_loaded()` check or just relying on autoload) and the PECL 
implementation gets used instead.  Poof, nice speed boost.

The questions I have are:

1) Is this even a viable approach?  It seems like it, but to my 
knowledge no one else has done this to any serious extent which makes me 
wonder if there's a reason the road less traveled is less traveled.

2) Is anyone else doing this?  No sense doing it ourselves if someone 
else already is.

3) What would be the cleanest way to do so?  We had the thought of 
partially automating the process by having PHP auto-generate at the very 
least the subs of any classes and functions that the module provides.  
However, when my colleague tried using the same parser as is used for 
generating documentation he says he got several times as many classes as 
the manual says the module has.  We were using the PECL HTTP module as 
our sample (http://www.php.net/http).  (I don't know the exact details 
of what he did at the moment.)  Is that not a viable approach?  Would we 
be better off using reflection?  Is there some other tool we're not 
aware of?

If viable I'd love if this would start a trend, but we'll see where it 
goes.  I know it wouldn't work for all PECL modules, obviously, but I 
suspect it could work for several, and provide an easy way for different 
PHP projects to share backend code without needing lots of C developers.

--Larry Garfield

---

## #52483 — johannes@schlueters.de — 2011-05-20T16:33:18+00:00

URL: https://externals.io/message/52482#52483 · In reply to: #52482

Hi,


> 1) Is this even a viable approach?  It seems like it, but to my 
> knowledge no one else has done this to any serious extent which makes me 
> wonder if there's a reason the road less traveled is less traveled.

This is a case by case thing. Most modules talk to a C library. You'd
have to recreate the functionality. Some things can be done. Some things
can be done but will be slooooowww. Some things won't work.

> 2) Is anyone else doing this?  No sense doing it ourselves if someone 
> else already is.

There are some things where a similar solution exists already in PEAR or
some other repository. They often have a different interface (leveraging
PHP while extensions are formed more or less after a C lib) but can
easily be swapped.

> 3) What would be the cleanest way to do so? 

Case by case.

>  We had the thought of 
> partially automating the process by having PHP auto-generate at the very 
> least the subs of any classes and functions that the module provides.  
> However, when my colleague tried using the same parser as is used for 
> generating documentation he says he got several times as many classes as 
> the manual says the module has.  We were using the PECL HTTP module as 
> our sample (http://www.php.net/http).  (I don't know the exact details 
> of what he did at the moment.)  Is that not a viable approach?  Would we 
> be better off using reflection?  Is there some other tool we're not 
> aware of?

Well, HTTP is a doable case, I assume. Creating stubs from reflection
shouldn't be too hard. While lots of HTTP classes already exist. Why
build yet another one?

In other cases I think you shouldn't follow necessarily the extensions
interface but build one that makes sense[tm].

> If viable I'd love if this would start a trend, but we'll see where it 
> goes.  I know it wouldn't work for all PECL modules, obviously, but I 
> suspect it could work for several, and provide an easy way for different 
> PHP projects to share backend code without needing lots of C developers.

Well, in general PHP is grown-up and usable enough that more and more
stuff can be done in pure PHP. Main reasons for using C are that
existing C libraries can be used and C code is still way faster. Look at
the stuff frameworks provide ... the way should actually be the other
way round than what you describe: Stuff is done in PHP andportedtoC when
needed.

johannes

---

## #52484 — Richard Quadling — 2011-05-20T16:34:37+00:00

URL: https://externals.io/message/52482#52484 · In reply to: #52482

> Hi all.
>
> I'm working with a fellow developer on an experimental project.  There are
> some PECL modules that we want to try and use in an open source project
> where we cannot guarantee that PECL modules will be available, since it's
> intended for widespread distribution on both shared hosts and custom
> hosting. The thought we had was to do a user-space port of the PECL module
> to include in the project and rely on that.  Then if the PECL module is
> installed, we don't include the library (either via an `extension_loaded()`
> check or just relying on autoload) and the PECL implementation gets used
> instead.  Poof, nice speed boost.
>
> The questions I have are:
>
> 1) Is this even a viable approach?  It seems like it, but to my knowledge no
> one else has done this to any serious extent which makes me wonder if
> there's a reason the road less traveled is less traveled.
>
> 2) Is anyone else doing this?  No sense doing it ourselves if someone else
> already is.
>
> 3) What would be the cleanest way to do so?  We had the thought of partially
> automating the process by having PHP auto-generate at the very least the
> subs of any classes and functions that the module provides.  However, when
> my colleague tried using the same parser as is used for generating
> documentation he says he got several times as many classes as the manual
> says the module has.  We were using the PECL HTTP module as our sample
> (http://www.php.net/http).  (I don't know the exact details of what he did
> at the moment.)  Is that not a viable approach?  Would we be better off
> using reflection?  Is there some other tool we're not aware of?
>
> If viable I'd love if this would start a trend, but we'll see where it goes.
>  I know it wouldn't work for all PECL modules, obviously, but I suspect it
> could work for several, and provide an easy way for different PHP projects
> to share backend code without needing lots of C developers.
>
> --Larry Garfield

If you using an extension that exists as glue to a third party
library, then you are going to need PHP to communicate at a very low
level. Something I'm not sure would be possible at the moment.

If the extension is self contained, then this would more likely be a
good candidate.

You can use reflection to build the stubs easily enough

But I'm not sure all the things you can do in C can be done in PHP,
not without a LOT of effort. Talking to third party libraries or
making c calls to the OS is out.

Richard.
-- 
Richard Quadling
Twitter : EE : Zend : PHPDoc
@RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY : bit.ly/lFnVea

---

## #52485 — Larry Garfield — 2011-05-20T16:45:38+00:00

URL: https://externals.io/message/52482#52485 · In reply to: #52483

>
> > […]
> Well, HTTP is a doable case, I assume. Creating stubs from reflection
> shouldn't be too hard. While lots of HTTP classes already exist. Why
> build yet another one?
>
> In other cases I think you shouldn't follow necessarily the extensions
> interface but build one that makes sense[tm].

> > […]
> Well, in general PHP is grown-up and usable enough that more and more
> stuff can be done in pure PHP. Main reasons for using C are that
> existing C libraries can be used and C code is still way faster. Look at
> the stuff frameworks provide ... the way should actually be the other
> way round than what you describe: Stuff is done in PHP andportedtoC when
> needed.
>
> johannes

Well a non-small part of the goal here is to use an existing interface 
that is (presumably) used by people outside of our project, and to not 
have to rethink the interface ourselves.  To an extent I'd prefer a 
non-perfect existing interface to a custom perfect interface.  While 
certainly some PECL modules wouldn't make any sense in user space (no 
one is porting runkit to user space, naturally), I've seen a fair number 
of modules that are mostly socket wrappers to Solr, oAuth, etc.  Those 
can be done, slower of course, in PHP.  That's the use case I mostly 
see, where there's existing work that we can leverage if we can work 
around the "must have root" limitation on most hosts.

If the long term result is that PECL gets more attention and usage as 
people realize the advantages of C-based code, even better.

--Larry Garfield

---

## #52487 — Andrew Curioso — 2011-05-20T16:47:18+00:00

URL: https://externals.io/message/52482#52487 · In reply to: #52482

AMF-PHP does something similar to this.
It is a PHP implementation of AMF (Action Message Format) so it allows you
to accept and respond to AMF messages from Flash apps.

Its native PHP encoder is extremely slow but it works. However, if you
install amfext from PECL it will use that for encoding instead.

AMF-PHP is more than just an encoder, though; whereas the extension is only
used for the encoding part.
So, in that regard, it may not be the best example to go by for what you
want to do.

- Andrew


On Fri, May 20, 2011 at 12:15 PM, Larry Garfield <larry@garfieldtech.com>wrote:

> Hi all.
>
> I'm working with a fellow developer on an experimental project.  There are
> some PECL modules that we want to try and use in an open source project
> where we cannot guarantee that PECL modules will be available, since it's
> intended for widespread distribution on both shared hosts and custom
> hosting. The thought we had was to do a user-space port of the PECL module
> to include in the project and rely on that.  Then if the PECL module is
> installed, we don't include the library (either via an `extension_loaded()`
> check or just relying on autoload) and the PECL implementation gets used
> instead.  Poof, nice speed boost.
>
> The questions I have are:
>
> 1) Is this even a viable approach?  It seems like it, but to my knowledge
> no one else has done this to any serious extent which makes me wonder if
> there's a reason the road less traveled is less traveled.
>
> 2) Is anyone else doing this?  No sense doing it ourselves if someone else
> already is.
>
> 3) What would be the cleanest way to do so?  We had the thought of
> partially automating the process by having PHP auto-generate at the very
> least the subs of any classes and functions that the module provides.
>  However, when my colleague tried using the same parser as is used for
> generating documentation he says he got several times as many classes as the
> manual says the module has.  We were using the PECL HTTP module as our
> sample (http://www.php.net/http).  (I don't know the exact details of what
> he did at the moment.)  Is that not a viable approach?  Would we be better
> off using reflection?  Is there some other tool we're not aware of?
>
> If viable I'd love if this would start a trend, but we'll see where it
> goes.  I know it wouldn't work for all PECL modules, obviously, but I
> suspect it could work for several, and provide an easy way for different PHP
> projects to share backend code without needing lots of C developers.
>
> --Larry Garfield
>
> --
> 
> To unsubscribe, visit: http://www.php.net/unsub.ph<http://www.php.net/unsub.php

---

## #52486 — johannes@schlueters.de — 2011-05-20T17:01:38+00:00

URL: https://externals.io/message/52482#52486 · In reply to: #52485

> If the long term result is that PECL gets more attention and usage as 
> people realize the advantages of C-based code, even better.

I think it's the exact opposite - the less C code we need the better.
Developing C code is more work. Maintaining C code is more work.
Distributing C code is more complicated. The less this is needed, the
faster PHP is, the better for everybody (except C programmers ;-) )

johannes

---

## #52488 — Brian Moon — 2011-05-20T18:14:03+00:00

URL: https://externals.io/message/52482#52488 · In reply to: #52486

> I think it's the exact opposite - the less C code we need the better.
> Developing C code is more work. Maintaining C code is more work.
> Distributing C code is more complicated. The less this is needed, the
> faster PHP is, the better for everybody (except C programmers ;-) )

I was with you until "The less this is needed, the faster PHP is, the 
better for everybody". PHP is just a glue language on top of an awesome 
library of C. If more people would embrace that and quit including 1000 
classes to show a simple web page then it would be better for everybody.

Brian.

---

## #52489 — Larry Garfield — 2011-05-20T18:21:42+00:00

URL: https://externals.io/message/52482#52489 · In reply to: #52486

> 
> > […]
> I think it's the exact opposite - the less C code we need the better.
> Developing C code is more work. Maintaining C code is more work.
> Distributing C code is more complicated. The less this is needed, the
> faster PHP is, the better for everybody (except C programmers ;-) )
>
> johannes

Unless the C code already exists and is already tested, as is the case 
with existing PECL modules.  Wasn't that the original intent with PHP, 
to stitch together C libraries that you could use more easily?

--Larry Garfield

---

## #52490 — Hannes Magnusson — 2011-05-20T21:16:07+00:00

URL: https://externals.io/message/52482#52490 · In reply to: #52488

2011/5/20 Brian Moon <brian@moonspot.net>:
> > […]
>
> I was with you until "The less this is needed, the faster PHP is, the better
> for everybody". PHP is just a glue language on top of an awesome library of
> C. If more people would embrace that and quit including 1000 classes to show
> a simple web page then it would be better for everybody.
>

Indeed. But how awesome would it be if you could talk to a c library
using userspace PHP?
We do have a large community that thinks c is scary and don't dare to
checkout the source to even fix a typo - but if they could write an
extension in PHP... :)

-Hannes

---

## #52491 — Pierre Joye — 2011-05-20T21:34:46+00:00

URL: https://externals.io/message/52482#52491 · In reply to: #52490

On Fri, May 20, 2011 at 11:16 PM, Hannes Magnusson
<hannes.magnusson@gmail.com> wrote:
> 2011/5/20 Brian Moon <brian@moonspot.net>:
> > […]
>
> Indeed. But how awesome would it be if you could talk to a c library
> using userspace PHP?


You can, it is possible via pecl's ffi. One has to give it some love.


Cheers,
-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #52493 — David Muir — 2011-05-21T07:06:42+00:00

URL: https://externals.io/message/52482#52493 · In reply to: #52482

> Hi all.
>
> I'm working with a fellow developer on an experimental project.  There
> are some PECL modules that we want to try and use in an open source
> project where we cannot guarantee that PECL modules will be available,
> since it's intended for widespread distribution on both shared hosts
> and custom hosting. The thought we had was to do a user-space port of
> the PECL module to include in the project and rely on that.  Then if
> the PECL module is installed, we don't include the library (either via
> an `extension_loaded()` check or just relying on autoload) and the PECL
> implementation gets used instead.  Poof, nice speed boost.
>
> The questions I have are:
>
> 1) Is this even a viable approach?  It seems like it, but to my
> knowledge no one else has done this to any serious extent which makes
> me wonder if there's a reason the road less traveled is less traveled.

One reason would be because it requires fixing bugs and implementing
features twice, so it effectively doubles the required work.

>
> 2) Is anyone else doing this?  No sense doing it ourselves if someone
> else already is.

There's a Drupal extension that does something like this, but in reverse:
http://drupal.org/project/drupal_php_ext

Cheers,
David

---

## #52522 — Larry Garfield — 2011-05-28T03:21:32+00:00

URL: https://externals.io/message/52482#52522 · In reply to: #52493

>
> > […]
> There's a Drupal extension that does something like this, but in reverse:
> http://drupal.org/project/drupal_php_ext
>
> Cheers,
> David

Yes, Drupal is the project I'm thinking of using this for. :-)  The goal 
is the same, but I'm trying to avoid having us develop yet-more-new-APIs 
when perfectly good ones already exist that have been banged on and 
tested by more people than just Drupal.  (That particular PECL module 
also, last time I checked, had little to no practical effect on 
performance.  It was just an experiement that didn't go very far.)

--Larry Garfield

---

## #52546 — Will Bond — 2011-05-30T15:48:44+00:00

URL: https://externals.io/message/52482#52546 · In reply to: #52482

On Fri, May 20, 2011 at 12:15 PM, Larry Garfield <larry@garfieldtech.com>wrote:

> I'm working with a fellow developer on an experimental project.  There are
> some PECL modules that we want to try and use in an open source project
> where we cannot guarantee that PECL modules will be available, since it's
> intended for widespread distribution on both shared hosts and custom
> hosting. The thought we had was to do a user-space port of the PECL module
> to include in the project and rely on that.  Then if the PECL module is
> installed, we don't include the library (either via an `extension_loaded()`
> check or just relying on autoload) and the PECL implementation gets used
> instead.  Poof, nice speed boost.
>

I do a fair amount of this with Flourish (http://flourishlib.com), since my
goal is to have all functionality work with only the extensions that come
with a standard PHP install (plus at least one database extension for the
desired database). I also back-port some functionality since my goal is for
PHP 5.1+ compatibility. When the PHP extension in question is available and
known not to contain show-stopper bugs, I'll use it. Otherwise I fall back
to a native PHP implementation of the functionality. Granted, the native PHP
is much slower, but people who are really concerned about performance can
always do the work of installing the extensions. In practice this seems to
work fairly well.

So far I have the following implemented with native PHP fallbacks:

mbstring (UTF-8 only): fUTF8
json (for 5.1): fJSON
bcmath: fNumber

I've also implemented imap/pop3 in native PHP to get around performance
issues, segfaults and similar issues. I've backported HTTP-only cookies to
5.1. There are also a whole bunch of other compatibility/portability
elements.


> The questions I have are:
>
> 1) Is this even a viable approach?  It seems like it, but to my knowledge
> no one else has done this to any serious extent which makes me wonder if
> there's a reason the road less traveled is less traveled.
>

It has seem to work fine for Flourish, although I don't have anywhere near
the traction/usage of any of the big frameworks or component libraries. I
think part of the reason this road is less traveled is because most
developers who care about more advanced functionality are more than capable
of finding an environment that has the extensions they need.


> 2) Is anyone else doing this?  No sense doing it ourselves if someone else
> already is.
>

As I mentioned, I do this is Flourish, however I do wrap the functionality
up in my own API since I am kind of particular about the usability of APIs.
Sometimes wrapping it in my own API provides the ability to skip 30% of the
functionality I don't see being frequently used, other times it allows me to
fix bugs in specific versions of PHP, such as with `stream_select()` in some
versions of PHP 5.2.


> 3) What would be the cleanest way to do so?  We had the thought of
> partially automating the process by having PHP auto-generate at the very
> least the subs of any classes and functions that the module provides.
>  However, when my colleague tried using the same parser as is used for
> generating documentation he says he got several times as many classes as the
> manual says the module has.  We were using the PECL HTTP module as our
> sample (http://www.php.net/http).  (I don't know the exact details of what
> he did at the moment.)  Is that not a viable approach?  Would we be better
> off using reflection?  Is there some other tool we're not aware of?
>

Like I said, I create my own APIs. I normally will have used the C
extensions some and have some ideas about how I think they can be improved
and how I normally see them used. This then informs me about what I think
the custom API should be. Sometimes I do a pretty straight-forward
implementation though. For instance, I've implemented all of the PHP string
functions to work against UTF-8, even things like `ucwords()`.


> If viable I'd love if this would start a trend, but we'll see where it
> goes.  I know it wouldn't work for all PECL modules, obviously, but I
> suspect it could work for several, and provide an easy way for different PHP
> projects to share backend code without needing lots of C developers.
>

I agree that not having to rely on C code is helpful, but in my opinion it
isn't because developing in C is hard/annoying/there are many C developers,
but because it is much harder to get special C extensions installed on many
shared hosts. Even for people who do have full access to install extensions,
it is often a pain to deal with compared to dropping in an updated PHP
script.

So from my experience implementing various semi-CPU-intensive work, the key
aspects to making native PHP work is trying to rely on built-in functions
that do as much as possible and to be sure to think about algorithms being
used. For instance, I found I was able to implement the porter stemming
algorithm with much better performance by using preg* functions for string
manipulation. For the diff/patch library I wrote, I found using the patience
diff algorithm it was possible to write a diffing library that wouldn't
consume 128MB of ram for 3000 line files, and it was an order of magnitude
faster to boot.

Will