Proposing inclusion of PCS in the 7.2 core distribution

Hi,

Le 05/06/2017 à 21:26, Fleshgrinder a écrit :

I skimmed through the documentation of yours. There is however one
question left. Is it possible to have C code that is accessible only to
the PHP code of an extensions, instead of all user-level code?

As extension PHP code is executed in the same context as user-level
code, there is no way to have such access restriction. The workaround,
IMO, would be to expose non-documented methods or functions. But nothing
will prevent user-level code to call them.

Regards

François

Le 05/06/2017 à 19:46, François Laupretre a écrit :

PCS provides a fast and easy mechanism to mix C and PHP code in PHP
extensions (more about PCS at http://pcs.tekwire.net). Thanks to the
PHP 7 performance improvement and the inclusion of opcache in the
core, a lot of existing non-performance-critical extension code may
now be converted to PHP without significant performance loss (this
must be measured case by case, of course, but tests show that
opcode-cached PHP code is often faster than C).

Sorry, but I don't like the idea of having PHP code bundled in C
extension.

Have low-level part written in C, and user-land part in PHP is indeed
a good way (e.g. mondodb, phpiredis + phredis...), but having the PHP
library distributed via composer or any other way is enough.

P.S. IIRC couchbase have tried this way, and revert it

We did it as well, in the early early days. It wasn't great and reverted
that too. I also believe PHP code should be distributed through
composer, and that is much easier to upgrade, as well as allow for
multiple versions running on the same server.

cheers,
Derick

Hi together,

Am 06.06.2017 um 14:43 schrieb Nikita Popov:

First of all: I think the ability to implement parts of PHP extensions in
PHP is extremely important and will be a game changer in our ability to
maintain and improve our standard library.

I agree with you here 100%.

1. As far as I understand, PCS relies on autoloading. There are two issues
   here: First, autoloading does not register symbols prior to autoloading.
   This means that functions like get_defined_classes() will not behave as
   expected. Second, autoloading does not support functions. I think both of
   these problems can be solved with some up-front symbol analysis. Lazily
   compiling internal functions should not run into any of the problems we
   have with userland function autoloading.

(disclaimer: I have not deep knowledge how the core really works)

From what I see there is a difference between writing extension code in
PHP and placing that plain PHP files somewhere when installing. So I
would argue when compiling the extension or the core, all these PHP
extension stuff should get compiled to OpCodes and put in a segment of
the resulting binary.
The OpCache could then be modified to allow it to use the OpCodes from
these read-only segments (one for the core, one optionally for each
extension). That would void the problems you mentioned above, but maybe
I am missing how the OpCache and symbol registration work in PHP.

Greets
Dennis

First of all: I think the ability to implement parts of PHP extensions in
PHP is extremely important and will be a game changer in our ability to
maintain and improve our standard library.

Ditto this. The argument for having a MUCH larger pool of maintainers
is #1. The reduced footprint for crash-bugs is a very close #2.

Anyway, to get back to the topic of PCS. First, I would recommend to target
PHP 7.3 for this change. Feature freeze for 7.2 is in a bit over a month
and I think we'll want to make some non-trivial changes to how this works
if we integrate it in PHP. If added to PHP, I think this should be
integrated into the core, rather than being an extension.

As an RM for 7.2 and someone who's been daydreaming of something
like PCS for awhile, I'm going to second this strongly. I literally
cut the alpha1 tag this morning (for Thursday's official release), and
something like PCS is pretty significant in its disruptive capacity
(not to mention being lower-value until we start actually moving
some of those extensions to PHP code). Save this for 7.3, please.

2. It has already been mentioned in the thread, but what seems to lack
   right now is a good way of integrating PHP and C portions. As far as I
   understand, PCS allows you to write an entire class in PHP, but it does not
   allow you to offload parts of the functionality to C without exposing
   additional public APIs. I think there are two things we can do here:

b) Allow binding some methods specified in PHP to an internal
implementation. The way this works in HHVM is that the PHP file contains
just a signature, with an attribute that signifies that an internal
implementation will be bound to that function:

As the author of HNI*, I look forward to iterating on this as a
potential design (maybe learning from HNI's experience and making
something better), but I regard it as a "version 2" thing. We can get
PCS good and stable first, THEN worry about bridging to C later.

This would be useful for other reasons as well. In particular, this could
become a replacement for the existing arginfo-based signature
specification, which is somewhat limited and causes discrepancies with
userland classes. For example, arginfo does not support default values.

100% this, though PHP's version of HNI will suffer a few shortcomings
due to the lack of a type_traits equivalent in C99. I'm not
suggesting we go C++11 just to get a better bridge, but it's a real
constraint in getting the same advantages that HNI has.

TL;DR - This is one giant "Me Too" response. :p

-Sara

• HHVM Native Interface. i.e. Exactly what Niki just described.

Hi Nikita,

Le 06/06/2017 à 14:43, Nikita Popov a écrit :

Anyway, to get back to the topic of PCS. First, I would recommend to
target PHP 7.3 for this change. Feature freeze for 7.2 is in a bit
over a month and I think we'll want to make some non-trivial changes
to how this works if we integrate it in PHP. If added to PHP, I think
this should be integrated into the core, rather than being an extension.

Agreed. My initial choice was to keep it as a separate extension. This
was fine for a proof of concept but we cannot solve the points raised
below without a tighter integration with the core. I would have liked to
keep the coupling as loose as possible but solving the remaining issues
is more important. So, let's consider 7.3 as the new target.

1. As far as I understand, PCS relies on autoloading. There are two
   issues here: First, autoloading does not register symbols prior to
   autoloading. This means that functions like get_defined_classes() will
   not behave as expected. Second, autoloading does not support
   functions. I think both of these problems can be solved with some
   up-front symbol analysis. Lazily compiling internal functions should
   not run into any of the problems we have with userland function
   autoloading.

Right. PCS implements a workaround by loading the code containing
functions at the beginning of each request. So, my plan was a future
extension of the autoloading mechanism to functions (and constants).

The PCS autoloader already extracts symbols from the PHP code at
registration time. So, PCS already has a list of the class, function,
and constant names it manages. I don't know what you mean with 'Lazily
compiling internal functions'. Autoloading, combined with opcache,
avoids to execute the whole code at the beginning of each request but
there are probably better alternatives.

2. It has already been mentioned in the thread, but what seems to lack
   right now is a good way of integrating PHP and C portions. As far as I
   understand, PCS allows you to write an entire class in PHP, but it
   does not allow you to offload parts of the functionality to C without
   exposing additional public APIs. I think there are two things we can
   do here:

a) Provide a mechanism that makes certain functions only available
inside extension PHP code. This would allow exposing some private PHP
functions which are only used in the internal implementation of the
extension.

b) Allow binding some methods specified in PHP to an internal
implementation. The way this works in HHVM is that the PHP file
contains just a signature, with an attribute that signifies that an
internal implementation will be bound to that function:

class Bar {
<<__Native>>
function foo($args);
}

This would be useful for other reasons as well. In particular, this
could become a replacement for the existing arginfo-based signature
specification, which is somewhat limited and causes discrepancies with
userland classes. For example, arginfo does not support default values.

Implementing this may be beyond my capacity but, if you agree, we can
work together to implement solutions to these issues. Other volunteers
are welcome too.

Regards

François

Hi,

-----Original Message-----
From: Nikita Popov [mailto:nikita.ppv@gmail.com]
Sent: Tuesday, June 6, 2017 2:43 PM
To: François Laupretre francois@tekwire.net
Cc: PHP internals internals@lists.php.net
Subject: Re: [PHP-DEV] Proposing inclusion of PCS in the 7.2 core distribution

On Mon, Jun 5, 2017 at 7:46 PM, François Laupretre francois@tekwire.net
wrote:

Hi,

PCS provides a fast and easy mechanism to mix C and PHP code in PHP
extensions (more about PCS at http://pcs.tekwire.net). Thanks to the
PHP
7 performance improvement and the inclusion of opcache in the core, a
lot of existing non-performance-critical extension code may now be
converted to PHP without significant performance loss (this must be
measured case by case, of course, but tests show that opcode-cached
PHP code is often faster than C).

Another motivation is the lack of extension maintainers. It may be
complex to convert a C extension to PHP but, once it's done,
maintenance becomes much easier.

As one of PCS goals is to allow converting parts of existing core
extensions to PHP, it seems natural to initiate the movement by an
inclusion of PCS in the core distribution. Then, I and others will
start proposing conversions of existing code. IMO, the PDO generic
layer is a perfect candidate, but there are many others.

Converting existing C code to PHP is not the only usage. With PCS,
adding an OO layer to a function-only extension becomes an easy task.
Sara recently told about a curl OOP layer
(https://gist.github.com/sgole mon/e95bfc34d34c4f63fa953ee9294ae02c).
Using PCS, adding such PHP code on top of the curl extension would take less
than one hour.

I hadn't proposed this so far because the 'cache_key' operation
currently proposed for 7.2 is a pre-requisite, as PCS exposes the PHP
code it manages via a stream wrapper.

So, please give me your thoughts. Suggestions of potential candidates
to be rewritten from C to PHP are welcome too.

Hi,

First of all: I think the ability to implement parts of PHP extensions in PHP is
extremely important and will be a game changer in our ability to maintain and
improve our standard library.

There are essentially only two good reasons for implementing functionality in C:
One is performance, the other is FFI. Unfortunately, the requirement to use C
for everything inside an extension means that we write a large amounts of C
code that does not fall into either of those categories. The resulting code is hard
to maintain, often subtly buggy and usually not consistent with ordinary userland
PHP code. Typical issues we see all the time are bad or completely absent
serialization support, lack of circular garbage collection, crashes when the
object is improperly initialized and bugs appearing when internal classes are
extended.

On top of that, implementing certain functionality in C actually makes the
resulting code slower than equivalent PHP code. While our virtual machine is
highly optimized, our internal APIs are often not, or not typically used in their
most efficient form. One case where internal code loses are invocations of
userland callbacks. Another is access to properties.

The current situation also has a large and somewhat hidden impact on our API
design. Due to the large maintenance burden that implementing "proper"
APIs imposes on us, we tend to go with the simplest possible API. Usually this
means that we end up directly exposing C binding APIs, even if they are a very
bad fit for PHP. As already noted in this thread, the current curl API is such an
example. (I know that some people will argue that its better to expose simple
procedural APIs rather than fancy object oriented APIs -- however, that's a
choice that should be made based on technical arguments, not due to technical
limitations.)

Some people have mentioned that this is better solved by shipping the PHP code
separately using composer. While this may be viable for 3rd party extensions
(and may be preferable if they have large fractions of PHP code), this option
does not exist for our standard library. We can hardly tell people that they
should go install a composer package in order to make use of some APIs in our
standard library.

Anyway, to get back to the topic of PCS. First, I would recommend to target PHP
7.3 for this change. Feature freeze for 7.2 is in a bit over a month and I think
we'll want to make some non-trivial changes to how this works if we integrate it
in PHP. If added to PHP, I think this should be integrated into the core, rather
than being an extension.

Here are some random thoughts:

1. As far as I understand, PCS relies on autoloading. There are two issues
   here: First, autoloading does not register symbols prior to autoloading.
   This means that functions like get_defined_classes() will not behave as
   expected. Second, autoloading does not support functions. I think both of these
   problems can be solved with some up-front symbol analysis. Lazily compiling
   internal functions should not run into any of the problems we have with userland
   function autoloading.

2. It has already been mentioned in the thread, but what seems to lack right now
   is a good way of integrating PHP and C portions. As far as I understand, PCS
   allows you to write an entire class in PHP, but it does not allow you to offload
   parts of the functionality to C without exposing additional public APIs. I think
   there are two things we can do here:

a) Provide a mechanism that makes certain functions only available inside
extension PHP code. This would allow exposing some private PHP functions
which are only used in the internal implementation of the extension.

b) Allow binding some methods specified in PHP to an internal implementation.
The way this works in HHVM is that the PHP file contains just a signature, with an
attribute that signifies that an internal implementation will be bound to that
function:

class Bar {
<<__Native>>
function foo($args);
}

This would be useful for other reasons as well. In particular, this could become a
replacement for the existing arginfo-based signature specification, which is
somewhat limited and causes discrepancies with userland classes. For example,
arginfo does not support default values.

The mechanism like HHVM has is what were surely useful. Where I see a concern regarding PHP is, that with the original proposal a PHP interpreter is needed for the partials processing. To the compilation time for the core, it is not expected to be available. In further, for example if the binary just compiled would be used, it has an issue potential with FDO/PGO. Reason - required preparation tasks would produce training data not necessarily desired. Perhaps that is solvable by extending the build time - an independent minimal binary could be produced just for the goal. That, however, might not suffice depending on complexity, fe like it were about a PECL ext depending on classes of another one and trying to use type hints, classes from dependent, etc. Perhaps this needs more evaluation for non core exts. It's somehow a chinken/egg issue.

IMO, in any case the pieces, that can be handed out into a PHP code, would be a huge win. That would add to complexity reduction of the actual C parts an make the actual dev faster and more qualitative. There are always cases, where moving the implementation partially to C is a win in speed or functionality, or where is moving C implementation into userland doesn't make things worse but simplifies a lot. Currently it's almost only one way - only moving parts to C is a win in most case. Flexibility is a huge win in having both, too.

I'd also see the topic as coupled tightly with the previous discussions about Opcache core integration. Looking at what Python does, a possibility to redistribute the opcode cached bins might make sense. Of course, there are differences in how it works, where we could introduce some naming/configuration conventions to pursue the goal. That might involve also a change to package.xml specs for PECL exts. In general, I'd put the Opcache integration in the foreground, as that is the long standing topic that would also make evermore sense when the JIT branch in integrated. A PECL ext can have sources only, but given it's compiled for and on a specific platform - having right the opcode bins is more optimal. Maybe having both ways of either pure PHP or opcode bins were useful, too. In any case, Opcache integration with the core could be a game changer in many topics, as for me. Another related topic might be the integration of libffi, also looking at the good examples from Python.

Thanks

Anatol

Hi Francois,

I'm in favour of shipping PHP code as part of either PHP core, and for
extensions.

I would use this for Imagick. There are several 'helper' functions
that would make this library easier to use, that are trivial to write
in PHP, that are incredibly not trivial to write in C.

Please give examples of core extensions that, in your opinion, are susceptible to be partially or fully converted to PHP.

Uh - I don't really agree that any existing core code should be
converted to PHP. If the code is already there and working as C code,
there are very few benefits converting it to PHP now. If we ever get
around to refactoring how strings are stored in PHP, it's at that
point that converting existing functionality to PHP code might be a
good idea.

Suggestions to complement existing core extensions (like an additional curl OO layer) are welcome too.

If you want this RFC to pass, I'd strongly recommend having
conversations about adding those as separate RFCs, once the "having
PHP code distributed with PHP" RFC is passed as otherwise the
signal-to-noise ratios will be terrible.

cheers
Dan
Ack