RFC: execution opcode file without php source code file

6 years ago by chopins.xiao@live.com — view source — reply

unread

I commit a new request path, the request is about execution opcode file without php source code file
this RFC provides similar to class file of java and pyo file of python.
patch is: https://github.com/php/php-src/pull/6146

6 years ago by michal.brzuchalski@gmail.com — view source — reply

unread

Hi,

czw., 1 paź 2020 o 11:36 肖鑫鑫 chopins.xiao@live.com napisał(a):

I commit a new request path, the request is about execution opcode file
without php source code file
this RFC provides similar to class file of java and pyo file of python.
patch is: https://github.com/php/php-src/pull/6146

I had exact the same feature in my mind and working implementation few
years ago
when PHP 7.0 appeared on horizon which didn't get a Zend Guard support.
I like the idea a lot.

One risk I see here is people putting whole framework or apps into one PHP
file before
calling opcache_compile_file to produce one big binary file.

Would it be possible to move this feature to separate function and pass
iterable list of files?
I am not sure if with current OPCache file format it's possible but if that
might reduce amount
of steps required to produce one huge OPCache file which then can be loaded
by interpreter.

Second thing worth of noting is that PR examples use CLI SAPI and lacks of
examples how
FPM can load it. Would it require at least an index.php with <?php include 'myapp.bin'; ?

Cheers,
Michał Marcin Brzuchalski

6 years ago by Henrik Skov — view source — reply

unread

Hi list !

I would support such a feature as well.

I am working on an implementation that would integrate PHAR support and
opcode cache binary files
(so that you could make a PHAR file that contains binary (opcode cache)
files)

I am voicing my opinion here because the current PHAR implementation
does not work with binary (opcode cache) files.
(The problem seems to be that the PHAR extension directly extracts/opens
the phar file directory, compiles the file into opcodes
and then executes them) - But compiling a file into opcodes that is
already a binary (opcode cache) file yields garbage...

So if this feature were to be implemented, the implementation would have
to (in my opinion) solve this problem in the PHAR
extension.

The way I have worked around this problem is to implement some extract
functions (require_once_bin(), require_bin(), include_bin() and
include_once_bin() and then make sure that autoloaders used in inside
the PHAR use those when reading from/including from that PHAR file that
contains opcode binary files.

/Henrik

Hi,

czw., 1 paź 2020 o 11:36 肖鑫鑫 chopins.xiao@live.com napisał(a):

I commit a new request path, the request is about execution opcode file
without php source code file
this RFC provides similar to class file of java and pyo file of python.
patch is: https://github.com/php/php-src/pull/6146

I had exact the same feature in my mind and working implementation few
years ago
when PHP 7.0 appeared on horizon which didn't get a Zend Guard support.
I like the idea a lot.

One risk I see here is people putting whole framework or apps into one PHP
file before
calling opcache_compile_file to produce one big binary file.

Would it be possible to move this feature to separate function and pass
iterable list of files?
I am not sure if with current OPCache file format it's possible but if that
might reduce amount
of steps required to produce one huge OPCache file which then can be loaded
by interpreter.

Second thing worth of noting is that PR examples use CLI SAPI and lacks of
examples how
FPM can load it. Would it require at least an index.php with <?php include 'myapp.bin'; ?

Cheers,
Michał Marcin Brzuchalski

6 years ago by Rowan Tommins — view source — reply

unread

Hi,

I commit a new request path, the request is about execution opcode file
without php source code file
this RFC provides similar to class file of java and pyo file of python.
patch is: https://github.com/php/php-src/pull/6146

I'm sure someone who knows the internals better can clarify, but my understanding is that PHP OpCodes don't currently have any stability guarantee, so you can't rely on a binary taken from one version will run on another, even within a release.

In order to be useful, this will therefore need two things:

a header in the file identifying the engine version it was compiled for, with an error raised on any mismatch
a policy of how to manage that version number, and how users can know which versions their binary files will work on

There's probably a limit to how stable we can (or want to) make the VM, so these files will never be as portable as a Java class file or .Net assembly.

Regards,

--
Rowan Tommins
[IMSoP]

6 years ago by Nikita Popov — view source — reply

unread

On Thu, Oct 1, 2020 at 1:44 PM Rowan Tommins rowan.collins@gmail.com
wrote:

Hi,

I commit a new request path, the request is about execution opcode file
without php source code file
this RFC provides similar to class file of java and pyo file of python.
patch is: https://github.com/php/php-src/pull/6146

I'm sure someone who knows the internals better can clarify, but my
understanding is that PHP OpCodes don't currently have any stability
guarantee, so you can't rely on a binary taken from one version will run on
another, even within a release.

In order to be useful, this will therefore need two things:

a header in the file identifying the engine version it was compiled for,
with an error raised on any mismatch

a policy of how to manage that version number, and how users can know
which versions their binary files will work on

There's probably a limit to how stable we can (or want to) make the VM, so
these files will never be as portable as a Java class file or .Net assembly.

This is correct. PHP uses the "system ID" to determine whether it is
possible to reuse compiled opcodes. This system ID is used by the existing
file cache.

The linked pull request however explicitly disables this check, which is
not safe. And with that check enabled, this feature would probably have
limited usefulness, as the cached file would be bound to a specific PHP
patch release (and to some degree to which extensions are loaded).

It would be good to know what the actual use-case for this is. If the goal
here is to distribute pre-compiled PHP code without the source code, then
this is not going to work without making the opcode format stable (which, I
think, is pretty unlikely.) If the goal is improving cold startup
performance, then I wonder what the benefit over the existing file cache is.

Regards,
Nikita

6 years ago by Dik Takken — view source — reply

unread

It would be good to know what the actual use-case for this is. If the goal
here is to distribute pre-compiled PHP code without the source code, then
this is not going to work without making the opcode format stable (which, I
think, is pretty unlikely.) If the goal is improving cold startup
performance, then I wonder what the benefit over the existing file cache is.

The only use case I see is to package commercial closed source
applications as Docker containers. That allows the packager to ship the
compiled code along with the exact PHP version and configuration that
can run it reliably.

Regards,
Dik Takken

6 years ago by Levi Morrison via internals — view source — reply

unread

It would be good to know what the actual use-case for this is. If the goal
here is to distribute pre-compiled PHP code without the source code, then
this is not going to work without making the opcode format stable (which, I
think, is pretty unlikely.) If the goal is improving cold startup
performance, then I wonder what the benefit over the existing file cache is.

The only use case I see is to package commercial closed source
applications as Docker containers. That allows the packager to ship the
compiled code along with the exact PHP version and configuration that
can run it reliably.

Regards,
Dik Takken

Note that for PHP 8.0 we made the system ID change even more than it
used to because doing so fixes sigsegvs when switching what extensions
are loaded (common when trialing an APM product, or loading and
unloading xdebug) when file cache is used. This means that it is even
more important to have sources than in the past, and necessarily so
because of real segmentation faults.

So I second Nikita's opinion: this is not going to work well without
stabilizing the opcode format. This may be possible, but it would be a
lot of work; it's not happening soon. If you do it anyway be prepared
for plenty of crash complaints.

6 years ago by Pierre — view source — reply

unread

Le 01/10/2020 à 18:32, Levi Morrison via internals a écrit :

Note that for PHP 8.0 we made the system ID change even more than it
used to because doing so fixes sigsegvs when switching what extensions
are loaded (common when trialing an APM product, or loading and
unloading xdebug) when file cache is used. This means that it is even
more important to have sources than in the past, and necessarily so
because of real segmentation faults.

So I second Nikita's opinion: this is not going to work well without
stabilizing the opcode format. This may be possible, but it would be a
lot of work; it's not happening soon. If you do it anyway be prepared
for plenty of crash complaints.

Hello,

I know this may sound naive, but would it be a bad idea to stabilize
opcodes (which probably is much harder than it looks), having the VM
being built following a thorough specification ? I could open the door
to many things, such as third party compilers or optimizers ?

I don't know much Zend VM internals, but having the possibility to have
PHP binaries other than phar sound compelling to me, I wish I could
deliver upgrades using a single binary instead of a 100k PHP files
delivery, especially in docker environments.

It may also open the door to autoload-less execution, link time-like
optimizations, and even open a slight door toward reflection around
modules/packages ?

Regards,

Pierre

6 years ago by Rowan Tommins — view source — reply

unread

The only use case I see is to package commercial closed source
applications as Docker containers. That allows the packager to ship the
compiled code along with the exact PHP version and configuration that
can run it reliably.

Could this not be achieved by freezing the existing OpCache file cache into
the Docker image and telling OpCache not to validate timestamps, so that it
always loads from the cache? That way, your application wouldn't even need
to be modified, as the require_once lines would still refer to the original
source paths.

Regards,

Rowan Tommins
[IMSoP]

6 years ago by David Rodrigues — view source — reply

unread

Hello folks,

Instead of an opcode without a php source file, that I imagine is to
protect the code itself, why not a method to encrypt phar files (not like a
password). I do not know if exists a secure method to decrypt to execute
only, without reveals the original source code, but maybe it could be done.
So opcode could be generated based on encrypted phar to give more speed.

Atenciosamente,
David Rodrigues

Em sex., 2 de out. de 2020 às 13:52, Rowan Tommins rowan.collins@gmail.com
escreveu:

The only use case I see is to package commercial closed source
applications as Docker containers. That allows the packager to ship the
compiled code along with the exact PHP version and configuration that
can run it reliably.

Could this not be achieved by freezing the existing OpCache file cache into
the Docker image and telling OpCache not to validate timestamps, so that it
always loads from the cache? That way, your application wouldn't even need
to be modified, as the require_once lines would still refer to the original
source paths.

Regards,

Rowan Tommins
[IMSoP]

6 years ago by Walter Parker — view source — reply

unread

Hello folks,

Instead of an opcode without a php source file, that I imagine is to
protect the code itself, why not a method to encrypt phar files (not like a
password). I do not know if exists a secure method to decrypt to execute
only, without reveals the original source code, but maybe it could be done.
So opcode could be generated based on encrypted phar to give more speed.

The flaw with decryption is that the image will have to get the
decryption key from somewhere. The history of the DVD industry over
the past 25 years has shown that getting that key to an untrusted
device and then keeping it secret is much harder than it looks (when
people care enough to go after the key, even hardware locks can be
made to fail). Or look at the phone industry, even phones with
hardware & software locked crypto get broken open.
Note, if the device is trusted, then encryption and decryption are not
needed (as you would then be able to trust the device to not give out
the source code).

Also, note that reverse engineering (decompling) raw binaries is a
thing (there are tools to do that). You need to decide on you threat
model for your code to decide on how much it is worth to lock your
code.

Walter

--
The greatest dangers to liberty lurk in insidious encroachment by men
of zeal, well-meaning but without understanding. -- Justice Louis D.
Brandeis

6 years ago by Rowan Tommins — view source — reply

unread

Opcode binaries can be published when binding to a PHP interpreter
(version dependent) release product, rather than just the PHP source
code(for example, JAVA and .NET is version-dependent)

Just to clarify the comparison with other languages:

Both the .Net CLR and the JVM are designed to be backwards compatible,
so that an assembly / class compiled for version 1 should run without
re-compilation on version 2 (but not the other way around)
Until recently, the .Net CLR was a very stable target, which had gone
through only 4 versions in nearly 20 years (confusingly numbered 1.0,
1.1, 2.0, and 4.0)
The JVM is specified independently of the Java language, and has
multiple interoperable implementations

This is very different from PHP OpCodes, which are not documented
outside the engine's source, and have no provision for backwards
compatibility. Not only are they likely to change between annual
releases (e.g. 7.3 to 7.4) but a change in 7.4.23 could conceivably
change the VM in such a way that code compiled on 7.4.22 would no longer
run correctly.

The Python .pyc (and .pyo, which is just a .pyc created with
optimisation turned on) is a better comparison - like PHP, it is an
implementation-dependent bytecode, although it appears there is some
guarantee of stability between bugfix releases. Its primary use seems to
be accelerating runtime on a single machine, not distributing a binary
elsewhere, and OpCache already provides that for PHP.

Regards,

--
Rowan Tommins (né Collins)
[IMSoP]

6 years ago by chopins.xiao@gmail.com — view source — reply

unread

For example, Java,.NET, etc., I mean that opcode compiled files can be bound to fixed versions without considering the compatibility of different versions.
The current opcode file load in Opcache is as follows:
[request]---->[find OP cache]----->[load OP]---->[verify OP]---->[exec OP]

Patch changed to:

[request]---+--->[is-OP-file]-------->[load-the-file-to-OP]-------------> [verify-OP]---->[exec-OP]
|--->[not-OP-file---->[find-OP-cache]------>[load-OP]------> /

This prejudgement process can be closed with the php.ini configuration item without affecting the existing system
So it's not divorced from the idea of Opcode as cached data.
If a different version of opcode is executed, the patch reports an E_STRICT error.

发件人: Rowan Tommins rowan.collins@gmail.com
发送时间: 2020年10月2日 2:46
收件人: PHP internals internals@lists.php.net
主题: [PHP-DEV] RFC: execution opcode file without php source code file

Opcode binaries can be published when binding to a PHP interpreter
(version dependent) release product, rather than just the PHP source
code(for example, JAVA and .NET is version-dependent)

Just to clarify the comparison with other languages:

Both the .Net CLR and the JVM are designed to be backwards compatible,
so that an assembly / class compiled for version 1 should run without
re-compilation on version 2 (but not the other way around)
Until recently, the .Net CLR was a very stable target, which had gone
through only 4 versions in nearly 20 years (confusingly numbered 1.0,
1.1, 2.0, and 4.0)
The JVM is specified independently of the Java language, and has
multiple interoperable implementations

Regards,

--
Rowan Tommins (né Collins)
[IMSoP]

To unsubscribe, visit: https://www.php.net/unsub.php

6 years ago by Rowan Tommins — view source — reply

unread

For example, Java,.NET, etc., I mean that opcode compiled files can be
bound to fixed versions without considering the compatibility of different
versions.

As I said already, this comparison is simply incorrect: both .net and Java
have been carefully engineered so that code compiled for one version can be
run on later versions.

PHP's VM does not even guarantee compatibility from one bugfix to the next,
or with and without a particular debugging or extension loaded.

If a different version of opcode is executed, the patch reports an
E_STRICT error.

To be clear, the possible consequences of loading opcodes on the incorrect
version would include crashing with a segmentation fault. That's why the
existing code, which your patch by-passes, refuses to load the file at all
if the "system_id" does not match. An E_STRICT is completely inappropriate
for this scenario.

Regards,

Rowan Tommins
[IMSoP]

RFC: execution opcode file without php source code file View as Markdown

Regards,

Regards,

Regards,