# RFC: execution opcode file without php source code file - Thread: https://externals.io/message/111965 - Original archive: https://news-web.php.net/php.internals/111965 - Started by chopins.xiao@live.com on 2020-10-01T09:36:20+00:00 - Messages: 14 - RFC: OPCache direct execution opcode file without php source code file (Declined) — https://externals.io/rfc/direct-execution-opcode.md - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · RFCs at https://externals.io/rfc.md · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #111965 — chopins.xiao@live.com — 2020-10-01T09:36:20+00:00 URL: https://externals.io/message/111965#111965 I commit a new request path, the request is about execution opcode file without php source code file this RFC provides similar to class file of java and pyo file of python. patch is: https://github.com/php/php-src/pull/6146 --- ## #111966 — michal.brzuchalski@gmail.com — 2020-10-01T10:36:12+00:00 URL: https://externals.io/message/111965#111966 · In reply to: #111965 Hi, czw., 1 paź 2020 o 11:36 肖 鑫鑫 napisał(a): > I commit a new request path, the request is about execution opcode file > without php source code file > this RFC provides similar to class file of java and pyo file of python. > patch is: https://github.com/php/php-src/pull/6146 I had exact the same feature in my mind and working implementation few years ago when PHP 7.0 appeared on horizon which didn't get a Zend Guard support. I like the idea a lot. One risk I see here is people putting whole framework or apps into one PHP file before calling `opcache_compile_file` to produce one big binary file. Would it be possible to move this feature to separate function and pass iterable list of files? I am not sure if with current OPCache file format it's possible but if that might reduce amount of steps required to produce one huge OPCache file which then can be loaded by interpreter. Second thing worth of noting is that PR examples use CLI SAPI and lacks of examples how FPM can load it. Would it require at least an index.php with ` Hi, > > czw., 1 paź 2020 o 11:36 肖 鑫鑫 napisał(a): > > > […] > > I had exact the same feature in my mind and working implementation few > years ago > when PHP 7.0 appeared on horizon which didn't get a Zend Guard support. > I like the idea a lot. > > One risk I see here is people putting whole framework or apps into one PHP > file before > calling `opcache_compile_file` to produce one big binary file. > > Would it be possible to move this feature to separate function and pass > iterable list of files? > I am not sure if with current OPCache file format it's possible but if that > might reduce amount > of steps required to produce one huge OPCache file which then can be loaded > by interpreter. > > Second thing worth of noting is that PR examples use CLI SAPI and lacks of > examples how > FPM can load it. Would it require at least an index.php with ` 'myapp.bin';` ? > > Cheers, > Michał Marcin Brzuchalski --- ## #111969 — Rowan Tommins — 2020-10-01T11:44:06+00:00 URL: https://externals.io/message/111965#111969 · In reply to: #111965 Hi, >I commit a new request path, the request is about execution opcode file >without php source code file >this RFC provides similar to class file of java and pyo file of python. >patch is: https://github.com/php/php-src/pull/6146 I'm sure someone who knows the internals better can clarify, but my understanding is that PHP OpCodes don't currently have any stability guarantee, so you can't rely on a binary taken from one version will run on another, even within a release. In order to be useful, this will therefore need two things: - a header in the file identifying the engine version it was compiled for, with an error raised on any mismatch - a policy of how to manage that version number, and how users can know which versions their binary files will work on There's probably a limit to how stable we can (or want to) make the VM, so these files will never be as portable as a Java class file or .Net assembly. Regards, -- Rowan Tommins [IMSoP] --- ## #111970 — Nikita Popov — 2020-10-01T12:30:28+00:00 URL: https://externals.io/message/111965#111970 · In reply to: #111969 On Thu, Oct 1, 2020 at 1:44 PM Rowan Tommins wrote: > Hi, > > > > […] > > I'm sure someone who knows the internals better can clarify, but my > understanding is that PHP OpCodes don't currently have any stability > guarantee, so you can't rely on a binary taken from one version will run on > another, even within a release. > > In order to be useful, this will therefore need two things: > > - a header in the file identifying the engine version it was compiled for, > with an error raised on any mismatch > - a policy of how to manage that version number, and how users can know > which versions their binary files will work on > > There's probably a limit to how stable we can (or want to) make the VM, so > these files will never be as portable as a Java class file or .Net assembly. > This is correct. PHP uses the "system ID" to determine whether it is possible to reuse compiled opcodes. This system ID is used by the existing file cache. The linked pull request however explicitly disables this check, which is not safe. And with that check enabled, this feature would probably have limited usefulness, as the cached file would be bound to a specific PHP patch release (and to some degree to which extensions are loaded). It would be good to know what the actual use-case for this is. If the goal here is to distribute pre-compiled PHP code without the source code, then this is not going to work without making the opcode format stable (which, I think, is pretty unlikely.) If the goal is improving cold startup performance, then I wonder what the benefit over the existing file cache is. Regards, Nikita --- ## #111976 — Dik Takken — 2020-10-01T15:13:41+00:00 URL: https://externals.io/message/111965#111976 · In reply to: #111970 > It would be good to know what the actual use-case for this is. If the goal > here is to distribute pre-compiled PHP code without the source code, then > this is not going to work without making the opcode format stable (which, I > think, is pretty unlikely.) If the goal is improving cold startup > performance, then I wonder what the benefit over the existing file cache is. The only use case I see is to package commercial closed source applications as Docker containers. That allows the packager to ship the compiled code along with the exact PHP version and configuration that can run it reliably. Regards, Dik Takken --- ## #111977 — Levi Morrison via internals — 2020-10-01T16:32:11+00:00 URL: https://externals.io/message/111965#111977 · In reply to: #111976 > > > > […] > > The only use case I see is to package commercial closed source > applications as Docker containers. That allows the packager to ship the > compiled code along with the exact PHP version and configuration that > can run it reliably. > > Regards, > Dik Takken Note that for PHP 8.0 we made the system ID change even more than it used to because doing so fixes sigsegvs when switching what extensions are loaded (common when trialing an APM product, or loading and unloading xdebug) when file cache is used. This means that it is even more important to have sources than in the past, and necessarily so because of real segmentation faults. So I second Nikita's opinion: this is not going to work well without stabilizing the opcode format. This may be possible, but it would be a lot of work; it's not happening soon. If you do it anyway be prepared for plenty of crash complaints. --- ## #111980 — Rowan Tommins — 2020-10-01T18:46:03+00:00 URL: https://externals.io/message/111965#111980 > 1. Opcode binaries can be published when binding to a PHP interpreter > (version dependent) release product, rather than just the PHP source > code(for example, JAVA and .NET is version-dependent) Just to clarify the comparison with other languages: - Both the .Net CLR and the JVM are designed to be backwards compatible, so that an assembly / class compiled for version 1 should run without re-compilation on version 2 (but not the other way around) - Until recently, the .Net CLR was a very stable target, which had gone through only 4 versions in nearly 20 years (confusingly numbered 1.0, 1.1, 2.0, and 4.0) - The JVM is specified independently of the Java language, and has multiple interoperable implementations This is very different from PHP OpCodes, which are not documented outside the engine's source, and have no provision for backwards compatibility. Not only are they likely to change between annual releases (e.g. 7.3 to 7.4) but a change in 7.4.23 could conceivably change the VM in such a way that code compiled on 7.4.22 would no longer run correctly. The Python .pyc (and .pyo, which is just a .pyc created with optimisation turned on) is a better comparison - like PHP, it is an implementation-dependent bytecode, although it appears there is some guarantee of stability between bugfix releases. Its primary use seems to be accelerating runtime on a single machine, not distributing a binary elsewhere, and OpCache already provides that for PHP. Regards, -- Rowan Tommins (né Collins) [IMSoP] --- ## #111981 — chopins.xiao@gmail.com — 2020-10-02T09:43:42+00:00 URL: https://externals.io/message/111965#111981 · In reply to: #111980 For example, Java,.NET, etc., I mean that opcode compiled files can be bound to fixed versions without considering the compatibility of different versions. The current opcode file load in Opcache is as follows: [request]---->[find OP cache]----->[load OP]---->[verify OP]---->[exec OP] Patch changed to: [request]---+--->[is-OP-file]-------->[load-the-file-to-OP]-------------> [verify-OP]---->[exec-OP] |--->[not-OP-file---->[find-OP-cache]------>[load-OP]------> / This prejudgement process can be closed with the php.ini configuration item without affecting the existing system So it's not divorced from the idea of Opcode as cached data. If a different version of opcode is executed, the patch reports an `E_STRICT` error. ________________________________ 发件人: Rowan Tommins 发送时间: 2020年10月2日 2:46 收件人: PHP internals 主题: [PHP-DEV] RFC: execution opcode file without php source code file > 1. Opcode binaries can be published when binding to a PHP interpreter > (version dependent) release product, rather than just the PHP source > code(for example, JAVA and .NET is version-dependent) Just to clarify the comparison with other languages: - Both the .Net CLR and the JVM are designed to be backwards compatible, so that an assembly / class compiled for version 1 should run without re-compilation on version 2 (but not the other way around) - Until recently, the .Net CLR was a very stable target, which had gone through only 4 versions in nearly 20 years (confusingly numbered 1.0, 1.1, 2.0, and 4.0) - The JVM is specified independently of the Java language, and has multiple interoperable implementations This is very different from PHP OpCodes, which are not documented outside the engine's source, and have no provision for backwards compatibility. Not only are they likely to change between annual releases (e.g. 7.3 to 7.4) but a change in 7.4.23 could conceivably change the VM in such a way that code compiled on 7.4.22 would no longer run correctly. The Python .pyc (and .pyo, which is just a .pyc created with optimisation turned on) is a better comparison - like PHP, it is an implementation-dependent bytecode, although it appears there is some guarantee of stability between bugfix releases. Its primary use seems to be accelerating runtime on a single machine, not distributing a binary elsewhere, and OpCache already provides that for PHP. Regards, -- Rowan Tommins (né Collins) [IMSoP] -- To unsubscribe, visit: https://www.php.net/unsub.php --- ## #111982 — Pierre — 2020-10-02T09:59:15+00:00 URL: https://externals.io/message/111965#111982 · In reply to: #111977 Le 01/10/2020 à 18:32, Levi Morrison via internals a écrit : > Note that for PHP 8.0 we made the system ID change even more than it > used to because doing so fixes sigsegvs when switching what extensions > are loaded (common when trialing an APM product, or loading and > unloading xdebug) when file cache is used. This means that it is even > more important to have sources than in the past, and necessarily so > because of real segmentation faults. > > So I second Nikita's opinion: this is not going to work well without > stabilizing the opcode format. This may be possible, but it would be a > lot of work; it's not happening soon. If you do it anyway be prepared > for plenty of crash complaints. > Hello, I know this may sound naive, but would it be a bad idea to stabilize opcodes (which probably is much harder than it looks), having the VM being built following a thorough specification ? I could open the door to many things, such as third party compilers or optimizers ? I don't know much Zend VM internals, but having the possibility to have PHP binaries other than phar sound compelling to me, I wish I could deliver upgrades using a single binary instead of a 100k PHP files delivery, especially in docker environments. It may also open the door to autoload-less execution, link time-like optimizations, and even open a slight door toward reflection around modules/packages ? Regards, Pierre --- ## #111986 — Rowan Tommins — 2020-10-02T16:41:25+00:00 URL: https://externals.io/message/111965#111986 · In reply to: #111981 > For example, Java,.NET, etc., I mean that opcode compiled files can be > bound to fixed versions without considering the compatibility of different > versions. > As I said already, this comparison is simply incorrect: both .net and Java have been carefully engineered so that code compiled for one version can be run on later versions. PHP's VM does not even guarantee compatibility from one bugfix to the next, or with and without a particular debugging or extension loaded. > If a different version of opcode is executed, the patch reports an > `E_STRICT` error. > To be clear, the possible consequences of loading opcodes on the incorrect version would include crashing with a segmentation fault. That's why the existing code, which your patch by-passes, refuses to load the file at all if the "system_id" does not match. An `E_STRICT` is completely inappropriate for this scenario. Regards, -- Rowan Tommins [IMSoP] --- ## #111987 — Rowan Tommins — 2020-10-02T16:51:32+00:00 URL: https://externals.io/message/111965#111987 · In reply to: #111976 > The only use case I see is to package commercial closed source > applications as Docker containers. That allows the packager to ship the > compiled code along with the exact PHP version and configuration that > can run it reliably. > Could this not be achieved by freezing the existing OpCache file cache into the Docker image and telling OpCache not to validate timestamps, so that it always loads from the cache? That way, your application wouldn't even need to be modified, as the require_once lines would still refer to the original source paths. Regards, -- Rowan Tommins [IMSoP] --- ## #111988 — David Rodrigues — 2020-10-02T17:07:54+00:00 URL: https://externals.io/message/111965#111988 · In reply to: #111987 Hello folks, Instead of an opcode without a php source file, that I imagine is to protect the code itself, why not a method to encrypt phar files (not like a password). I do not know if exists a secure method to decrypt to execute only, without reveals the original source code, but maybe it could be done. So opcode could be generated based on encrypted phar to give more speed. Atenciosamente, David Rodrigues Em sex., 2 de out. de 2020 às 13:52, Rowan Tommins escreveu: > > > > […] > > > Could this not be achieved by freezing the existing OpCache file cache into > the Docker image and telling OpCache not to validate timestamps, so that it > always loads from the cache? That way, your application wouldn't even need > to be modified, as the require_once lines would still refer to the original > source paths. > > Regards, > -- > Rowan Tommins > [IMSoP] --- ## #111989 — Walter Parker — 2020-10-02T19:29:28+00:00 URL: https://externals.io/message/111965#111989 · In reply to: #111988 > > Hello folks, > > Instead of an opcode without a php source file, that I imagine is to > protect the code itself, why not a method to encrypt phar files (not like a > password). I do not know if exists a secure method to decrypt to execute > only, without reveals the original source code, but maybe it could be done. > So opcode could be generated based on encrypted phar to give more speed. > > The flaw with decryption is that the image will have to get the decryption key from somewhere. The history of the DVD industry over the past 25 years has shown that getting that key to an untrusted device and then keeping it secret is much harder than it looks (when people care enough to go after the key, even hardware locks can be made to fail). Or look at the phone industry, even phones with hardware & software locked crypto get broken open. Note, if the device is trusted, then encryption and decryption are not needed (as you would then be able to trust the device to not give out the source code). Also, note that reverse engineering (decompling) raw binaries is a thing (there are tools to do that). You need to decide on you threat model for your code to decide on how much it is worth to lock your code. Walter -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis