# alternative to the fopen() hack in autoloaders - Thread: https://externals.io/message/45992 - Original archive: https://news-web.php.net/php.internals/45992 - Started by Lukas Kahwe Smith on 2009-11-10T16:45:20+00:00 - Messages: 44 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/45992.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #45992 — Lukas Kahwe Smith — 2009-11-10T16:45:20+00:00 URL: https://externals.io/message/45992#45992 Ahoi, I have written an RFC for a more efficient solution to get rid of the common `fopen()` hack inside autoloaders: if ($fp = @fopen($file, 'r', true)) { fclose($fp); include $file; } Here is the gist of the proposal: In order to solve the above issues this RFC proposes the addition of a new construct/function for now called “autoload_include” for lack of a better name that largely behaves like the “include” does today with the following differences, that when the include failed because of a missing file no warning is raised and php null is returned. Further details can be found on the wiki: http://wiki.php.net/rfc/autoload_include As stated in the RFC, I am not happy with the name "autoload_include". Suggestions welcome! regards, Lukas Kahwe Smith mls@pooteeweet.org --- ## #45993 — Guilherme Blanco — 2009-11-10T17:00:34+00:00 URL: https://externals.io/message/45992#45993 · In reply to: #45992 include_silent is the name I though most intuitive once I finished to read the RFC. But it may not be the best too. It just need to be verbose. Cheers, > Ahoi, > > I have written an RFC for a more efficient solution to get rid of the common > `fopen()` hack inside autoloaders: > if ($fp = @fopen($file, 'r', true)) { > fclose($fp); > include $file; > } > > Here is the gist of the proposal: > In order to solve the above issues this RFC proposes the addition of a new > construct/function for now called “autoload_include” for lack of a better > name that largely behaves like the “include” does today with the following > differences, that when the include failed because of a missing file no > warning is raised and php null is returned. > > Further details can be found on the wiki: > http://wiki.php.net/rfc/autoload_include > > As stated in the RFC, I am not happy with the name "autoload_include". > Suggestions welcome! > > regards, > Lukas Kahwe Smith > mls@pooteeweet.org > > > > > -- > > > > -- Guilherme Blanco - Web Developer CBC - Certified Bindows Consultant Cell Phone: +55 (16) 9215-8480 MSN: guilhermeblanco@hotmail.com URL: http://blog.bisna.com São Paulo - SP/Brazil --- ## #45997 — Lukas Kahwe Smith — 2009-11-10T17:53:02+00:00 URL: https://externals.io/message/45992#45997 > > What's the problem with `file_exists()` ? i have explained this in the RFC. for one it does not support the include path setting. furthermore this does not solve the race condition and requires more FS operations than my proposal would. regards Lukas --- ## #45999 — Stanislav Malyshev — 2009-11-10T18:16:58+00:00 URL: https://externals.io/message/45992#45999 Hi! > Alternatively include() could be extended to allow resources, so the > above would turn info > > if ($fp = @fopen($file, 'r', true)) { > include($fp); > fclose($fp); > } This would break security distinction between file ops and include ops, when URLs are allowed for open but not include. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #46000 — Rasmus Lerdorf — 2009-11-10T18:20:43+00:00 URL: https://externals.io/message/45992#46000 · In reply to: #45999 Stanislav Malyshev wrote: > Hi! > > > […] > > This would break security distinction between file ops and include ops, > when URLs are allowed for open but not include. Yes, I was going to say, this sounds like a very bad idea security-wise. We'd be back to the local vs. remote resource identification problem. -Rasmus --- ## #46001 — Igor Feghali — 2009-11-10T18:37:34+00:00 URL: https://externals.io/message/45992#46001 · In reply to: #45992 > As stated in the RFC, I am not happy with the name "autoload_include". > Suggestions welcome! That reminds me of the set theory... how about: contain $file; or superset $file; there are also the not so original (as found in many other languages) keywords "import" / "load". Best, ~IF. --- ## #46002 — Stanislav Malyshev — 2009-11-10T18:49:28+00:00 URL: https://externals.io/message/45992#46002 · In reply to: #45993 Hi! > > […] Maybe it'd be easier to just add parameter to file_exists that allows it to use include_path? I think that's at least what ZF is trying to do with that fopen - it tries to find out if such file exists in the path and if so - include it. I don't think you need to create special language construct for that - you can shut off warning with @. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #46003 — Lukas Kahwe Smith — 2009-11-10T18:57:36+00:00 URL: https://externals.io/message/45992#46003 · In reply to: #46002 > Hi! > > > […] > > Maybe it'd be easier to just add parameter to file_exists that > allows it to use include_path? I think that's at least what ZF is > trying to do with that fopen - it tries to find out if such file > exists in the path and if so - include it. > I don't think you need to create special language construct for that > - you can shut off warning with @. yes that would solve the issue partially. the race condition would still remain, but its admitedly a rare case .. well I guess not so rare if you have a busy site and try to implement some caching for generated files .. then again these kinds of files are rarely loaded via autoload (probably more used for some generated arrays and stuff like that). it would also not solve the imho needless file system operations. that being said, i brought up adding such a flag to `file_exists()` a few times before and each attempt has been shot down saying that fopen () should have never gotten this flag and this portion of the code should not mess with the include path. regards, Lukas Kahwe Smith mls@pooteeweet.org --- ## #46004 — Stanislav Malyshev — 2009-11-10T19:07:42+00:00 URL: https://externals.io/message/45992#46004 · In reply to: #46003 Hi! > yes that would solve the issue partially. the race condition would still > remain, but its admitedly a rare case .. well I guess not so rare if you I would have hard time thinking of application that deletes its own include files frequently from other processes and we are supposed to handle that deterministically. But even then worst thing would happen is that include fails. > (probably more used for some generated arrays and stuff like that). it > would also not solve the imho needless file system operations. You could cache file_exists using all kinds of external caching mechanisms if you want to. > that being said, i brought up adding such a flag to `file_exists()` a few > times before and each attempt has been shot down saying that `fopen()` > should have never gotten this flag and this portion of the code should > not mess with the include path. Why `fopen()` should have never gotten this flag? I don't remember any argument on that. Also, if we have include path I see no reason why we shouldn't have code that can work with it. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #46005 — Lukas Kahwe Smith — 2009-11-10T19:16:52+00:00 URL: https://externals.io/message/45992#46005 · In reply to: #46004 > Hi! > > > […] > > I would have hard time thinking of application that deletes its own > include files frequently from other processes and we are supposed to > handle that deterministically. But even then worst thing would > happen is that include fails. there are many approaches to caching, one of which is delete to invalidate and regenerate before the next use. again as the RFC makes it clear .. the purpose is to be able to differentiate between a syntax error and a missing file. > > […] > > You could cache file_exists using all kinds of external caching > mechanisms if you want to. sure .. but that requires yet more code. but in that case i might as well iterate over the include path to be able to determine the absolute path and cache that, which is something i also want to enable in an efficient manner with the optional addition of returning the file loaded instead of true in case the file does not return anything explicitly. again something that i mentioned in the RFC. > > […] > > Why `fopen()` should have never gotten this flag? I don't remember any > argument on that. Also, if we have include path I see no reason why > we shouldn't have code that can work with it. IIRC Derick was one of the most vocal against adding this to `file_exists()` and complaining about the fact that its even available for `fopen()`. regards, Lukas Kahwe Smith mls@pooteeweet.org --- ## #46006 — Stanislav Malyshev — 2009-11-10T19:38:35+00:00 URL: https://externals.io/message/45992#46006 · In reply to: #46005 Hi! > there are many approaches to caching, one of which is delete to > invalidate and regenerate before the next use. again as the RFC makes it > clear .. the purpose is to be able to differentiate between a syntax > error and a missing file. If you writing your own cache basing on includes, you can write sequence that handles deletion correctly, knows if your own data exist or not and doesn't need include path functions for that. > sure .. but that requires yet more code. but in that case i might as More functionality requires more code. I think it's better than turning the language into kitchen sink of similar language constructs, each has a little tweak to suit one particular use case. Next thing we would have framework_include, database_include, template_system_include, my_personal_homepage_include and so on. I think we need to do very generic stuff in the language, less generic in functions and yet less generic - in user code. > well iterate over the include path to be able to determine the absolute > path and cache that, which is something i also want to enable in an > efficient manner with the optional addition of returning the file loaded > instead of true in case the file does not return anything explicitly. > again something that i mentioned in the RFC. We could also have function file_find() (or any other name, let the bikeshedding begin) or something that would resolve filename against include path and return full name if such exists or false if none exists. That seems generic enough operation to have a function for. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #46007 — Lukas Kahwe Smith — 2009-11-10T19:48:46+00:00 URL: https://externals.io/message/45992#46007 · In reply to: #46006 > > […] > > If you writing your own cache basing on includes, you can write > sequence that handles deletion correctly, knows if your own data > exist or not and doesn't need include path functions for that. > > > […] > > More functionality requires more code. I think it's better than > turning the language into kitchen sink of similar language > constructs, each has a little tweak to suit one particular use case. > Next thing we would have framework_include, database_include, > template_system_include, my_personal_homepage_include and so on. I > think we need to do very generic stuff in the language, less generic > in functions and yet less generic - in user code. well imho neither require nor include have been written with the "right" API in mind to solve todays needs. for all i care we could also break BC for include and adjust it according to my proposal in PHP6. anyways .. you are glossing over several draw backs in the current possible approaches, each saying you can solve that in user land with more code. yet my proposal would be a tiny change in php core (probably alot of code could be shared) and would still surpass the user land approaches. i do not know if there is anything possible to make include itself more flexible since its a language construct and not a "normal" function. > > […] > > We could also have function file_find() (or any other name, let the > bikeshedding begin) or something that would resolve filename against > include path and return full name if such exists or false if none > exists. That seems generic enough operation to have a function for. ok .. so your objection to the RFC is solely because it introduces a new language construct? regards, Lukas Kahwe Smith mls@pooteeweet.org --- ## #46008 — Greg Beaver — 2009-11-10T19:56:57+00:00 URL: https://externals.io/message/45992#46008 · In reply to: #45999 Stanislav Malyshev wrote: > Hi! > > > […] > > This would break security distinction between file ops and include ops, > when URLs are allowed for open but not include. > Not really - the wrapper used to open the file pointer is stored in the resource, so we can just check it against the same restrictions we would for static urls. I think this idea deserves another look. Greg --- ## #46010 — Stanislav Malyshev — 2009-11-10T20:28:26+00:00 URL: https://externals.io/message/45992#46010 · In reply to: #46008 Hi! > > […] Additional thing - it probably would be a problem for bytecode caches, since they'd have to: a. watch all file opens, in case some of these will later be used for include b. somehow be able to get filename back from open stream to get the cached file. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #46011 — Stanislav Malyshev — 2009-11-10T20:31:17+00:00 URL: https://externals.io/message/45992#46011 · In reply to: #46007 Hi! > ok .. so your objection to the RFC is solely because it introduces a new > language construct? No, my objection is that it is not necessary to introduce a language construct, and the construct introduced is not the right one. If frameworks want to find out if file exists or get its name - we should give them API to do that. If they want just to silence the errors - they already have the construct to do that. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #46012 — Lukas Kahwe Smith — 2009-11-10T20:56:47+00:00 URL: https://externals.io/message/45992#46012 · In reply to: #46011 > > […] > > No, my objection is that it is not necessary to introduce a language > construct, and the construct introduced is not the right one. If > frameworks want to find out if file exists or get its name - we > should give them API to do that. If they want just to silence the > errors - they already have the construct to do that. They dont want to determine if a file exists. They want to be able to handle the case of a missing file being included differently than a syntax error. Since php core does not provide such features, the only way to do this is using the `fopen()` hack, iterating over the include path .. or I guess track errors with @. My proposal actually makes it possible to do what the frameworks need with less overhead. There is no way to do what these frameworks need inside userland without the above mentioned hacks. The fact that almost all frameworks are forced to use these hacks and quite a significant number of developers use frameworks (resulting in quite a number of apps end users install also using these frameworks) implies that we are not talking about an edge case here. Now I also made it clear that its not about blindly silencing "errors" but there is a need to differentiate between different error causes. Of course having to use @ for such a common use case is also not ideal. Its just that the error handling we provide internally isnt really able to handle this scenario well: 1) checking before adds overhead and opens issue with potential race conditions 2) silencing the error and using track errors adds overhead and potential issues with custom error handlers Anyways, lets see if there are other comments in the coming days. I think you have made your priorities clear. At least I understand them, but do not share them. regards, Lukas Kahwe Smith mls@pooteeweet.org --- ## #46013 — Stanislav Malyshev — 2009-11-10T21:28:12+00:00 URL: https://externals.io/message/45992#46013 · In reply to: #46012 Hi! > They dont want to determine if a file exists. They want to be able to > handle the case of a missing file being included differently than a > syntax error. Since php core does not provide such features, the only Let's see why they need to handle missing file being included differently? Or, even more interestingly - why they need to include file that is missing if they know they'd have to do something different? Maybe they would want to know if file exists or nor before trying to include it, because they don't know which exactly file they are going to include? The goal here, at least for ZF, is to find the right file and include it. Both APIs that I proposed allow to express it directly - find file and include it. Your API would make them do it through chain of trial and error by trying to include a string of non-existing files before actually trying a successful one, thus mixing three functions - finding files, including files and error reporting - into one. How it's better? > possible to do what the frameworks need with less overhead. There is no > way to do what these frameworks need inside userland without the above > mentioned hacks. The fact that almost all frameworks are forced to use I just showed you at least two different ways to do it, with different APIs. You may not agree it's right APIs but please don't make it appear as your solution is only possible one in existence. It is not. > Now I also made it clear that its not about blindly silencing "errors" > but there is a need to differentiate between different error causes. Of How you are going to differentiate? Your proposal only has one return value (null) which is conveniently coincides with return value that can be produced by include too by doing return;. You don't know why it failed, you don't even really know did it fail or not. You just know you got null in return. > 1) checking before adds overhead and opens issue with potential race > conditions I don't think it's real use case - please show me one framework that uses plugin includes through include path while other processes are deleting their plugins randomly. I think you are trying to solve non-existing problem here. Caching/templating APIs usually know exactly where their files are and don't need to look for them in include path. > 2) silencing the error and using track errors adds overhead and > potential issues with custom error handlers Again, what frameworks need is to find the right file among variety of possibilities in include path and then include it. I think it can be well served by using existing "include" with addition of "find" API and there's no necessity for new language construct to serve one specific use case. In a month, some framework would have slightly different use case and you'd invent another packaged language construct for this? There's a reason why language constructs tend to be primitive - because you can assemble a lot of use cases from simple primitives, but it's hard to deal with use-case-tailored constructs. We have four inclusion constructs in language as it is, I think it's enough. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #46014 — Alban — 2009-11-10T21:41:34+00:00 URL: https://externals.io/message/45992#46014 · In reply to: #46012 Le Tue, 10 Nov 2009 21:56:47 +0100, Lukas Kahwe Smith a écrit : > > > > […] > > > They dont want to determine if a file exists. They want to be able to > handle the case of a missing file being included differently than a > syntax error. Since php core does not provide such features, the only > way to do this is using the `fopen()` hack, iterating over the include > path .. or I guess track errors with @. My proposal actually makes it > possible to do what the frameworks need with less overhead. There is no > way to do what these frameworks need inside userland without the above > mentioned hacks. The fact that almost all frameworks are forced to use > these hacks and quite a significant number of developers use frameworks > (resulting in quite a number of apps end users install also using these > frameworks) implies that we are not talking about an edge case here. > > Now I also made it clear that its not about blindly silencing "errors" > but there is a need to differentiate between different error causes. Of > course having to use @ for such a common use case is also not ideal. Its > just that the error handling we provide internally isnt really able to > handle this scenario well: > > 1) checking before adds overhead and opens issue with potential race > conditions > 2) silencing the error and using track errors adds overhead and > potential issues with custom error handlers > > Anyways, lets see if there are other comments in the coming days. I > think you have made your priorities clear. At least I understand them, > but do not share them. > > regards, > Lukas Kahwe Smith > mls@pooteeweet.org Hi, I think if PHP throws exception instead of warning/error then this problem will not have existence reason : try { include 'file'; // if we are sure about data sort in get_included_files, just do : $fullFileName = array_pop(get_included_files()); // ... } catch(phpParseIncludeException $e) { // ... } catch(missingIncludeException $e) { // ... } catch(includeException $e) { // ... } regards, Alban Leroux seza@paradoxal.org --- ## #46015 — Lukas Kahwe Smith — 2009-11-10T21:59:30+00:00 URL: https://externals.io/message/45992#46015 · In reply to: #46014 > I think if PHP throws exception instead of warning/error then this > problem will not have existence reason : sure with exceptions we could provide more contextual information and also give a local way to handle the situation locally. that being said for now it was decided to only use exception for constructor errors in core php. regards, Lukas Kahwe Smith mls@pooteeweet.org --- ## #46016 — Lukas Kahwe Smith — 2009-11-10T22:10:37+00:00 URL: https://externals.io/message/45992#46016 · In reply to: #46013 > Hi! > > > […] > > Let's see why they need to handle missing file being included > differently? Or, even more interestingly - why they need to include > file that is missing if they know they'd have to do something > different? Maybe they would want to know if file exists or nor > before trying to include it, because they don't know which exactly > file they are going to include? > The goal here, at least for ZF, is to find the right file and > include it. Both APIs that I proposed allow to express it directly - > find file and include it. Your API would make them do it through > chain of trial and error by trying to include a string of non- > existing files before actually trying a successful one, thus mixing > three functions - finding files, including files and error reporting > - into one. How it's better? Because its tons more efficient, requires less code from the end user and all the various other reasons i have stated before. Again we seem to have different priorities but reiterating them doesnt provide value for this mailinglist. So your points are taken, I will add them to the RFC. We will see if other people have an opinion and then someone might call for a vote. > > […] > > How you are going to differentiate? Your proposal only has one > return value (null) which is conveniently coincides with return > value that can be produced by include too by doing return;. You > don't know why it failed, you don't even really know did it fail or > not. You just know you got null in return. I stated the differences to include. As such I did not state that syntax errors should be handled differently, so in the case of a syntax error false would be returned and a warning raised. > > […] > > I don't think it's real use case - please show me one framework that > uses plugin includes through include path while other processes are > deleting their plugins randomly. I think you are trying to solve non- > existing problem here. Caching/templating APIs usually know exactly > where their files are and don't need to look for them in include path. I never claimed that the race condition is a common scenario. Do note that I always called this an edge case, both on this list an in the RFC. The performance overhead however is real, both from additional function calls as well as additional filesystem access. > > […] > > Again, what frameworks need is to find the right file among variety > of possibilities in include path and then include it. I think it can > be well served by using existing "include" with addition of "find" > API and there's no necessity for new language construct to serve one > specific use case. In a month, some framework would have slightly > different use case and you'd invent another packaged language > construct for this? You are repeating yourself in the same email. You are not addressing 2) so just leave it out on your reply. To address your comment about if I am going to invent another construct. I have as well as other framework developers brought up this limitation in PHP for a while. You might have also noticed that this construct aka hack has been around for a while. So I am not trying to patch up the issues I discovered yesterday, I am trying to find a solution for a problem that has been know since years. As a result I do not share your concern that tomorrow we will need yet another different "magic" method to handle includes. regards, Lukas Kahwe Smith mls@pooteeweet.org --- ## #46017 — Greg Beaver — 2009-11-11T00:24:41+00:00 URL: https://externals.io/message/45992#46017 · In reply to: #46010 Stanislav Malyshev wrote: > Hi! > > > […] > > Additional thing - it probably would be a problem for bytecode caches, > since they'd have to: > a. watch all file opens, in case some of these will later be used for > include > b. somehow be able to get filename back from open stream to get the > cached file. Hi again, A good point. Fortunately, most streams do store the filename in the stream struct, so this would still probably be possible in the majority of cases. If not available, the original requested filename is used (the one from the fopen call), so opcode caches probably could just use that. Greg --- ## #46018 — Stanislav Malyshev — 2009-11-11T00:37:42+00:00 URL: https://externals.io/message/45992#46018 · In reply to: #46017 Hi! > A good point. Fortunately, most streams do store the filename in the > stream struct, so this would still probably be possible in the majority > of cases. If not available, the original requested filename is used > (the one from the fopen call), so opcode caches probably could just use > that. The problem is, by then the stream will already be opened, i.e. extra filesystem call is done. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #46019 — Greg Beaver — 2009-11-11T00:50:00+00:00 URL: https://externals.io/message/45992#46019 · In reply to: #46018 Stanislav Malyshev wrote: > Hi! > > > […] > > The problem is, by then the stream will already be opened, i.e. extra > filesystem call is done. Hi, Yes, but let's not forget we're talking about making this: into this: Of course 1 stat is worse than no stats. Perhaps better would be to introduce a function: if (can_include($file)) { include $file; } opcode caches could easily snag this, as we could provide a simple hook the way we do with stream_resolve_path(). That would actually make a 0-stat smart autoloader a possibility. Just in case it isn't obvious, can_include() would be the equivalent to an include_path search followed by `is_readable()`, which is essentially what the `fopen()` hack does now. can_include() would also remove the unnecessary opening of the file that `fopen()` performs. Greg --- ## #46020 — Stanislav Malyshev — 2009-11-11T01:20:29+00:00 URL: https://externals.io/message/45992#46020 · In reply to: #46019 Hi! > Yes, but let's not forget we're talking about making this: While with proper caching, it can be 0 filesystem accesses. > if (can_include($file)) { > include $file; > } That's exactly what I was proposing - to have function to find file in include path and probably return the resulting filename (in case framework wants to keep it, or cache, or use it as a key or whatever). -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #46021 — Lukas Kahwe Smith — 2009-11-11T07:27:07+00:00 URL: https://externals.io/message/45992#46021 · In reply to: #46019 > > Perhaps better would be to introduce a function: > > if (can_include($file)) { > include $file; > } > > opcode caches could easily snag this, as we could provide a simple > hook the way we do with stream_resolve_path(). That would actually > make a 0-stat smart autoloader a possibility. Just in case it isn't > obvious, can_include() would be the equivalent to an include_path > search followed by `is_readable()`, which is essentially what the fopen > () hack does now. can_include() would also remove the unnecessary > opening of the file that `fopen()` performs. so you are suggesting that the byte code cache also cache file searches performed by this function. this is starting to sound more interesting from my pov. regards lukas --- Next page: https://externals.io/message/45992.md?page=2