# Performance Consideration of 1 class per file - Thread: https://externals.io/message/9525 - Original archive: https://news-web.php.net/php.internals/9525 - Started by Alan Knowles on 2004-04-24T03:37:42+00:00 - Messages: 13 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #9525 — Alan Knowles — 2004-04-24T03:37:42+00:00 URL: https://externals.io/message/9525#9525 PEAR is considering stipulating 1 class per file, for the packages. - one of the concerns raised is performance. While this is true for a non-cache compiled situation (where performance is less of an issue) - would anyone care to comment on the implications when using the Zend Optimizer or APC, on how significant the impact of this would be. - looking at typical application where the number of files included in the code with may jump from 20->30 included files.. From my understanding, if you where looking for performance, you would probably have a cache expiry which would mean very few stat (if any) calls based on these addition includes.. - hence negating the issue.. Thanks Alan -- Can you help out? Need Consulting Services or Know of a Job? http://www.akbkhome.com --- ## #9526 — Rasmus Lerdorf — 2004-04-24T04:58:06+00:00 URL: https://externals.io/message/9525#9526 · In reply to: #9525 It can be significant. There are a couple of issues: 1. The included_files list gets updated each time you include a file. In order to make sure that the same file included by different paths or symlinks don't conflict we do a `realpath()` on the file to be included. That means stats on every component up to and including the file itself. I can't speak for all the opcode caches, but at least APC doesn't do anything to alleviate this. 2. APC uses the file's device and inode as the lookup key into shared memory to find the opcodes for the file, so a stat has to be done on each and every file that is cached. There is currently no such thing as a request-spanning stat cache. So yes, jumping from 20 to 30 include files could very well bring a rather significant performance hit. Anybody looking for real performance out of PHP pretty much has to address the first issue. It is easy enough to do, simply get rid of the realpath lookup in that situation and just live with the fact that if someone includes the same file via different paths it won't catch it. I can see that it is convenient to have PHP figure this out for you, but the price of that convenience is not worth it in my opinion. -Rasmus > PEAR is considering stipulating 1 class per file, for the packages. - > one of the concerns raised is performance. While this is true for a > non-cache compiled situation (where performance is less of an issue) - > would anyone care to comment on the implications when using the Zend > Optimizer or APC, on how significant the impact of this would be. > > - looking at typical application where the number of files included in > the code with may jump from 20->30 included files.. > > From my understanding, if you where looking for performance, you would > probably have a cache expiry which would mean very few stat (if any) > calls based on these addition includes.. - hence negating the issue.. > > Thanks > Alan > > > > -- > Can you help out? > Need Consulting Services or Know of a Job? > http://www.akbkhome.com --- ## #9528 — Christian Schneider — 2004-04-24T07:13:16+00:00 URL: https://externals.io/message/9525#9528 · In reply to: #9526 Rasmus Lerdorf wrote: > 1. The included_files list gets updated each time you include a file. In > order to make sure that the same file included by different paths or > symlinks don't conflict we do a `realpath()` on the file to be included. That's done by PHP, not APC, right? Does this only apply to require_once or require as well? > 2. APC uses the file's device and inode as the lookup key into shared > memory to find the opcodes for the file, so a stat has to be done on Hmm.. If the stat on the file and the check for device/inode would be done first then you wouldn't have to do a realpath, right? But I guess that's not easily done until after the realpath check. > So yes, jumping from 20 to 30 include files could very well bring a rather > significant performance hit. I guess that's only important if your PHP code is really simple and you don't do something like e.g. DB queries because otherwise that'd be 90% of the running time anyway, right? I guess someone _that_ considered about performance could easily do a cat *.php | grep -v require | php -w >app.lib or the like and include app.lib. - Chris --- ## #9529 — Rasmus Lerdorf — 2004-04-24T07:42:35+00:00 URL: https://externals.io/message/9525#9529 · In reply to: #9528 > Rasmus Lerdorf wrote: > > […] > > That's done by PHP, not APC, right? Does this only apply to require_once > or require as well? Right, it is done by PHP which is why APC can't do anything about it. It happens on all files parsed by PHP no matter how they are included. > > […] > > Hmm.. If the stat on the file and the check for device/inode would be > done first then you wouldn't have to do a realpath, right? But I guess > that's not easily done until after the realpath check. Well, the included_files list is pathname based and when checking to see if a file has already been included we do a string compare. We could potentially follow apc's lead and use device+inode instead and thereby not need the realpath call and still maintain the current functionality. There are some other issues with this though. Like, for example, when you edit a file most editors will actually create a new file and delete the old, so you end up with the same pathname having a new inode. We could state that you should never edit a file on a live production web server, but everyone does and I can just see the stream of bug reports this might spur. > > […] > > I guess that's only important if your PHP code is really simple and you > don't do something like e.g. DB queries because otherwise that'd be 90% > of the running time anyway, right? Sure, you could argue that the stat calls will get lost in the noise of a complex script. But they really can add up fast. Bouncing along the include_path looking for include files adds extra stats, open_basedir is another killer when it comes to stats. Even just having "." as the first piece of the include_path is going to cost you an extra stat when including stuff from PEAR (unless you hardcode the pear path). And the sort of people that need to worry about this tend to cache stuff from backends like crazy which, if done right, means that the backend is only touched once in a while and the bulk of the requests will pull data right out of a fast memory-based session mechanism or use other similar tricks. However, if at the same time you are stuck with 30 includes and each of these cost you 10 stat calls, that is 300 disk-touching system calls per request that you really could do without. > I guess someone _that_ considered about performance could easily do a > cat *.php | grep -v require | php -w >app.lib > or the like and include app.lib. Yes, and I know a number of folks that do pre-processing like this on their code before pushing it to their production servers. I am all for pre-runtime content management systems that allow authors to structure their code in whatever manner they see fit and when they push to the production servers the content is optimized for the delivery mechanism. For the majority of users PHP is plenty fast enough. We could stick a sleep(1) in there and I doubt anybody would notice. When you are only serving up a few hundred thousand requests per day it really doesn't matter how you architect things. Regardless of this basic fact, I don't think we should be sticking sleep(1)'s in the code, and we should be thinking about reducing the system call overhead where we can, and projects like PEAR should carefully evaluate the cost vs. convenience ratio when making decisions like the one which prompted this. -Rasmus --- ## #9530 — Alan Knowles — 2004-04-24T07:50:40+00:00 URL: https://externals.io/message/9525#9530 · In reply to: #9526 Thanks - I just had a look through zend_comple.c:zend_include_or_eval_handler it appears that zend_stream_open .. and hence fopen is called on every require_once/include_once would it make sense to add something as simple as if (opline->op2.....lval == ZEND_REQUIRE_ONCE) { locations = get_potential_locations(file,PG('include_path')); foreach (locations as location) if (zend_hash_find(&EG(included_files), location)) { // i've already got it ! NEXT_OPCODE(); } } } reduce the stat calls substancially?? - especially where we have cases that alot of the pear classes already have "require_once 'PEAR.php'" .. even though it may have already been loaded by another package... Or did I miss some other optimization that is already there for that..? .. Yeah, I see from APC in CVS, it only wraps the compile_file() call, which has recieves a file pointer, so the stating/realpath damage etc. has already been done) - so unless it does something similar to the above check, and overrides the include handler, it wont help much either, I guess.. Regards Alan Rasmus Lerdorf wrote: > It can be significant. There are a couple of issues: > > 1. The included_files list gets updated each time you include a file. In > order to make sure that the same file included by different paths or > symlinks don't conflict we do a `realpath()` on the file to be included. > That means stats on every component up to and including the file > itself. I can't speak for all the opcode caches, but at least APC > doesn't do anything to alleviate this. > > 2. APC uses the file's device and inode as the lookup key into shared > memory to find the opcodes for the file, so a stat has to be done on > each and every file that is cached. There is currently no such thing > as a request-spanning stat cache. > > So yes, jumping from 20 to 30 include files could very well bring a rather > significant performance hit. > > Anybody looking for real performance out of PHP pretty much has to address > the first issue. It is easy enough to do, simply get rid of the realpath > lookup in that situation and just live with the fact that if someone > includes the same file via different paths it won't catch it. I can see > that it is convenient to have PHP figure this out for you, but the price > of that convenience is not worth it in my opinion. > > -Rasmus > > > > > > […] > > -- Can you help out? Need Consulting Services or Know of a Job? http://www.akbkhome.com --- ## #9531 — Rasmus Lerdorf — 2004-04-24T08:49:20+00:00 URL: https://externals.io/message/9525#9531 · In reply to: #9530 > Thanks - I just had a look through > zend_comple.c:zend_include_or_eval_handler > > it appears that zend_stream_open .. and hence fopen is called on every > require_once/include_once > > would it make sense to add something as simple as > > if (opline->op2.....lval == ZEND_REQUIRE_ONCE) { > locations = get_potential_locations(file,PG('include_path')); > foreach (locations as location) > if (zend_hash_find(&EG(included_files), location)) { > // i've already got it ! > NEXT_OPCODE(); > } > } > } I am not sure this will help all that much. You are taking a hit on the common case in order to optimize what I believe to be the less common case of a *_once including a file already included. For PEAR it might be common, but outside of PEAR I don't think it is. We still need to do the realpath stats prior to this and for the common codepath the fopen still needs to be done after. Then again, your check may be fast enough that it could be worth adding to speed up the failure case. Overall I think you would get more bang for the buck by coming up with an intelligent caching stat/realpath implementation. The obvious drawback would be that symlinks and other filesystem changes done while the server is running would not be seen until a restart. -Rasmus --- ## #9534 — Michael Walter — 2004-04-24T12:14:14+00:00 URL: https://externals.io/message/9525#9534 · In reply to: #9528 Christian Schneider wrote: > [...] > I guess someone _that_ considered about performance could easily do a > cat *.php | grep -v require | php -w >app.lib > or the like and include app.lib. Well yeah, it gets slightly harder when you dynamically require modules. Cheers, Michael --- ## #9537 — Sebastian Bergmann — 2004-04-24T14:18:12+00:00 URL: https://externals.io/message/9525#9537 · In reply to: #9528 Christian Schneider wrote: > I guess someone _that_ considered about performance could easily do a > cat *.php | grep -v require | php -w >app.lib > or the like and include app.lib. Maybe the PEAR Packager / Installer could be changed to (optionally?) create a single source file (without require/include statements) comments, whitespace) from all role="php" s and use this for deployment. -- Sebastian Bergmann http://sebastian-bergmann.de/ http://phpOpenTracker.de/ Das Buch zu PHP 5: http://professionelle-softwareentwicklung-mit-php5.de/ --- ## #9538 — George Schlossnagle — 2004-04-24T14:26:24+00:00 URL: https://externals.io/message/9525#9538 · In reply to: #9537 > Christian Schneider wrote: > > […] > > Maybe the PEAR Packager / Installer could be changed to (optionally?) > create a single source file (without require/include statements) > comments, whitespace) from all role="php" s and use this for > deployment. Having naively tried this in the past, a couple comments: 1) it's not as simple as it sounds, especially if you have includes that do more than just declare functions and classes. 2) it's hard to debug when things go wrong, as the sources aren't where you had them. 3) if you have any conditional includes, 3a) things can easliy break 3b) you often end up doing more work and being slower than using the includes I don't think this is a truly awful idea, but should be approached with a high degree of caution, as it's pretty brittle. George --- ## #9539 — Sebastian Bergmann — 2004-04-24T14:29:17+00:00 URL: https://externals.io/message/9525#9539 · In reply to: #9538 George Schlossnagle wrote: > 1) it's not as simple as it sounds, especially if you have includes > that do more than just declare functions and classes. Who said anything about this beeing easy? Easy is boring ;-) As far as 1) is concerned: this should not be a problem with PEAR packages, right? -- Sebastian Bergmann http://sebastian-bergmann.de/ http://phpOpenTracker.de/ Das Buch zu PHP 5: http://professionelle-softwareentwicklung-mit-php5.de/ --- ## #9541 — Stephan Schmidt — 2004-04-24T15:09:53+00:00 URL: https://externals.io/message/9525#9541 · In reply to: #9537 Hi, > Maybe the PEAR Packager / Installer could be changed to (optionally?) > create a single source file (without require/include statements) > comments, whitespace) from all role="php" s and use this for > deployment. This should not be done, as a lot of packages are driver based. If you create one large file from all DB drivers, the compile time surely will be longer than the time needed to open the needed files. Just my 2cts. Stephan --- ## #9542 — George Schlossnagle — 2004-04-24T15:29:10+00:00 URL: https://externals.io/message/9525#9542 > George Schlossnagle wrote: > > […] > > Who said anything about this beeing easy? Easy is boring ;-) > > As far as 1) is concerned: this should not be a problem with PEAR > packages, right? No idea. George --- ## #9574 — Jeff Moore — 2004-04-27T00:30:33+00:00 URL: https://externals.io/message/9525#9574 · In reply to: #9525 > PEAR is considering stipulating 1 class per file, for the packages. - > one of the concerns raised is performance. While this is true for a > non-cache compiled situation (where performance is less of an issue) - > would anyone care to comment on the implications when using the Zend > Optimizer or APC, on how significant the impact of this would be. The Mojavi framework combines 19(?) single class files into a single file to include. I benchmarked this and if I recall correctly, the combined file loaded in roughly half the time of including the individual 19 files. The timing was only for including, no actual work was performed. No cache was used. The benchmarks were done under OS X, which as has been discussed on this list before, has a particularly inefficient implementation of realpath (used in the including process).