Greetings,
This seemed like the appropriate list to discuss this - let me know if it isn't.
We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
mostly PHP over NFS (we have separate servers for static content). I
have been trying to figure out why we're seeing so many getattr
requests from our Apache/PHP servers. I think I've narrowed it down to
how PHP/APC gets information about a file to see if it has been
changed since it was last cached:
https://bugs.php.net/bug.php?id=59372
Rasmus said:
"Just because you see an open() syscall doesn't mean the cache isn't
used. The open() is there because PHP by default does open+fstat for
performance reasons. If you look carefully you will see there is just
an open() and no read() calls. So it opens the file, uses the stat
struct from the open fd to get the inode and fetches the op_array from
the opcode cache. This can be made more efficient, but this report is
incorrect in assuming the cache isn't being used."
This is exactly what I'm seeing when I strace an httpd process -
mostly open() and fstat() calls w/o any read() calls. This makes sense
to me, however, the problem with this is as far as I understand it,
that an open() on an NFS file system causes an automatic gettattr()
call to the server. Doing this completely bypasses the NFS attribute
cache, which is why we're seeing so many getattr requests to our NFS
server despite having an attribute cache timeout of 60 seconds. For
those familiar with NFS, we could just as well disable our attribute
cache at this point, which is almost never recommended because of
performance reasons.
I found a good explaination of why this happens in this Redhat document:
"Avoid unnecessarily opening and closing of files. If you access a
file frequently, opening and closing that file triggers the
opentoclose cache
consistency mechanism in the NFS file system. Triggering this
mechanism causes NFS getattr requests to be generated, which can slow
performance. By keeping the file open, traffic is kept to a minimum."
I've tried doing apc.stat=0 and about every other combination of
options, but PHP still does an open()+3 fstat() commands on every
single page load no matter what I do. We have autoload set, so that
equates to several automatic open() (and subsequent NFS getattr())
calls with every single PHP request. To give you an idea what kind of
load this is generating for us:
-Out of 7,000 total NFS operations per second right now:
-About 45% are getattr calls
-Our Apache/PHP servers are generating 52% of all NFS calls
-An average of 82% of those calls are getattr requests
-As a comparison, our far busier content servers (many times the req/s
of the Apache/PHP servers) are generating half the number of NFS
requests our PHP servers are, and 98% or so of those requests are read
requests with just 1% getattr
I am pretty proficient with patching/compiling/profiling, but sadly I
am no C coder. I've tried digging into this myself but I'm pretty out
of my element here. I'm curious if anyone on this list could give me
any pointers on what I can do to change this behavior and reduce the
load on our NFS server. If we can't change PHP behavior, we may have
to abandon NFS for our PHP files due to the load PHP appears to be
generating on our NFS server. I'd hate to do this, so please let me
know if any of you have any ideas as to what I can do to change this
behavior.
Brendon
Rasmus said:
"Just because you see an open() syscall doesn't mean the cache isn't
used. The open() is there because PHP by default does open+fstat for
performance reasons. If you look carefully you will see there is just
an open() and no read() calls. So it opens the file, uses the stat
struct from the open fd to get the inode and fetches the op_array from
the opcode cache. This can be made more efficient, but this report is
incorrect in assuming the cache isn't being used."This is exactly what I'm seeing when I strace an httpd process -
mostly open() andfstat()calls w/o any read() calls. This makes sense
to me, however, the problem with this is as far as I understand it,
that an open() on an NFS file system causes an automatic gettattr()
call to the server. Doing this completely bypasses the NFS attribute
cache, which is why we're seeing so many getattr requests to our NFS
server despite having an attribute cache timeout of 60 seconds. For
those familiar with NFS, we could just as well disable our attribute
cache at this point, which is almost never recommended because of
performance reasons.
Wow, people are still serving web files over NFS? Sounds painful.
But, this is what APC's apc.include_once_override setting tries to
address. Try turning that on.
-Rasmus
Rasmus said:
Wow, people are still serving web files over NFS? Sounds painful.But, this is what APC's apc.include_once_override setting tries to
address. Try turning that on.
Yep - I think it would be a lot more painful if we didn't use a very
expensive NetApp Filer to serve NFS (the solution before the Filer
would lock up every few weeks). I'm really surprised more people
haven't complained about this. I've seen a few complaints about people
seeing poor PHP app performance on NFS, but they usually move to
gluster or something else before ever figuring it out. Maybe very few
people use NFS to serve web files - who knows.
I tried apc.include_once_override - no change. I set up Apache to one
process and straced it. I clicked around on the site a few times and
see the typical pattern:
open("[PATH]/CacheFile.class.php", O_RDONLY) = 10
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
Out of 486 open() calls, 263 were for PHP files (54%). Here is my
current APC config:
extension=/usr/local/lib/php/extensions/apc.so
apc.enabled=1
apc.shm_segments=1
apc.optimization=0
apc.shm_size=256M
apc.ttl=1200
apc.user_ttl=1200
apc.num_files_hint=300
apc.mmap_file_mask=/apc.shm.XXXXXX
apc.slam_defense=0
apc.write_lock=1
apc.include_once_override=1
apc.stat=0 doesn't help. Every PHP request causes an open() and
subsequent getattr() to the Filer no matter what options I set.
I tried apc.include_once_override - no change. I set up Apache to one
process and straced it. I clicked around on the site a few times and
see the typical pattern:open("[PATH]/CacheFile.class.php", O_RDONLY) = 10
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
Assuming that those are relative includes, can you try with:
apc.canonicalize=0
apc.stat=0
Damien
Assuming that those are relative includes, can you try with:
apc.canonicalize=0
apc.stat=0
Paths are absolute. stat=0 (and canonicalize=0 just to try it)
produced the same result.
Brendon
Assuming that those are relative includes, can you try with:
apc.canonicalize=0
apc.stat=0
Paths are absolute. stat=0 (and canonicalize=0 just to try it)
produced the same result.Brendon
Brendon, are your scripts doing a log of include_once / require_once
calls? In you look at ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER() in
Zend/zend_vm_execute.h then you will see that this Zend handler does
zend_resolve_path() for any xxx_ONCE instructions and zend_stream_open()
on the first request of the same. Yes APC rehooks this handler with a
wrapper but this is only for lazy loading. When it honors the xxx_once
instructions, it will still open the streams even if the code itself is
fully cached in APC and the I/O is entirely nugatory. I suspect that
this could generate the NFS traffic that you are discussing.
This would be easy to avoid, but this would require replacing this
handler entirely or doing dynamic code patching, nether of which APC
currently does, I believe. Incidentally because this is Zend feature
and nothing directly to with APC, O+ will also have this runtime
characteristic.
//Terry
Brendon, are your scripts doing a log of include_once / require_once calls?
Yep, but we also changed our autoload script (auto_prepend_file) to
use require instead of require_once, and we still saw the same
behavior (open calls for the PHP files required by the
auto_prepend_file same as before).
In you look at ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER() in
Zend/zend_vm_execute.h then you will see that this Zend handler does
zend_resolve_path() for any xxx_ONCE instructions and zend_stream_open() on
the first request of the same. Yes APC rehooks this handler with a wrapper
but this is only for lazy loading. When it honors the xxx_once
instructions, it will still open the streams even if the code itself is
fully cached in APC and the I/O is entirely nugatory. I suspect that this
could generate the NFS traffic that you are discussing.
Yes if the streams are opened whether cached or not, then this will
result in an automatic getattr request to the NFS server which is
exactly what we're seeing.
Brendon
Hi!
Wow, people are still serving web files over NFS? Sounds painful.
Sometimes, there's not much (better) choices if you need to keep
writeable files in sync over a number of machines. There are other
shared FSes but they would lead to pretty much the same issues.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Hi!
Wow, people are still serving web files over NFS? Sounds painful.
Sometimes, there's not much (better) choices if you need to keep
writeable files in sync over a number of machines. There are other
shared FSes but they would lead to pretty much the same issues.
Yeah, but NFS, especially without the realpath cache, which you lose if
you turn on open_basedir, is deathly slow because of all the stats.
Typically PHP scripts are not actually "writable files" and the way to
keep them in synch across multiple machines is to use a deploy
mechanism/rsynch to deploy them. You may very well have some writable
files that need to be shared at the app-level, but then we wouldn't be
talking about PHP stat calls and the realpath cache. And most people
have moved to something like memcache for sharing writable data across
machines.
-Rasmus
Hi!
Yeah, but NFS, especially without the realpath cache, which you lose if
you turn on open_basedir, is deathly slow because of all the stats.Typically PHP scripts are not actually "writable files" and the way to
keep them in synch across multiple machines is to use a deploy
mechanism/rsynch to deploy them. You may very well have some writable
rsync may be way too slow if you need the change appear more or less
"immediately" on all servers. Unless you run rsync each second if which
case you're probably better off with NFS :)
files that need to be shared at the app-level, but then we wouldn't be
talking about PHP stat calls and the realpath cache. And most people
You would still if the data is stored as php file. Which last time I
tested - given APC or something like - was the fastest way to get a
bunch of data into PHP (it was faster than loading and unserializing,
even with APC/memcache).
have moved to something like memcache for sharing writable data across
machines.
memcache is non-persistent. It's a cache, not a storage solution, so you
can not use it for anything that actually needs to be stored permanently.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Hi!
Yeah, but NFS, especially without the realpath cache, which you lose if
you turn on open_basedir, is deathly slow because of all the stats.Typically PHP scripts are not actually "writable files" and the way to
keep them in synch across multiple machines is to use a deploy
mechanism/rsynch to deploy them. You may very well have some writablersync may be way too slow if you need the change appear more or less
"immediately" on all servers. Unless you run rsync each second if which
case you're probably better off with NFS :)files that need to be shared at the app-level, but then we wouldn't be
talking about PHP stat calls and the realpath cache. And most peopleYou would still if the data is stored as php file. Which last time I
tested - given APC or something like - was the fastest way to get a
bunch of data into PHP (it was faster than loading and unserializing,
even with APC/memcache).have moved to something like memcache for sharing writable data across
machines.memcache is non-persistent. It's a cache, not a storage solution, so you
can not use it for anything that actually needs to be stored permanently.
Sure, but then you can go with something like Redis.
But, again, if you go back to the original question, this has nothing to
do with often-changing data in a couple of PHP include files:
We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
mostly PHP over NFS (we have separate servers for static content)."
So they are serving up all their PHP over NFS for some reason.
-Rasmus
The point that this thread highlights is that apps developers /
administrators at both ends of the scale -- the enterprise and the
shared service user -- normally have little say over the infrastructure
architecture on which their application runs. In both these cases the
infrastructure will be hosting 100s if not 1000s of apps, and the
environment cannot be tailored to any one app. So issues like storage
architecture and security architecture (e.g. UID-based enforcement of
app separation) are a given as NFRs (non-functional requirements). Use
of NFS with server / storage separation is still a standard
implementation design pattern, when enterprises require simple-to-manage
scalability on these tiers. We aren't going to change this, and neither
will the O/P.
Surely PHP will achieve better penetration and the greater acceptance if
it can offer more robust performance in this sort of environment?
Sure, but then you can go with something like Redis.
But, again, if you go back to the original question, this has nothing to
do with often-changing data in a couple of PHP include files:We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
mostly PHP over NFS (we have separate servers for static content)."So they are serving up all their PHP over NFS for some reason.
The point that this thread highlights is that apps developers /
administrators at both ends of the scale -- the enterprise and the
shared service user -- normally have little say over the
infrastructure architecture on which their application runs. In both
these cases the infrastructure will be hosting 100s if not 1000s of
apps, and the environment cannot be tailored to any one app. So
issues like storage architecture and security architecture (e.g.
UID-based enforcement of app separation) are a given as NFRs
(non-functional requirements). Use of NFS with server / storage
separation is still a standard implementation design pattern, when
enterprises require simple-to-manage scalability on these tiers. We
aren't going to change this, and neither will the O/P.Surely PHP will achieve better penetration and the greater acceptance
if it can offer more robust performance in this sort of environment?Sure, but then you can go with something like Redis.
But, again, if you go back to the original question, this has nothing to
do with often-changing data in a couple of PHP include files:We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
mostly PHP over NFS (we have separate servers for static content)."So they are serving up all their PHP over NFS for some reason.
Brendon,
Just to follow up with a bit more detail, apart from the obvious NFS
tuning with things like the actimeo mount parameters, you can get a
better idea of what is going on if you use a local copy of one of your
apps running under a local linux apache server. This is an example from
my Ubuntu laptop, but it works just as well on a test VM if you still
use WinXX on your PC.
trace_wget () {
sleep 1
coproc sudo strace -p $(ps -C apache2 --no-headers -o pid) -tt -o /tmp/strace$1.log
sleep 1; ST_PID=$(ps -C strace --no-headers -o pid)
wget "http://localhost/$2" -O /dev/null -o /dev/null
sleep 2; sudo kill $ST_PID
}
# start Apache in debug mode
sudo service apache2 stop
sudo bash -c ". /etc/apache2/envvars; coproc apache2 -X"
# trace three gets
trace_wget 0 phpinfo.php
trace_wget 1 mwiki/index.php?title=Main_Page
trace_wget 2 mwiki/index.php?title=Main_Page
#restart normally
sudo service apache2 stop
sudo service apache2 start
sudo chmod 777 /tmp/strace?.log
grep -c "open(" /tmp/strace?.log
grep -c "stat(" /tmp/strace?.log
The first get is just to load and prime the mod_php5 thread (Ubuntu has
a crazy localtime implementation), the second loads and compiles the PHP
MediaWiki modules needed to render a page, the third repeats this now
fully cached in ACP. In this case,
we have:
open() `fstat()`/lstat()
/tmp/strace1.log 108 650 (Priming the APC cache)
/tmp/strace2.log 27 209 (Using the APC cache)
So APC does materially reduce the I/O calls, and (if you look at the
traces) it removes most of the mmaps and compilation. The MWiki
autoloader uses require() to load classes but in the code as a whole the
most common method of loading modules is require_once() (414 uses as
opposed to 44 in total of the other include or require functions) and as
I said in my previous post, this I/O is avoidable by recoding the
ZEND_INCLUDE_OR_EVAL hander to work cooperatively with the opcode cache
when present.
Note that even when the cache does this, it still can't optimize coding
patterns like:
if ( file_exists( "$IP/AdminSettings.php" ) ) {
require( "$IP/AdminSettings.php" );
}
and IIRC mwiki does 4 of these on a page render. Here a pattern based
on (@include($file) == 1) is more cache-friendly.
Now this isn't a material performance issue when the code tree is
mounted on local storage (as is invariably the case for a developer test
configuration) as the stat / open overhead of a pre-(file) cached file
is microseconds, but it is material on scalable production
infrastructure which uses network attached NFS or NTFS file systems.
So in summary, this is largely avoidable but unfortunately I don't think
that we can persuade the dev team to address this issue, even if I gave
them the solution as a patch :-(
Regards
Terry Ellison
Hi!
and IIRC mwiki does 4 of these on a page render. Here a pattern based
on (@include($file) == 1) is more cache-friendly.
This goes back to the fact that @ does not really disable errors - it
only disables reporting of the errors, but the whole message is
generated and goes all the cycle up to the actual error reporting before
being suppressed. I've tried to address this problem a number of times
but looks like there's really no way to do it without sacrificing some
parts of current error reporting such as track_errors.
So in summary, this is largely avoidable but unfortunately I don't think
that we can persuade the dev team to address this issue, even if I gave
them the solution as a patch :-(
Could you explain what you're talking about here?
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Hi!
and IIRC mwiki does 4 of these on a page render. Here a pattern based
on (@include($file) == 1) is more cache-friendly.
This goes back to the fact that @ does not really disable errors - it
only disables reporting of the errors, but the whole message is
generated and goes all the cycle up to the actual error reporting before
being suppressed. I've tried to address this problem a number of times
but looks like there's really no way to do it without sacrificing some
parts of current error reporting such as track_errors.
Yup, in this scenario we're trading off two things. What the pattern is
saying is if the source file exists then compile it. However, if the
file exists, in say 90+% of the cases, and the code is often cached then
the file_exists check generates I/O that isn't needed if the file is
already APC (or O+ / xcache) cached. Yup if the file is missing then
the error is generated and trapped in the reporting cycle, but the final
result is that the result is false and the false==1 fails. Other than
the wasted cycles in running up this stack, there aren't any other
side-effects that I am aware of. Are there any?
Yes, this is an overhead, but it is small beer compared to doing a
getattr() RPC across the server room fabric to a NAS server backend.
So in summary, this is largely avoidable but unfortunately I don't think
that we can persuade the dev team to address this issue, even if I gave
them the solution as a patch :-(
Could you explain what you're talking about here?
The fact that you are engaging in this dialogue is great and I really
appreciate it. What I am still trying to work out is how to interact
with you guys in a way that is mutually productive. I did make the
mistake of choosing the mainstay stable version iN live prod systems use
-- 5.3 -- to take this whole issue apart and I guess that the dev team
regards this as history.
I guess that I should bite the bullet and switch to 5.5. I've been
working on an evaluatorfork of APC optimized for CLI/GCI which tackles a
lot of these issues head on and performs reasonable well, but I realise
that this is a dead-end and will never get deployed, but I am currently
considering regressing some of this technology into 5.5 and O+. Are you
interested in a version of O+ which supports all SAPIs?
In architectural terms I feel that having a universal cache option is
important. It changes the mindset from something which is only used in
niche performance use cases to a standard option. It also means that
the cache can be stress tested by the entire php test suite. However,
to do some of this you don't start with a patch, but with an RFC
informed by evidence, and that's my real reason for doing this demonstrator.
//Terry
I guess that I should bite the bullet and switch to 5.5.
Yes.
I've been working on an evaluatorfork of APC optimized for CLI/GCI
which tackles a lot of these issues head on and performs reasonable
well, but I realise that this is a dead-end and will never get
deployed, but I am currently considering regressing some of this
technology into 5.5 and O+. Are you interested in a version of O+
which supports all SAPIs?
Yes, but there are some SAPIs that are more common, so these should be
prioritized if trade-offs are required.
My particular desire is for "embed" SAPI support. We use this in our
Tuxedo application server, but it is a relatively niche use case.
In architectural terms I feel that having a universal cache option
is important. It changes the mindset from something which is only
used in niche performance use cases to a standard option. It also
means that the cache can be stress tested by the entire php test
suite. However, to do some of this you don't start with a patch,
but with an RFC informed by evidence, and that's my real reason for
doing this demonstrator.
I'm looking forward to seeing your suggestions and patches.
Chris
--
christopher.jones@oracle.com http://twitter.com/ghrd
Newly updated, free PHP & Oracle book:
http://www.oracle.com/technetwork/topics/php/underground-php-oracle-manual-098250.html
Hi!
Yes, this is an overhead, but it is small beer compared to doing a
getattr() RPC across the server room fabric to a NAS server backend.
That depends of what your error handlers do. Some may write to log
files, etc. if not configured properly (since error_reporting setting
doesn't have to be considered in it).
IIRC, for most of the cases O+ should be able to resolve all
includes/requires on cached files without syscalls - but file_exists is
a different matter since caching it in generic case can be very dangerous.
I guess that I should bite the bullet and switch to 5.5. I've been
working on an evaluatorfork of APC optimized for CLI/GCI which tackles a
lot of these issues head on and performs reasonable well, but I realise
that this is a dead-end and will never get deployed, but I am currently
considering regressing some of this technology into 5.5 and O+. Are you
interested in a version of O+ which supports all SAPIs?
I think right now O+ can support CLI (provided enable_cli is set) but
for most cases it's kind of useless since scripts are rarely re-included
in common CLI scenarios. So if you know how to improve common CLI
scenarios it may be interesting, though I imagine it's not the most
common use case. But if it adds there without problems for anything
else, why not.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
That depends of what your error handlers do. Some may write to log
files, etc. if not configured properly (since error_reporting setting
doesn't have to be considered in it). IIRC, for most of the cases O+
should be able to resolve all includes/requires on cached files
without syscalls - but file_exists is a different matter since caching
it in generic case can be very dangerous.
I am not suggesting caching file_exists but rather encouraging coding
patterns which avoid its use /if/ the application is intended to give
good cached performance -- e.g. apps like mwiki, wordpress and drupal.
I guess that I should bite the bullet and switch to 5.5. I've been
working on an evaluatorfork of APC optimized for CLI/GCI which tackles a
lot of these issues head on and performs reasonable well, but I realise
that this is a dead-end and will never get deployed, but I am currently
considering regressing some of this technology into 5.5 and O+. Are you
interested in a version of O+ which supports all SAPIs?
I think right now O+ can support CLI (provided enable_cli is set) but
for most cases it's kind of useless since scripts are rarely re-included
in common CLI scenarios. So if you know how to improve common CLI
scenarios it may be interesting, though I imagine it's not the most
common use case. But if it adds there without problems for anything
else, why not.
Let me get a dev stack for 5.5 and O+ up and then I can comment on
this. As far as APC's support for CLI goes -- the cache doesn't connect
to the apache2 SMA even if you run the cli process in the same UID, and
since the SMA gets released when the process count goes to zero, caching
doesn't really do anything as the cache discarded between cli
executions, so if practice you always run uncached.
As to the frequency of the use case, maybe not in the case of cli, but
cgi is still tremendously common, as shared hosting providers still need
to implement UID-based separation of scripting environments for
different user accounts. There are still significant scaling issues
with php-fpm in this usecase, so only a few SHPs offer FastCGI support
and then only as a premium option. BTW, I saw that you recommended in
one of your blog posts that you to use a bytecode cache if you care
about performance should do. However, a lot of app maintainers who use a
shared hosting service do care about performance but don't have this as
an option :-(
For example with my cli-based LPC opcode cache, I can do a series of
"make test" runs (1) with the opcache disabled; (2) with it enabled but
the cache empty and (3) with it enabled and the cache(s) primed.
Ideally all three (*) should give the same test results, and this gives
you confidence that the cache is working properly. I would like to do
this with APC or O+, but in practice AFAIK, I can't do (3) so how do I
or the developers know what PHP and extension features aren't being
supported / working properly when the code is cached?
(*) With the tests as currently written, some fail with case (3) because
the runs are missing compiler warnings which are only generated if the
code isn't cached (and in case of my cache at its current build quite a
few fail because it still doesn't play nicely with other extensions like
phar.)
Brendon,
Just to follow up with a bit more detail, apart from the obvious NFS tuning
with things like the actimeo mount parameters, you can get a better idea of
what is going on if you use a local copy of one of your apps running under a
local linux apache server. This is an example from my Ubuntu laptop, but it
works just as well on a test VM if you still use WinXX on your PC.trace_wget () {
sleep 1
coproc sudo strace -p $(ps -C apache2 --no-headers -o pid) -tt -o
/tmp/strace$1.log
sleep 1; ST_PID=$(ps -C strace --no-headers -o pid)
wget "http://localhost/$2"; -O /dev/null -o /dev/null
sleep 2; sudo kill $ST_PID
}start Apache in debug mode
sudo service apache2 stop
sudo bash -c ". /etc/apache2/envvars; coproc apache2 -X"trace three gets
trace_wget 0 phpinfo.php
trace_wget 1 mwiki/index.php?title=Main_Page
trace_wget 2 mwiki/index.php?title=Main_Page#restart normally
sudo service apache2 stop
sudo service apache2 start
sudo chmod 777 /tmp/strace?.loggrep -c "open(" /tmp/strace?.log
grep -c "stat(" /tmp/strace?.logThe first get is just to load and prime the mod_php5 thread (Ubuntu has a
crazy localtime implementation), the second loads and compiles the PHP
MediaWiki modules needed to render a page, the third repeats this now fully
cached in ACP. In this case,
we have:open() `fstat()`/lstat()/tmp/strace1.log 108 650 (Priming the APC cache)
/tmp/strace2.log 27 209 (Using the APC cache)So APC does materially reduce the I/O calls, and (if you look at the
traces) it removes most of the mmaps and compilation. The MWiki autoloader
uses require() to load classes but in the code as a whole the most common
method of loading modules is require_once() (414 uses as opposed to 44 in
total of the other include or require functions) and as I said in my
previous post, this I/O is avoidable by recoding the ZEND_INCLUDE_OR_EVAL
hander to work cooperatively with the opcode cache when present.Note that even when the cache does this, it still can't optimize coding
patterns like:if ( file_exists( "$IP/AdminSettings.php" ) ) { require( "$IP/AdminSettings.php" ); }and IIRC mwiki does 4 of these on a page render. Here a pattern based on
(@include($file) == 1) is more cache-friendly.Now this isn't a material performance issue when the code tree is mounted on
local storage (as is invariably the case for a developer test configuration)
as the stat / open overhead of a pre-(file) cached file is microseconds, but
it is material on scalable production infrastructure which uses network
attached NFS or NTFS file systems.So in summary, this is largely avoidable but unfortunately I don't think
that we can persuade the dev team to address this issue, even if I gave them
the solution as a patch :-(Regards
Terry Ellison
Terry,
Thanks for your detailed input. This is essentially what I did in my test setup.
APC is definitely critical to the performance of our site. I'm just
very curious to see how much impact the high number of getattr
operations coming from the Apache/PHP servers is having on the filer.
It looks like we might be able to move our PHP files to local storage
to at least measure the impact this is having.
Brendon
Terry,
Thanks for your detailed input. This is essentially what I did in my test setup.
APC is definitely critical to the performance of our site. I'm just
very curious to see how much impact the high number of getattr
operations coming from the Apache/PHP servers is having on the filer.
It looks like we might be able to move our PHP files to local storage
to at least measure the impact this is having.Brendon
Brendon,
I think that PHP is a great platform; it's just that PHP's sweet spot is
just a little off the optimum for typical scalable enterprise
infrastructures. IMO, the main issue that you should consider and
discuss with your infrastructure support team is how to mitigate the
impacts for such high-throughput business critical applications. As you
(and Rasmus previously, IIRC) mention one key issue is what is on local
and what is on network attached storage.
On one system that I got hauled in to troubleshoot, it turned out that
the main problem was that the PHP session data was being written to a
directory in shared storage, causing write-though overload that ended up
hitting half a dozen key apps. It was a trivial configuration change to
move it to local storage transforming performance. (I apologise if I am
telling how "to suck eggs" but this is really for others tracking this
thread) the main issue here is that the file hierarchies that must have
some integrity over the application tier need to be on the NAS,
everything else is a matter of convenience vs performance, so for
example having a cron job to rsync the apps PHP hierarchy onto local
storage might well transform your app performance and give an indirect
boost to cohosted apps. Happy hunting :)
Wow, people are still serving web files over NFS? Sounds painful.
So they are serving up all their PHP over NFS for some reason.
I've been thinking about this and wanted to make a few final points. I
think this is potentially a pretty major issue and not something to be
dismissed.
We've invested a great deal of money in a high performance shared
storage system, so of course we want to use this system for absolutely
everything we can within reason, including sharing 1500+ PHP files to
our many web servers. I don't think most systems administrators would
find this unreasonable.
Why would we want to split our architecture up so just our PHP files
are being synced from local disk and everything else is shared via
NFS? Doing this seems far more unreasonable to me and just doesn't
make any sense.
NFS is so common for sharing files that saying "Wow, people are still
serving web files over NFS?" is like saying "Wow, people are still
using the ls command to list directory contents on Linux?" I think NFS
is still very widely used, even for sharing web files.
This being said, you stated in bug #59372 that "PHP by default does
open+fstat for performance reasons" and while doing an open+fstat
might result in better performance on local file systems, doing so is
not optimal when using NFS. NFS "close-to-open cache consistency" is
designed such that opening a file stored on an NFS server causes an
automatic getattr or access request. This could also be true for other
shared file systems as others have mentioned.
All of these open+fstats are definitely causing a measurable impact to
our systems. Right now, for example, 50% of the 6-8K+ ops/s being
processed by our filer are getattr operations, and a majority of these
getattr calls are coming from our Apache/PHP servers. Doing open+fstat
bypasses the NFS attribute cache similar to disabling it altogether,
where a simple stat would likely utilize the attribute cache.
Disabling the attribute cache is generally not recommended due to the
performance impact this has on the filer (and the application).
I don't think the appropriate answer is "don't use NFS" because this
is ridiculous as a long term solution (NFS is common, and people are
going to use it or something similar). I think the appropriate answer
is to update PHP to use stat vs. open+fstat or doing something similar
that would be optimized for both local AND shared file systems (I
would be writing a patch instead of this email if I could).
Until then, I think it's important that people know that PHP is NOT
optimized for use with NFS (and possilby other shared file systems)
even when using an opcode cache like APC. I think this is a very
important piece of information that people need to know. Up until this
week, I wouldn't have thought twice about sharing PHP files over NFS,
and I've been a Unix admin for over twelve years (only new to this
environment).
Brendon
NFS is so common for sharing files that saying "Wow, people are still
serving web files over NFS?" is like saying "Wow, people are still
using the ls command to list directory contents on Linux?" I think NFS
is still very widely used, even for sharing web files.
This is simply not true. I do have a fair bit of experience in this
field, and I don't know of any major sites that do this and I have
worked with a good chunk of the largest sites out there.
I don't think the appropriate answer is "don't use NFS" because this
is ridiculous as a long term solution (NFS is common, and people are
going to use it or something similar). I think the appropriate answer
is to update PHP to use stat vs. open+fstat or doing something similar
that would be optimized for both local AND shared file systems (I
would be writing a patch instead of this email if I could).
If it is of such importance to you and you are not able to do it
yourself, then hire someone to do it. We may or may not get around to
it, but like most things in PHP, we work on what we need ourselves and I
don't think anybody here would even consider putting all their PHP files
on an NFS share when performance was important.
-Rasmus
NFS is so common for sharing files that saying "Wow, people are still
serving web files over NFS?" is like saying "Wow, people are still
using the ls command to list directory contents on Linux?" I think NFS
is still very widely used, even for sharing web files.
This is simply not true. I do have a fair bit of experience in this
field, and I don't know of any major sites that do this and I have
worked with a good chunk of the largest sites out there.
Eh??? Fortune 500 enterprises and governmental departments are pretty
conservative. NAS and SAN based iSCSI and FCoE based elastic block
storage give great performance for server-specific file-systems, but
Brendon is right: for distributed file systems, NFS and CIFS still
dominate.I don't think the appropriate answer is "don't use NFS" because this
is ridiculous as a long term solution (NFS is common, and people are
going to use it or something similar). I think the appropriate answer
is to update PHP to use stat vs. open+fstat or doing something similar
that would be optimized for both local AND shared file systems (I
would be writing a patch instead of this email if I could).
If it is of such importance to you and you are not able to do it
yourself, then hire someone to do it. We may or may not get around to
it, but like most things in PHP, we work on what we need ourselves and I
don't think anybody here would even consider putting all their PHP files
on an NFS share when performance was important.
Again wrong. Apps developers don't do this because that want to; they
do it because the IT services group that runs the production
infrastructure has mandated standard templates for live deployment to
keep the through-life cost of providing the application infrastructure
manageable, with all sorts of bureaucratic exception processes if you
think that your app is a special case. If you are lucky then they offer
a range of EC2-like standard VM templates so that you can deploy an
EBS-based approach, but most are still in catch-up mode compared to
Amazons offerings.
If you are a GM or a American Airlines, or the NIH for example then you
will have 1,000s of applications customer facing and internal and you've
got to adopt this approach for 90+% of these applications.
What Brendon is asking for is reasonable, sensible and in relative terms
easy to implement. However, I agree that it may be more sensible to use
community effort to achieve this.
NFS is so common for sharing files that saying "Wow, people are still
serving web files over NFS?" is like saying "Wow, people are still
using the ls command to list directory contents on Linux?" I think NFS
is still very widely used, even for sharing web files.
This is simply not true. I do have a fair bit of experience in this
field, and I don't know of any major sites that do this and I have
worked with a good chunk of the largest sites out there.
Eh??? Fortune 500 enterprises and governmental departments are pretty
conservative. NAS and SAN based iSCSI and FCoE based elastic block
storage give great performance for server-specific file-systems, but
Brendon is right: for distributed file systems, NFS and CIFS still
dominate.
By major I meant traffic-wise, not Fortune-500, although there are some
of those on the list too. I mostly work with medium-to-large scale
Internet companies. Think Yahoo, Facebook, Flickr, Digg, Etsy, WePay,
Room77. These types of companies would never consider serving all their
Web traffic from NFS. Yes, Yahoo had a ton of Netapp filers as well, but
this was for shared data storage, they would never consider putting
their application logic on them.
This is also something that has been like this for 10+ years and nobody
has stepped up to fix it so far. It shouldn't be news to anyone that
stats and opens over NFS are slow. I am not sure why it should suddenly
be an urgent problem for us at this point. But like I said, we may get
to it. If the integrated opcode cache happens it becomes easier to
manage the flow between the compiler, the cache and the executor and we
can probably optimize some things there.
And as I mentioned in another thread, let's see some RFCs proposing how
to fix some of these things rather than simply posting "I wish the PHP
devs would do this.." type of messages. These go over really badly with
most of the longtime contributors here and they even tend to have the
opposite of the desired effect.
-Rasmus
Ramus, thanks for your detailed response.
NFS is so common for sharing files that ...
This is simply not true. I do have a fair bit of experience in this
field, and I don't know of any major sites that do this and I have
worked with a good chunk of the largest sites out there.
Eh??? Fortune 500 enterprises and governmental departments are pretty
conservative. NAS and SAN based iSCSI and FCoE based elastic block
storage give great performance for server-specific file-systems, but
Brendon is right: for distributed file systems, NFS and CIFS still
dominate.
By major I meant traffic-wise, not Fortune-500, although there are some
of those on the list too. I mostly work with medium-to-large scale
Internet companies. Think Yahoo, Facebook, Flickr, Digg, Etsy, WePay,
Room77. These types of companies would never consider serving all their
Web traffic from NFS. Yes, Yahoo had a ton of Netapp filers as well, but
this was for shared data storage, they would never consider putting
their application logic on them.
Now I agree with you: for this sector of Internet B2C companies, their
business is centred around a small number of apps that dominate their
revenue streams, so of course they are free to design their
infrastructure architecture to optimize the performance of these apps. I
also accept that this sector was and is directly or indirectly the major
funder of PHP development effort.
However, my counter point is that this is no longer the only
infrastructure usecase for PHP. Now mature, it has entered other
sectors and Brendon and Daniel posts highlight two of them:
-
Enterprise use as Brendon raises. Enterprises have moved to use
internet based technologies to automate internal business
processes. These apps work on the company intranet, not on the
internet. So when you book a car or go to your bank or order a part
from a manufacturer, the assistant may well be sitting in front of a
PHP app that never sees the internet but is still core to that
business. Thanks partly to the flexibility of cloud resources, CIOs
and CTOs are increasingly looking at open technologies such PHP to
replace MS ones. Incidentally IMO, its this sort of business stream
that will provide hard funding to value-add companies such as Zend. -
The hosting service providers as Daniel raises. In terms of sheer
numbers this is that largest community of PHP users who buy their
+/- $100pa service from a hosting provider. They still care about
performance. The providers care about the efficiency of their
infrastructure. They (initially) using PHP because Wordpress,
Mediawiki, ... are written in it. But this is also a major entry
vehicle for a new generation of PHP developers to get an initial
internet presence. If PHP runs 3x slower than language X, and X is
just as flexible then we are putting up unnecessary barriers to
their entry and turning away that new cadre.
This is also something that has been like this for 10+ years and nobody
has stepped up to fix it so far. It shouldn't be news to anyone that
stats and opens over NFS are slow. I am not sure why it should suddenly
be an urgent problem for us at this point. But like I said, we may get
to it.
It's not suddenly urgent but perhaps this is more a question of maybe
hitting a tipping point where it might now be wise to address this issue.
If the integrated opcode cache happens it becomes easier to
manage the flow between the compiler, the cache and the executor and we
can probably optimize some things there.
+1
And as I mentioned in another thread, let's see some RFCs proposing how
to fix some of these things rather than simply posting "I wish the PHP
devs would do this.." type of messages. These go over really badly with
most of the longtime contributors here and they even tend to have the
opposite of the desired effect.
As I have posted separately, I forked and then rewrote APC to address
this sweet spot. OK my LPC is very a much bug-ridden alpha code that
fails 10% of the PHP test suite largely due to extension
interoperability issues, and I've had other things to do this last month
-- including deciding whether to switch to a proper O+ delta. However,
my aim was for me to use this as an evaluation test bed, not a serious
production contender. However, now that I've written an opcode cache
which runs Mediawiki under php-cgi (with ~ 5% of the NFS getattrs, BTW),
rolling some key tweeks into the Zend compiler, execution environment
and APC -- which I understand well and should be straight forward -- or
O+ -- which I don't as yet.
My challenge is deciding (i) do I work on PHP 5.6 / 5.7 and the
corresponding beta APC version which at current rates of adoption might
have begin to have an impact in the community sometime in the next 5
years, or (ii) work on a performance patch to the stable APC version
which is typically installed with PHP 5.3 which these guys could apply
within a few months.
I'll draft the RFCs if the webmaster gives me the karma to do so.
Regards Terry
My challenge is deciding (i) do I work on PHP 5.6 / 5.7 and the
corresponding beta APC version which at current rates of adoption might
have begin to have an impact in the community sometime in the next 5 years,
or (ii) work on a performance patch to the stable APC version which is
typically installed with PHP 5.3 which these guys could apply within a few
months.
or contribute those patches back and integrate them into the vanilla apc?
for me it seems that Rasmus opinion was that having improvements in this
area would be welcome, but personally it was never a high priority for the
current authors and there is a chance that it won't change, so if nobody
steps up and create a patch, there is no guarantee that this will be
addressed soon.
I'll draft the RFCs if the webmaster gives me the karma to do so.
sure, just drop a mail to the php-webmaster list (as far as I can see you
have yet to do that).
--
Ferenc Kovács
@Tyr43l - http://tyrael.hu
My challenge is deciding (i) do I work on PHP 5.6 / 5.7 and the corresponding beta APC version which at current rates of adoption might have begin to have an impact in the community sometime in the next 5 years, or (ii) work on a performance patch to the stable APC version which is typically installed with PHP 5.3 which these guys could apply within a few months.or contribute those patches back and integrate them into the vanilla apc?
Humm. I think that we are sort of saying the same thing, but at cross
purposes. Of course I should offer any up patches for mainstream APC and
at best these will go into 3.1.14 or 3.1.15 and may then get adopted
sometime for production systems whenever -- that's only if the release
of a core O+ doesn't drop APC into legacy status.
However Ubuntu 12.04-LTS is a good example of a stable production stack
and this uses PHP 5.3.10 and APC 3.1.7. Debian Squeeze is even further
behind and it runs PHP 5.3.3 and APC 3.1.3.
A performance patch could also be made available based on the last
stable version of APC, say 3.1.9 -- that is before the attempts to
support the new PHP 5.4 features destabilised it. With this patch, then
at least individual system admins would have the option to download a
stable version from PECL + patch it to use with their production stacks
within the next 3-6 months.
Regards
Terry
On Fri, Feb 22, 2013 at 1:52 PM, Terry Ellison Terry@ellisons.org.ukwrote:
My challenge is deciding (i) do I work on PHP 5.6 / 5.7 and the
corresponding beta APC version which at current rates of adoption might
have begin to have an impact in the community sometime in the next 5 years,
or (ii) work on a performance patch to the stable APC version which is
typically installed with PHP 5.3 which these guys could apply within a few
months.or contribute those patches back and integrate them into the vanilla apc?
Humm. I think that we are sort of saying the same thing, but at cross
purposes. Of course I should offer any up patches for mainstream APC and at
best these will go into 3.1.14 or 3.1.15 and may then get adopted sometime
for production systems whenever -- that's only if the release of a core O+
doesn't drop APC into legacy status.
no comment on that, but I'm expecting/hoping that we will keep apc alive at
least until we have supported versions where O+ isn't in the core (5.3 and
5.4).
However Ubuntu 12.04-LTS is a good example of a stable production stack
and this uses PHP 5.3.10 and APC 3.1.7. Debian Squeeze is even further
behind and it runs PHP 5.3.3 and APC 3.1.3.A performance patch could also be made available based on the last stable
version of APC, say 3.1.9 -- that is before the attempts to support the new
PHP 5.4 features destabilised it. With this patch, then at least
individual system admins would have the option to download a stable version
from PECL + patch it to use with their production stacks within the next
3-6 months.
I still think that it would be better kept as a release, and that we should
try to have a release which is stable with both 5.3 and 5.4, but it's
easier said than done.
--
Ferenc Kovács
@Tyr43l - http://tyrael.hu
Quoting Brendon Colby brendon@newgrounds.com:
We've invested a great deal of money in a high performance shared
storage system, so of course we want to use this system for absolutely
everything we can within reason, including sharing 1500+ PHP files to
our many web servers. I don't think most systems administrators would
find this unreasonable.
Great points Brendon!
We have a similar architectural to your setup running NetApp filers,
Apache/PHP web servers performing shared hosting for over 40,000+
websites.
We pull over 35k IOPS and 80% are from PHP stats and getattr's.
Someone earlier mentioned that people now use GlusterFS as an
alternative. GlusterFS native client runs like absolute crap with
Apache/PHP applications in order to get reasonable performance you
must use the GlusterFS NFS implementation and mount the GlusterFS
using NFS.
You end up in the same performance issues using GlusterFS and PHP.
More and more clustered PHP applications are using GlusterFS via NFS.
Due to the getattr and stat issues in PHP it doesn't scale well in
these sorts of deployments.
In order to recover some of the performance on our NetApp's we have
invested in 1TB of Flash Cache which caches the majority of getattr
and stat requests on the filer however there is still a performance
hit as the requests still go via the network and not cached on the
local client.
It would be great to see the PHP devs address this issue.
Regards,
Dan Zoltak
Quoting daniel@zoltak.com:
Great points Brendon!
We have a similar architectural to your setup running NetApp filers,
Apache/PHP web servers performing shared hosting for over 40,000+
websites.We pull over 35k IOPS and 80% are from PHP stats and getattr's.
Someone earlier mentioned that people now use GlusterFS as an
alternative. GlusterFS native client runs like absolute crap with
Apache/PHP applications in order to get reasonable performance you
must use the GlusterFS NFS implementation and mount the GlusterFS
using NFS.You end up in the same performance issues using GlusterFS and PHP.
More and more clustered PHP applications are using GlusterFS via
NFS. Due to the getattr and stat issues in PHP it doesn't scale well
in these sorts of deployments.
I would also like to add that newer Cloud concentric application are
developed so that they do not persistent data to local storage. This
then negates the use of NFS i.e. we can deploy code via git to the
nodes running on local storage. All dynamically generated user data is
persisted to object storage and sessions are persisted using the
memcache session handler.
However in order to support legacy applications that are not Cloud
centric then a network file system is the only viable option in shared
hosting environments.
Regards,
Dan Zoltak
Hi Brendon,
In our environment, we use NFS for shared storage, we are using APC as well
with stat=0. In our setting, we also experiencing high number of stat()
calls on our file system. My initial finding of this problem is we enabled
the open_basedir setting. And there is already a bug report for this,
https://bugs.php.net/bug.php?id=52312
We tested the issue in 5.2.x, 5.3.x and 5.4.x, all of them experiencing
same issue.
I m not sure if this is a same problem as yours. But I do hope the
community can find out a good solutions to handle the balance of better
security and good performance.
Cheers
On Tue, Feb 19, 2013 at 7:26 AM, Brendon Colby brendon@newgrounds.comwrote:
Greetings,
This seemed like the appropriate list to discuss this - let me know if it
isn't.We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
mostly PHP over NFS (we have separate servers for static content). I
have been trying to figure out why we're seeing so many getattr
requests from our Apache/PHP servers. I think I've narrowed it down to
how PHP/APC gets information about a file to see if it has been
changed since it was last cached:https://bugs.php.net/bug.php?id=59372
Rasmus said:
"Just because you see an open() syscall doesn't mean the cache isn't
used. The open() is there because PHP by default does open+fstat for
performance reasons. If you look carefully you will see there is just
an open() and no read() calls. So it opens the file, uses the stat
struct from the open fd to get the inode and fetches the op_array from
the opcode cache. This can be made more efficient, but this report is
incorrect in assuming the cache isn't being used."This is exactly what I'm seeing when I strace an httpd process -
mostly open() andfstat()calls w/o any read() calls. This makes sense
to me, however, the problem with this is as far as I understand it,
that an open() on an NFS file system causes an automatic gettattr()
call to the server. Doing this completely bypasses the NFS attribute
cache, which is why we're seeing so many getattr requests to our NFS
server despite having an attribute cache timeout of 60 seconds. For
those familiar with NFS, we could just as well disable our attribute
cache at this point, which is almost never recommended because of
performance reasons.I found a good explaination of why this happens in this Redhat document:
"Avoid unnecessarily opening and closing of files. If you access a
file frequently, opening and closing that file triggers the
opentoclose cache
consistency mechanism in the NFS file system. Triggering this
mechanism causes NFS getattr requests to be generated, which can slow
performance. By keeping the file open, traffic is kept to a minimum."I've tried doing apc.stat=0 and about every other combination of
options, but PHP still does an open()+3fstat()commands on every
single page load no matter what I do. We have autoload set, so that
equates to several automatic open() (and subsequent NFS getattr())
calls with every single PHP request. To give you an idea what kind of
load this is generating for us:-Out of 7,000 total NFS operations per second right now:
-About 45% are getattr calls
-Our Apache/PHP servers are generating 52% of all NFS calls
-An average of 82% of those calls are getattr requests
-As a comparison, our far busier content servers (many times the req/s
of the Apache/PHP servers) are generating half the number of NFS
requests our PHP servers are, and 98% or so of those requests are read
requests with just 1% getattrI am pretty proficient with patching/compiling/profiling, but sadly I
am no C coder. I've tried digging into this myself but I'm pretty out
of my element here. I'm curious if anyone on this list could give me
any pointers on what I can do to change this behavior and reduce the
load on our NFS server. If we can't change PHP behavior, we may have
to abandon NFS for our PHP files due to the load PHP appears to be
generating on our NFS server. I'd hate to do this, so please let me
know if any of you have any ideas as to what I can do to change this
behavior.Brendon
In our environment, we use NFS for shared storage, we are using APC as well
with stat=0. In our setting, we also experiencing high number ofstat()
calls on our file system. My initial finding of this problem is we enabled
the open_basedir setting. And there is already a bug report for this,
https://bugs.php.net/bug.php?id=52312We tested the issue in 5.2.x, 5.3.x and 5.4.x, all of them experiencing
same issue.
Kevin, I've just walked through this in 5.3 and 54 and updated this
bugrep. In short there is some silly coding here which should be
addressed. Even if we accept that PHP should comply with
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-5178 if
open_basedir is set, then the cache should only be ignored on the actual
open itself, as this is the only one that is exploitable, but let's have
this debate on the bugrep. Let me think about the security and other
NFRs and propose a patch.
Brendon, in our environment, we use NFS for shared storage, we are
using APC as well with stat=0. In our setting, we are experiencing
high number of stat() calls on the NFS. My initial finding of this
problem is we enabled the open_basedir setting. And there is already a
bug report for this, https://bugs.php.net/bug.php?id=52312
We tested the issue in 5.2.x, 5.3.x and 5.4.x, all of them
experiencing same issue.
I m not sure if this is a same problem as yours. But I do hope the
community can find out a good solutions to handle the balance of
better security and good performance.