# PHP causing high number of NFS getattr operations?

- Thread: https://externals.io/message/65908
- Original archive: https://news-web.php.net/php.internals/65908
- Started by Brendon Colby on 2013-02-18T20:26:13+00:00
- Messages: 33
- Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/65908.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #65908 — Brendon Colby — 2013-02-18T20:26:13+00:00

URL: https://externals.io/message/65908#65908

Greetings,

This seemed like the appropriate list to discuss this - let me know if it isn't.

We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
mostly PHP over NFS (we have separate servers for static content). I
have been trying to figure out why we're seeing so many getattr
requests from our Apache/PHP servers. I think I've narrowed it down to
how PHP/APC gets information about a file to see if it has been
changed since it was last cached:

https://bugs.php.net/bug.php?id=59372

Rasmus said:

"Just because you see an open() syscall doesn't mean the cache isn't
used. The open() is there because PHP by default does open+fstat for
performance reasons. If you look carefully you will see there is just
an open() and no read() calls. So it opens the file, uses the stat
struct from the open fd to get the inode and fetches the op_array from
the opcode cache. This can be made more efficient, but this report is
incorrect in assuming the cache isn't being used."

This is exactly what I'm seeing when I strace an httpd process -
mostly open() and `fstat()` calls w/o any read() calls. This makes sense
to me, however, the problem with this is as far as I understand it,
that an open() on an NFS file system causes an automatic gettattr()
call to the server. Doing this completely bypasses the NFS attribute
cache, which is why we're seeing so many getattr requests to our NFS
server despite having an attribute cache timeout of 60 seconds. For
those familiar with NFS, we could just as well disable our attribute
cache at this point, which is almost never recommended because of
performance reasons.

I found a good explaination of why this happens in this Redhat document:

http://www.redhat.com/rhecm/rest-rhecm/jcr/repository/collaboration/jcr:system/jcr:versionStorage/5e75391d7f00000110a97b91452f0cee/1/jcr:frozenNode/rh:pdfFile.pdf

"Avoid unnecessarily opening and closing of files. If you access a
file frequently, opening and closing that file triggers the
open­to­close cache
consistency mechanism in the NFS file system.  Triggering this
mechanism causes NFS getattr requests to be generated, which can slow
performance. By keeping the file open, traffic is kept to a minimum."

I've tried doing apc.stat=0 and about every other combination of
options, but PHP still does an open()+3 `fstat()` commands on every
single page load no matter what I do. We have autoload set, so that
equates to several automatic open() (and subsequent NFS getattr())
calls with every single PHP request. To give you an idea what kind of
load this is generating for us:

-Out of 7,000 total NFS operations per second right now:
-About 45% are getattr calls
-Our Apache/PHP servers are generating 52% of all NFS calls
-An average of 82% of those calls are getattr requests
-As a comparison, our far busier content servers (many times the req/s
of the Apache/PHP servers) are generating half the number of NFS
requests our PHP servers are, and 98% or so of those requests are read
requests with just 1% getattr

I am pretty proficient with patching/compiling/profiling, but sadly I
am no C coder. I've tried digging into this myself but I'm pretty out
of my element here. I'm curious if anyone on this list could give me
any pointers on what I can do to change this behavior and reduce the
load on our NFS server. If we can't change PHP behavior, we may have
to abandon NFS for our PHP files due to the load PHP appears to be
generating on our NFS server. I'd hate to do this, so please let me
know if any of you have any ideas as to what I can do to change this
behavior.

Brendon

---

## #65909 — Rasmus Lerdorf — 2013-02-18T20:35:20+00:00

URL: https://externals.io/message/65908#65909 · In reply to: #65908

> Rasmus said:
> 
> "Just because you see an open() syscall doesn't mean the cache isn't
> used. The open() is there because PHP by default does open+fstat for
> performance reasons. If you look carefully you will see there is just
> an open() and no read() calls. So it opens the file, uses the stat
> struct from the open fd to get the inode and fetches the op_array from
> the opcode cache. This can be made more efficient, but this report is
> incorrect in assuming the cache isn't being used."
> 
> This is exactly what I'm seeing when I strace an httpd process -
> mostly open() and `fstat()` calls w/o any read() calls. This makes sense
> to me, however, the problem with this is as far as I understand it,
> that an open() on an NFS file system causes an automatic gettattr()
> call to the server. Doing this completely bypasses the NFS attribute
> cache, which is why we're seeing so many getattr requests to our NFS
> server despite having an attribute cache timeout of 60 seconds. For
> those familiar with NFS, we could just as well disable our attribute
> cache at this point, which is almost never recommended because of
> performance reasons.

Wow, people are still serving web files over NFS? Sounds painful.

But, this is what APC's apc.include_once_override setting tries to
address. Try turning that on.

-Rasmus

---

## #65910 — Brendon Colby — 2013-02-18T21:03:26+00:00

URL: https://externals.io/message/65908#65910 · In reply to: #65909

> 
>
> > […]
> Wow, people are still serving web files over NFS? Sounds painful.
>
> But, this is what APC's apc.include_once_override setting tries to
> address. Try turning that on.

Yep - I think it would be a lot more painful if we didn't use a very
expensive NetApp Filer to serve NFS (the solution before the Filer
would lock up every few weeks). I'm really surprised more people
haven't complained about this. I've seen a few complaints about people
seeing poor PHP app performance on NFS, but they usually move to
gluster or something else before ever figuring it out. Maybe very few
people use NFS to serve web files - who knows.

I tried apc.include_once_override - no change. I set up Apache to one
process and straced it. I clicked around on the site a few times and
see the typical pattern:

open("[PATH]/CacheFile.class.php", O_RDONLY) = 10
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0

Out of 486 open() calls, 263 were for PHP files (54%). Here is my
current APC config:

extension=/usr/local/lib/php/extensions/apc.so
apc.enabled=1
apc.shm_segments=1
apc.optimization=0
apc.shm_size=256M
apc.ttl=1200
apc.user_ttl=1200
apc.num_files_hint=300
apc.mmap_file_mask=/apc.shm.XXXXXX
apc.slam_defense=0
apc.write_lock=1
apc.include_once_override=1

apc.stat=0 doesn't help. Every PHP request causes an open() and
subsequent getattr() to the Filer no matter what options I set.

---

## #65911 — Damien Tournoud — 2013-02-18T21:32:29+00:00

URL: https://externals.io/message/65908#65911 · In reply to: #65910

> I tried apc.include_once_override - no change. I set up Apache to one
> process and straced it. I clicked around on the site a few times and
> see the typical pattern:
>
> open("[PATH]/CacheFile.class.php", O_RDONLY) = 10
> fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
> fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
> fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0

Assuming that those are relative includes, can you try with:

  apc.canonicalize=0
  apc.stat=0

Damien

---

## #65912 — Brendon Colby — 2013-02-18T21:47:36+00:00

URL: https://externals.io/message/65908#65912 · In reply to: #65911

> Assuming that those are relative includes, can you try with:
>
>   apc.canonicalize=0
>   apc.stat=0

Paths are absolute. stat=0 (and canonicalize=0 just to try it)
produced the same result.

Brendon

---

## #65915 — Terry Ellison — 2013-02-19T00:10:55+00:00

URL: https://externals.io/message/65908#65915 · In reply to: #65912

> 
> > […]
> Paths are absolute. stat=0 (and canonicalize=0 just to try it)
> produced the same result.
>
> Brendon
>

Brendon, are your scripts doing a log of include_once / require_once 
calls?  In you look at ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER() in 
Zend/zend_vm_execute.h then you will see that this Zend handler does 
zend_resolve_path() for any xxx_ONCE instructions and zend_stream_open() 
on the first request of the same.  Yes APC rehooks this handler with a 
wrapper but this is only for lazy loading.  When it honors the xxx_once 
instructions, it will still open the streams even if the code itself is 
fully cached in APC and the I/O is entirely nugatory.  I suspect that 
this could generate the NFS traffic that you are discussing.

This would be easy to avoid, but this would require replacing this 
handler entirely or doing dynamic code patching, nether of which APC 
currently does, I believe.  Incidentally because this is Zend feature 
and nothing directly to with APC,  O+ will also have this runtime 
characteristic.

//Terry

---

## #65917 — Kevin Yung — 2013-02-19T01:30:02+00:00

URL: https://externals.io/message/65908#65917 · In reply to: #65908

Hi Brendon,

In our environment, we use NFS for shared storage, we are using APC as well
with stat=0. In our setting, we also experiencing high number of `stat()`
calls on our file system. My initial finding of this problem is we enabled
the open_basedir setting. And there is already a bug report for this,
https://bugs.php.net/bug.php?id=52312

We tested the issue in 5.2.x, 5.3.x and 5.4.x, all of them experiencing
same issue.

I m not sure if this is a same problem as yours. But I do hope the
community can find out a good solutions to handle the balance of better
security and good performance.

Cheers

On Tue, Feb 19, 2013 at 7:26 AM, Brendon Colby <brendon@newgrounds.com>wrote:

> Greetings,
>
> This seemed like the appropriate list to discuss this - let me know if it
> isn't.
>
> We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
> mostly PHP over NFS (we have separate servers for static content). I
> have been trying to figure out why we're seeing so many getattr
> requests from our Apache/PHP servers. I think I've narrowed it down to
> how PHP/APC gets information about a file to see if it has been
> changed since it was last cached:
>
> https://bugs.php.net/bug.php?id=59372
>
> Rasmus said:
>
> "Just because you see an open() syscall doesn't mean the cache isn't
> used. The open() is there because PHP by default does open+fstat for
> performance reasons. If you look carefully you will see there is just
> an open() and no read() calls. So it opens the file, uses the stat
> struct from the open fd to get the inode and fetches the op_array from
> the opcode cache. This can be made more efficient, but this report is
> incorrect in assuming the cache isn't being used."
>
> This is exactly what I'm seeing when I strace an httpd process -
> mostly open() and `fstat()` calls w/o any read() calls. This makes sense
> to me, however, the problem with this is as far as I understand it,
> that an open() on an NFS file system causes an automatic gettattr()
> call to the server. Doing this completely bypasses the NFS attribute
> cache, which is why we're seeing so many getattr requests to our NFS
> server despite having an attribute cache timeout of 60 seconds. For
> those familiar with NFS, we could just as well disable our attribute
> cache at this point, which is almost never recommended because of
> performance reasons.
>
> I found a good explaination of why this happens in this Redhat document:
>
>
> http://www.redhat.com/rhecm/rest-rhecm/jcr/repository/collaboration/jcr:system/jcr:versionStorage/5e75391d7f00000110a97b91452f0cee/1/jcr:frozenNode/rh:pdfFile.pdf
>
> "Avoid unnecessarily opening and closing of files. If you access a
> file frequently, opening and closing that file triggers the
> open­to­close cache
> consistency mechanism in the NFS file system.  Triggering this
> mechanism causes NFS getattr requests to be generated, which can slow
> performance. By keeping the file open, traffic is kept to a minimum."
>
> I've tried doing apc.stat=0 and about every other combination of
> options, but PHP still does an open()+3 `fstat()` commands on every
> single page load no matter what I do. We have autoload set, so that
> equates to several automatic open() (and subsequent NFS getattr())
> calls with every single PHP request. To give you an idea what kind of
> load this is generating for us:
>
> -Out of 7,000 total NFS operations per second right now:
> -About 45% are getattr calls
> -Our Apache/PHP servers are generating 52% of all NFS calls
> -An average of 82% of those calls are getattr requests
> -As a comparison, our far busier content servers (many times the req/s
> of the Apache/PHP servers) are generating half the number of NFS
> requests our PHP servers are, and 98% or so of those requests are read
> requests with just 1% getattr
>
> I am pretty proficient with patching/compiling/profiling, but sadly I
> am no C coder. I've tried digging into this myself but I'm pretty out
> of my element here. I'm curious if anyone on this list could give me
> any pointers on what I can do to change this behavior and reduce the
> load on our NFS server. If we can't change PHP behavior, we may have
> to abandon NFS for our PHP files due to the load PHP appears to be
> generating on our NFS server. I'd hate to do this, so please let me
> know if any of you have any ideas as to what I can do to change this
> behavior.
>
> Brendon

---

## #65916 — Kevin Yung — 2013-02-19T01:44:13+00:00

URL: https://externals.io/message/65908#65916 · In reply to: #65908

Brendon, in our environment, we use NFS for shared storage, we are
using APC as well with stat=0. In our setting, we are experiencing
high number of `stat()` calls on the NFS. My initial finding of this
problem is we enabled the open_basedir setting. And there is already a
bug report for this, https://bugs.php.net/bug.php?id=52312

We tested the issue in 5.2.x, 5.3.x and 5.4.x, all of them
experiencing same issue.

I m not sure if this is a same problem as yours. But I do hope the
community can find out a good solutions to handle the balance of
better security and good performance.

---

## #65918 — Stas Malyshev — 2013-02-19T01:50:09+00:00

URL: https://externals.io/message/65908#65918 · In reply to: #65909

Hi!

> Wow, people are still serving web files over NFS? Sounds painful.

Sometimes, there's not much (better) choices if you need to keep
writeable files in sync over a number of machines. There are other
shared FSes but they would lead to pretty much the same issues.

-- 
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

---

## #65919 — Rasmus Lerdorf — 2013-02-19T01:55:53+00:00

URL: https://externals.io/message/65908#65919 · In reply to: #65918

> Hi!
> 
> > […]
> 
> Sometimes, there's not much (better) choices if you need to keep
> writeable files in sync over a number of machines. There are other
> shared FSes but they would lead to pretty much the same issues.

Yeah, but NFS, especially without the realpath cache, which you lose if
you turn on open_basedir, is deathly slow because of all the stats.

Typically PHP scripts are not actually "writable files" and the way to
keep them in synch across multiple machines is to use a deploy
mechanism/rsynch to deploy them. You may very well have some writable
files that need to be shared at the app-level, but then we wouldn't be
talking about PHP stat calls and the realpath cache. And most people
have moved to something like memcache for sharing writable data across
machines.

-Rasmus

---

## #65921 — Stas Malyshev — 2013-02-19T02:14:39+00:00

URL: https://externals.io/message/65908#65921 · In reply to: #65919

Hi!

> Yeah, but NFS, especially without the realpath cache, which you lose if
> you turn on open_basedir, is deathly slow because of all the stats.
> 
> Typically PHP scripts are not actually "writable files" and the way to
> keep them in synch across multiple machines is to use a deploy
> mechanism/rsynch to deploy them. You may very well have some writable

rsync may be way too slow if you need the change appear more or less
"immediately" on all servers. Unless you run rsync each second if which
case you're probably better off with NFS :)

> files that need to be shared at the app-level, but then we wouldn't be
> talking about PHP stat calls and the realpath cache. And most people

You would still if the data is stored as php file. Which last time I
tested - given APC or something like - was the fastest way to get a
bunch of data into PHP (it was faster than loading and unserializing,
even with APC/memcache).

> have moved to something like memcache for sharing writable data across
> machines.

memcache is non-persistent. It's a cache, not a storage solution, so you
can not use it for anything that actually needs to be stored permanently.
-- 
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

---

## #65922 — Rasmus Lerdorf — 2013-02-19T02:32:49+00:00

URL: https://externals.io/message/65908#65922 · In reply to: #65921

> Hi!
> 
> > […]
> 
> rsync may be way too slow if you need the change appear more or less
> "immediately" on all servers. Unless you run rsync each second if which
> case you're probably better off with NFS :)
> 
> > […]
> 
> You would still if the data is stored as php file. Which last time I
> tested - given APC or something like - was the fastest way to get a
> bunch of data into PHP (it was faster than loading and unserializing,
> even with APC/memcache).
> 
> > […]
> 
> memcache is non-persistent. It's a cache, not a storage solution, so you
> can not use it for anything that actually needs to be stored permanently.

Sure, but then you can go with something like Redis.

But, again, if you go back to the original question, this has nothing to
do with often-changing data in a couple of PHP include files:

  We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
  mostly PHP over NFS (we have separate servers for static content)."

So they are serving up all their PHP over NFS for some reason.

-Rasmus

---

## #65927 — Terry Ellison — 2013-02-19T09:36:09+00:00

URL: https://externals.io/message/65908#65927 · In reply to: #65922

The point that this thread highlights is that apps developers / 
administrators at both ends of the scale -- the enterprise and the 
shared service user -- normally have little say over the infrastructure 
architecture on which their application runs.  In both these cases the 
infrastructure will be hosting 100s if not 1000s of apps, and the 
environment cannot be tailored to any one app.  So issues like storage 
architecture and security architecture (e.g. UID-based enforcement of 
app separation) are a given as NFRs (non-functional requirements). Use 
of NFS with server / storage separation is still a standard 
implementation design pattern, when enterprises require simple-to-manage 
scalability on these tiers.  We aren't going to change this, and neither 
will the O/P.

Surely PHP will achieve better penetration and the greater acceptance if 
it can offer more robust performance in this sort of environment?
> Sure, but then you can go with something like Redis.
>
> But, again, if you go back to the original question, this has nothing to
> do with often-changing data in a couple of PHP include files:
>
>    We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
>    mostly PHP over NFS (we have separate servers for static content)."
>
> So they are serving up all their PHP over NFS for some reason.

---

## #65960 — Terry Ellison — 2013-02-19T16:06:22+00:00

URL: https://externals.io/message/65908#65960 · In reply to: #65927

> The point that this thread highlights is that apps developers / 
> administrators at both ends of the scale -- the enterprise and the 
> shared service user -- normally have little say over the 
> infrastructure architecture on which their application runs.  In both 
> these cases the infrastructure will be hosting 100s if not 1000s of 
> apps, and the environment cannot be tailored to any one app.  So 
> issues like storage architecture and security architecture (e.g. 
> UID-based enforcement of app separation) are a given as NFRs 
> (non-functional requirements). Use of NFS with server / storage 
> separation is still a standard implementation design pattern, when 
> enterprises require simple-to-manage scalability on these tiers.  We 
> aren't going to change this, and neither will the O/P.
>
> Surely PHP will achieve better penetration and the greater acceptance 
> if it can offer more robust performance in this sort of environment?
> > […]
>
Brendon,

Just to follow up with a bit more detail, apart from the obvious NFS 
tuning with things like the actimeo mount parameters, you can get a 
better idea of what is going on if you use a local copy of one of your 
apps running under a local linux apache server.  This is an example from 
my Ubuntu laptop, but it works just as well on a test VM if you still 
use WinXX on your PC.

    trace_wget () {
    	sleep 1
    	coproc sudo strace -p $(ps -C apache2 --no-headers -o pid) -tt -o /tmp/strace$1.log
    	sleep 1; ST_PID=$(ps -C strace --no-headers -o pid)
    	wget "http://localhost/$2" -O /dev/null -o /dev/null
    	sleep 2; sudo kill $ST_PID
    }
    # start Apache in debug mode
    sudo service apache2 stop
    sudo bash -c ". /etc/apache2/envvars; coproc apache2 -X"

    # trace three gets
    trace_wget 0 phpinfo.php
    trace_wget 1 mwiki/index.php?title=Main_Page
    trace_wget 2 mwiki/index.php?title=Main_Page

    #restart normally
    sudo service apache2 stop
    sudo service apache2 start
    sudo chmod 777 /tmp/strace?.log

    grep -c "open(" /tmp/strace?.log
    grep -c "stat(" /tmp/strace?.log

The first get is just to load and prime the mod_php5 thread (Ubuntu has 
a crazy localtime implementation), the second loads and compiles the PHP 
MediaWiki modules needed to render a page, the third repeats this now 
fully cached in ACP.  In this case,
we have:

                       open()    `fstat()`/lstat()
    /tmp/strace1.log   108           650      (Priming the APC cache)
    /tmp/strace2.log    27           209      (Using the APC cache)

So APC *does* materially reduce the I/O calls, and (if you look at the 
traces) it removes most of the mmaps and compilation.  The MWiki 
autoloader uses require() to load classes but in the code as a whole the 
most common method of loading modules is require_once() (414 uses as 
opposed to 44 in total of the other include or require functions) and as 
I said in my previous post, this I/O is avoidable by recoding the 
ZEND_INCLUDE_OR_EVAL hander to work cooperatively with the opcode cache 
when present.

Note that even when the cache does this, it still can't optimize coding 
patterns like:

         if ( file_exists( "$IP/AdminSettings.php" ) ) {
             require( "$IP/AdminSettings.php" );
         }

and IIRC mwiki does 4 of these on a page render.  Here a pattern based 
on (@include($file) == 1) is more cache-friendly.

Now this isn't a material performance issue when the code tree is 
mounted on local storage (as is invariably the case for a developer test 
configuration) as the stat / open overhead of a pre-(file) cached file 
is microseconds, but it is material on scalable production 
infrastructure which uses network attached NFS or NTFS file systems.

So in summary, this is largely avoidable but unfortunately I don't think 
that we can persuade the dev team to address this issue, even if I gave 
them the solution as a patch :-(

Regards
Terry Ellison

---

## #65992 — Stas Malyshev — 2013-02-19T20:32:18+00:00

URL: https://externals.io/message/65908#65992 · In reply to: #65960

Hi!

> and IIRC mwiki does 4 of these on a page render.  Here a pattern based
> on (@include($file) == 1) is more cache-friendly.

This goes back to the fact that @ does not really disable errors - it
only disables reporting of the errors, but the whole message is
generated and goes all the cycle up to the actual error reporting before
being suppressed. I've tried to address this problem a number of times
but looks like there's really no way to do it without sacrificing some
parts of current error reporting such as track_errors.

> So in summary, this is largely avoidable but unfortunately I don't think
> that we can persuade the dev team to address this issue, even if I gave
> them the solution as a patch :-(

Could you explain what you're talking about here?

-- 
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

---

## #65999 — Terry Ellison — 2013-02-19T23:08:34+00:00

URL: https://externals.io/message/65908#65999 · In reply to: #65992

> Hi!
>
> > […]
> This goes back to the fact that @ does not really disable errors - it
> only disables reporting of the errors, but the whole message is
> generated and goes all the cycle up to the actual error reporting before
> being suppressed. I've tried to address this problem a number of times
> but looks like there's really no way to do it without sacrificing some
> parts of current error reporting such as track_errors.
Yup, in this scenario we're trading off two things.  What the pattern is 
saying  is if the source file exists then compile it. However, if the 
file exists, in say 90+% of the cases, and the code is often cached then 
the file_exists check generates I/O that isn't needed if the file is 
already APC (or O+ / xcache) cached.  Yup if the file is missing then 
the error is generated and trapped in the reporting cycle, but the final 
result is that the result is false and the false==1 fails.  Other than 
the wasted cycles in running up this stack, there aren't any other 
side-effects that I am aware of. Are there any?

Yes, this is an overhead, but it is small beer compared to doing a 
getattr() RPC across the server room fabric to a NAS server backend.
> > […]
> Could you explain what you're talking about here?
The fact that you are engaging in this dialogue is great and I really 
appreciate it.  What I am still trying to work out is how to interact 
with you guys in a way that is mutually productive.  I did make the 
mistake of choosing the mainstay stable version iN live prod systems use 
-- 5.3 -- to take this whole issue apart and I guess that the dev team 
regards this as history.

I guess that I should bite the bullet and switch to 5.5.  I've been 
working on an evaluatorfork of APC optimized for CLI/GCI which tackles a 
lot of these issues head on and performs reasonable well, but I realise 
that this is a dead-end and will never get deployed, but I am currently 
considering regressing some of this technology into 5.5 and O+.  Are you 
interested in a version of O+ which supports all SAPIs?

In architectural terms I feel that having a universal cache option is 
important.  It changes the mindset from something which is only used in 
niche performance use cases to a standard option.  It also means that 
the cache can be stress tested by the entire php test suite.  However, 
to do some of this you don't start with a patch, but with an RFC 
informed by evidence, and that's my real reason for doing this demonstrator.

//Terry

---

## #66016 — Christopher Jones — 2013-02-20T01:51:29+00:00

URL: https://externals.io/message/65908#66016 · In reply to: #65999

> I guess that I should bite the bullet and switch to 5.5.

Yes.

> I've been working on an evaluatorfork of APC optimized for CLI/GCI
> which tackles a lot of these issues head on and performs reasonable
> well, but I realise that this is a dead-end and will never get
> deployed, but I am currently considering regressing some of this
> technology into 5.5 and O+.  Are you interested in a version of O+
> which supports all SAPIs?

Yes, but there are some SAPIs that are more common, so these should be
prioritized if trade-offs are required.

My particular desire is for "embed" SAPI support.  We use this in our
Tuxedo application server, but it is a relatively niche use case.

> In architectural terms I feel that having a universal cache option
> is important.  It changes the mindset from something which is only
> used in niche performance use cases to a standard option.  It also
> means that the cache can be stress tested by the entire php test
> suite.  However, to do some of this you don't start with a patch,
> but with an RFC informed by evidence, and that's my real reason for
> doing this demonstrator.

I'm looking forward to seeing your suggestions and patches.

Chris

-- 
christopher.jones@oracle.com  http://twitter.com/ghrd
Newly updated, free PHP & Oracle book:
http://www.oracle.com/technetwork/topics/php/underground-php-oracle-manual-098250.html

---

## #66027 — Stas Malyshev — 2013-02-20T08:26:58+00:00

URL: https://externals.io/message/65908#66027 · In reply to: #65999

Hi!

> Yes, this is an overhead, but it is small beer compared to doing a
> getattr() RPC across the server room fabric to a NAS server backend.

That depends of what your error handlers do. Some may write to log
files, etc. if not configured properly (since error_reporting setting
doesn't have to be considered in it).
IIRC, for most of the cases O+ should be able to resolve all
includes/requires on cached files without syscalls - but file_exists is
a different matter since caching it in generic case can be very dangerous.

> I guess that I should bite the bullet and switch to 5.5.  I've been
> working on an evaluatorfork of APC optimized for CLI/GCI which tackles a
> lot of these issues head on and performs reasonable well, but I realise
> that this is a dead-end and will never get deployed, but I am currently
> considering regressing some of this technology into 5.5 and O+.  Are you
> interested in a version of O+ which supports all SAPIs? 

I think right now O+ can support CLI (provided enable_cli is set) but
for most cases it's kind of useless since scripts are rarely re-included
in common CLI scenarios. So if you know how to improve common CLI
scenarios it may be interesting, though I imagine it's not the most
common use case. But if it adds there without problems for anything
else, why not.

-- 
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

---

## #66061 — Terry Ellison — 2013-02-20T15:00:07+00:00

URL: https://externals.io/message/65908#66061 · In reply to: #66027

> That depends of what your error handlers do. Some may write to log 
> files, etc. if not configured properly (since error_reporting setting 
> doesn't have to be considered in it). IIRC, for most of the cases O+ 
> should be able to resolve all includes/requires on cached files 
> without syscalls - but file_exists is a different matter since caching 
> it in generic case can be very dangerous. 
I am not suggesting caching file_exists but rather encouraging coding 
patterns which avoid its use /if/ the application is intended to give 
good cached performance -- e.g. apps like mwiki, wordpress and drupal.

> > […]
> I think right now O+ can support CLI (provided enable_cli is set) but
> for most cases it's kind of useless since scripts are rarely re-included
> in common CLI scenarios. So if you know how to improve common CLI
> scenarios it may be interesting, though I imagine it's not the most
> common use case. But if it adds there without problems for anything
> else, why not.
Let me get a dev stack for 5.5 and O+ up and then I can comment on 
this.  As far as APC's support for CLI goes -- the cache doesn't connect 
to the apache2 SMA even if you run the cli process in the same UID, and 
since the SMA gets released when the process count goes to zero, caching 
doesn't really do anything as the cache discarded between cli 
executions, so if practice you always run uncached.

As to the frequency of the use case, maybe not in the case of cli, but 
cgi is still tremendously common, as shared hosting providers still need 
to implement UID-based separation of scripting environments for 
different user accounts.  There are still significant scaling issues 
with php-fpm in this usecase, so only a few SHPs offer FastCGI support 
and then only as a premium option. BTW, I saw that you recommended in 
one of your blog posts that you to use a bytecode cache if you care 
about performance should do. However, a lot of app maintainers who use a 
shared hosting service do care about performance but don't have this as 
an option :-(

For example with my cli-based LPC opcode cache, I can do a series of  
"make test" runs (1) with the opcache disabled; (2) with it enabled but 
the cache empty and (3) with it enabled and the cache(s) primed.  
Ideally all three (*) should give the same test results, and this gives 
you confidence that the cache is working properly.  I would like to do 
this with APC or O+, but in practice AFAIK, I can't do (3) so how do I 
or the developers know what PHP and extension features aren't being 
supported / working properly when the code is cached?

(*) With the tests as currently written, some fail with case (3) because 
the runs are missing compiler warnings which are only generated if the 
code isn't cached (and in case of my cache at its current build quite a 
few fail because it still doesn't play nicely with other extensions like 
phar.)

---

## #66071 — Brendon Colby — 2013-02-20T19:26:25+00:00

URL: https://externals.io/message/65908#66071 · In reply to: #65915

>
> Brendon, are your scripts doing a log of include_once / require_once calls?

Yep, but we also changed our autoload script (auto_prepend_file) to
use require instead of require_once, and we still saw the same
behavior (open calls for the PHP files required by the
auto_prepend_file same as before).

> In you look at ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER() in
> Zend/zend_vm_execute.h then you will see that this Zend handler does
> zend_resolve_path() for any xxx_ONCE instructions and zend_stream_open() on
> the first request of the same.  Yes APC rehooks this handler with a wrapper
> but this is only for lazy loading.  When it honors the xxx_once
> instructions, it will still open the streams even if the code itself is
> fully cached in APC and the I/O is entirely nugatory.  I suspect that this
> could generate the NFS traffic that you are discussing.

Yes if the streams are opened whether cached or not, then this will
result in an automatic getattr request to the NFS server which is
exactly what we're seeing.

Brendon

---

## #66082 — Brendon Colby — 2013-02-20T23:52:22+00:00

URL: https://externals.io/message/65908#66082 · In reply to: #65960

> Brendon,
>
> Just to follow up with a bit more detail, apart from the obvious NFS tuning
> with things like the actimeo mount parameters, you can get a better idea of
> what is going on if you use a local copy of one of your apps running under a
> local linux apache server.  This is an example from my Ubuntu laptop, but it
> works just as well on a test VM if you still use WinXX on your PC.
>
> trace_wget () {
> 	sleep 1
> 	coproc sudo strace -p $(ps -C apache2 --no-headers -o pid) -tt -o
> /tmp/strace$1.log
> 	sleep 1; ST_PID=$(ps -C strace --no-headers -o pid)
> 	wget "http://localhost/$2" -O /dev/null -o /dev/null
> 	sleep 2; sudo kill $ST_PID
> }
> # start Apache in debug mode
> sudo service apache2 stop
> sudo bash -c ". /etc/apache2/envvars; coproc apache2 -X"
>
> # trace three gets
> trace_wget 0 phpinfo.php
> trace_wget 1 mwiki/index.php?title=Main_Page
> trace_wget 2 mwiki/index.php?title=Main_Page
>
> #restart normally
> sudo service apache2 stop
> sudo service apache2 start
> sudo chmod 777 /tmp/strace?.log
>
> grep -c "open(" /tmp/strace?.log
> grep -c "stat(" /tmp/strace?.log
>
> The first get is just to load and prime the mod_php5 thread (Ubuntu has a
> crazy localtime implementation), the second loads and compiles the PHP
> MediaWiki modules needed to render a page, the third repeats this now fully
> cached in ACP.  In this case,
> we have:
>
>                   open()    `fstat()`/lstat()
> /tmp/strace1.log   108           650      (Priming the APC cache)
> /tmp/strace2.log    27           209      (Using the APC cache)
>
> So APC *does* materially reduce the I/O calls, and (if you look at the
> traces) it removes most of the mmaps and compilation.  The MWiki autoloader
> uses require() to load classes but in the code as a whole the most common
> method of loading modules is require_once() (414 uses as opposed to 44 in
> total of the other include or require functions) and as I said in my
> previous post, this I/O is avoidable by recoding the ZEND_INCLUDE_OR_EVAL
> hander to work cooperatively with the opcode cache when present.
>
> Note that even when the cache does this, it still can't optimize coding
> patterns like:
>
>         if ( file_exists( "$IP/AdminSettings.php" ) ) {
>             require( "$IP/AdminSettings.php" );
>         }
>
> and IIRC mwiki does 4 of these on a page render.  Here a pattern based on
> (@include($file) == 1) is more cache-friendly.
>
> Now this isn't a material performance issue when the code tree is mounted on
> local storage (as is invariably the case for a developer test configuration)
> as the stat / open overhead of a pre-(file) cached file is microseconds, but
> it is material on scalable production infrastructure which uses network
> attached NFS or NTFS file systems.
>
> So in summary, this is largely avoidable but unfortunately I don't think
> that we can persuade the dev team to address this issue, even if I gave them
> the solution as a patch :-(
>
> Regards
> Terry Ellison

Terry,

Thanks for your detailed input. This is essentially what I did in my test setup.

APC is definitely critical to the performance of our site. I'm just
very curious to see how much impact the high number of getattr
operations coming from the Apache/PHP servers is having on the filer.
It looks like we might be able to move our PHP files to local storage
to at least measure the impact this is having.

Brendon

---

## #66084 — Terry Ellison — 2013-02-21T00:26:32+00:00

URL: https://externals.io/message/65908#66084 · In reply to: #66082

> Terry,
>
> Thanks for your detailed input. This is essentially what I did in my test setup.
>
> APC is definitely critical to the performance of our site. I'm just
> very curious to see how much impact the high number of getattr
> operations coming from the Apache/PHP servers is having on the filer.
> It looks like we might be able to move our PHP files to local storage
> to at least measure the impact this is having.
>
> Brendon
>
Brendon,

I think that PHP is a great platform; it's just that PHP's sweet spot is 
just a little off the optimum for typical scalable enterprise 
infrastructures.  IMO, the main issue that you should consider and 
discuss with your infrastructure support team is how to mitigate the 
impacts for such high-throughput business critical applications. As you 
(and Rasmus previously, IIRC) mention one key issue is what is on local 
and what is on network attached storage.

On one system that I got hauled in to troubleshoot, it turned out that 
the main problem was that the PHP session data was being written to a 
directory in shared storage, causing write-though overload that ended up 
hitting half a dozen key apps.  It was a trivial configuration change to 
move it to local storage transforming performance.  (I apologise if I am 
telling how "to suck eggs" but this is really for others tracking this 
thread) the main issue here is that the file hierarchies that must have 
some integrity over the application tier need to be on the NAS, 
everything else is a matter of convenience vs performance, so for 
example having a cron job to rsync the apps PHP hierarchy onto local 
storage might well transform your app performance and give an indirect 
boost to cohosted apps.  Happy hunting :)

---

## #66139 — Brendon Colby — 2013-02-21T23:15:44+00:00

URL: https://externals.io/message/65908#66139 · In reply to: #65922

> Wow, people are still serving web files over NFS? Sounds painful.



> So they are serving up all their PHP over NFS for some reason.

I've been thinking about this and wanted to make a few final points. I
think this is potentially a pretty major issue and not something to be
dismissed.

We've invested a great deal of money in a high performance shared
storage system, so of course we want to use this system for absolutely
everything we can within reason, including sharing 1500+ PHP files to
our many web servers. I don't think most systems administrators would
find this unreasonable.

Why would we want to split our architecture up so just our PHP files
are being synced from local disk and everything else is shared via
NFS? Doing this seems far more unreasonable to me and just doesn't
make any sense.

NFS is so common for sharing files that saying "Wow, people are still
serving web files over NFS?" is like saying "Wow, people are still
using the ls command to list directory contents on Linux?" I think NFS
is still very widely used, even for sharing web files.

This being said, you stated in bug #59372 that "PHP by default does
open+fstat for performance reasons" and while doing an open+fstat
might result in better performance on local file systems, doing so is
not optimal when using NFS. NFS "close-to-open cache consistency" is
designed such that opening a file stored on an NFS server causes an
automatic getattr or access request. This could also be true for other
shared file systems as others have mentioned.

All of these open+fstats are definitely causing a measurable impact to
our systems. Right now, for example, 50% of the 6-8K+ ops/s being
processed by our filer are getattr operations, and a majority of these
getattr calls are coming from our Apache/PHP servers. Doing open+fstat
bypasses the NFS attribute cache similar to disabling it altogether,
where a simple stat would likely utilize the attribute cache.
Disabling the attribute cache is generally not recommended due to the
performance impact this has on the filer (and the application).

I don't think the appropriate answer is "don't use NFS" because this
is ridiculous as a long term solution (NFS is common, and people are
going to use it or something similar). I think the appropriate answer
is to update PHP to use stat vs. open+fstat or doing something similar
that would be optimized for both local AND shared file systems (I
would be writing a patch instead of this email if I could).

Until then, I think it's important that people know that PHP is NOT
optimized for use with NFS (and possilby other shared file systems)
even when using an opcode cache like APC. I think this is a very
important piece of information that people need to know. Up until this
week, I wouldn't have thought twice about sharing PHP files over NFS,
and I've been a Unix admin for over twelve years (only new to this
environment).

Brendon

---

## #66141 — Rasmus Lerdorf — 2013-02-21T23:38:49+00:00

URL: https://externals.io/message/65908#66141 · In reply to: #66139

> NFS is so common for sharing files that saying "Wow, people are still
> serving web files over NFS?" is like saying "Wow, people are still
> using the ls command to list directory contents on Linux?" I think NFS
> is still very widely used, even for sharing web files.

This is simply not true. I do have a fair bit of experience in this
field, and I don't know of any major sites that do this and I have
worked with a good chunk of the largest sites out there.

> I don't think the appropriate answer is "don't use NFS" because this
> is ridiculous as a long term solution (NFS is common, and people are
> going to use it or something similar). I think the appropriate answer
> is to update PHP to use stat vs. open+fstat or doing something similar
> that would be optimized for both local AND shared file systems (I
> would be writing a patch instead of this email if I could).

If it is of such importance to you and you are not able to do it
yourself, then hire someone to do it. We may or may not get around to
it, but like most things in PHP, we work on what we need ourselves and I
don't think anybody here would even consider putting all their PHP files
on an NFS share when performance was important.

-Rasmus

---

## #66142 — daniel@zoltak.com — 2013-02-21T23:42:55+00:00

URL: https://externals.io/message/65908#66142 · In reply to: #66139

Quoting Brendon Colby <brendon@newgrounds.com>:
> We've invested a great deal of money in a high performance shared
> storage system, so of course we want to use this system for absolutely
> everything we can within reason, including sharing 1500+ PHP files to
> our many web servers. I don't think most systems administrators would
> find this unreasonable.

Great points Brendon!

We have a similar architectural to your setup running NetApp filers,  
Apache/PHP web servers performing shared hosting for over 40,000+  
websites.

We pull over 35k IOPS and 80% are from PHP stats and getattr's.

Someone earlier mentioned that people now use GlusterFS as an  
alternative. GlusterFS native client runs like absolute crap with  
Apache/PHP applications in order to get reasonable performance you  
must use the GlusterFS NFS implementation and mount the GlusterFS  
using NFS.

You end up in the same performance issues using GlusterFS and PHP.  
More and more clustered PHP applications are using GlusterFS via NFS.  
Due to the getattr and stat issues in PHP it doesn't scale well in  
these sorts of deployments.

In order to recover some of the performance on our NetApp's we have  
invested in 1TB of Flash Cache which caches the majority of getattr  
and stat requests on the filer however there is still a performance  
hit as the requests still go via the network and not cached on the  
local client.

It would be great to see the PHP devs address this issue.

Regards,
Dan Zoltak

---

Next page: https://externals.io/message/65908.md?page=2