Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:65908
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain newgrounds.com designates 209.85.214.45 as permitted sender)
MIME-Version: 1.0
Date: Mon, 18 Feb 2013 15:26:13 -0500
Message-ID: <CAHHbRnEUVcGqH6+++CWXM6XWobJn5MzKxLdfX9RHxoJ0uXaZdg@mail.gmail.com>
To: internals@lists.php.net
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Subject: PHP causing high number of NFS getattr operations?
From: brendon@newgrounds.com (Brendon Colby)

Greetings,

This seemed like the appropriate list to discuss this - let me know if it i=
sn't.

We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
mostly PHP over NFS (we have separate servers for static content). I
have been trying to figure out why we're seeing so many getattr
requests from our Apache/PHP servers. I think I've narrowed it down to
how PHP/APC gets information about a file to see if it has been
changed since it was last cached:

https://bugs.php.net/bug.php?id=3D59372

Rasmus said:

"Just because you see an open() syscall doesn't mean the cache isn't
used. The open() is there because PHP by default does open+fstat for
performance reasons. If you look carefully you will see there is just
an open() and no read() calls. So it opens the file, uses the stat
struct from the open fd to get the inode and fetches the op_array from
the opcode cache. This can be made more efficient, but this report is
incorrect in assuming the cache isn't being used."

This is exactly what I'm seeing when I strace an httpd process -
mostly open() and fstat() calls w/o any read() calls. This makes sense
to me, however, the problem with this is as far as I understand it,
that an open() on an NFS file system causes an automatic gettattr()
call to the server. Doing this completely bypasses the NFS attribute
cache, which is why we're seeing so many getattr requests to our NFS
server despite having an attribute cache timeout of 60 seconds. For
those familiar with NFS, we could just as well disable our attribute
cache at this point, which is almost never recommended because of
performance reasons.

I found a good explaination of why this happens in this Redhat document:

http://www.redhat.com/rhecm/rest-rhecm/jcr/repository/collaboration/jcr:sys=
tem/jcr:versionStorage/5e75391d7f00000110a97b91452f0cee/1/jcr:frozenNode/rh=
:pdfFile.pdf

"Avoid unnecessarily opening and closing of files. If you access a
file frequently, opening and closing that file triggers the
open=ADto=ADclose cache
consistency mechanism in the NFS file system.  Triggering this
mechanism causes NFS getattr requests to be generated, which can slow
performance. By keeping the file open, traffic is kept to a minimum."

I've tried doing apc.stat=3D0 and about every other combination of
options, but PHP still does an open()+3 fstat() commands on every
single page load no matter what I do. We have autoload set, so that
equates to several automatic open() (and subsequent NFS getattr())
calls with every single PHP request. To give you an idea what kind of
load this is generating for us:

-Out of 7,000 total NFS operations per second right now:
-About 45% are getattr calls
-Our Apache/PHP servers are generating 52% of all NFS calls
-An average of 82% of those calls are getattr requests
-As a comparison, our far busier content servers (many times the req/s
of the Apache/PHP servers) are generating half the number of NFS
requests our PHP servers are, and 98% or so of those requests are read
requests with just 1% getattr

I am pretty proficient with patching/compiling/profiling, but sadly I
am no C coder. I've tried digging into this myself but I'm pretty out
of my element here. I'm curious if anyone on this list could give me
any pointers on what I can do to change this behavior and reduce the
load on our NFS server. If we can't change PHP behavior, we may have
to abandon NFS for our PHP files due to the load PHP appears to be
generating on our NFS server. I'd hate to do this, so please let me
know if any of you have any ideas as to what I can do to change this
behavior.

Brendon