We have a bit of a problem with how Linux handles huge pages when you
run out of them.
I've been discussing it with Dmitry and he recently committed a way to
disable Huge Pages in the main allocator to master:
https://github.com/php/php-src/commit/945a661912612cdecd221cd126feda7d4335c33c
Unfortunately it looks like Linux is very unhappy when a CoW happens to
a huge page allocated in its parent process and it tries to re-map this
page. If there are no available huge pages you get random seg
faults/bus errors and everything blows up. I don't understand why Linux
can't fall back to a regular page here in this case, but it doesn't. It
seems like an OS bug to me, but nonetheless this affects
mod_php/php-fpm since we have a huge page allocated before the fork,
but it also affects any cli script that does a pcntl_fork(). As far as
I can tell there is no way to fix it. We could try to make sure we
never do a MAP_HUGETLB prior to the fork in mod_php/php-fpm but that
still doesn't solve the pcntl_fork() case.
Anatol, we should merge Dmitry's patch into PHP-7.0 and further, I
think we should flip the default to zend_mm_use_huge_pages = 0 in it.
If you have well-behaved code that never tries to allocate a lot of
memory you can turn on huge pages, but I think we should default to
safety here. Those SIGBUS storms are quite alarming and hard to figure
out.
There have also been some other issues with huge pages as per these
bugs:
https://bugs.php.net/70984
https://bugs.php.net/71272
https://bugs.php.net/71355
This is very much an expert feature that we need to document better and
people should know what they are stepping in when they turn this
feature on.
-Rasmus
Hi Rasmus,
-----Original Message-----
From: Rasmus [mailto:rasmus@lerdorf.com]
Sent: Sunday, March 20, 2016 5:29 AM
To: php-internals internals@lists.php.net
Subject: [PHP-DEV] Huge PagesWe have a bit of a problem with how Linux handles huge pages when you run
out of them.I've been discussing it with Dmitry and he recently committed a way to disable
Huge Pages in the main allocator to master:https://github.com/php/php-
src/commit/945a661912612cdecd221cd126feda7d4335c33cUnfortunately it looks like Linux is very unhappy when a CoW happens to a huge
page allocated in its parent process and it tries to re-map this page. If there are
no available huge pages you get random seg faults/bus errors and everything
blows up. I don't understand why Linux can't fall back to a regular page here in
this case, but it doesn't. It seems like an OS bug to me, but nonetheless this
affects mod_php/php-fpm since we have a huge page allocated before the fork,
but it also affects any cli script that does apcntl_fork(). As far as I can tell there
is no way to fix it. We could try to make sure we never do a MAP_HUGETLB prior
to the fork in mod_php/php-fpm but that still doesn't solve thepcntl_fork()case.Anatol, we should merge Dmitry's patch into PHP-7.0 and further, I think we
should flip the default to zend_mm_use_huge_pages = 0 in it.If you have well-behaved code that never tries to allocate a lot of memory you
can turn on huge pages, but I think we should default to safety here. Those
SIGBUSstorms are quite alarming and hard to figure out.There have also been some other issues with huge pages as per these
bugs:https://bugs.php.net/70984 https://bugs.php.net/71272 https://bugs.php.net/71355This is very much an expert feature that we need to document better and people
should know what they are stepping in when they turn this feature on.
Dmitry has already merged this patch into the 7.0 branch. I also know from Remi that Fedora doesn't play well with the huge pages. Huge pages require special system settings which are usually not enabled in OSes by default. Having it off by default in PHP would therefore not cause any functional disadvantage while addressing the stability concerns. Thus reversing the default setting sounds feasible, it could be already done for the upcoming 7.0.5 if there are no objections.
Regards
Anatol
Dmitry has already merged this patch into the 7.0 branch. I also know from Remi that Fedora doesn't play well with the huge pages. Huge pages require special system settings which are usually not enabled in OSes by default. Having it off by default in PHP would therefore not cause any functional disadvantage while addressing the stability concerns. Thus reversing the default setting sounds feasible, it could be already done for the upcoming 7.0.5 if there are no objections.
Yes, and since the special system settings, namely setting
vm.nr_hugepages is a system-wide shared setting and not PHP-specific it
makes even more sense to not enable huge pages by default. If you happen
to install PHP on a system that has vm.nr_hugepages set to non-zero for
some other service on that machine, PHP is going blow up as soon as it
runs out of huge pages.
And the 2nd reason to disable huge pages by default is that, as you say,
on most systems vm.nr_hugepages is going to be 0 but MAP_HUGETLB will be
defined because every modern OS supports huge pages even if they are not
actually enabled. But the allocator doesn't know that. It will try to do
a MAP_HUGETLB mmap every single time here:
https://github.com/php/php-src/blob/master/Zend/zend_alloc.c#L470
which is going to fail with an ENOMEM error and fall back to a
non-HUGETLB mmap. So every alloc there incurs an extra syscall in the
default case.
So we have 3 things to do:
- Default it to off
- Provide a way to specify the huge page size as per bug 71355
- Document the use of USE_ZEND_ALLOC_HUGE_PAGES and how to set
vm.nr_hugepages appropriately along with a warning about what
might happen if don't allocate enough
1 and 3 are trivial. 2 is a little trickier if we want to try to detect
the page size. We could just not try and let the user specify it.
Perhaps even use ZEND_ALLOC_HUGE_PAGES directly. As in:
ZEND_ALLOC_HUGE_PAGES=2048
would turn on huge pages in the allocator with 2k pages.
-Rasmus
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Le 20/03/2016 05:29, Rasmus a écrit :
We have a bit of a problem with how Linux handles huge pages when
you run out of them.
Yes, especially on old kernel (2.6.32 on RHEL-6)
For the build in my repository :
-
- RHEL-6 : build with --disable-huge-code-pages
-
- RHEL-7 and Fedora : only enabled on x86_64 (php.ini)
I think we should flip the default to zend_mm_use_huge_pages = 0 in
it.
+1
Remi.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlbvnskACgkQYUppBSnxahhifACgjys/mVvAwJEiT40njsSRnVYC
ZrUAoMKxgxkdktKDZnNsLNUkqYUqv2o5
=nBDN
-----END PGP SIGNATURE
We have a bit of a problem with how Linux handles huge pages when you
run out of them.I've been discussing it with Dmitry and he recently committed a way to
disable Huge Pages in the main allocator to master:https://github.com/php/php-src/commit/945a661912612cdecd221cd126feda7d4335c33c
Unfortunately it looks like Linux is very unhappy when a CoW happens to
a huge page allocated in its parent process and it tries to re-map this
page. If there are no available huge pages you get random seg faults/bus
errors and everything blows up. I don't understand why Linux can't fall
back to a regular page here in this case, but it doesn't. It seems like
an OS bug to me, but nonetheless this affects mod_php/php-fpm since we
have a huge page allocated before the fork, but it also affects any cli
script that does apcntl_fork(). As far as I can tell there is no way to
fix it. We could try to make sure we never do a MAP_HUGETLB prior to the
fork in mod_php/php-fpm but that still doesn't solve thepcntl_fork()case.Anatol, we should merge Dmitry's patch into PHP-7.0 and further, I think
we should flip the default to zend_mm_use_huge_pages = 0 in it.If you have well-behaved code that never tries to allocate a lot of
memory you can turn on huge pages, but I think we should default to
safety here. ThoseSIGBUSstorms are quite alarming and hard to figure out.There have also been some other issues with huge pages as per these bugs:
https://bugs.php.net/70984 https://bugs.php.net/71272 https://bugs.php.net/71355This is very much an expert feature that we need to document better and
people should know what they are stepping in when they turn this feature
on.-Rasmus
This explains a recent issue reported for GearmanManager on PHP 7 as well.
PHP Warning: pcntl_fork(): Error 12 in
/usr/local/share/gearman-manager/GearmanManager.php on line 929
Error 12 is a memory allocation error.
https://github.com/brianlmoon/GearmanManager/issues/136
--