Hi Dmitry,
-----Original Message-----
From: Dmitry Stogov [mailto:dmitry@zend.com]
Sent: Monday, September 14, 2015 12:58 PM
To: Anatol Belski ab@php.net; Kalle Sommer Nielsen kalle@php.net;
Rasmus Lerdorf rasmus@lerdorf.com; Xinchen Hui xinchen.h@zend.com;
Nikita Popov nikita.ppv@gmail.com; Zeev Suraski zeev@zend.com; Andi
Gutmans andi@zend.com; PHP Internals internals@lists.php.net
Subject: [PHP-DEV] Re: [PHP-CVS] com php-src: Added an experemental ability
to move PHP code pages (PHP TEXT segment) into HUGE pages. PHP should be
configured and built with --enable-huge-code-pages, OS should be configured to
provide huge pages. It's possible toThe effect of this patch is really exiting to keep it disabled by default... :)
$ perf stat -e instructions,iTLB-loads,iTLB-load-misses,cycles
sapi/cgi/php-cgi -d opcache.enable_huge_code_pages=0 -T 10000
/var/www/html/bench/wordpress-3.6/index.php > /dev/null191,778,620,621 instructions # 0.81 insns per
cycle
178,497,016
iTLB-loads
87,351,276 iTLB-load-misses # 48.94% of all iTLB
cache hits
238,136,402,797
cycles110.221114219 seconds time elapsed
$ perf stat -e instructions,iTLB-loads,iTLB-load-misses,cycles
sapi/cgi/php-cgi -d opcache.enable_huge_code_pages=1 -T 10000
/var/www/html/bench/wordpress-3.6/index.php > /dev/null191,780,181,536 instructions # 0.83 insns per
cycle
39,017,004
iTLB-loads
41,631,359 iTLB-load-misses # 106.70% of all iTLB
cache hits
231,287,491,307
cycles107.742837269 seconds time elapsed
May be we should enable this by default in RC (may be also file cache) and make
decision based on feedback?
The file cache feature has proven itself as stable already, IMHO it should be built by default in RC3.
With the new huge memory pages feature I'd suggest a bit softer - enabling its building, but setting opcache.enable_huge_code_pages=0 by default. Then it can be easy enable and give one more intermediate step for pre testing and go into RC4 completely if ok. What do you think?
Regards
Anatol
On Mon, Sep 14, 2015 at 11:05 PM, Anatol Belski anatol.php@belski.net
wrote:
Hi Dmitry,
-----Original Message-----
From: Dmitry Stogov [mailto:dmitry@zend.com]
Sent: Monday, September 14, 2015 12:58 PM
To: Anatol Belski ab@php.net; Kalle Sommer Nielsen kalle@php.net;
Rasmus Lerdorf rasmus@lerdorf.com; Xinchen Hui xinchen.h@zend.com;
Nikita Popov nikita.ppv@gmail.com; Zeev Suraski zeev@zend.com; Andi
Gutmans andi@zend.com; PHP Internals internals@lists.php.net
Subject: [PHP-DEV] Re: [PHP-CVS] com php-src: Added an experemental
ability
to move PHP code pages (PHP TEXT segment) into HUGE pages. PHP should be
configured and built with --enable-huge-code-pages, OS should be
configured to
provide huge pages. It's possible toThe effect of this patch is really exiting to keep it disabled by
default... :)$ perf stat -e instructions,iTLB-loads,iTLB-load-misses,cycles
sapi/cgi/php-cgi -d opcache.enable_huge_code_pages=0 -T 10000
/var/www/html/bench/wordpress-3.6/index.php > /dev/null191,778,620,621 instructions # 0.81 insns per
cycle
178,497,016
iTLB-loads
87,351,276 iTLB-load-misses # 48.94% of all iTLB
cache hits
238,136,402,797
cycles110.221114219 seconds time elapsed
$ perf stat -e instructions,iTLB-loads,iTLB-load-misses,cycles
sapi/cgi/php-cgi -d opcache.enable_huge_code_pages=1 -T 10000
/var/www/html/bench/wordpress-3.6/index.php > /dev/null191,780,181,536 instructions # 0.83 insns per
cycle
39,017,004
iTLB-loads
41,631,359 iTLB-load-misses # 106.70% of all iTLB
cache hits
231,287,491,307
cycles107.742837269 seconds time elapsed
May be we should enable this by default in RC (may be also file cache)
and make
decision based on feedback?The file cache feature has proven itself as stable already, IMHO it should
be built by default in RC3.
OK.
With the new huge memory pages feature I'd suggest a bit softer - enabling
its building, but setting opcache.enable_huge_code_pages=0 by default.
OK.
Then it can be easy enable and give one more intermediate step for pre
testing and go into RC4 completely if ok.
In general we may get build problems on some platforms...
What do you think?
I didn't get, do you like to delay this for RC4? or enable it at compile
time and disable at run-time by default for RC3?
I agree with both.
Thanks. Dmitry.
Regards
Anatol
-----Original Message-----
From: Dmitry Stogov [mailto:dmitry@zend.com]
Sent: Monday, September 14, 2015 10:37 PM
To: Anatol Belski anatol.php@belski.net
Cc: Anatol Belski ab@php.net; Kalle Sommer Nielsen kalle@php.net;
Rasmus Lerdorf rasmus@lerdorf.com; Xinchen Hui xinchen.h@zend.com;
Nikita Popov nikita.ppv@gmail.com; Zeev Suraski zeev@zend.com; Andi
Gutmans andi@zend.com; PHP Internals internals@lists.php.net
Subject: Re: [PHP-DEV] Re: [PHP-CVS] com php-src: Added an experemental
ability to move PHP code pages (PHP TEXT segment) into HUGE pages. PHP
should be configured and built with --enable-huge-code-pages, OS should be
configured to provide huge pages. It's possiblWith the new huge memory pages feature I'd suggest a bit softer -
enabling its building, but setting opcache.enable_huge_code_pages=0 by
default.OK.
Then it can be easy enable and give one more intermediate step for pre
testing and go into RC4 completely if ok.In general we may get build problems on some platforms...
What do you think?
I didn't get, do you like to delay this for RC4? or enable it at compile time and
disable at run-time by default for RC3?
I agree with both.
I was suggesting the latter - build and disable at run-time. With RC4 I meant - as it's run-time enabled by default currently, that could be turned on by default in RC4 if everything was well in RC3.
Regarding possible build issues you mention - probably some magic in config.m4 could disable huge pages? I wasn't aware, but probably yep - like FreeBSD or OSX likely to not to have the sys/prctl.h . This could be a reason for delaying till RC4, but it's actually easy solved doing some test compilation in config.m4 . I'd see build + disable on run-time in RC3 as a better move because it turns the test circle earlier which is good considering there are only 4 RCs to go yet.
Regards
Anatol
-----Original Message-----
From: Dmitry Stogov [mailto:dmitry@zend.com]
Sent: Monday, September 14, 2015 10:37 PM
To: Anatol Belski anatol.php@belski.net
Cc: Anatol Belski ab@php.net; Kalle Sommer Nielsen kalle@php.net;
Rasmus Lerdorf rasmus@lerdorf.com; Xinchen Hui xinchen.h@zend.com;
Nikita Popov nikita.ppv@gmail.com; Zeev Suraski zeev@zend.com; Andi
Gutmans andi@zend.com; PHP Internals internals@lists.php.net
Subject: Re: [PHP-DEV] Re: [PHP-CVS] com php-src: Added an experemental
ability to move PHP code pages (PHP TEXT segment) into HUGE pages. PHP
should be configured and built with --enable-huge-code-pages, OS should be
configured to provide huge pages. It's possiblWith the new huge memory pages feature I'd suggest a bit softer -
enabling its building, but setting opcache.enable_huge_code_pages=0 by
default.OK.
Then it can be easy enable and give one more intermediate step for pre
testing and go into RC4 completely if ok.In general we may get build problems on some platforms...
What do you think?
I didn't get, do you like to delay this for RC4? or enable it at compile time and
disable at run-time by default for RC3?
I agree with both.I was suggesting the latter - build and disable at run-time. With RC4 I meant - as it's run-time enabled by default currently, that could be turned on by default in RC4 if everything was well in RC3.
Regarding possible build issues you mention - probably some magic in config.m4 could disable huge pages? I wasn't aware, but probably yep - like FreeBSD or OSX likely to not to have the sys/prctl.h . This could be a reason for delaying till RC4, but it's actually easy solved doing some test compilation in config.m4 . I'd see build + disable on run-time in RC3 as a better move because it turns the test circle earlier which is good considering there are only 4 RCs to go yet.
As long as we fail gracefully when compiled with hugepage support and
then run on a system that doesn't have any hugepages configured. These
days the build box and the production execution box are rarely the same
machine.
Redhat enables THP by default and has for a while, but I know lots of
admins that disable this feature right away on production machines. I
don't think it is default on Ubuntu, but at least they have hugepage
tools to make it easy to configure. And from a quick look at Debnian
(jessie) it wasn't obvious how to get hugepages working. I don't see a
package for hugeadm offhand. The kernel supports it, obviously, but you
need to take some steps to configure them.
Which means we also need to spend a bit of time documenting how to
enable hugepages at the OS level for the various operating systems to go
along with this feature.
-Rasmus
-----Original Message-----
From: Dmitry Stogov [mailto:dmitry@zend.com]
Sent: Monday, September 14, 2015 10:37 PM
To: Anatol Belski anatol.php@belski.net
Cc: Anatol Belski ab@php.net; Kalle Sommer Nielsen kalle@php.net;
Rasmus Lerdorf rasmus@lerdorf.com; Xinchen Hui xinchen.h@zend.com;
Nikita Popov nikita.ppv@gmail.com; Zeev Suraski zeev@zend.com; Andi
Gutmans andi@zend.com; PHP Internals internals@lists.php.net
Subject: Re: [PHP-DEV] Re: [PHP-CVS] com php-src: Added an experemental
ability to move PHP code pages (PHP TEXT segment) into HUGE pages. PHP
should be configured and built with --enable-huge-code-pages, OS should
be
configured to provide huge pages. It's possiblWith the new huge memory pages feature I'd suggest a bit softer -
enabling its building, but setting opcache.enable_huge_code_pages=0 by
default.OK.
Then it can be easy enable and give one more intermediate step for pre
testing and go into RC4 completely if ok.In general we may get build problems on some platforms...
What do you think?
I didn't get, do you like to delay this for RC4? or enable it at
compile time and
disable at run-time by default for RC3?
I agree with both.I was suggesting the latter - build and disable at run-time. With RC4 I
meant - as it's run-time enabled by default currently, that could be turned
on by default in RC4 if everything was well in RC3.Regarding possible build issues you mention - probably some magic in
config.m4 could disable huge pages? I wasn't aware, but probably yep - like
FreeBSD or OSX likely to not to have the sys/prctl.h . This could be a
reason for delaying till RC4, but it's actually easy solved doing some test
compilation in config.m4 . I'd see build + disable on run-time in RC3 as a
better move because it turns the test circle earlier which is good
considering there are only 4 RCs to go yet.As long as we fail gracefully when compiled with hugepage support and
then run on a system that doesn't have any hugepages configured. These
days the build box and the production execution box are rarely the same
machine.
Yeah. This is transparent solution. At first it's disabled by default by
opcache.huge_code_pages=0. At second, it should work without problem if OS
doesn't provide huge pages.
Redhat enables THP by default and has for a while, but I know lots of
admins that disable this feature right away on production machines. I
don't think it is default on Ubuntu, but at least they have hugepage
tools to make it easy to configure. And from a quick look at Debnian
(jessie) it wasn't obvious how to get hugepages working. I don't see a
package for hugeadm offhand. The kernel supports it, obviously, but you
need to take some steps to configure them.
PHP prefers using regular Huge Pages and may fail back to THP (Transparent
Huge Pages).
So THP don't have to be enabled.
Which means we also need to spend a bit of time documenting how to
enable hugepages at the OS level for the various operating systems to go
along with this feature.
Right. Manual OS configuration is required. See
https://wiki.debian.org/Hugepages#Enabling_HugeTlbPage
Steps (3) and (4) are optional.
Thanks. Dmitry.
-Rasmus
On Tue, Sep 15, 2015 at 4:31 AM, Rasmus Lerdorf <rasmus@lerdorf.com
mailto:rasmus@lerdorf.com> wrote:
Redhat enables THP by default and has for a while, but I know lots of
admins that disable this feature right away on production machines. I
don't think it is default on Ubuntu, but at least they have hugepage
tools to make it easy to configure. And from a quick look at Debnian
(jessie) it wasn't obvious how to get hugepages working. I don't see a
package for hugeadm offhand. The kernel supports it, obviously, but you
need to take some steps to configure them.PHP prefers using regular Huge Pages and may fail back to THP
(Transparent Huge Pages).
So THP don't have to be enabled.Which means we also need to spend a bit of time documenting how to enable hugepages at the OS level for the various operating systems to go along with this feature.
Right. Manual OS configuration is required. See
https://wiki.debian.org/Hugepages#Enabling_HugeTlbPage
Steps (3) and (4) are optional.
Yup. I did get it working here. Perhaps it would be a good idea to add a
row to the phpinfo()
OpCache section indicating whether huge pages are
currently being used. It would make it easier to debug. I know you can
check /proc/meminfo directly, but it isn't obvious what exactly is using
the hugepages from that.
-Rasmus
On Tue, Sep 15, 2015 at 4:31 AM, Rasmus Lerdorf <rasmus@lerdorf.com
mailto:rasmus@lerdorf.com> wrote:
Redhat enables THP by default and has for a while, but I know lots of
admins that disable this feature right away on production machines. I
don't think it is default on Ubuntu, but at least they have hugepage
tools to make it easy to configure. And from a quick look at Debnian
(jessie) it wasn't obvious how to get hugepages working. I don't see
a
package for hugeadm offhand. The kernel supports it, obviously, but
you
need to take some steps to configure them.PHP prefers using regular Huge Pages and may fail back to THP
(Transparent Huge Pages).
So THP don't have to be enabled.Which means we also need to spend a bit of time documenting how to enable hugepages at the OS level for the various operating systems
to go
along with this feature.
Right. Manual OS configuration is required. See
https://wiki.debian.org/Hugepages#Enabling_HugeTlbPage
Steps (3) and (4) are optional.Yup. I did get it working here. Perhaps it would be a good idea to add a
row to thephpinfo()
OpCache section indicating whether huge pages are
currently being used. It would make it easier to debug. I know you can
check /proc/meminfo directly, but it isn't obvious what exactly is using
the hugepages from that.
It's possible to get detailed information form /proc/<pid>/smaps
Actually, PHP may use huge pages for Zend MM, opcache SHM and now for a
part of PHP code segment.
Thanks. Dmitry.
-Rasmus