Zend JIT Open Sourced

On Fri, Feb 27, 2015 at 10:36 PM, Anthony Ferrara ircmaxell@gmail.com
wrote:

Dmitry,

That's not to say there's anything wrong with this approach, nor that
there isn't a ton we can learn from it. I think it's a fantastic
research effort and plan on digging through it myself. Thank you for
open sourcing it.

Thanks for good words :)

This work may be adopted for some specific cases.
25-30 times speedup on Mandelbrot allows usage for numeric calculation
instead of C.

https://gist.github.com/dstogov/12323ad13d3240aee8f1

anyone may repeat the language battle :)

These tests seem really odd. A 15% speed advantage over GCC -O2? Sure,
it's possible. But I don't think it's likely. It really smells to me
like bias in the testing methodology. (and the lack of an -O3 result
is suspicious as well).

No. it true, but of course it's not 100% fair.
gcc compiles files for x86 or x86_64 platform in general.
When we compile in run-time we may relay on knowledge of our CPU.
In this case LLVM generates AVX instructions, while gcc SSE2.
Looking into assembler code, you may see that PHP even not inferred type of
all variables and makes few unnecessary check in the loop, but modern CPU
are so smart that the code looking mach worse work with the same speed as
(gcc -O2). Unfortunately it works in the other direction as well.

gcc -O2 -mavx will outperform us :)

And looking at the code, I can see why. The PHP version is writing to
an internal buffer, while every other version has to write to STDOUT
on every single iteration.

So you are intentionally not benchmarking the output in the PHP
version (you even explicitly call ob_start()) but are benchmarking it
in every other version. So in fact, the PHP code does something
different than the rest of the code.

Sneaky sneaky. Also completely fake. A proper methodology would have
explicitly disabled any buffer so that the tests all tested the same
thing. Or even better, build up an internal buffer in all of the
implementations. That way you can compare the computation and not rely
on STDOUT (terminal) response.

this also may make some difference, but I think PHP stream layer is not as
good as C.
just profile it with Linux perf, oprofile, callgrind, ...

perf record <command>

perf report -n

Thanks. Dmitry

Anthony

Dmitry,

Sneaky sneaky. Also completely fake.

It's been brought to my attention that some people have taken what I
said completely out of context and insinuated it as a direct insult to
you. I assure you that was not the intent (I called the benchmark
sneaky and fake, which it is).

So if you interpreted it as an insult, I apologize sincerely. I did
not intend to insult you personally at all with that remark. It was
only meant as a comment about the code that was posted.

Anthony

On Fri, Feb 27, 2015 at 10:36 PM, Anthony Ferrara ircmaxell@gmail.com
wrote:

Dmitry,

That's not to say there's anything wrong with this approach, nor that
there isn't a ton we can learn from it. I think it's a fantastic
research effort and plan on digging through it myself. Thank you for
open sourcing it.

Thanks for good words :)

This work may be adopted for some specific cases.
25-30 times speedup on Mandelbrot allows usage for numeric calculation
instead of C.

https://gist.github.com/dstogov/12323ad13d3240aee8f1

anyone may repeat the language battle :)

These tests seem really odd. A 15% speed advantage over GCC -O2? Sure,
it's possible. But I don't think it's likely. It really smells to me
like bias in the testing methodology. (and the lack of an -O3 result
is suspicious as well).

And looking at the code, I can see why. The PHP version is writing to
an internal buffer, while every other version has to write to STDOUT
on every single iteration.

So you are intentionally not benchmarking the output in the PHP
version (you even explicitly call ob_start()) but are benchmarking it
in every other version. So in fact, the PHP code does something
different than the rest of the code.

Sneaky sneaky. Also completely fake.

Please, be polite. We opened sources, and the sources of benchmarks. Anyone
can repeat this.
Smart people may analyze results themselves before claiming others.
I think you are smart person, and I respect the things you are doing.

Thanks. Dmitry.

A proper methodology would have

explicitly disabled any buffer so that the tests all tested the same
thing. Or even better, build up an internal buffer in all of the
implementations. That way you can compare the computation and not rely
on STDOUT (terminal) response.

Anthony

-----Original Message-----
From: Anthony Ferrara [mailto:ircmaxell@gmail.com]
Sent: Friday, February 27, 2015 9:36 PM
To: Dmitry Stogov
Cc: Zeev Suraski; Jordi Boggiano; PHP Internals
Subject: Re: [PHP-DEV] Re: Zend JIT Open Sourced

And looking at the code, I can see why. The PHP version is writing to an
internal buffer, while every other version has to write to STDOUT on every
single iteration.

Except stdout is buffered too (www.turnkeylinux.org/blog/unix-buffering).
Perhaps there's some difference there, but it's not nearly as obvious as you
claim (let alone the other stuff in the other email, but let's leave that
aside).
Personally, my money would be on glibc stdout's buffering being more
efficient than our output buffering layer. But you're welcome to test.

Zeev

Dmitry,

That's not to say there's anything wrong with this approach, nor that
there isn't a ton we can learn from it. I think it's a fantastic
research effort and plan on digging through it myself. Thank you for
open sourcing it.

Thanks for good words :)

This work may be adopted for some specific cases.
25-30 times speedup on Mandelbrot allows usage for numeric calculation
instead of C.

https://gist.github.com/dstogov/12323ad13d3240aee8f1

anyone may repeat the language battle :)

These tests seem really odd. A 15% speed advantage over GCC -O2? Sure,
it's possible. But I don't think it's likely. It really smells to me
like bias in the testing methodology. (and the lack of an -O3 result
is suspicious as well).

And looking at the code, I can see why. The PHP version is writing to
an internal buffer, while every other version has to write to STDOUT
on every single iteration.

So you are intentionally not benchmarking the output in the PHP
version (you even explicitly call ob_start()) but are benchmarking it
in every other version. So in fact, the PHP code does something
different than the rest of the code.

We actually discussed this at the time of the results.
IIRC it really has nothing to do with the output mechanism, etc.. The benchmark does enough iterations and very little output that the impact there is negligible (you can test this yourself to see if I am right but I am pretty sure I am).
It is due to the fact that at runtime LLVM can optimize better to the architecture than a static standard gcc build. Constraining gcc with the right architecture dependent switches upfront will also improve the gcc results. Anyway, still pretty cool to see this although it has very little impact (if any) on real world apps ala Magent, WordPress, Drupal, ...

I think the important learning is that faster synthetic benchmarks have very little impact on overall application performance. Sure it can have an impact on specific algorithmic pieces of code but that’s the exception not the norm. No doubt there are other ways to write JIT including tracing JITs etc. but I do think we found that we are more bound by I/O and memory/caches than the quality of the machine code as the engine is already quite tight. And with apps consuming more and more Cloud services the I/O bottleneck issue looks grimmer than ever! :) That also comes across consistently in benchmarks of PHP 7 vs. hhvm on real-world apps - you see a JIT and non-JIT platform pretty much head to head on performance and actually on the complex stuff PHP 7 is often faster.

Anyway, definitely makes sense to continue to look at these kind of opportunities down the road but PHP 7 is such a huge step-up on real world application performance I think getting that out the door is the biggest possible short-term win when it comes to performance. Looking forward to seeing folks dig into the code and have ideas down the road!!

Andi