Hi,
it's the second time I'm running into this: In a rather big project I'm
using gearman to enable semi-parallel processing of many small jobs. I'm
using PHP 5.6.22-0+deb8u1 with OPcache 7.0.6-dev on Debian.
All was well before the code needed refactoring for a new feature: 30
workers processed 4000 jobs/second at around 5% CPU usage.
After refactoring (no functional changes), some times after starting the
workers, processing is getting slower and slower. After about 2 hours I
hit 100% CPU usage, at about 300 jobs/second.
Memory usage only doubles from start to stall, which is what I observed
before when everything was fine.
I suspect this to be a problem of the cycle collector, or with garbage
collecting in general. So at any given time, I want PHP to dump all the
zval's around, which I have to evaluate manually to find "problematic"
ones. This could be triggered by either a function call, or by
adding/extending PHP's signal handler and sending that signal to the
interpreter process.
Where do I have to look at in the sources? Or is there already a tool
which does what I want, but didn't find yet?
I also learned this could be hash collisions related to reordering and
renaming the data in my code, but as I'm currently stuck with 5.6 I
can't use the HashDoS related patches floating around here.
Any hints on where to find the culprit?
Thanks,
Thomas
Hi,
it's the second time I'm running into this: In a rather big project I'm
using gearman to enable semi-parallel processing of many small jobs. I'm
using PHP 5.6.22-0+deb8u1 with OPcache 7.0.6-dev on Debian.All was well before the code needed refactoring for a new feature: 30
workers processed 4000 jobs/second at around 5% CPU usage.
After refactoring (no functional changes), some times after starting the
workers, processing is getting slower and slower. After about 2 hours I hit
100% CPU usage, at about 300 jobs/second.Memory usage only doubles from start to stall, which is what I observed
before when everything was fine.I suspect this to be a problem of the cycle collector, or with garbage
collecting in general. So at any given time, I want PHP to dump all the
zval's around, which I have to evaluate manually to find "problematic"
ones. This could be triggered by either a function call, or by
adding/extending PHP's signal handler and sending that signal to the
interpreter process.Where do I have to look at in the sources? Or is there already a tool
which does what I want, but didn't find yet?I also learned this could be hash collisions related to reordering and
renaming the data in my code, but as I'm currently stuck with 5.6 I can't
use the HashDoS related patches floating around here.Any hints on where to find the culprit?
Thanks,
Thomas
Before trying anything more sophisticated, I'd suggest to use "perf record"
to check whether the GC is indeed the culprit. If it is, you'll see
functions like "gc_mark_grey" etc taking the majority of the time.
Nikita