Gearman support for php-fpm?

15 years ago by Rasmus Lerdorf — view source

unread

Has anybody thought about adding Gearman support to the fpm sapi?
Managing Gearman workers is pretty much identical to managing fastcgi
workers, so it doesn't seem like much of a stretch.

-Rasmus

15 years ago by Reilly — view source

unread

Rasmus,

I can work on it if no one has it already.

Warm Regards,
Patrick

Has anybody thought about adding Gearman support to the fpm sapi?
Managing Gearman workers is pretty much identical to managing fastcgi
workers, so it doesn't seem like much of a stretch.

-Rasmus

15 years ago by Brian Moon — view source

unread

The big difference is that different processes do different jobs in
Gearman. All PHP workers in fpm just run what ever code is handed to
them. How do you handle that?

Brian.

http://brian.moonspot.net/

Has anybody thought about adding Gearman support to the fpm sapi?
Managing Gearman workers is pretty much identical to managing fastcgi
workers, so it doesn't seem like much of a stretch.

-Rasmus

15 years ago by Tjerk Anne Meesters — view source

unread

All "fpm" workers would be called using the same function name (e.g.
"runphp"). This allows you to reuse a gearmand instance for other
workers too.

The advantages of fpm vs gearman would be mostly performance (it uses
ping and requires an underlying layer on top of tcp/ip), followed by
dynamic spawning and "crash recovery". Think there are more but I
can't access the website from over here ;)

The big difference is that different processes do different jobs in
Gearman. All PHP workers in fpm just run what ever code is handed to
them. How do you handle that?

Brian.

http://brian.moonspot.net/

Has anybody thought about adding Gearman support to the fpm sapi?
Managing Gearman workers is pretty much identical to managing fastcgi
workers, so it doesn't seem like much of a stretch.

-Rasmus

--

--

Tjerk

15 years ago by Rasmus Lerdorf — view source

unread

All "fpm" workers would be called using the same function name (e.g.
"runphp"). This allows you to reuse a gearmand instance for other
workers too.

I don't think we would need to require that. Obviously you could build
your Gearman worker such that it registers "runphp" and has a big switch
in it to run various types of jobs and fpm would just need to manage a
single pool of workers. But, assuming fpm could manage multiple pools,
I don't see the problem with having different types of workers.

-Rasmus

15 years ago by Rasmus Lerdorf — view source

unread

The big difference is that different processes do different jobs in
Gearman. All PHP workers in fpm just run what ever code is handed to
them. How do you handle that?
Threading ;) ?

Definitely not. Gearman is actually the answer to a lot of problems
where people think they need threads.

As far as different workers doing different jobs, I don't see the issue
here. fpm doesn't need to know the details. All it has to know is that
it needs to maintain pools of php processes. Each pool would correspond
to a Gearman job. The workers can take care of registering themselves
with gearmand. All fpm has to do is start up and maintain N processes
per worker pool.

[gearman]
worker1.path = /path/to/worker1.php
worker1.pm = dynamic
worker1.pm.max_children = 10
worker1.pm.start_servers = 5

worker2.path = /path/to/worker2.php
worker2.pm = dynamic
worker2.pm.max_children = 4
worker2.pm.start_servers = 2

...

or however you do multiple nested conf blocks in the fpm .ini file.

In fact, this may not need to be gearman-specific. We could just call
it cli job management or something like that if we left it completely up
to the workers to register with the gearmand. Alternatively, we could
make fpm Gearman-aware and register the workers from there, but I am not
sure I see much of an advantage to that other than making the worker
code slightly simpler.

-Rasmus

15 years ago by Brian Moon — view source

unread

[gearman]
worker1.path = /path/to/worker1.php
worker1.pm = dynamic
worker1.pm.max_children = 10
worker1.pm.start_servers = 5

worker2.path = /path/to/worker2.php
worker2.pm = dynamic
worker2.pm.max_children = 4
worker2.pm.start_servers = 2

So, this is a proposed new entry in the ini format? I am a bit ignorant
about fpm, I admit. This would require a good bit of fiddling for our
gearman set up to work with it. We run X workers that know how to do all
jobs available and then 1 + n (defined by ini file) that only do one job
for each available job. This gives us good coverage with controlled
process load. I guess it would just require a lot more configuration
than we have now. For example, could we pass parameters to the script?
Having a separate script plus a separate class for every function
quickly makes a lot of files.

I will simply stay tuned and see where it goes.

FWIW, this is my GearmanManager that basically does this already via
PHP+pcntl. http://github.com/brianlmoon/GearmanManager. It can not
dynamically ramp up more workers like FPM can. But, I would never run it
that way anyhow. I like my processes under control.

Brian.

15 years ago by Rasmus Lerdorf — view source

unread

[gearman]
worker1.path = /path/to/worker1.php
worker1.pm = dynamic
worker1.pm.max_children = 10
worker1.pm.start_servers = 5

worker2.path = /path/to/worker2.php
worker2.pm = dynamic
worker2.pm.max_children = 4
worker2.pm.start_servers = 2

So, this is a proposed new entry in the ini format? I am a bit ignorant
about fpm, I admit. This would require a good bit of fiddling for our
gearman set up to work with it. We run X workers that know how to do all
jobs available and then 1 + n (defined by ini file) that only do one job
for each available job. This gives us good coverage with controlled
process load. I guess it would just require a lot more configuration
than we have now. For example, could we pass parameters to the script?
Having a separate script plus a separate class for every function
quickly makes a lot of files.

I will simply stay tuned and see where it goes.

FWIW, this is my GearmanManager that basically does this already via
PHP+pcntl. http://github.com/brianlmoon/GearmanManager. It can not
dynamically ramp up more workers like FPM can. But, I would never run it
that way anyhow. I like my processes under control.

Yes, I know, I have seen it. Process management and signal handling
like that really isn't something that should be done in PHP as far as I
am concerned. And FPM already does all of this and more. It just needs
a slight tweak to be able to manage any sort of php process.

-Rasmus