Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:116916
MIME-Version: 1.0
References: <c2f04ec1-d62e-47ab-ac19-f43ab28101d7@Spark>
In-Reply-To: <c2f04ec1-d62e-47ab-ac19-f43ab28101d7@Spark>
Date: Sun, 23 Jan 2022 20:39:57 +0000
Message-ID: <CAEKnhAGH0Yq2a+t1z9-O9Jpjug=eeRzEimjF6VTd9ZjYHQ2X1A@mail.gmail.com>
To: Adam Hamsik <adam.hamsik@lablabs.io>
Cc: PHP internals list <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="00000000000092f5aa05d645db3b"
Subject: Re: [PHP-DEV] Best way to monitor php-fpm container liveness on Kubernetes
From: bukka@php.net (Jakub Zelenka)

--00000000000092f5aa05d645db3b
Content-Type: text/plain; charset="UTF-8"

Hi

On Sat, Jan 22, 2022 at 8:00 PM Adam Hamsik <adam.hamsik@lablabs.io> wrote:

> Hello,
>
> We are using PHP for our application backends, this works very well as we
> have developed s imple way to clone them with minimal effort(they can be
> very similar). For our orchestration we are using Kubernetes (>= 1.21). Our
> application pod generally contains NGINX + php-fpm and fluentbit for log
> shipping. We generally want to have LivenessProbe(for an simple explanation
> this is a simple check which is run against our pod to verify if it's
> alive, if it fails particular container will be restarted).
>
> This works very we(we are also using swoole which is roughly 80-70%
> better)l, but in certain unstable situations when we see higher application
> latency (db problem or a bug in our application). We often experience
> problems, because pods are falsely marked as dead (failed liveness probe
> and restarted by kubelet). This happens all processes in our static pool
> are allocated to application requests. For our livenessProbe we tried to
> use both fpm.ping and fpm.status endpoints but both of them behave in a
> same way as they are managed with worker processes.
>
> I had a look at pgp-src repo if e.g. we can use signals to verify if
> application server is running as a way to go around our issue. When looking
> at this I saw fpm-systemd.c which is a SystemD specific check. This check
> reports fpm status every couple seconds(configurable to systemd). Would you
> be willing to integrate similar feature for kubernetes. This would be based
> on a pull model probably with and REST interface.
>
> My idea is following:
>
> 1) During startup if this is enabled php-fpm master will open a secondary
> port pm.health_port(9001) and listen for a pm.health_path(/healtz)[2].
> 2) If we receive GET request fpm master process will respond with HTTP
> code 200 and string ok. If anything is wrong (we can later add some
> checks/metrics to make sure fpm is in a good state).  If we do not respond
> or fpm is not ok our LivenessProbe will fail. based on configuration this
> will trigger container restart.
>
> Would you be interested to integrate feature like this ? or is there any
> other way how we can achieve similar results ?
>
>
This is already being tracked as https://bugs.php.net/bug.php?id=68678 .
You could probably already use pm.status_listen but which should cover what
you need. We should eventually add support for ping.listen . It's a bit
lower priority on my list because status already sort of cover it but if
someone wants to send a PR for that, than I will be happy to review it - I
would imagine that the implementation will be pretty much the same like
pm.status_listen.

I'm wondering if it would be also useful to add HTTP support for ping
because currently it's just FCGI so you still need to hit it through web
server, right? Adding HTTP support could be a bit harder as it would need
to be a different listener...

Regards

Jakub

--00000000000092f5aa05d645db3b--