# php daemons, memory

- Thread: https://externals.io/message/42546
- Original archive: https://news-web.php.net/php.internals/42546
- Started by Thomas Koch on 2009-01-10T17:30:34+00:00
- Messages: 9
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #42546 — Thomas Koch — 2009-01-10T17:30:34+00:00

URL: https://externals.io/message/42546#42546

Hi,

I wrote some daemon scripts for a web crawler in PHP. Now I spent one
day to work around the growing memory consumtion of these scripts.

Since I use a MySQL connection, Syslog and many classes, I wanted to let
the script run a while before restarting. So the scripts had lines like:

$i = 1000;
while( --$i )
{
  while( run() ); // run returns false if there are no pending jobs
  `gc_collect_cycles()`;
  // echo get_memory_usage();
  sleep( 20 );
}

I thought, that most memory should be cleaned up after each termination
of the run() function, but the reported memory usage grew rapidly.

- Was it a mistake to use PHP for such scripts? What language should I've been 
choosing instead?

- --enable-debug did report some small leaks, but much less then the
  consumption grow.

- My suspicion is, that either pdo_mysql or dom are not freeing their
  used memory during a request. Is that possible?

- It would help me a lot, if I could easily get an overview, what is
  consuming my memory. Which Zvals are known? Which extension / line did 
allocate how much memory?

- I'm using http://libslack.org/daemon now to control the script
  execution. This gave me the idea for a special kind of PHP binary
"php-daemon":
  - php-daemon is an executable that restarts a given php script in a
    loop
  - php-daemon can be combined with apc/xcache to store the bytecode
  Unfortunately I'm still to much a newbie to write this myself.

Thanks for your time reading this,
-- 
Thomas Koch, http://www.koch.ro
YMC AG, http://www.ymc.ch

---

## #42547 — Brian Moon — 2009-01-10T18:39:18+00:00

URL: https://externals.io/message/42546#42547 · In reply to: #42546

First, this is not an internals question. But, I would be not many 
people on the general list could help you.

> I wrote some daemon scripts for a web crawler in PHP. Now I spent one
> day to work around the growing memory consumtion of these scripts.

I have PHP daemons running for weeks that don't use up a lot of memory. 
  You have to be careful what you do in a PHP script.  I would guess you 
are running Linux.  Assuming you are, the linux memory system never 
frees allocated memory back to the OS.  Not until the process dies at 
least.  So, if any part of your code ever needs a lot of memory, that 
will be allocated to the process until it ends.

> - Was it a mistake to use PHP for such scripts? What language should I've been
> choosing instead?

Any language has its caveats.  And you can waste (leak is a strong word) 
memory with any language.

> - My suspicion is, that either pdo_mysql or dom are not freeing their
>    used memory during a request. Is that possible?

Are you telling pdo to free its results?  If not, that is bad 
programming.  You have to unset vars and free db result sets yourself to 
ensure they are not building up.  PHP uses a lazy garbage collector that 
is optimized for short lived web scripts.  You have to overcome that 
when working with PHP in a non-web environment.

> - I'm using http://libslack.org/daemon now to control the script
>    execution. This gave me the idea for a special kind of PHP binary
> "php-daemon":
>    - php-daemon is an executable that restarts a given php script in a
>      loop
>    - php-daemon can be combined with apc/xcache to store the bytecode
>    Unfortunately I'm still to much a newbie to write this myself.

My standard way to handle scripts that need a lot of memory is to use 
the pcntl functions to fork children that run and do the work.  They can 
end after a certain time or memory usage.  Again, you have to make sure 
the main, parent script is well written and does not waste file 
descriptors, connections, etc.

Hmm, maybe I will write a blog post about this.

-- 

Brian Moon
Senior Web Engineer
------------------------------
When you care enough to spend the very least.
http://dealnews.com/

---

## #42548 — jvlad — 2009-01-11T00:21:23+00:00

URL: https://externals.io/message/42546#42548 · In reply to: #42546

> Since I use a MySQL connection, Syslog and many classes, I wanted to let
> the script run a while before restarting. So the scripts had lines like:
>
> $i = 1000;
> while( --$i )
> {
>  while( run() ); // run returns false if there are no pending jobs
>  `gc_collect_cycles()`;
>  // echo get_memory_usage();
>  sleep( 20 );
> }
>
> - Was it a mistake to use PHP for such scripts? What language should I've 
> been
> choosing instead?

Almost.
Try to compile your very-own-version of php, using configure argument
--disable-zend-memory-manager
also, disable _all_ php extensions that your scripts do not depend on.

>
> - --enable-debug did report some small leaks, but much less then the
>  consumption grow.

zend memory manager won't release memory to the system. Memory held by ZMM
quickly becomes fragmented into smaller chunks than needed, so it takes from 
the system
more and more "fresh" memory. Definitely this approach is not for your 
situation (php daemons).

-jv

---

## #42552 — Markus Fischer — 2009-01-11T05:10:04+00:00

URL: https://externals.io/message/42546#42552 · In reply to: #42546

Hi,

albeit *not* as a daemon, we've successfully developed a Crawler in PHP
within our company. It can run for hours without a leak, if I remember
correctly it's peak memory consumption is below 64MB. However we're
crawling only a small amount of URLs, just around 10.000 .

As Brian mentioned: free your database resources, unset unused
variables. We've had one major rewrite which, besides re-architecturing
the whole thing for plugin/modularity, involved auditing every step to
make sure resources are properly freed. Usually a PHP developer doesn't
have to pay much attention to it because of the wide-used process-fork
model (but I guess I don't need to tell you that :).

But you'll get often beaten by PHP itself:
it has quite some leaks and finding/tracking them done costs time,
sometimes requires skill at the C level of PHP to properly
understand/diagnose things and if you were (unfortunately) successful in
identifying a PHP problem you've report a bug, preferable attach provide
a patch/workaround.

For example, we've had to fight http://bugs.php.net/bug.php?id=43450 .
Tracking this PHP problem was quite time consuming, involving multiple
developers, etc. Luckily we could work around this, but it was pretty
annoying.

We actually planned to release this as open source, donate it to Zend,
whatever. Legally it's done within the company, just no one had the time
for the publishing process, going over things, etc. :/

As a sidenote: We've hit the current limit of our Crawler implementation
in PHP itself: we can't to parallel fetching/processing of URLs in a
efficient manner. You can get things quick running in PHP, but doing
things with style and a serious architecture hits its limits. We've gone
to Java for such cases, made sense for us anyway as we had to move away
from Zend_Search_Lucene as it had performance problems with our index
where as Lucene/Solr was still mostly bored. Will be interesting to see
if http://code.google.com/p/marjory/ can handle this. Ops, off-topic.



HTH,
- Markus

---

## #42553 — Antony Dovgal — 2009-01-11T09:07:21+00:00

URL: https://externals.io/message/42546#42553 · In reply to: #42548

> Almost.
> Try to compile your very-own-version of php, using configure argument
> --disable-zend-memory-manager

There is no such configure option since ages.
Zend MM can be disabled by setting USE_ZEND_ALLOC env var to 0, but only when 
--enable-debug is used (since nobody is supposed to do that except for debugging 
purposes).

> also, disable _all_ php extensions that your scripts do not depend on.

That won't affect memory usage very much.

> - --enable-debug did report some small leaks, but much less then the
> > […]
> 
> zend memory manager won't release memory to the system. 

That's plain wrong.
Of course it does free() memory whenever it thinks the memory should be free()-ed. 
Though that doesn't guarantee that the OS is be able to reuse this memory.

-- 
Wbr, 
Antony Dovgal

---

## #42558 — jvlad — 2009-01-11T14:51:36+00:00

URL: https://externals.io/message/42546#42558 · In reply to: #42553

> 
> > […]
>
> There is no such configure option since ages.

seems you're flying too high.
RHEL5 still ships php5.1.6 and it has this option and it undefines 
ZEND_USE_ZEND_ALLOC.
For the newer versions (5.2.x) it can be set at the environment, which is 
not that good btw.

> That won't affect memory usage very much.

I gives appox 2MB per php-cli instance. Sometimes it's good enough, if for 
example you need to run 200 instances.


> > […]
>
> That's plain wrong.
> Of course it does free() memory whenever it thinks the memory should be 
> free()-ed.
> Though that doesn't guarantee that the OS is be able to reuse this memory.

Antony, I think you're wrong with this. ZMM does never release memory to the 
system. Instead it puts free chunks into its own free memory chain.
See how _zend_mm_free_int() works in 5.2.5 and it's the only function that 
is called from for example _efree, at least when ZMM is enabled.

-regards,
jv.

---

## #42559 — jvlad — 2009-01-11T15:13:37+00:00

URL: https://externals.io/message/42546#42559 · In reply to: #42558

> ZMM does never release memory to the system.

my appologies, "never" is a too strong word for that. I'd use "almost 
never".
Indeed ZMM does release segment to the OS if all the memory blocks in it are 
freed.
I'd only add that with an average script, it won't happen.

jv

---

## #42560 — Antony Dovgal — 2009-01-11T15:40:06+00:00

URL: https://externals.io/message/42546#42560 · In reply to: #42558

> > […]
> 
> seems you're flying too high.
> RHEL5 still ships php5.1.6 and it has this option and it undefines 
> ZEND_USE_ZEND_ALLOC.

PHP 5.2.0 has been released more than 2 (two) years ago, which should be 
enough for a distro to catch up, don't you think so?

> For the newer versions (5.2.x) it can be set at the environment, which is 
> not that good btw.

Please elaborate.
 
> > […]
> 
> I gives appox 2MB per php-cli instance. Sometimes it's good enough, if for 
> example you need to run 200 instances.

Shared libs are, well, shared among the processes, so I don't think it's "per instance".

-- 
Wbr, 
Antony Dovgal

---

## #42561 — jvlad — 2009-01-11T16:04:53+00:00

URL: https://externals.io/message/42546#42561 · In reply to: #42560

> > […]
>
> PHP 5.2.0 has been released more than 2 (two) years ago, which should be
> enough for a distro to catch up, don't you think so?

Perhaps. But what they ship with RHEL5 is up to Redhat.

> > […]
>
> Please elaborate.

Ok. With default memory allocation strategy I can run relatively short 
scripts very efficiently and that's why I'd not change it for the web site.
On the other hand, for long-running scripts that I run using php-cli, I'd 
always prefer resource saving priorities.
In particular, I'd prefer ZMM not to "cache" memory blocks.
With only one system-wide setting, I have no choise. Either all php will use 
ZMM or neither.

> > […]
>
> Shared libs are, well, shared among the processes, so I don't think it's 
> "per instance".

That's truth. Code segments will be shared. But even with 2MB of code, CPU 
uses 512 entries in TLB for _each_ process and it's clear worthless stuff. 
In case of hundred instances running simultaneously, it may save some time. 
Also, please add time for the initialization of the modules that won't be 
used.

Anyway, if php is to be recompiled from the sources in order to build 
long-term running daemons, the configure options should be carefully tuned.

-jv