Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:42546
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain koch.ro from 195.34.83.107 cause and error)
Organization: Young Media Concepts
To: internals@lists.php.net
Date: Sat, 10 Jan 2009 18:30:34 +0100
User-Agent: KMail/1.9.9
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-ID: <200901101830.35403.thomas@koch.ro>
Subject: php daemons, memory
From: thomas@koch.ro (Thomas Koch)

Hi,

I wrote some daemon scripts for a web crawler in PHP. Now I spent one
day to work around the growing memory consumtion of these scripts.

Since I use a MySQL connection, Syslog and many classes, I wanted to let
the script run a while before restarting. So the scripts had lines like:

$i = 1000;
while( --$i )
{
  while( run() ); // run returns false if there are no pending jobs
  gc_collect_cycles();
  // echo get_memory_usage();
  sleep( 20 );
}

I thought, that most memory should be cleaned up after each termination
of the run() function, but the reported memory usage grew rapidly.

- Was it a mistake to use PHP for such scripts? What language should I've been 
choosing instead?

- --enable-debug did report some small leaks, but much less then the
  consumption grow.

- My suspicion is, that either pdo_mysql or dom are not freeing their
  used memory during a request. Is that possible?

- It would help me a lot, if I could easily get an overview, what is
  consuming my memory. Which Zvals are known? Which extension / line did 
allocate how much memory?

- I'm using http://libslack.org/daemon now to control the script
  execution. This gave me the idea for a special kind of PHP binary
"php-daemon":
  - php-daemon is an executable that restarts a given php script in a
    loop
  - php-daemon can be combined with apc/xcache to store the bytecode
  Unfortunately I'm still to much a newbie to write this myself.

Thanks for your time reading this,
-- 
Thomas Koch, http://www.koch.ro
YMC AG, http://www.ymc.ch