Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:42547
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain dealnews.com from 72.5.90.27 cause and error)
Message-ID: <4968EB56.5060902@dealnews.com>
Date: Sat, 10 Jan 2009 12:39:18 -0600
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b1pre) Gecko/20081006 Shredder/3.0a3
MIME-Version: 1.0
To: Thomas Koch <thomas@koch.ro>
CC: internals@lists.php.net
References: <200901101830.35403.thomas@koch.ro>
In-Reply-To: <200901101830.35403.thomas@koch.ro>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] php daemons, memory
From: brianm@dealnews.com (Brian Moon)

First, this is not an internals question. But, I would be not many 
people on the general list could help you.

> I wrote some daemon scripts for a web crawler in PHP. Now I spent one
> day to work around the growing memory consumtion of these scripts.

I have PHP daemons running for weeks that don't use up a lot of memory. 
  You have to be careful what you do in a PHP script.  I would guess you 
are running Linux.  Assuming you are, the linux memory system never 
frees allocated memory back to the OS.  Not until the process dies at 
least.  So, if any part of your code ever needs a lot of memory, that 
will be allocated to the process until it ends.

> - Was it a mistake to use PHP for such scripts? What language should I've been
> choosing instead?

Any language has its caveats.  And you can waste (leak is a strong word) 
memory with any language.

> - My suspicion is, that either pdo_mysql or dom are not freeing their
>    used memory during a request. Is that possible?

Are you telling pdo to free its results?  If not, that is bad 
programming.  You have to unset vars and free db result sets yourself to 
ensure they are not building up.  PHP uses a lazy garbage collector that 
is optimized for short lived web scripts.  You have to overcome that 
when working with PHP in a non-web environment.

> - I'm using http://libslack.org/daemon now to control the script
>    execution. This gave me the idea for a special kind of PHP binary
> "php-daemon":
>    - php-daemon is an executable that restarts a given php script in a
>      loop
>    - php-daemon can be combined with apc/xcache to store the bytecode
>    Unfortunately I'm still to much a newbie to write this myself.

My standard way to handle scripts that need a lot of memory is to use 
the pcntl functions to fork children that run and do the work.  They can 
end after a certain time or memory usage.  Again, you have to make sure 
the main, parent script is well written and does not waste file 
descriptors, connections, etc.

Hmm, maybe I will write a blog post about this.

-- 

Brian Moon
Senior Web Engineer
------------------------------
When you care enough to spend the very least.
http://dealnews.com/