Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:42552
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain fischer.name from 62.179.121.33 cause and error)
Message-ID: <49697F2C.3050900@fischer.name>
Date: Sun, 11 Jan 2009 06:10:04 +0100
User-Agent: Thunderbird 2.0.0.19 (Windows/20081209)
MIME-Version: 1.0
To: Thomas Koch <thomas@koch.ro>
CC: internals@lists.php.net
References: <200901101830.35403.thomas@koch.ro>
In-Reply-To: <200901101830.35403.thomas@koch.ro>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] php daemons, memory
From: markus@fischer.name (Markus Fischer)

Hi,

albeit *not* as a daemon, we've successfully developed a Crawler in PHP
within our company. It can run for hours without a leak, if I remember
correctly it's peak memory consumption is below 64MB. However we're
crawling only a small amount of URLs, just around 10.000 .

As Brian mentioned: free your database resources, unset unused
variables. We've had one major rewrite which, besides re-architecturing
the whole thing for plugin/modularity, involved auditing every step to
make sure resources are properly freed. Usually a PHP developer doesn't
have to pay much attention to it because of the wide-used process-fork
model (but I guess I don't need to tell you that :).

But you'll get often beaten by PHP itself:
it has quite some leaks and finding/tracking them done costs time,
sometimes requires skill at the C level of PHP to properly
understand/diagnose things and if you were (unfortunately) successful in
identifying a PHP problem you've report a bug, preferable attach provide
a patch/workaround.

For example, we've had to fight http://bugs.php.net/bug.php?id=43450 .
Tracking this PHP problem was quite time consuming, involving multiple
developers, etc. Luckily we could work around this, but it was pretty
annoying.

We actually planned to release this as open source, donate it to Zend,
whatever. Legally it's done within the company, just no one had the time
for the publishing process, going over things, etc. :/

As a sidenote: We've hit the current limit of our Crawler implementation
in PHP itself: we can't to parallel fetching/processing of URLs in a
efficient manner. You can get things quick running in PHP, but doing
things with style and a serious architecture hits its limits. We've gone
to Java for such cases, made sense for us anyway as we had to move away
from Zend_Search_Lucene as it had performance problems with our index
where as Lucene/Solr was still mostly bored. Will be interesting to see
if http://code.google.com/p/marjory/ can handle this. Ops, off-topic.



HTH,
- Markus