Multithreaded C Gearman worker to run PHP scripts

16 years ago by Thomas Koch — view source

unread

Good evening Europe, good morning America,

I hacked a gearman worker today that runs PHP scripts. As you can easily spot
on first sight, I don't have many clues on C, threads and PHP internals. But
it seems to kind of work. :-)

So I'm happy for anybody interested to look over my work and help me to become
better:
http://github.com/thkoch2001/gearman-php-worker/tree/master

More to come tomorrow...

Thomas Koch, http://www.koch.ro
YMC AG, http://www.ymc.ch

16 years ago by johannes@php.net — view source

unread

Hi,

So I'm happy for anybody interested to look over my work and help me to become
better:
http://github.com/thkoch2001/gearman-php-worker/tree/master

You are wrapping every call inside PHP_EMBED_START_BLOCK/_END_BLOCK
calls. By this you have a full PHP startup and shutdown on every single
request to the worker. This costs quite some time and that
startup/shutdown is not thread-safe which means multiple threads
starting and finishing PHP will end with quite some trouble.

For doing this you should write your own SAPI, for that take the sapi
structure and create the helper functions (for a start take them from
php_embed.c). In your main you then do the following (pseudo-code)

void main() {
/* param parsing and stuff ... */

tsrm_startup(1, 1, 0, NULL);
sapi_startup(&gearman_module);

/* now PHP is ready to handle requests */
startWorkers( host, port, scripts_list, numberOfWorkers );

/* after the last request finished and while shutting down: /
php_module_shutdown(TSRMLS_C);
sapi_shutdown();
tsrm_shutdown();

}

and then in your runPHP it looks, using shortened and pseudo-code, like
this:

char *runPHP( char *scriptName )
{
void *tsrm_ls = NULL;
tsrm_ls = ts_resource(0); / actually you would do this when
creating the thread and pass the TSRMLS macros along/

if (php_request_startup(TSRMLS_C)==FAILURE) {
    return NULL;
}
/* prepare script handle and environment */

zend_try_first {
    php_execute_script(&scriptHandle TSRMLS_CC); 
} zend_catch {
    /* handle error */
} zend_end_try();

php_request_shutdown((void *) 0);

}

That's not tested and based on my mind and quick checking of my code,
not the perfect final implementation but hopefully a good start. Please
double check with the implementation of other SAPIs!

For making development simple you might link such code against the embed
sapi (and ignore php_embed.[ch]) at least the plumbing code should be
moved into php-src/sapi/gearman or something and properly built without
embed dependency.

johannes

16 years ago by Thomas Koch — view source

unread

As Johannes suggested (thx!) I made a gearman_worker sapi:

http://github.com/thkoch2001/php-
src/tree/857657ddce9aad9709cc39832fb716555021725c/sapi/gearman_worker

(or http://tinyurl.com/gearmanworkersapi )

and linked the old code against it:

http://github.com/thkoch2001/gearman-php-worker/

It starts now, waits for tasks, but segfaults as soon as a task is received.
Need to debug it tomorrow.

If anybody wants to guide me... :-) It seems that something with opening the
script files goes wrong:

#0 0x00007ffff6ffd620 in ?? () from /lib/libc.so.6
#1 0x00007ffff6fff838 in malloc () from /lib/libc.so.6
#2 0x00007ffff7838c58 in __zend_malloc (ht=0x896ab0, arKey=0x61dc40
"finfo_open", nKeyLength=11, h=13873911580025231368, pData=0x61dc60,
nDataSize=232,
pDest=0x7ffff48b79b0, flag=1) at /var/checkouts/php/Zend/zend_alloc.h:81
#3 _zend_hash_quick_add_or_update (ht=0x896ab0, arKey=0x61dc40 "finfo_open",
nKeyLength=11, h=13873911580025231368, pData=0x61dc60, nDataSize=232,
pDest=0x7ffff48b79b0, flag=1) at /var/checkouts/php/Zend/zend_hash.c:319
#4 0x00007ffff783910a in zend_hash_copy (target=0x896ab0, source=0x604030,
pCopyConstructor=0, tmp=<value optimized out>, size=232)
at /var/checkouts/php/Zend/zend_hash.c:786
#5 0x00007ffff782b338 in compiler_globals_ctor (compiler_globals=0x896840,
tsrm_ls=0x848650) at /var/checkouts/php/Zend/zend.c:494
#6 0x00007ffff77c9ba3 in allocate_new_resource
(thread_resources_ptr=0x603138, thread_id=140737296173392) at
/var/checkouts/php/TSRM/TSRM.c:294
#7 0x00007ffff77c9d1c in ts_resource_ex (id=0, th_id=<value optimized out>)
at /var/checkouts/php/TSRM/TSRM.c:361
#8 0x0000000000401525 in runPHP (scriptName=0x849b45 "test.php") at
worker.c:60
#9 0x000000000040163c in runJob (job=0x849a40, cb_arg=0x0,
result_size=0x8492b0, ret_ptr=0x7ffff48b809c) at worker.c:96
#10 0x00007ffff72f40a4 in gearman_worker_work (worker=0x8492a0) at
worker.c:593
#11 0x000000000040176e in workerThreadFunction (workerPointer=0x8492a0) at
worker.c:135
#12 0x00007ffff6d74faa in start_thread () from /lib/libpthread.so.0
#13 0x00007ffff705729d in clone () from /lib/libc.so.6
#14 0x0000000000000000 in ?? ()

For making development simple you might link such code against the embed
sapi (and ignore php_embed.[ch]) at least the plumbing code should be
moved into php-src/sapi/gearman or something and properly built without
embed dependency.

johannes
Thomas Koch, http://www.koch.ro

16 years ago by Thomas Koch — view source

unread

I defeated the segfaults!

Now the monster in the next level is named smart_str. I have a smart_str to
buffer the SAPI output.

On the first request no output comes out.
Second request gives the correct output.
All following request return some bytes of garbarge.

The relevant code parts:

typedef struct _gearman_worker_request_struct {
smart_str result;
} gearman_worker_request_struct;

static void gearman_worker_globals_ctor(gearman_worker_request_struct *request
TSRMLS_DC)
{
memset( &request->result, 0, sizeof(smart_str) );
}

static void gearman_worker_globals_dtor(gearman_worker_request_struct *request
TSRMLS_DC)
{
smart_str_free(&GWR(result));
}

static int php_embed_ub_write(const char *str, uint str_length TSRMLS_DC)
{
smart_str_appendl(&GWR(result), str, (int) str_length);

return str_length;

}

char *php_gearman_worker_get_result(TSRMLS_D)
{
char *result;
smart_str_0(&GWR(result));

result = strndup( GWR(result).c, GWR(result).len );

return result;

}

and in the sapi startup:

ts_allocate_id(&gearman_worker_id, sizeof(gearman_worker_request_struct),

(ts_allocate_ctor) gearman_worker_globals_ctor, (ts_allocate_dtor)
gearman_worker_globals_dtor);

Thanks for your patience!!!

Thomas Koch, http://www.koch.ro

16 years ago by Thomas Koch — view source

unread

It took me a day to get a binary gearman worker out of the PHP make system.
But it works (somehow).

Now I'd like to ask, whether it could be included in upstream PHP source? I'd
need a sponsor to guide me cleaning up everything.

Tomorrow I'll try to get source code caching with APC to work. It works to
cache variables, but returns in these lines in ext/apc/apc_main.c, lines
~526++:

/* try to create a cache key; if we fail, give up on caching */
if (!apc_cache_make_file_key(&key, h->filename, PG(include_path), t

TSRMLS_CC)) {
return old_compile_file(h, type TSRMLS_CC);
}

Source code at:
http://github.com/thkoch2001/php-
src/tree/857657ddce9aad9709cc39832fb716555021725c/sapi/gearman_worker

(or http://tinyurl.com/gearmanworkersapi )

Thomas Koch, http://www.koch.ro

16 years ago by Andrei Zmievski — view source

unread

Can you explain how this is different from the worker class provided by
pecl/gearman?

-Andrei

Good evening Europe, good morning America,

I hacked a gearman worker today that runs PHP scripts. As you can easily
spot
on first sight, I don't have many clues on C, threads and PHP internals.
But
it seems to kind of work. :-)

So I'm happy for anybody interested to look over my work and help me to
become
better:
http://github.com/thkoch2001/gearman-php-worker/tree/master

More to come tomorrow...

Thomas Koch, http://www.koch.ro
YMC AG, http://www.ymc.ch