Persistent zvals

I've spent the last few days pouring over the Zend engine source code. I
think I have a basic understanding on the memory management. Likely what I
say may be incorrect, so I apologize in advance.

What I'm trying to do is write an extension to persist PHP variables past
the end of the request in the SAPI to be used on the next request (different
user, same process). I'd like to do this without serialization. The reason
is certain objects in PHP (in a framework, for example) are not request
specific and it would be useful to have these objects already loaded and
usable without having to construct them on every request. (Sample use case:
Dependency injection like Guice without having to write physical containers

• using reflection, written in PHP)

From what I've gathered thus far, it is impossible to do without copying the
non-persistent memory into persistent memory, and then back again. I'm
assuming this is because all the memory associated with PHP variables use
emalloc, which places it onto a stack that is disposed of at the end of the
request.

So there seems to only be two ways to do this:

1 - Copy non-persistent memory into persistent memory (and back) using a
deep copy. Probably not very efficient. May not be much better than
serialization.

2 - Modify the Zend engine to flag objects/zvals as persistent so they
aren¹t thrown away until the process ends.

#2 seems to be the better way to go. #1 seems to be the only way to do it as
an extension (maintaining its own stack).

There seems to have been some discussion (7 years ago) of this mentioned
here under 6.9:

http://www.php.net/~derick/meeting-notes.html

I've been able to do it somewhat with string zvals, but objects are a
different story (given that a zval contains a handle index referring an
entry in a bucket). The "goal", at least with objects, is the objects
doesn't destruct until the end of the process. With copying memory it looks
like I'd probably have to copy the object into my own bucket, modify the
original in zend's bucket so the destructor isn't called (destructor_callled
= 1), and then at the start of the request copy what's in my bucket into
zend's bucket.

At this point I feel light a mad scientist. I'm hoping to gain some insight
on how this might be done properly with PHP 5.3/5.4 (or just 5.4) - even if
it involves modifying the Zend engine. Have you guys had any recent
discussions about doing this?

Luke

Hi!

From what I've gathered thus far, it is impossible to do without copying the

non-persistent memory into persistent memory, and then back again. I'm

assuming this is because all the memory associated with PHP variables use

emalloc, which places it onto a stack that is disposed of at the end of the

request.

The stack part is wrong, emalloc does not use stack, but the general

sense of it is correct - all memory allocated by emalloc is freed at the

end of the request, so anything you would want to preserve would have to

be copied to some other memory area.

How is memory tracked with emalloc? From my observations anything
created with emalloc, with or without a zval wrapper, is freed at the
end of the request. Things created with pemalloc, which seems to be a
wrapper of malloc, isn't thrown away.

I saw some references (off the top if my head - on my phone) to
something like G(mm_stack)->_malloc(...), which is called by emalloc.

Could you briefly explain how it does work or point be to a link that
does? Up until now I've been pouring over the source code - and there
is a lot, tracing down macro after macro.

2 - Modify the Zend engine to flag objects/zvals as persistent so they

aren¹t thrown away until the process ends.

The problem with that is that these variables can contain references to

other variables and other things (like class in object, etc.) that will

be freed, so the data will point to freed memory. Unless the variable is

a simple scalar or you do not allow associating it with any other

variables in any way you would have this problem. If it's a simple

scalar, you can do better by using existing caches. Not allowing

associations with other variables means there's not much reason for this

thing to actually be a PHP variable.

From what I've seen this is because the zvals, the values inside, the
object bucket, etc... is all created by emalloc. Correct me if I'm
wrong, but even if i could persist the zval (wrapper) the value inside
would be freed unless both the zval and the value were created with
pemalloc (alias of malloc). It goes a step further with the objects
because a zend_object is just a handle ID (pointing to a spot in the
bucket) and a pointer to object handlers.

I've been able to do it somewhat with string zvals, but objects are a

different story (given that a zval contains a handle index referring an

entry in a bucket). The "goal", at least with objects, is the objects

doesn't destruct until the end of the process. With copying memory it looks

With objects you have a problem - do you also mean to keep the class?

What about the methods code? If you intend to keep all this in memory -

and remember you'd also have to guard all of it so no "local" variable

gets anywhere into any of object's properties, or sub-values of these

properties - I'm not sure you'd do better than serializing.

At this point I feel light a mad scientist. I'm hoping to gain some insight

on how this might be done properly with PHP 5.3/5.4 (or just 5.4) - even if

it involves modifying the Zend engine. Have you guys had any recent

discussions about doing this?

Doing it is much harder than you think, since all structures in Zend

Engine are assumed to be temporary and you'd need to copy a lot of stuff

to make your object work. I wouldn't really advise it.

Don't get me wrong - I totally get it. Trying to copy the zvals, the
value inside, and the object out of the bucket, and then the
descendants is not a good solution. It would probably be buggy at best
and not much better than serialization. I'd like to avoid copying
memory if possible. A flag on the zval would be ideal.

With the continued assumption that memory structures are temporary
something like this will never be feasible. In order for something
like this to work (at least from my understanding. - I'm probably
totally wrong) values inside a zval would have to be created with
malloc and the life cycle of that object would have to be determined
by the wrapping zval. A zval would have a persistent flag and a
recount the value inside would be released when the recount reaches
zero or at the end of the request if the persistent flag is zero....
But this is something totally different than it is now.

Luke

--

Stanislav Malyshev, Software Architect

SugarCRM: http://www.sugarcrm.com/

(408)454-6900 ext. 227

Hi,

From what I've gathered thus far, it is impossible to do without
copying the non-persistent memory into persistent memory, and then
back again. I'm assuming this is because all the memory associated
with PHP variables use emalloc, which places it onto a stack that is
disposed of at the end of the request.

So there seems to only be two ways to do this:

1 - Copy non-persistent memory into persistent memory (and back) using
a deep copy. Probably not very efficient. May not be much better than
serialization.

Yes, see apc_store() and friends to see all the small parts needed to
copy it as properly as possible. While this still won't work for all
cases (resources, internal classes, ..) but well, this might be what
you're looking for in a ready state ;-)

Apc serializes the data. Unless the data is a cache or session
information there is almost no difference between unseralizing a bunch
of objects verses and initializing the objects in code.

It would be ideal of you could initialize a bunch of objects once and
carry them over to the next request. No serialization, no copying. A
lot of framework objects would benefit from this.

2 - Modify the Zend engine to flag objects/zvals as persistent so they
aren¹t thrown away until the process ends.

Which is a major undertaking, taking copy-on-write and friends into
account. As in a case like

mark_persistent($persistent);
$persistent['some key'] = function_returning_lots_of_data();

would suddenly require to create a copy of all the data. Such things can
quickly cost more than recreating the structures more frequently ...

When having to copy data, yes thing yet messy very quickly. But if
there were a persistent flag on the zval the new zval would have the
persistent flag. The old one would be discarded when it's reference
count reached zero.

Right now everything is thrown out at the end of the request because
it's created with emalloc, regardless of the recount. With a
persistent flag (1 or 0) and a recount of 1 or greater it wouldn't be
until the end of the process. Easier said than done - for sure.

johannes

Hi!

How is memory tracked with emalloc? From my observations anything
created with emalloc, with or without a zval wrapper, is freed at the
end of the request. Things created with pemalloc, which seems to be a
wrapper of malloc, isn't thrown away.

Zend Engine has its own memory manager, see zend_alloc.c. Unfortunately,
I'm not sure I'll be able to explain in short how it works, but TLDR
version of it is that it's like malloc, but private to Zend Engine, so
everything is freed at the end of the request.

From what I've seen this is because the zvals, the values inside, the
object bucket, etc... is all created by emalloc. Correct me if I'm
wrong, but even if i could persist the zval (wrapper) the value inside
would be freed unless both the zval and the value were created with
pemalloc (alias of malloc). It goes a step further with the objects
because a zend_object is just a handle ID (pointing to a spot in the
bucket) and a pointer to object handlers.

This is correct, all internal memory allocations are done by emalloc.
zend_object, however, is not just a handle - this handle refers to
object structure, which has class pointer, properties, etc.

and not much better than serialization. I'd like to avoid copying
memory if possible. A flag on the zval would be ideal.

That's what I am trying to explain - flag on one zval would not help
you, since this zval uses dozens, if not hundreds (for bigger object)
other resources, including other zvals, hashtables, object structures,
class structures, function structures, etc. etc.
Add to that that if you allocated something in emalloc arena you can not
just mark it "do not free it" - that would create a huge fragmentation
problem if you had something non-freeable at the middle of freeable
region. So you can not just "flag" random variable as non-freeable
without messing up whole memory manager (btw, not advisable with malloc
either - fragmentation can inflate you memory usage very easily). Not
that it would work even if you could, see above.

I think I understand what you're getting at. So to avoid fragmentation
you would have to have two independent memory spaces. Making non
persistent memory persistent would require copying from one space to
another. Is that correct?

How is this handled in other platforms where you have an application
instance (with state, if that's the correct terminology)?

Are there any other options for reliably persisting objects/variables
(without having to recreate)?

(Re-sending to the list)

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227