# Performance of buffer based functionality (JSON, AES, serialize())

- Thread: https://externals.io/message/50557
- Original archive: https://news-web.php.net/php.internals/50557
- Started by Andi Gutmans on 2010-11-25T17:47:10+00:00
- Messages: 11
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #50557 — Andi Gutmans — 2010-11-25T17:47:10+00:00

URL: https://externals.io/message/50557#50557

Hi,

Completely different topic :)

I've been looking a bit into performance around json encoding, hashing+encryption (aes) and `serialize()`/unserialize(). Data that is marshaled and often transmitted over the wire.

I know there have been some high-end apps that have benefited from some custom serializers, etc... (typically platform dependent).
I wonder if people here think improvements in these areas would move the needle for the majority of mainstream apps or not.

Thanks,

Andi

---

## #50560 — Ilia Alshanetsky — 2010-11-25T17:51:50+00:00

URL: https://externals.io/message/50557#50560 · In reply to: #50557

I think there is much to gain by improving the serialization speed in
PHP. It is used everywhere from caches like memcache, to sessions or
manual data input into DB. I would say that there are very few
non-trivial apps that would not benefit from a more compact and faster
serializer.

In our specific work-use-case switching to igbinary improved the speed
of the overall page generation by 2-3%.


> Hi,
>
> Completely different topic :)
>
> I've been looking a bit into performance around json encoding, hashing+encryption (aes) and `serialize()`/unserialize(). Data that is marshaled and often transmitted over the wire.
>
> I know there have been some high-end apps that have benefited from some custom serializers, etc... (typically platform dependent).
> I wonder if people here think improvements in these areas would move the needle for the majority of mainstream apps or not.
>
> Thanks,
>
> Andi

---

## #50571 — Michel Bartz — 2010-11-25T19:12:50+00:00

URL: https://externals.io/message/50557#50571 · In reply to: #50557

Hi,

I think it would, a lot of sites/apps nowadays rely a lot on JSON encoding/decoding, plus a lot of technologies are relying on serialization/json encoding (Memcached, Redis, to name a few) at the PHP level, which can be a really big performance eater if you use it a lot.
On the other hand, it's not too difficult to get IGBinary setup as serializer instead of the default one, and it does some pretty good improvement.
Ever considered making it a part of PHP? What would be the implication (as I'm not too familiar with all the internals of PHP besides extensions...)?

Michel Bartz

Lead Developer
Manwin Canada
Skype: michel.php

-----Original Message-----
From: Andi Gutmans [mailto:andi@zend.com]
Sent: Thursday, November 25, 2010 12:47 PM
To: internals@lists.php.net
Subject: [PHP-DEV] Performance of buffer based functionality (JSON, AES, `serialize()`)

Hi,

Completely different topic :)

I've been looking a bit into performance around json encoding, hashing+encryption (aes) and `serialize()`/unserialize(). Data that is marshaled and often transmitted over the wire.

I know there have been some high-end apps that have benefited from some custom serializers, etc... (typically platform dependent).
I wonder if people here think improvements in these areas would move the needle for the majority of mainstream apps or not.

Thanks,

Andi

This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.

Ce courrier ?lectronique est confidentiel et prot?g?. L'exp?diteur ne renonce pas aux droits et obligations qui s'y rapportent. Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les) destinataire(s) d?sign?(s) est interdite. Si vous recevez ce courrier ?lectronique par erreur, veuillez m'en aviser imm?diatement, par retour de courrier ?lectronique ou par un autre moyen.

---

## #50572 — Pierre Joye — 2010-11-25T19:14:24+00:00

URL: https://externals.io/message/50557#50572 · In reply to: #50557

hi,

For the record here, igbinary is a very good example of such optimization:

http://opensource.dynamoid.com/


> Hi,
>
> Completely different topic :)
>
> I've been looking a bit into performance around json encoding, hashing+encryption (aes) and `serialize()`/unserialize(). Data that is marshaled and often transmitted over the wire.
>
> I know there have been some high-end apps that have benefited from some custom serializers, etc... (typically platform dependent).
> I wonder if people here think improvements in these areas would move the needle for the majority of mainstream apps or not.
>
> Thanks,
>
> Andi
>



-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #50573 — Jonah H. Harris — 2010-11-25T19:47:10+00:00

URL: https://externals.io/message/50557#50573 · In reply to: #50572

> For the record here, igbinary is a very good example of such optimization:
>
> http://opensource.dynamoid.com/


igbinary is a nice extension indeed.  However, for those of us who have
environments which include multiple programming languages, custom
serializations become a PITA.  As such, we generally go with something more
portable such as Avro or straight JSON.  Awhile back, I had done some work
rewriting the JSON serialization functions to use the fast (and BSD
licensed) yajl JSON parser (https://github.com/lloyd/yajl).  Initial
benchmarks showed a 4-7% performance improvement in
serialization/deserialization.

I'll see if I can dig it up--hopefully it's not on my dead computer.

-- 
Jonah H. Harris
Blog: http://www.oracle-internals.com/

---

## #50574 — Pierre Joye — 2010-11-25T19:51:35+00:00

URL: https://externals.io/message/50557#50574 · In reply to: #50573

> 
> > […]
>
> igbinary is a nice extension indeed.  However, for those of us who have
> environments which include multiple programming languages, custom
> serializations become a PITA.  As such, we generally go with something more
> portable such as Avro or straight JSON.  Awhile back, I had done some work
> rewriting the JSON serialization functions to use the fast (and BSD
> licensed) yajl JSON parser (https://github.com/lloyd/yajl).  Initial
> benchmarks showed a 4-7% performance improvement in
> serialization/deserialization.

Good point indeed. That makes me think about bson
(http://bsonspec.org/), which is used by mongodb for example.

-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #50580 — Ilia Alshanetsky — 2010-11-26T00:15:47+00:00

URL: https://externals.io/message/50557#50580 · In reply to: #50574

Just read over the BSON spec, looks fairly interesting, the only bit
that appears to be missing for PHP purposes is object support. We
would need to introduce custom type on top of standard BSON. However
from compactness and consistency standpoint it looks fairly appealing.


> 
> > […]
>
> Good point indeed. That makes me think about bson
> (http://bsonspec.org/), which is used by mongodb for example.
>
> --
> Pierre
>
> @pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #50618 — Andi Gutmans — 2010-11-27T07:13:50+00:00

URL: https://externals.io/message/50557#50618 · In reply to: #50580

It's nice but as long as the browsers don't implement it natively then it's less useful for server to client communication.
Of course can still be quite useful with custom I/O or data sources that implement it natively i.e. mongodb.

> -----Original Message-----
> From: Ilia Alshanetsky [mailto:ilia@prohost.org]
> Sent: Thursday, November 25, 2010 4:16 PM
> To: Pierre Joye
> Cc: Jonah H. Harris; Andi Gutmans; internals@lists.php.net
> Subject: Re: [PHP-DEV] Performance of buffer based functionality (JSON, AES,
> `serialize()`)
> 
> Just read over the BSON spec, looks fairly interesting, the only bit that appears
> to be missing for PHP purposes is object support. We would need to introduce
> custom type on top of standard BSON. However from compactness and
> consistency standpoint it looks fairly appealing.
> 
> 
> > […]
> wrote:
> > […]
> wrote:
> > […]

---

## #50666 — Jonathan Bond-Caron — 2010-11-28T16:15:16+00:00

URL: https://externals.io/message/50557#50666 · In reply to: #50557

> 
> I know there have been some high-end apps that have benefited from 
> some custom serializers, etc... (typically platform dependent).
> I wonder if people here think improvements in these areas would move 
> the needle for the majority of mainstream apps or not.
> 

Like people have mentioned, improving (un)serialize speed would be a huge
benefit, especially for caching data sets or large objects.

From experience, it would seem valuable to have:
1) serialize_text($var)

The existing `serialize()` minus the `NULL` bytes on private properties. It has
been a source problems for developers serializing an object with private
properties and storing it in a database (the string may get cutoff).

I'm not sure why there's a `NULL` byte in 'zend_mangle_property_name', instead
the char "_" could be used to mark a private property in the serialized
text.
The unserialize could be BC compatible accepting both `NULL` and "_" around a
private property.

2) serialize_binary($var)

An efficient and compact serialization using techniques from igbinary.

A potential problem with igbinary I've noticed is it packs a double as a 64
bit integer. 
That could be a problem if you serialize on a platform that has an IEEE 754
binary representation and unserialize on a non-IEEE platform but I don't
know if php compiles on architectures that are non-IEEE.

It could also be interesting to pack integers as varints:
http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints
http://protobuf-c.googlecode.com/svn/trunk/src/google/protobuf-c/protobuf-c.
c

That's most likely slower though then what igbinary does with integers

---

## #50710 — Julien Pauli — 2010-11-30T08:26:51+00:00

URL: https://externals.io/message/50557#50710 · In reply to: #50666

I guess serialize mechanism cant use any char that can be part of a
PHP variable. And "_" can. As property names respect binary
compatibility, the only char that can be used to mark private
properties is actually the `NULL` byte. Ping me if I'm wrong.

But I'm +1 for improving the `serialize()` speed, I had problems
recently with it, and igbinary came to save me as well :)

Julien.Pauli


> 
> > […]
>
> Like people have mentioned, improving (un)serialize speed would be a huge
> benefit, especially for caching data sets or large objects.
>
> From experience, it would seem valuable to have:
> 1) serialize_text($var)
>
> The existing `serialize()` minus the `NULL` bytes on private properties. It has
> been a source problems for developers serializing an object with private
> properties and storing it in a database (the string may get cutoff).
>
> I'm not sure why there's a `NULL` byte in 'zend_mangle_property_name', instead
> the char "_" could be used to mark a private property in the serialized
> text.
> The unserialize could be BC compatible accepting both `NULL` and "_" around a
> private property.
>
> 2) serialize_binary($var)
>
> An efficient and compact serialization using techniques from igbinary.
>
> A potential problem with igbinary I've noticed is it packs a double as a 64
> bit integer.
> That could be a problem if you serialize on a platform that has an IEEE 754
> binary representation and unserialize on a non-IEEE platform but I don't
> know if php compiles on architectures that are non-IEEE.
>
> It could also be interesting to pack integers as varints:
> http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints
> http://protobuf-c.googlecode.com/svn/trunk/src/google/protobuf-c/protobuf-c.
> c
>
> That's most likely slower though then what igbinary does with integers

---

## #50830 — Jonathan Bond-Caron — 2010-12-02T17:11:32+00:00

URL: https://externals.io/message/50557#50830 · In reply to: #50710

> I guess serialize mechanism cant use any char that can be part of a 
> PHP variable. And "_" can. As property names respect binary 
> compatibility, the only char that can be used to mark private 
> properties is actually the `NULL` byte. Ping me if I'm wrong.
> 

Right, what I was proposing didn't make sense. After digging through the source, say we have:
class Foo {
	public $a = 1;
	protected $b = 2;
	private $c = 3;
}

Currently this is:
O:3:"Foo":3:{s:1:"a";i:1;s:4:"�*�b";i:2;s:6:"�Foo�c";i:3;}

An alternative could be:

O:3:"Foo":3:{s:1:"a";i:1;*;s:4:"b";i:2;_;s:6:"c";i:3;}

Where "*;" is a marker for protected, "_;" is a marker for private

It would involve some trickery in ext/standard/var_unserializer.re :
"*;"	{
      /* prepend �*� to the next key so that we have zend_symtable_find("�*�b") */
}

"_;"	{
      /* prepend �Foo� to the next key so that we have zend_symtable_find("�Foo�c") */
}

Just a thought if someone wants to refactor it / look into performance, I believe that approach would support both:

O:3:"Foo":3:{s:1:"a";i:1;*;s:4:"b";i:2;_;s:6:"c";i:3;}
O:3:"Foo":3:{s:1:"a";i:1;s:4:"�*�b";i:2;s:6:"�Foo�c";i:3;}