# Activation of IGBinary serialization extension in 5.4 by default - Thread: https://externals.io/message/54677 - Original archive: https://news-web.php.net/php.internals/54677 - Started by Paul Dragoonis on 2011-08-18T11:12:01+00:00 - Messages: 34 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/54677.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #54677 — Paul Dragoonis — 2011-08-18T11:12:01+00:00 URL: https://externals.io/message/54677#54677 Hey guys, After a brief discussion with Pierre I'm taking this discussion to here. We have the igbinary pecl extension available [1], however not many of our userbase are familiar that it even exists. Here [2] is the performance boost we get from using igbinary with `serialize()` and `unserialize()`. Here we get a great perf boost, and what's more excellent as we don't have to break BC to get there! This seems like a no-brainer and a good investment in the speed increase of PHP by default for the future users of the platform. [1] http://pecl.php.net/package/igbinary [2] http://www.php.net/~pierre/vcqa/apcigninary_perf.png Comments please. Regards, Paul Dragoonis. --- ## #54678 — Arpad Ray — 2011-08-18T11:30:15+00:00 URL: https://externals.io/message/54677#54678 · In reply to: #54677 Hi, > Hey guys, > > After a brief discussion with Pierre I'm taking this discussion to here. > We have the igbinary pecl extension available [1], however not many of our > userbase are familiar that it even exists. > Here [2] is the performance boost we get from using igbinary with > `serialize()` and `unserialize()`. > > Here we get a great perf boost, and what's more excellent as we don't have > to break BC to get there! It would break BC for anyone with serialized strings stored in the current format. Maybe bundle it in core and provide an ini setting like session.serialize_handler, but I don't think making it the default in 5.4 is an option. Regards, Arpad > > This seems like a no-brainer and a good investment in the speed increase of > PHP by default for the future users of the platform. > > [1] http://pecl.php.net/package/igbinary > [2] http://www.php.net/~pierre/vcqa/apcigninary_perf.png > > Comments please. > > Regards, > Paul Dragoonis. --- ## #54679 — Roger Llopart Pla — 2011-08-18T11:38:41+00:00 URL: https://externals.io/message/54677#54679 · In reply to: #54678 I agree with bundling it in the core aswell, but having it as default, as stated, would be a huge BC break. 2011/8/18 Arpad Ray > Hi, > > On Thu, Aug 18, 2011 at 12:12 PM, Paul Dragoonis > wrote: > > […] > our > > […] > have > > […] > > It would break BC for anyone with serialized strings stored in the > current format. > > Maybe bundle it in core and provide an ini setting like > session.serialize_handler, but I don't think making it the default in > 5.4 is an option. > > Regards, > > Arpad > > > […] > of > > […] --- ## #54680 — Ferenc Kovacs — 2011-08-18T11:48:22+00:00 URL: https://externals.io/message/54677#54680 · In reply to: #54678 > Hi, > > > > […] > > It would break BC for anyone with serialized strings stored in the > current format. > > Maybe bundle it in core and provide an ini setting like > session.serialize_handler, but I don't think making it the default in > 5.4 is an option. agree with the BC part, just wanted to mention that that igbinary can already be set as a session handler via session.serialize_handler if the extension present. -- Ferenc Kovács @Tyr43l - http://tyrael.hu --- ## #54681 — Paul Dragoonis — 2011-08-18T11:48:24+00:00 URL: https://externals.io/message/54677#54681 · In reply to: #54679 On Thu, Aug 18, 2011 at 12:38 PM, Roger Llopart Pla wrote: > I agree with bundling it in the core aswell, but having it as default, as > stated, would be a huge BC break. > > 2011/8/18 Arpad Ray > > > […] > Sorry guys, I didn't realise there was an actual BC break in the serialized structure. I agree with bundling and providing a .ini setting to activate, but off by default. > > > […] --- ## #54682 — Kingsquare.nl - Robin Speekenbrink — 2011-08-18T12:02:19+00:00 URL: https://externals.io/message/54677#54682 · In reply to: #54680 Hello everyone, My first post after watching the list for a while: my opionion on this would also be incorporate it into the core but with BC compatibility: the serializer could (easily from what i understand) check to see if the serialized format is in the `old` format and read that accordingly, but write it back in the new... That way the problem would 'clear' itself up.. ? (or is this just simple 'user'-level thinking? :P) With kind regards, Robin Speekenbrink Kingsquare Information Services Op 18-8-2011 13:48, Ferenc Kovacs schreef: > > > […] > > agree with the BC part, just wanted to mention that that igbinary can > already be set as a session handler via session.serialize_handler if > the extension present. --- ## #54683 — Pierre Joye — 2011-08-18T12:22:35+00:00 URL: https://externals.io/message/54677#54683 · In reply to: #54677 hi! > After a brief discussion with Pierre I'm taking this discussion to here. For the record here, I'm not in favor of bundling igbinary (or other custom serializer). My main concerns are: . the PHP serializer can be overridden at runtime (as in after the build) . there will always have some new faster, better, sexier serializer, depending on the app/usages . we can't enable it by default (as other stated too here) for BC reason (and interop) . not enabling it by default defeats the purpose of bundling it . easier to support it while being only in pecl ('do you use the core version or latest pecl?") Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #54684 — Kiall Mac Innes — 2011-08-18T12:25:25+00:00 URL: https://externals.io/message/54677#54684 · In reply to: #54681 Some others systems use something similar to syntax checking (ldap password hashes). Baisically, prefix a code to the serialized string to indicate what method was used to serialize. Eg no prefix, or $O$ for the old style. Or $I$ for the igbinary syntax. This has the advantage of allowing additional formats to be added over time without breaking BC. Kiall > On Thu, Aug 18, 2011 at 12:38 PM, Roger Llopart Pla wrote: > > > […] here. > > […] > Sorry guys, I didn't realise there was an actual BC break in the serialized > structure. > I agree with bundling and providing a .ini setting to activate, but off by > default. > > > > […] increase > > […] --- ## #54685 — johannes@schlueters.de — 2011-08-18T13:16:36+00:00 URL: https://externals.io/message/54677#54685 · In reply to: #54681 > I agree with bundling and providing a .ini setting to activate, but off by > default. An ini setting is evil for this. People use serialized data as transport format having that depend on an ini setting is bad. suggestions: - The decode function could check the format ("old" format will always begin by one of aisfb or so followed by a colon, igbinary something else) - `serialize()` might get an extra parameter to set the format - lateron, if we want: if old format is being use a deprecation notice is shown when the older fomrat is being used) johannes --- ## #54686 — Arvids Godjuks — 2011-08-18T13:25:31+00:00 URL: https://externals.io/message/54677#54686 · In reply to: #54684 Just add a check on what format is used on read, write with the new format. Deprecate the old format in next version after 5.4 (or drop it at all in 6.0 or whatever the new major version will be). Make a noise about the change (I can do that for the Russian speaking community with ease). It will be transparent. People using serialized caches can just drop them and re-init. People storing data can recode it with ease - just read it and write it back - it will be in new format. Problem solved :) --- ## #54687 — Ferenc Kovacs — 2011-08-18T13:33:32+00:00 URL: https://externals.io/message/54677#54687 · In reply to: #54685 2011/8/18 Johannes Schlüter : > > > […] > > An ini setting is evil for this. People use serialized data as transport > format having that depend on an ini setting is bad. > we already do this (session.serialize_handler, defaults to our own format, but can be set to wddx or igbinary) the userland session handler should be handler agnostic, it shouldn't really break the application if the serialization format change, as the handler should know or care whats inside the $data as long as it can be passed around. but imo this sucks. your handler cannot look into the data, the only available methods for packing and unpacking the session data can only used to read and write the $_SESSION superglobal, so for example if you to load another session for some reason, you have to save the value of the $_SESSION, empty $_SESSION, load the encoded session, call session_decode, fetch the $_SESSION then restore it from your copy. and I really hate that the php session format is almost compatible with serialize/unserialize. if I remember correctly only lacks the a:number_of_items_in_the_session:{} around the session data. why did we did this? to save a few bytes? /o\ sorry for the rant. -- Ferenc Kovács @Tyr43l - http://tyrael.hu --- ## #54688 — johannes@schlueters.de — 2011-08-18T13:42:16+00:00 URL: https://externals.io/message/54677#54688 · In reply to: #54687 > we already do this (session.serialize_handler, defaults to our own > format, but can be set to wddx or igbinary) Well the session handler doesn't matter too much and is more of an admin thing, unlike `serialize()`/unserialize(). Those are the ones which shouldn't be set by configuration as they are used programmatically. The format is relevant for session though, too, as people use memcache or something to share sessions from multiple hosts. > and I really hate that the php session format is almost compatible > with serialize/unserialize. if I remember correctly only lacks the > a:number_of_items_in_the_session:{} around the session data. > why did we did this? to save a few bytes? /o\ History with register globals, back then each variable had to be serialized individually based on the list of registered vars. We resisted from changing the format to keep compatibility. (some people also want to stay sessions alive while updating) johannes --- ## #54689 — Ferenc Kovacs — 2011-08-18T13:50:13+00:00 URL: https://externals.io/message/54677#54689 · In reply to: #54688 2011/8/18 Johannes Schlüter : > > > […] > > Well the session handler doesn't matter too much and is more of an admin > thing, unlike `serialize()`/unserialize(). Those are the ones which > shouldn't be set by configuration as they are used programmatically. > you are right, sorry for the offtopic. -- Ferenc Kovács @Tyr43l - http://tyrael.hu --- ## #54696 — Stas Malyshev — 2011-08-18T17:27:41+00:00 URL: https://externals.io/message/54677#54696 · In reply to: #54683 Hi! > . not enabling it by default defeats the purpose of bundling it Actually, I don't think it does. It's one thing to tell people "put this option into php.ini" and quite another is "download it from PECL (?), compile it (?!) and install it". The first is within the limits of any literate person, the second is sysadmin territory. > . easier to support it while being only in pecl ('do you use the core > version or latest pecl?") It is a concern, if this extension is not stable yet. Is it? -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #54701 — Pierre Joye — 2011-08-18T19:39:56+00:00 URL: https://externals.io/message/54677#54701 · In reply to: #54696 hi Stas! > > […] > > Actually, I don't think it does. It's one thing to tell people "put this > option into php.ini" and quite another is "download it from PECL (?), > compile it (?!) and install it". The first is within the limits of any > literate person, the second is sysadmin territory. All major distros have it already, so it is the same. Now, if it is not enabled by default, then many admins won't install it, back to the same problem. It is very unlikely that someone with root or admin perms won't be able to install an extra extension from the distro repository or using the DLL on windows. > > […] > > It is a concern, if this extension is not stable yet. Is it? It is stable. But the concern is not about its stability but the ease of support. Having it in core means than some users may use (for example) 5.4.2 version while we are at 5.4.6 or pecl has 3 new versions. Not a big deal, just easier. That being said, I'm not opposed to include it, I only don't see it as a must go :) Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #54702 — Kingsquare.nl - Robin Speekenbrink — 2011-08-19T07:48:49+00:00 URL: https://externals.io/message/54677#54702 · In reply to: #54701 Dear All, Again with my 2cents into the basket: I was hoping Ilia Alshanetsky would also support this idea since in his talks (i.e. http://ilia.ws/archives/241-IPC-Hidden-Features-of-PHP-Slides.html) he has been promoting the exact same use: switch the standard serializer to igBinary... Also the proposed include it -> read the data -> convert on write if in previous format (possibly an ini setting?) would provide full BC and would allow a painless upgrade... wouldnt it? -- With kind regards, Robin Speekenbrink Kingsquare Information Services > hi Stas! > > > > > […] thin the limits of any > > […] > > All major distros have it already, so it is the same. Now, if it is > not enabled by default, then many admins won't install it, back to the > same problem. It is very unlikely that someone with root or admin > perms won't be able to install an extra extension from the distro > repository or using the DLL on windows. > > > […] > > It is stable. But the concern is not about its stability but the ease > of support. Having it in core means than some users may use (for > example) 5.4.2 version while we are at 5.4.6 or pecl has 3 new > versions. Not a big deal, just easier. > > That being said, I'm not opposed to include it, I only don't see it as > a must go :) > > Cheers, --- ## #54707 — Sebastian Krebs — 2011-08-19T08:56:50+00:00 URL: https://externals.io/message/54677#54707 · In reply to: #54702 Hi, One pitfall I see is, that two separate system can communicate using serialized data. They both must understand the same format and (in this case) must be configured equal. Regards, Sebastian 2011/8/19 Kingsquare.nl - Robin Speekenbrink > Dear All, > > Again with my 2cents into the basket: I was hoping Ilia Alshanetsky would > also support this idea since in his talks (i.e. > http://ilia.ws/archives/241-**IPC-Hidden-Features-of-PHP-**Slides.html) > he has been promoting the exact same use: switch the standard serializer to > igBinary... > > Also the proposed include it -> read the data -> convert on write if in > previous format (possibly an ini setting?) would provide full BC and would > allow a painless upgrade... wouldnt it? > > -- > With kind regards, > > Robin Speekenbrink > Kingsquare Information Services > > > > > […] > > > […] --- ## #54710 — John Carter — 2011-08-19T09:15:01+00:00 URL: https://externals.io/message/54677#54710 · In reply to: #54677 All, We're fairly heavy users of igbnary and we've found that the latest 1.1.1 from pecl has issues with values-by-reference and therefore can't be used as a drop-in replacement for the standard serialiser as it stands. I suspect it's the same issue as this https://github.com/igbinary/igbinary/issues/6 John. > Hey guys, > > After a brief discussion with Pierre I'm taking this discussion to here. > We have the igbinary pecl extension available [1], however not many of our > userbase are familiar that it even exists. > Here [2] is the performance boost we get from using igbinary with > `serialize()` and `unserialize()`. > > Here we get a great perf boost, and what's more excellent as we don't have > to break BC to get there! > > This seems like a no-brainer and a good investment in the speed increase of > PHP by default for the future users of the platform. > > [1] http://pecl.php.net/package/igbinary > [2] http://www.php.net/~pierre/vcqa/apcigninary_perf.png > > Comments please. > > Regards, > Paul Dragoonis. -- John Carter Development Manager Identity Networks --- ## #54717 — Jonathan Bond-Caron — 2011-08-19T11:15:22+00:00 URL: https://externals.io/message/54677#54717 · In reply to: #54702 > > Also the proposed include it -> read the data -> convert on write if > in previous format (possibly an ini setting?) would provide full BC > and would allow a painless upgrade... wouldnt it? > 100% against enabling it by default. - still a BC break since some apps in the wild assume serialize($foo) is ascii text (whether that's good or bad). - the default `serialize()` is readable, easy to debug, doesn't assume you can read binary. PHP is also about simplicity not performance. - igbinary_serialize_double() is not cross-platform so serializing/unserializing a 'float' won't always work http://www.dmh2000.com/cpp/dswap.shtml (2. use a 'union' type to overlay a byte array on top of the data) My preference would be to have: serialize_text() -- ascii text only (standard/var.c) but remove 'NULL' byte (working on a patch for this) serialize_binary() -- igbinary bundled into core... `serialize()` -- based on php.ini setting, serialize_text by default --- ## #54718 — Stefan Neufeind — 2011-08-19T11:24:19+00:00 URL: https://externals.io/message/54677#54718 · In reply to: #54717 > > > […] > > 100% against enabling it by default. > > - still a BC break since some apps in the wild assume serialize($foo) is ascii text (whether that's good or bad). > - the default `serialize()` is readable, easy to debug, doesn't assume you can read binary. PHP is also about simplicity not performance. > - igbinary_serialize_double() is not cross-platform so serializing/unserializing a 'float' won't always work > http://www.dmh2000.com/cpp/dswap.shtml (2. use a 'union' type to overlay a byte array on top of the data) > > My preference would be to have: > > serialize_text() -- ascii text only (standard/var.c) but remove 'NULL' byte (working on a patch for this) > serialize_binary() -- igbinary bundled into core... > > `serialize()` -- based on php.ini setting, serialize_text by default In general fine with me. But why not just give it a second (optional) parameter? Regards, Stefan --- ## #54719 — Sebastian Bergmann — 2011-08-19T11:28:54+00:00 URL: https://externals.io/message/54677#54719 · In reply to: #54677 Am 18.08.2011 13:12, schrieb Paul Dragoonis: > Here we get a great perf boost, and what's more excellent as we don't have > to break BC to get there! Are you sure it does not break BC? Does http://bit.ly/nIRxqz still work with IGBinary, for instance? -- Sebastian Bergmann Co-Founder and Principal Consultant http://sebastian-bergmann.de/ http://thePHP.cc/ --- ## #54720 — Pierre Joye — 2011-08-19T11:36:01+00:00 URL: https://externals.io/message/54677#54720 · In reply to: #54719 > Am 18.08.2011 13:12, schrieb Paul Dragoonis: > > […] > >  Are you sure it does not break BC? Does http://bit.ly/nIRxqz still work >  with IGBinary, for instance? That obviously won't work as long as igbinary is used. It is covered by the "interop" or BC break parts in my initial reply. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #54721 — Paul Dragoonis — 2011-08-19T11:42:05+00:00 URL: https://externals.io/message/54677#54721 · In reply to: #54720 > On Fri, Aug 19, 2011 at 1:28 PM, Sebastian Bergmann > wrote: > > […] > have > > […] > > That obviously won't work as long as igbinary is used. It is covered > by the "interop" or BC break parts in my initial reply. > So far, it looks like this "transparent" upgrade isn't that transparent anymore and will result in BC breaks. New approaches must be thought of. Like Pierre said, there will be new serialization engines in the future so it makes no sense to add more functions like serialize_binary() or whatnot. I think we need to keep our code base identical and maintain the serialize_handler via our php.ini configuration. Cheers, Paul. > > Cheers, > -- > Pierre > > @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #54723 — Ferenc Kovacs — 2011-08-19T12:04:31+00:00 URL: https://externals.io/message/54677#54723 · In reply to: #54721 > > > > […] > > So far, it looks like this "transparent" upgrade isn't that transparent > anymore and will result in BC breaks. New approaches must be thought of. > > Like Pierre said, there will be new serialization engines in the future so > it makes no sense to add more functions like serialize_binary() or whatnot. > > I think we need to keep our code base identical and maintain the > serialize_handler via our php.ini configuration. this could still cause problems, as pointed out before (when you change your handler, or when you pass serialized data between different php installs, etc.), so I really like what Kiall Mac Innes suggested: prefix the data with the serializer identifier. if it is missing, then it was serialized with the old method, if it is present, the appropriate method should be used, if the serializer ident is there but unknown, then exit with a fatal error telling the user that the custom handler is missing. the downside would be that if you want to serialize/unserialize the data outside of php, your implementation should take care of this prefix. just a wild idea, but maybe useful: instead of creating a prefix, we could serialize the passed data with the given(php, igbinary, etc.) handler, then wrap the whole stuff into an array which holds the name of the used handler and the serialized data, and serialize this array with the old(php) serialize method. this way the datablob would be always a valid serialized string, and would be easier to get the serialize method than with the prefixing. -- Ferenc Kovács @Tyr43l - http://tyrael.hu --- ## #54724 — Arpad Ray — 2011-08-19T12:31:46+00:00 URL: https://externals.io/message/54677#54724 · In reply to: #54723 > this could still cause problems, as pointed out before (when you > change your handler, or when you pass serialized data between > different php installs, etc.), so I really like what Kiall Mac Innes > suggested: prefix the data with the serializer identifier. > if it is missing,  then it was serialized with the old method, if it > is present, the appropriate method should be used, if the serializer > ident is there but unknown, then exit with a fatal error telling the > user that the custom handler is missing. This came up when we discussing changing the default session serialization format a while ago. A prefix is a nice idea but care would have to be taken not to collide with any of the bytes which are valid for the current format to start with. Might be more of a curiosity than anything useful, but IIRC `unserialize()` currently ignores everything after the end of the data. > the downside would be that if you want to serialize/unserialize the > data outside of php, your implementation should take care of this > prefix. > just a wild idea, but maybe useful: > instead of creating a prefix, we could serialize the passed data with > the given(php, igbinary, etc.) handler, then wrap the whole stuff into > an array which holds the name of the used handler and the serialized > data, and serialize this array with the old(php) serialize method. > this way the datablob would be always a valid serialized string, and > would be easier to get the serialize method than with the prefixing. If my old app couldn't read some newly serialized string, I'd rather it failed hard than apparently succeed but have the wrong data. Regards, Arpad --- Next page: https://externals.io/message/54677.md?page=2