# unserialize() & unicode issues - Thread: https://externals.io/message/18738 - Original archive: https://news-web.php.net/php.internals/18738 - Started by Antony Dovgal on 2005-09-09T09:49:05+00:00 - Messages: 17 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #18738 — Antony Dovgal — 2005-09-09T09:49:05+00:00 URL: https://externals.io/message/18738#18738 Hello all. I'm currently working on unicode support in `serialize()`/unserialize() and stuck with some issues. Here they are: 1) What to do with unserializing serialized unicode strings when unicode_semantics is Off? I presume it's safe to create & return IS_UNICODE in this case ? 2) Classnames are serialized without U: or s: prefix, but I can detect unicode string by it's leading "\". It's looks kinda tricky, but on the other hand forward slash can't appear there if it's not unicode. Or should I change it to use U:/s: prefixes? (Didn't try it yet, so I can't say how difficult it would be). The other problem here is that we can't use unicode class names when unicode_semantics is Off because in this case class_table stores them as IS_STRING and we won't be able to find class entry by it's unicode name (thanks to Val for noticing this). 3) Currently `serialize()` produces valid \u0000 sequences, which can be parsed/restored perfectly fine when reading them from a file or returning from `serialize()`. But specifying them as a const string won't work as these sequences get parsed in compile time. Short example: IMO the best way here is to change `serialize()` output to produce something else (for example \pu0000 instead of \u0000) - in this case it works just fine. Comments? -- Wbr, Antony Dovgal --- ## #18742 — Andrei Zmievski — 2005-09-09T13:44:38+00:00 URL: https://externals.io/message/18738#18742 · In reply to: #18738 Yes, serialization is a problem. I would actually advocate putting a marker in the serialized file that indicates what the value of unicode_semantics switch was during the serialization, and if the value is different during deserialization, refuse to load it or start a new session. One really should not be changing that switch on a whim in-between sessions. -Andrei > Hello all. > > I'm currently working on unicode support in `serialize()`/unserialize > () and stuck with some issues. > Here they are: > > 1) What to do with unserializing serialized unicode strings when > unicode_semantics is Off? > I presume it's safe to create & return IS_UNICODE in this case ? > > 2) Classnames are serialized without U: or s: prefix, but I can > detect unicode string by it's leading "\". > It's looks kinda tricky, but on the other hand forward slash can't > appear there if it's not unicode. > Or should I change it to use U:/s: prefixes? (Didn't try it yet, so > I can't say how difficult it would be). > > The other problem here is that we can't use unicode class names > when unicode_semantics is Off because in this case class_table > stores them as IS_STRING and we won't be able to find class entry > by it's unicode name (thanks to Val for noticing this). > > 3) Currently `serialize()` produces valid \u0000 sequences, which can > be parsed/restored perfectly fine when reading them from a file or > returning from `serialize()`. > But specifying them as a const string won't work as these sequences > get parsed in compile time. > > Short example: > var_dump(unserialize('U:2:"\u0061\u0061";')); // won't work > var_dump(unserialize(serialize("aa"))); // works > var_dump('U:2:"\u0061\u0061";'); //produces unicode(9) "U:2:"aa";" > ?> > IMO the best way here is to change `serialize()` output to produce > something else (for example \pu0000 instead of \u0000) - in this > case it works just fine. > > Comments? > > -- > Wbr, Antony Dovgal --- ## #18744 — Antony Dovgal — 2005-09-09T13:58:15+00:00 URL: https://externals.io/message/18738#18744 · In reply to: #18742 > Yes, serialization is a problem. I would actually advocate putting a > marker in the serialized file that indicates what the value of > unicode_semantics switch was during the serialization, and if the > value is different during deserialization, refuse to load it or start > a new session. One really should not be changing that switch on a > whim in-between sessions. Why? It loads/works perfectly fine except for the problems I've mentioned. Also, you can't put any markers to the serialized _text_ (at least it sounds very bad to me), so it won't help you in this case. -- Wbr, Antony Dovgal --- ## #18745 — Andrei Zmievski — 2005-09-09T14:09:50+00:00 URL: https://externals.io/message/18738#18745 · In reply to: #18744 The problems you encountered are fairly big, I wouldn't just dismiss them. -Andrei > > > > […] > > Why? It loads/works perfectly fine except for the problems I've > mentioned. > Also, you can't put any markers to the serialized _text_ (at least > it sounds very bad to me), so it won't help you in this case. > > -- > Wbr, Antony Dovgal --- ## #18807 — Andi Gutmans — 2005-09-13T00:54:28+00:00 URL: https://externals.io/message/18738#18807 · In reply to: #18742 Not coming with a solution, but I believe this would be a bad idea. I do think some people will be using IS_UNICODE strings when unicode_semantics=off, mainly for existing applications. They may want to serialize Unicode strings even though their classes are IS_STRING. It might make sense to raise an error though if a "class" is used, but if it's just a value or a hash key, then those are valid in unicode_semantics=off. Andi At 06:44 AM 9/9/2005, Andrei Zmievski wrote: >Yes, serialization is a problem. I would actually advocate putting a >marker in the serialized file that indicates what the value of >unicode_semantics switch was during the serialization, and if the >value is different during deserialization, refuse to load it or start >a new session. One really should not be changing that switch on a >whim in-between sessions. > >-Andrei > > > > > > […] --- ## #18812 — Antony Dovgal — 2005-09-13T08:21:21+00:00 URL: https://externals.io/message/18738#18812 · In reply to: #18807 Even if the class name is in Unicode, we can try to convert it to ASCII and fail only in the case when we can't find its class entry in the list. So, I don't see any need in markers and other fairly major changes. > Not coming with a solution, but I believe this would be a bad idea. I > do think some people will be using IS_UNICODE strings when > unicode_semantics=off, mainly for existing applications. They may > want to serialize Unicode strings even though their classes are > IS_STRING. It might make sense to raise an error though if a "class" > is used, but if it's just a value or a hash key, then those are valid > in unicode_semantics=off. > > Andi > > At 06:44 AM 9/9/2005, Andrei Zmievski wrote: > > […] -- Wbr, Antony Dovgal --- ## #18813 — val khokhlov — 2005-09-13T09:32:21+00:00 URL: https://externals.io/message/18738#18813 · In reply to: #18812 Hello Antony, Tuesday, September 13, 2005, 11:21:21 AM, you wrote: AD> Even if the class name is in Unicode, we can try to convert it to ASCII AD> and fail only in the case when we can't find its class entry in the list. I think, it's not the only way. If we don't care about being compatible with previous PHP's `serialize()`, a more portable way is to store class/property names in unicode (if unicode_semantics=off when serializing, convert hash keys to unicode). Since we do know script encoding, we can always downgrade unicoded names into local encoding. -- Best regards, val mailto:val@vk.kiev.ua --- ## #18814 — Antony Dovgal — 2005-09-13T09:42:57+00:00 URL: https://externals.io/message/18738#18814 · In reply to: #18813 > Hello Antony, > > Tuesday, September 13, 2005, 11:21:21 AM, you wrote: > > AD> Even if the class name is in Unicode, we can try to convert it to ASCII > AD> and fail only in the case when we can't find its class entry in the list. > I think, it's not the only way. > If we don't care about being compatible with previous PHP's > `serialize()`, a more portable way is to store class/property names in > unicode (if unicode_semantics=off when serializing, convert hash keys to > unicode). Since we do know script encoding, we can always downgrade > unicoded names into local encoding. So you propose to store strings/hash keys/class names in Unicode even if unicode_semantics is Off ? It looks like adding unnecessary overhead to me. -- Wbr, Antony Dovgal --- ## #18815 — Derick Rethans — 2005-09-13T09:52:00+00:00 URL: https://externals.io/message/18738#18815 · In reply to: #18814 > > > […] > > So you propose to store strings/hash keys/class names in Unicode even if > unicode_semantics is Off ? > It looks like adding unnecessary overhead to me. But needed, as even with the semantics off, you can get unicode strings. Which can end up as array keys. Derick -- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org --- ## #18816 — Antony Dovgal — 2005-09-13T10:07:29+00:00 URL: https://externals.io/message/18738#18816 · In reply to: #18815 > > > > […] > > But needed, as even with the semantics off, you can get unicode strings. > Which can end up as array keys. Yes, in this case there is no way to avoid converting (while doing `unserialize()`), but I don't see any point in creating Unicode strings when serializing with unicode_semantics is Off. -- Wbr, Antony Dovgal --- ## #18818 — val khokhlov — 2005-09-13T10:42:29+00:00 URL: https://externals.io/message/18738#18818 · In reply to: #18814 Hello Antony, Tuesday, September 13, 2005, 12:42:57 PM, you wrote: AD> So you propose to store strings/hash keys/class names in Unicode AD> even if unicode_semantics is Off ? yes - those items that are encoded into unicode when unicode_semantics is on (afair, class names and property names for serialize) AD> It looks like adding unnecessary overhead to me. it's an overhead when you serialize and unserialize data on the same system with the same php.ini unicode settings; but when transferring data to other systems or changing something, you need to either use unicode, or specify encoding for the serialized data (so, you can convert them to the actual encoding when unserializing) -- Best regards, val mailto:val@vk.kiev.ua --- ## #18819 — Pierre Joye — 2005-09-13T10:43:40+00:00 URL: https://externals.io/message/18738#18819 · In reply to: #18816 > Yes, in this case there is no way to avoid converting (while doing `unserialize()`), > but I don't see any point in creating Unicode strings when serializing with unicode_semantics is Off. If I use serialized data on different hosts with different php, I can see a need of having unicode strings in serialize even if unicode_semantics is off. --Pierre --- ## #18881 — Derick Rethans — 2005-09-14T12:45:13+00:00 URL: https://externals.io/message/18738#18881 · In reply to: #18816 > > […] > > Yes, in this case there is no way to avoid converting (while doing > `unserialize()`), but I don't see any point in creating Unicode strings when > serializing with unicode_semantics is Off. Why not? An array can have a unicode string element, you want to properly serialize that, as you can't always downconvert. Derick -- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org --- ## #18899 — Marcus Boerger — 2005-09-14T14:02:40+00:00 URL: https://externals.io/message/18738#18899 · In reply to: #18744 Hello Antony, why can't we put a marker there? shouldn't we be able to add a flag and allow that flag in older php versions disallowing to load those serialized data in case it shows unicode semantics? Right now we'd simply fail in case of unicode usage in any serialized data with old php versions. Adding those types now would at least to show a more specific and thus helpful error message. On te other hand havin the semantics flag being optional would allow HEAD to unserialize data from older version without any problem. regards marcus Friday, September 9, 2005, 3:58:15 PM, you wrote: > > > […] > Why? It loads/works perfectly fine except for the problems I've mentioned. > Also, you can't put any markers to the serialized _text_ (at least it > sounds very bad to me), so it won't help you in this case. Best regards, Marcus --- ## #18921 — Antony Dovgal — 2005-09-14T18:58:31+00:00 URL: https://externals.io/message/18738#18921 · In reply to: #18899 > Hello Antony, > > why can't we put a marker there? shouldn't we be able to add a flag > and allow that flag in older php versions disallowing to load those > serialized data in case it shows unicode semantics? You mean adding a marker to be able to fail with a nice error msg? I don't think the marker is needed for that (see below). > Right now we'd > simply fail in case of unicode usage in any serialized data with old > php versions. Yes. And I think it's reasonable as nobody told that the old versions will be forward compatible. > Adding those types now would at least to show a more > specific and thus helpful error message. I suppose it's better to change 4.4.1/5.0.6/5.1 to fail gracefully when they find an unknown prefix ("U:"). > On te other hand havin the > semantics flag being optional would allow HEAD to unserialize data > from older version without any problem. Didn't get that paragraph... =| Do we have any problems with serialized data from older versions? I don't see any of them. -- Wbr, Antony Dovgal --- ## #18928 — Marcus Boerger — 2005-09-14T20:12:04+00:00 URL: https://externals.io/message/18738#18928 · In reply to: #18921 Hello Antony, Wednesday, September 14, 2005, 8:58:31 PM, you wrote: > > > […] > You mean adding a marker to be able to fail with a nice error msg? > I don't think the marker is needed for that (see below). > > […] > Yes. > And I think it's reasonable as nobody told that the old versions will be forward compatible. > > […] > I suppose it's better to change 4.4.1/5.0.6/5.1 to fail gracefully when they find an unknown prefix ("U:"). > > […] > Didn't get that paragraph... =| > Do we have any problems with serialized data from older versions? > I don't see any of them. Well right now we don't fail gracefully and i don't think we should unless we are dealing with something introduced in later versions that doesn't hurt or to generate helpfull error messages that explicitly tell you what new stuff the serialized data contains the old version being runned cannot deal with (e.g. Unicode data). Best regards, Marcus --- ## #18930 — Antony Dovgal — 2005-09-14T20:22:09+00:00 URL: https://externals.io/message/18738#18930 · In reply to: #18928 > Well right now we don't fail gracefully Right, but it could be done easily. > and i don't think we should unless > we are dealing with something introduced in later versions that doesn't hurt > or to generate helpfull error messages that explicitly tell you what new > stuff the serialized data contains the old version being runned cannot deal > with (e.g. Unicode data). Yes, that's exactly what I'm talking about. -- Wbr, Antony Dovgal